The “have you ever tried spending [time] with them, without alcohol?” isn't referring directly to the amount of alcohol the others are drinking, but dumbing yourself down with alcohol, so you aren't as bothered by their behaviour.

Of course the behaviours that would bother you while more sober might be induced or exacerbated by excessive alcohol, but often I find people who are difficult to deal with after a few drinks are people I'd rather not interact with much when they are sober either.