Agreed. And ideally that shouldn't matter. But in the real world there are other consequences and side effects. Maybe I don't want to make them uncomfortable. Maybe I just don't want to deal with the nagging, "busting my chops", or the questions about why I'm not drinking. Maybe I want to avoid some preconceived notions certain people may have about people who don't drink. So even if it's on them, there are still personal reasons I may want to avoid the situation.
People will take their cue from you as to how to respond. If you act embarrassed about it, they’ll react accordingly. If it’s no big deal to you, same thing for them.
I quit cold turkey years ago. I was worried about the same things. But even my most macho & hard-drinking friends just accepted it without comment.
I encountered a lot of peer pressure to drink when I was in college. (I didn't drink much because a lot of alcohol makes my stomach upset.)
Don't surround yourself with people who make you uncomfortable. It's a lesson I realized as I got older: It has little to do with drinking; but if the people around you make you feel uncomfortable not drinking, then you're around the wrong people.