> They ignore that refusing someone is also part of the freedom of association.

And this is undesirable with no restrictions which is why societies constrain the freedom of association, in particular, in the US, via various "Civil Rights" acts.