It’s because they are casting a wide net on people and stereotyping them. What they say isn’t strictly false, but it’s also perpetuating a specific stereotype—the kind that if directed at others would be rightfully suppressed.
Saying that everyone there is racist would be casting a wide net. Saying that a large enough % of people are racist such that it would materially impact your quality of life if you are a PoC is a perfectly reasonable thing to say.