“You are fascist” actually isn’t just an insult. If you display fascist tendencies then you’re a fascist, and he displays many of those typical tendencies.
There are actual fascists (and not as few as I would like) and they need to be called out, but using the term inaccurately and provocatively on a broad group makes it easier to oppose the usage outright. Optics are important to politics, like it or not.