Sure, it didn’t work—probably because enough people weren’t convinced that it was true enough (and also because they didn’t care)—but it's not unreasonable to think that such an argument should have been enough.
“You are fascist” actually isn’t just an insult. If you display fascist tendencies then you’re a fascist, and he displays many of those typical tendencies.
There are actual fascists (and not as few as I would like) and they need to be called out, but using the term inaccurately and provocatively on a broad group makes it easier to oppose the usage outright. Optics are important to politics, like it or not.
It actually is, though.
Sure, it didn’t work—probably because enough people weren’t convinced that it was true enough (and also because they didn’t care)—but it's not unreasonable to think that such an argument should have been enough.