This is a major and serious point. A large part of the discourse has become slapping the "extremist" label on every dissenting argument from the other side, regardless of where the position actually is.
If you ban extremist speech, then what you've done is create incentive to label everything as extremist.
I consider myself a centrist; I have been registered as an Independent since the first election in which I voted (21 years ago), and have voted for Republicans and Democrats. Politically, I try to evaluate each idea on the merits rather than which party espouses it. So at least in my case, there is no "other side" which I can mislabel as extremist.
Hahaha, of course there is. There is much more to politics than only Republicans and Democrats - in many countries they would both the be considered as on the same side.
If you ban extremist speech, then what you've done is create incentive to label everything as extremist.