I agree that there's enormous value in carving out mathematics from other linguistic reasoning, but I don't see defining as something as mathematic rather than linguistic is generally useful. You use the same skills to look for incoherency in both situations, but human language is generally expected to be incoherent on some level.

Besides, a lot of what people mean when they say they're bad at math is that they're bad at arithmetic, which is honestly understandable.