In the Laundry Files by Charles Stross, doing too much computations in your head may give you a form of brain damage by inviting extra-dimensional feeders to feast on your central nervous system. I feel like while the details may differ, as far as the big picture is concerned, he is not wrong.
Funnily enough, it was M. Stross who put me onto the fuss about the X11Libre's project's politics, and the GitHub issue over its README, just under a week ago. M. Stross had some rather strong words to say on the subject.
