I'll chip in with my main reason, as a London cycling veteran of nearly 10 years: shoulder checks.

I whip my head back and forth over my right shoulder constantly to check the state of traffic behind me.

Before I maneuvre, I check, check and check again. Always checking. Always fully aware of what's behind me.

Having something attached to my head makes it harder and less comfortable to do these rapid checks. I think this makes me less safe overall.