In most serif typefaces, one of the glyphs is significantly thicker than the other, and they don't meet exactly at 50% height. If one rotates the letter X 180 degrees, it would look out of place.

https://upload.wikimedia.org/wikipedia/commons/e/e6/Capitali...