To some people gender is basically sex. Simple as that. So a person can see themselves as a woman but others are free to see them as a man, because this isn't a scientific term as you say.

Its like how some call men who didn't conform to male norms "girlies" or so, these things are so ill defined that its dumb to argue over it. Of course it is rude to call something they don't wanna be called though.