It's not just that, it's also that men have higher skin temps than women. If a man and woman were wearing the exact same clothes, in the exact same room, the man would still almost certainly be feeling warmer.
It is directly affected by body fat at least. Fat is insulating (hence whales, seals, etc. having a thick fat layer) and definitely being on the larger size with a higher % of body fat makes someone feel warmer.
I'd say far less than the layers of wool in a suit.
I work from home and don't do business suits. I've always kept the a/c on 26 because I don't like the shock that you get when moving even from 19 to 30. Looks like other people are finally noticing that...