I heard Viswanathan Anand on TV a long time ago (when he had just become world champion), he claimed getting physically fitter had a definite impact on his game. Maybe that implicitly means there is physical fitness is a factor, hence women are disadvantaged?
(I'm just trying to reason against you, I don't think the fitness Anand was talking about requires crossing the boundary where being male is an advantage.)
The physical training is for endurance, as at high levels chess can be an exhausting sport. This is because winning chess competitions involves concentrating intensely for hours, days, or even (in rare cases like Kasparov vs Karpov) weeks while being under enormous and constant stress. That can be very draining if you're not prepared for it both physically and mentally.
I would liken competitive chess to an endurance sport like marathon running instead of to something like boxing or sumo wrestling, where physical mass and brute force can be deciding factors.
There's no reason that women can't be as physically or mentally prepared for such as sport as men.
Physical fitness is certainly important but I doubt for chess it's on the level that's not achievable for women. Unless you are talking about chess boxing which is an interesting sport.
(I'm just trying to reason against you, I don't think the fitness Anand was talking about requires crossing the boundary where being male is an advantage.)