I think the problem is that your definition of male and female do not match someone else's definitions. The presence of lack of dangly bits has remarkably little to do with what gender someone associates with (not even counting the fact that a person can have both sets of genitalia or neither).

For example, what gender would you assign to someone who was born with both sets of genitalia, and had the choice made by their parents to remove the vagina? She may choose to identify as female because that's how her brain (and possibly breasts) evolved, regardless of the choice her parents made for her as a baby.