You're talking about gametes and OP and GP are talking about dimorphism, which is generally considered the difference in forms not including gametes. It's a very specific word to do with size differences, body hair etc. That is absolutely a bimodal distribution. Even just in height it is not binary.

Sex does have several meanings, and form is one of them.