“gametes as the unifying definition of the sexes”

Agree entirely. The foundational trait. The “necessary & sufficient condition” to qualify as “male” and “female”. Though you are probably reluctant to go that far.

But you might take a gander at a Wikipedia article on extensional and intensional definitions:

An intensional definition gives meaning to a term by specifying necessary and sufficient conditions for when the term should be used. In the case of nouns, this is equivalent to specifying the properties that an object needs to have in order to be counted as a referent of the term.

If you can’t or won’t say what EVERY member of EVERY anisogamous species MUST have to qualify as male or female then the terms are pretty much useless.

Standard biological definitions stipulate that “functional gonads” are those conditions and properties: