Biologists do distinguish between the sexes (in humans and other species) based on gametes (in humans these are called sperm and eggs). However, I suggest a different standard be use in the Olympics and other sporting competitions. Swyer’s syndrome is quite rare and I know of no evidence that anyone involved with sports suffers from it. However, persons with Swyer’s syndrome have 46-XY (not 46-XX) chromosomes. They have fallopian tubes and a uterus. However, they lack functional ovaries (and testis). With medical help (IVF) they can have children. I consider them to be “female” even though they produce no gametes.

Using chromosomes (and in rare cases ultrasounds) for sex determination (in sports and elsewhere) has the huge advantage(s) of being cheap, easy, and non-invasive. Actually checking for gametes would require major surgery in some (rare) cases and would be highly invasive.