Again, you're begging the question by assuming that you can always tell simply because most of the time you can. What about Michelle (Mike) Obama?

I stood about 3ft away from Blaire White at a Walk Away rally a few years back and I'm pretty sure that, if I hadn't known to begin with, I wouldn't have questioned that Blaire was female.

Do a websearch for 'Buck Angel' and see if you don't think she passes as a man. Of course, since she is a pre-op pornstar, you can see her vagina for yourself if you're so inclined. FTMs pass a lot more easily than MTFs.