Consumer-facing medical AI is in a lose-lose situation: Tune your model to make sure it catches any possible severe issue and it will tell you to go to the ER for everything; tune it to be more conservative and people could get seriously injured. This is because making the right calls in medicine depends on expert judgment and experience accumulated from a million factors, some not available in any dataset. IMO, if these tools can’t get the balance right—and they probably never will—we shouldn’t allow or encourage them for medical use cases.