I wonder how will we train AI to become truly 'smarter' than humans. Not just the average human, but smarter than the smartest human in every possible field of knowledge?

Because so far I see that AI outputs are only great when 'orchestrated' by humans who are experts in that field.

It is not that people who cannot build apps are not able to build apps with AI, or that people who are not doctors cannot still generate some diagnosis out of some symptoms using AI. But any output generated by non-experts (who cannot verify the overall and directional validity of the output), is unreliable to build "on top of".

Like you take an app made by someone who has never built apps before - it is usually a minefield, which if you try to extend or build more features on top of, it starts blowing up.

Hallucinations take many shapes, and tiny hallucinated assumptions injected into different stages of a process compound when someone is not keeping an eye on it. I can control an AI doing work that I am not 100% capable of but at least 80% capable of doing. The moment I am using the AI to generate things that I am 0% understanding - it is just a wobbly foundation. At that point it also feels like "omg wow I am generating a maths proof", but I am too dumb to even disprove what the AI is creating, so I keep piling more and more on top of it.

Now I wonder that how exactly will be make AI 'smarter' than us when we run out of the topmost expert in every possible field. What data will we train the AI on?

How do we know an AI is smarter than Donald Knuth and Terrence Tao when there is no one to call its bullshit at that level anymore?