One of the biggest areas of AI hype right now is the notion that it will hyperaccelerate scientific progress. I understand why people think this — AI is already accelerating scientific production. But the big problem is that production seems to have gotten decoupled from any useful measure of progress a long time ago. There is a whole science-of-science literature hand wringing about the fact that scientific production has increased ~exponentially but progress has not accelerated and has even slowed down.

Producing papers, for the most part, is a game researchers must play for status and career progress. It's value is relative. It's like thinking that AI is going to help traders make a lot more money. If everyone has access to the same capabilities, there is no alpha. In every scientific field I'm familiar with, the amount of published stuff exceeds the community's collective bandwidth to absorb and build upon ideas by a factor of 100x or more. Inevitably, the vast majority of what's published makes zero impact. Yet we pretend that publication itself has some value. It doesn't.

I do think AI will help, but ironically, it's by stretching an already creaking system to its breaking point and forcing it to reinvent itself. For example, AI is getting to the point where it is able to produce papers that aren't easily distinguishable from human-authored ones. Dropping the cost of production to zero will force us to stop attaching value to publication, and look for ways to identify actual intellectual value.

There's also the fact that in many/most fields, there's a limit to AI's impact on production as well. It may be able to do the cognitive parts but the bottleneck may be experiments on humans or some other form of real-world interaction that's hard to automate. I suspect AI researchers are misled by the fact that AI research itself is mostly cognitive/computational, and forget how different most scientific fields are.