This is Work in Progress, a newsletter by Derek Thompson about work, technology, and how to solve some of America’s biggest problems. Sign up here to get it every week.
We should be living in a golden age of creativity in science and technology. We know more about the universe and ourselves than we did in any other period in history, and with easy access to superior research tools, our pace of discovery should be accelerating. But, as I wrote in the first edition of this newsletter, America is running out of new ideas.
“Everywhere we look we find that ideas … are getting harder to find,” a group of researchers from Stanford University and MIT famously concluded in a 2020 paper. Another paper found that “scientific knowledge has been in clear secular decline since the early 1970s,” and yet another concluded that “new ideas no longer fuel economic growth the way they once did.”
In the past year, I’ve traced the decline of scientific breakthroughs and entrepreneurship, warned that some markets can strangle novelty, and investigated the domination of old movies and songs in the film and music industries. This year, a new study titled “Papers and Patents Are Becoming Less Disruptive Over Time” inches us closer to an explanation for why the pace of knowledge has declined. The upshot is that any given paper today is much less likely to become influential than a paper in the same field from several decades ago. “Our study is the first to show that progress is slowing down, not just in one or two places, but across many domains of science and technology,” Michael Park, a co-author and professor at the University of Minnesota, told me.
The researchers relied on a metric called the Consolidation-Disruption Index—or CD Index—which measures the influence of new research. For example, if I write a crummy literature review and no scientist ever mentions my work because it’s so basic, my CD Index will be extremely low. If I publish a paradigm-shifting study and future scientists exclusively cite my work over the research I rendered irrelevant, my CD Index will be very high.
This new paper found that the CD Index of just about every academic domain today is in full-on mayday! mayday! descent. Across broad landscapes of science and technology, the past is eating the present, progress is plunging, and truly disruptive work is hard to come by. Despite an enormous increase in scientists and papers since the middle of the 20th century, the number of highly disruptive studies each year hasn’t increased.
Why is this happening?
One possibility is that disruptive science is becoming less productive as each field becomes more advanced and the amount of knowledge new scientists have to acquire increases. This is sometimes called the “burden of knowledge” theory. Just as picking apples from a tree becomes harder after you harvest the low-hanging fruit, science becomes harder after researchers solve the easiest mysteries. This must be true, in some cases: Calculating gravity in the 1600s basically required a telescope, pen, and paper. Discovering the Higgs boson in the 21st century required constructing a $10 billion particle collider and spending billions more firing subatomic particles at one another at near–light speed. Pretending these things are the same is not useful.
A related theory is Johan S. G. Chu’s concept of “durable dominance”—a phenomenon where highly competitive fields create a small number of dominant winners. Chu and the University of Chicago scholar James Evans found that progress has slowed in many fields because scientists are so overwhelmed by the glut of information in their domain that they’re reading and riffing on the same limited canon of famous papers. It’s more or less the same principle as a weekend couch potato overwhelmed by streaming options who opts to just watch the top-ranked TV show on Netflix. In both science and streaming, a surplus of options might be entrenching a small number of massive hits.
When I spoke with the disruption paper’s co-authors last week, they seemed interested in explanations beyond the burden-of-knowledge theory. “If the low-hanging-fruit theory were sufficient, then I think we’d expect to see the oldest fields stagnate most dramatically,” said Russell Funk, a co-author and professor at the Carlson School of Management. “But the fact that the decline in disruption is happening across so many fields of science and technology points to something broader about scientific practice, and the corporatization of science, and the decline of scientific exploration in the last few decades.”
In other words, if science is getting less productive, it’s not just because we know too much about the world. It’s because we know too little about science itself. Or, more specifically, we know too little about how to conduct research in a way that gets the best, most groundbreaking results.
According to the rules of modern academia, a young academic should build status by publishing as many papers in prestigious journals as she can, harvest the citations for clout, and solicit funding institutions for more money to keep it all going. These rules may have been created with the best intentions—to fund the most promising projects and ensure the productivity of scientists. But they have created a market logic that has some concerning consequences.
First, these rules might discourage truly free exploration. As the number of Ph.D. students has grown, National Institutes of Health funding has struggled to keep up. Thus, the success rate for new project grants has mostly declined in the past 30 years. As grants have become more competitive, savvy lab directors have strategically aimed for research that seems plausible but not too radical—optimally new rather than totally new, as one researcher put it. This approach may create a surplus of papers that are designed to advance knowledge only a little. A 2020 paper suggested that the modern emphasis on citations to measure scientific productivity has shifted rewards and behavior toward incremental science and “away from exploratory projects that are more likely to fail, but which are the fuel for future breakthroughs.” As attention given to new ideas has decreased, science has stagnated.
Second, at the far extreme, these incentives might create a surplus of papers that just aren’t any good—that is, they exist purely to advance careers, not science.
When I suggested that there might just be a lot more “bullshit papers out there,” Funk told me it was a distinct possibility. But rather than blame individual scientists, he said the fault lies in a system that encourages volume over quality: “There are journals, which I’d consider predatory journals, that make researchers pay money to publish their papers there, with only symbolic peer review, and then the journals play games by making the authors cite articles from the same journal.”
Funk’s predatory-journal story reminded me of the dark side of Moneyball: When any industry focuses too much on one metric, it can render the metric meaningless and warp the broader purpose of the industry. Just as we are living in a platinum age of television—more quantity but perhaps not more quality—we seem to be in a platinum age of science, in which the best you can say about the industry is that there certainly seems to be more of everything, including crap.
A year ago, I pitched the idea of an abundance agenda, arguing that the U.S. suffers from a scarcity mindset in health care, housing, and beyond. The crisis in science offers an interesting test of this thesis in that researchers are struggling with a superabundance of knowledge and studies. It’s a useful reminder that abundance is not a sufficient end point; rather, it’s an input. Science may have a deficit of disruption precisely because the industry doesn’t know how to navigate its crisis of plenty—too much knowledge to synthesize, and too many papers bolstering their authors’ reputation without expanding the frontier of science.