on bio anchors

“Bio Anchors” is a long document by Open Philanthropy researcher Ajeya Cotra.

Many people have written more accessible summaries of this document. My favorite of these is Scott’s, and he links to a few of the others.

I have thought a lot about this report, and I would summarize it in a very different way. Here is the argument of Bio Anchors, as I understand it. (Formatted like a quotation, but it’s my own words.)

Computers keep getting faster, exponentially. How far will this trend go before hitting a limit? It’s hard to know.

Hardware experts believe the current trend can’t go on indefinitely, at least as long as we’re still using silicon chips. And then if we switch to more exotic hardware, that’ll be so different that it’s difficult to assess now how much it will help.

But in the past, people have often claimed Moore’s Law will stop due to some fundamental limit or other, and yet people keep finding ways to make computers faster, or faster at specific applications. So it seems reasonable to imagine that computers will get many orders of magnitude faster in the coming century, even if they hit a limit sometime in the middle. So in 2060 or 2100, we might have computers that are (say) 10 million times faster than today’s computers.

This has extreme implications. Computers like that could run brute force calculations of such large scale they sound like jokes, like a “replay of brain evolution”: doing the same amount of calculation as all nervous systems of all organisms that ever lived on earth, taken together. [The report implies a ~10% probability you could do this with 1 trillion dollars worth of these future computers, which is a lot of money, but not necessarily an absurd % of future-GDP to spend on a big science project.]

These mind-boggling capabilities would no doubt change the world in numerous ways. One of the implications is that building true AI may be easier than you might think. Natural selection “built intelligence the hard way.” AI researchers are trying to build it an easier way, but with those computers, they wouldn’t have to be as clever – they could just offload a lot of the work onto massive brute force trial-and-error, like natural selection did. Makes you think!

Now, I don’t think this is a bad argument! It really is worth thinking about how we don’t know when Moore’s Law-style growth will stop, and what the world might be like if it keeps going for a long time.

But – this is not the way the report frames itself. It doesn’t resemble any of the other summaries of the report, either. This argument is not easy to read off the text of the report. You have to do a lot of digging and thinking to find it.

—-

What does Bio Anchors say that it is? It says it’s about forecasting when “transformative AI” will occur, meaning AI that will transform the world economy.

It uses a very complicated methodology, with a lot of moving parts. I won’t try to summarize all of them. But briefly, the key ingredients are

1. a future trajectory for falling computation cost over time (projecting Moore’s Law up to some limit)
2. a future trajectory for how much money people will be willing to spend on a transformative AI project
3. a number for the amount of computation needed to create a transformative AI

To find out when transformative AI happens, you look for when (amount of money spent) times (cost of computation) equal (computation needed for transformative AI).

The report spends the majority of its ~200 pages of discussion and calculation on ingredient #3, the amount of computation needed. Most of the report is about producing a (very wide) probability distribution over this number. It proposes six different ways of thinking about what this number might be, and estimates a probability distribution based on them, and then does a weighted average of all those distributions.

The report seems to assume that I really, really care about the exact shape of this distribution. But it’s not clear to me why this matters. The most important thing about it, by far, is just one number:

What is the probability that the amount of computation needed for transformative AI is feasible given the upper limit where Moore’s Law stops?

That is, assume some upper bound on spending like $1 trillion or even $10 trillion, and imagine spending that amount on the future computers that have hit the Moore’s Law limit. What is the probability that this is enough?

This probability is what the report emits for “the probability that transformative AI will be developed by 2100.” (Since the report only goes up to 2100, and also [sort of] assumes Moore’s Law will hit the limit before then.)

And this estimate, about AI by 2100, is the most important number that comes out of the report.

Here is the punchline of the report, as presented in some of the other summaries:

Conclusions of Bio Anchors

Bio Anchors estimates a >10% chance of transformative AI by 2036, a ~50% chance by 2055, and an ~80% chance by 2100. [OpenPhil Co-CEO Holden Karnofsky]

So When Will We Get Human-Level AI?

The report gives a long distribution of dates based on weights assigned to the six different models […]. But the median of all of that is 10% chance by 2031, 50% chance by 2052, and almost 80% chance by 2100. [Scott Alexander]

In a sense, the other numbers mentioned are just consequences of the “by 2100” number.

The “by 2100” number is set by the upper limit on Moore’s Law. If you go to earlier dates, you get a lower probability, in a manner that climbs smoothly because we’re combining uncertain estimates with smooth declines in compute cost. If Moore’s Law won’t “get us there” by 2100, it won’t get us there any earlier either; if it will get us there by 2100, then because of all the uncertainty involved, there’s some chance it will get us there by 2050, and some lower chance by 2030, etc.

The exact shape of the “how much compute?” distribution can affect two things: the “by 2100” probability, and the shape of the probability trend across earlier years.

But the latter really does not matter much. If the “by 2100” number is 80%, then the “by 2050” number is inevitably going to be something much greater than 0% and much less than 80%. So it’s 50%. If it had been 40% or 60%, would anyone have cared about the difference?

—-

So really, all that effort the report pours into the exact shape of the “how much compute?” distribution has little impact. Whether it knows it or not, all of that stuff is really just estimating a single number.

And that number doesn’t just depend on the “how much compute?” distribution. It also depends on how much money people will spend, and on how far Moore’s Law will continue. Let’s focus on the latter.

Where will Moore’s Law hit a limit? This estimate drives the whole conclusion of the report. But the report spends very little time on it. Indeed, most of the details are relegated to an appendix, and the appendix is full of comments like:

Because they have not been the primary focus of my research, I consider these estimates unusually unstable, and expect that talking to a hardware expert could easily change my mind. […]

[…] After an extremely cursory look into this topic (entirely via talking with Paul and simple Google queries), my tentative best guess is […]

[…] I am very unsure where the balance of considerations should fall.

Ultimately, the analysis has two parts.

First, Cotra estimates (“after an extremely cursory look into [the] topic”) that if you make silicon chips as transistor-dense as possible, and add on all other specific foreseeable mechanisms for better efficiency, you get a speedup of 144x, for a FLOPS/$ value of 1.7e19.

Then, Cotra notes that people might switch to non-silicon computers, and says that this may let us continue the trend, but maybe not as far as we went in the 20th century:

The above reasoning was focused on listing all the foreseeable improvements on the horizon for silicon-based chips, but I believe there is substantial possibility for both a) “unknown unknown” sources of improvements to silicon chips and b) transition to an exotic form of hardware. For example, at least some companies are actively working on optical computing in particular […]

My expectation is that “unknown unknown” factors and especially new hardware paradigms will at least drive the continuation of the recent ~3-4 year doubling time trend well past 2040 […]

With that said, my understanding from discussions with technical advisors is that further improvement in hardware prices is likely to be slower than Moore’s law and there will likely be much less total improvement over the next century than we have seen in the past century, even taking into account the various potential options for exotic computing paradigms – this is partly due to the likelihood of running up against fundamental physical limits. […]

and then she … uh … just guesses a number:

I am very unsure where the balance of considerations should fall. For now, I have assumed that hardware prices will fall at a rate somewhere in between Moore’s law and the more recent trend, halving once every 2.5 years. I have also assumed that there is room for about 6 OOM of further progress in hardware cost in this century, which is a little over half as much as the 11 OOM of progress that was made from the 1960s to 2020.

That 6 OOM improvement translates into an upper limit of 1e24 FLOPS/$. This means we’re assuming we’ll get an extra ~60,000x speedup over the previous estimate from maxing out silicon chips.

Is this too high? Too low? I have no idea – I don’t know how far Moore’s Law will go, any more than Cotra does. Any number is just a guess.

But of course, this estimate determines what is predicted.

With the 60,000x extra speedup, we get Cotra’s 78% chance of TAI by 2100.

Assume a 600x extra speedup, and you get a 66% chance. (This is Cotra’s “conservative” forecast, except with none of the other conservative assumptions.)

Assume a 6x extra speedup, and you get a 52% chance. (Which is still pretty high, to be fair.)

Assume no extra speedup, and also no speedup at all, just the same computers we have now, and you get a 34% chance … wait, what?!

Well, Cotra has a whole other forecast I didn’t mention for “algorithmic progress,” and the last number is what you get from just algorithmic progress and no Moore’s Law.

So depending on how much you trust that forecast, you might want to take all these numbers with an even bigger grain of salt than you’d expected from everything else we’ve seen.

How much should you trust Cotra’s algorithmic progress forecast? She writes:

Note: I have done very little research into algorithmic progress trends. Of the four main components of my model (2020 compute requirements, algorithmic progress, compute price trends, and spending on computation) I have spent the least time thinking about algorithmic progress. 

…and bases the forecast on one paper about ImageNet classifiers.

I want to be clear that when I quote these parts about Cotra not spending much time on something, I’m not trying to make fun of her. It’s good to be transparent about this kind of thing! I wish more people would do that. My complaint is not that she tells us what she spent time on, it’s that she spent time on the wrong things.

—-

The model in the report has many moving pieces. Most of them turn out to not matter much, including the ones Cotra spent the majority of her time thinking about.

Cotra encourages the reader to play around with an interactive spreadsheet and try changing the various assumptions. If you do this, you find that indeed, most of them don’t matter much. I think we’re supposed to draw the conclusion that the results are robust: they come out largely the same way across a range of assumptions.

But as we’ve seen, there are some assumptions that really do matter, like the Moore’s Law limit. It’s not a virtue of the analysis that it adds on all sorts of bells and whistles that don’t matter. They just obscure the underlying argument, and confer a misleading impression of robustness.

I think my fundamental objection to the report is that it doesn’t seem aware of what argument it’s making, or even than it is making an argument.

My summary at the top was presented like an argument. It encouraged you to think about how Moore’s Law might continue, and spelled out some of the implications. “X will do Y in the future, and that will cause Z.” A simple story.

That same argument is at the core of Bio Anchors, once you prune away the bells and whistles. It’s really just a story about how X will do Y in the future, and that will cause Z – except that X, Y, and Z are surrounded by a crowd of other factors P, Q, R, S … and you have to work to pick out the story.

Bio Anchors doesn’t tell you which factors play the deciding role in its argument. It sees itself not as making an argument, but as computing an estimate, where adding more nuances and side considerations can only (it is imagined) make the estimate more precise and accurate.

But if you’re making an estimate about a complex, poorly understood real-world phenomenon, you are making an argument, whether you know it or not. Economists understand this, I think. That’s why they spend so much time with simple stylized models, so they understand what basic claim they’re making about the nature of the phenomenon – the core dynamic, the “cast of characters,” the X that does Y and causes Z – before they start adding bells and whistles.

Your model is going to make an argument – somewhere inside, implicitly – whether or not you know what it is. And if you don’t know what it is, you don’t know whether it’s any good.