Can you clarify how you will resolve the "In 2028, will AI be at least as big a political issue as abortion?" market? I worry abortion will end up a bigger issue by metrics like funding and single issue voters but it might still resolve yes based on vibes ("how much of the "political conversation" seems to be about either").
Current LLMs aren't stuck with a limitation of being boring and politically correct, that's an extra limitation placed upon them by the needs of corporate PR. You can get access to less RLHF'd models that will be more weird and also more happy to say controversial things.
I didn’t see separate results without reinforcement learning in “Sparks of Artificial General Intelligence” either. But more broadly, those papers are testing performance on very different endpoints (domains like maths problems where the “correct approach” is pretty clear, with limited if any prompt engineering and no dedicated training dataset/reasoning template).
Huh I just re-downloaded the Sparks paper and you're right, I can't find any mention of this either. Am I going mad?! I don't think so because I did find this page:
> Why unicorns? Well, the inspiration comes from the paper Sparks of Artificial General Intelligence: Early experiments with GPT-4 by Sébastien Bubeck and others, in which they use the prompt "Draw a unicorn in TiKZ" to compare unicorn drawings and note how fine-tuning the GPT-4 model for safety caused the image quality to deteriorate.
Are we both mad? No, after more digging it turns out that we're both remembering this anecdote from the talk that accompanied the paper. Oddly, they didn't mention this in the paper (probably it's embarrassing) but the lead author did mention it here:
Maximum-Harmless-RLHF results in <50 Helpfulness Elo, while Maximum-Helpfulness-RLHF gives >150. For comparison: pretrained base is around -150 Helpfulness Elo.
I agree (this is my field). One could do a different kind of RLHF that is only concerned about accuracy and not about being politically cautious, and it would do better at predicting.
...Frankly, I'm less impressed that AI is beating forecasters and more disappointed that forecasters are losing to AI. This is the same AI that constantly comes up with absolute bullshit responses to basic questions. I guess the point is that when pretty much everyone's making bullshit bets, you'll get slightly more people making less bullshit bets than complete bullshit bets, but still...
Yeah if anyone's interested, I sometimes record forecast data across different sources (prediction markets, expert forecasters, and traditional betting sites), evaluate them for accuracy, and publish the results (even when they're boring) along with the raw data and analysis code. I did this for the 2022 midterms, some sports events, the Oscars, and am planning to do it for the 2024 election.
If anyone is part of the very tiny percentage of people who actually finds this interesting, please consider subscribing 🙂
The 0.12% versus 20% result for AI doomer/skepticism seems like it's mostly a result of selection bias. You pick the most skeptical superforecasters and the most doomer AI safety experts and you, not-at-all-shockingly, find they leave the room still being the most skeptical and most doomer, respectively.
I'd probably treat the 20% number as actually lower than it is, just because superforecasters are better at calibrating - understanding the difference between 0.1% and 0.12% - but the overall result seems... really obvious? I guess you think people should update more after the arguments, but should they? I feel like a huge chunk of effective forecasting is being able to quickly find, locate, and evaluate the best arguments/evidence for the various sides of a position, and rarely is the best argument going to be hidden somewhere in a Reddit thread posted six years ago.
Probably part of the 0.12% chance is also just "Yeah okay the AI god wiped out 99.9% of the human race by detonating cobalt bombs and releasing bioweapons, but the remaining 0.1% are evolving good radiation resistance quite quickly and the AI god has moved operations to the asteroid belt where it can get the PGMs it wants easier!"
Well there is Aumann's agreement theorem [1] which says that rational agents should converge to the same probabilistic beliefs after sharing all their information with one another. So I guess either the people involved weren't rational agents or they didn't succeed in sharing all their relevant intuitions even after 80 hours of debate.
Forecasters are most concerned with getting the right answer. Their status and reputation incentives align with being visibly right about a lot of stuff when asked to forecast it, or at least be wrong in the right direction.
AI domain experts mostly become prominent not because they personally built a really good AI tool, but because they put together the most fantastic and compelling vision of an AI future that captured the interest of the public. Their incentives align with maintaining that reputation even if it means they make some predictions that are badly wrong.
It is absolutely obvious to me that forecasters are in this instance much more likely to be reasoning correctly about the evidence.
Being perfectly honest, I’m not sure most “AI domain experts” deserve that name. They may be some of the people most well-read about subjects labeled “AI” but I think that the person using AI toolchains for pragmatic purposes today (e.g., a software engineer at FAANG struggling desperately to figure out how to replace 5k enterprise chat tech support agents with an augmented LLM) is much more accurately described as an AI domain expert. This is not to say that these people would do better at forecasting. They won’t either, but they can probably tell you a lot more about what current AI tooling is and can do than, say, Yudkowsky can (assuming Yudkowsky is considered an example of an AI domain expert), since the perception of Yudkowsky’s expertise is largely based on people thinking his theories are neat and he seems smart. Really there appear to be lots of people who are publicly identified as AI domain experts yet have very little object-level practice on implementing AI for something useful.
To add, even the example software engineer may not be enough of an expert to produce a relevant forecast if he/she is not deeply versed in the hardware issues.
The hardware limitations are the reason I put probability of a superintelligent AI wiping out humanity by itself by 2100 at 0.0001% (I'd put 0, but "0 is not a number"). Because:
Where would the chips needed for this come from? What technology node will they run on? Where would their power be supplied from?
The last one seems to be roundly ignored by the "experts". Converter efficiency is stuck in mid-90% range, mostly because of magnetics, which no one has figured out how to replace (and oh boy are they trying!). This means, say, 5% of the power going to the chips is wasted. But wait, this is only the last step on the board itself, you have to convert from 400 000 V transmission line all the way down to 1.2 V, with power losses at each step, approximately boiling the oceans in the process. Scale that :)
Yudkowsky couldn't predict a way out of a paper bag even if Altman stood there with a flashlight pointing the way.
Everyone involved in the AI or AI risk discussion is aware of the hardware limitation. The two reasons I think it's not going to actually be the bottleneck are:
1: Computing overhang: Early techniques in machine learning tend to be brute force and high inefficient. It's very difficult to scale up hardware by multiple orders of magnitude in a few years. But making algorithms faster by multiple orders of magnitude happens all the time. Usually when a breakthrough in some machine learning domain happens, there is a huge amount of untapped potential (ie. overhang) in existing hardware.
2: Incentives: Governments and private investors are throwing billions of dollars at making more hardware. If an X-risk scenario happens, it doesn't need to go from humans running the world to AI running the world directly. There could be the intermediate step of humans utilizing AI running the world, and then when there is sufficient infrastructure for AI to sustain itself without humans, then AI runs the world.
If "everyone is aware" of the hardware limitations, they seem to not be discussing them all that much. Are you sure Yudkowsky knows what an inductor is?
1: "making algorithms faster by multiple orders of magnitude happens all the time" - by OoM's? all the time? can you point to three examples?
2: "billions of dollars" will not buy you a second ASML, nor get you thousands more semiconductor specialists, nor make chip geometries keep shrinking at the same rate they have been in the last few decades. Phrases like "it doesn't need to", "there could be", "if there's sufficient infrastructure" sound more like wishful thinking than a description of a probable scenario.
>"making algorithms faster by multiple orders of magnitude happens all the time" - by OoM's? all the time? can you point to three examples?
Stockfish is hundreds times more efficient than Deepblue. Katago is dozes of times more efficient than Google's AlphaGo. Modern linear programming algorithms are 43000 times more efficient than those from the 90s (https://www.usna.edu/Users/cs/wcbrown/courses/S18SI335/notes/01/notes.html). EfficientNet (image classification) is 10 times more efficient than ResNet.
On the other hand, an incentive to be visibly right would not necessarily lead to accuracy on this question. People would only be around to observe their correctness if it resolves negative, so to maximize chance of being visibly right, the superforcasters should choose a very small probability regardless of the evidence or reasoning.
I assume that forecasters forecast this question using the same knowledge, tools, reasoning, and habits that they use elsewhere, because it is pretty easy to arrive at that conclusion independently in good faith rather than cynically, and a universal adoption of a very cynical position seems… unlikely.
Actually I think there's a non-negligible chance here that most of the AI domain experts don't actually believe the probability they're giving either, because if they did, they'd be arguing strenuously and publicly for targeted assassinations of leading AI figures, destruction of infrastructure, etc., rather than happily going to parties, raising money, dropping E, etc. with people who are supposedly leading a charge to destroy all known life in a few decades.
I am not as negative on the domain experts as you, but the update from 25% to 20% based on unrelated policy victories to me is a red flag. What recent policy victories could possibly have shifted this complex future event by 5%. It indicates, as you say, they are optimizing for something else like drawing attention to policy interventions they favor.
If we're talking about the cynical status-based incentives of forecasters, wouldn't that push them to wholly discount AI *or anything else* as an existential risk? In 100.0% of futures where you exist to have status, having said "no, X is not going to kill us all" is the correct answer that gives you positive status.
The amount of hedging you'd want to do to get maximum status across the ensemble of possible outcomes is a more interesting question - a fallback position of "hey, I never said X wouldn't kill 99% of us" could be useful. But it's not really relevant to a poll that only allows a simple numerical response.
My assumption is that overall behavior is affected strongly by status incentives, but that individual forecasts are mostly not: I.e., you don’t cynically change your answers to accord with whatever gets you the most status, but rather subconscious status motivation pushes you into approaching, researching, and deciding all questions a certain way. That way for forecasters leads them to more disciplined reasoning which works well across domains, but the incentives of AI demagogues leads them to very motivated reasoning when it comes to AI risk questions, even if you might or might not trust their judgment and accomplishment in other domains.
That theorem doesn't seem to fit with people having distinct priors from one another (e.g. if I have a 5% prior of this universe being a simulation, and you have a 0.001% prior of being in a simulation, the same evidence that would persuade me won't necessarily persuade you), and fixing an "incorrect" prior seems extremely non-trivial, if not outright impossible.
But if you both have all the same information about the world, you should both have the same prior of this universe being a simulation. Otherwise, one or both of you miscalculated the true probability of the universe being a simulation.
>Probably part of the 0.12% chance is also just "Yeah okay the AI god wiped out 99.9% of the human race by detonating cobalt bombs and releasing bioweapons, but the remaining 0.1% are evolving good radiation resistance quite quickly and the AI god has moved operations to the asteroid belt where it can get the PGMs it wants easier!"
Agreed. I am _not_ happy with the way FRI phrased the extinction question.
>“Extinction” meant reducing total human population below 5,000 (it didn’t require literal extinction).
I'll take this situation, with 4 orders of magnitude more humans than chimps, and the remaining chimps surviving basically only because some humans take an interest in them, as "Chimps lost to humans". Under FRI's criteria, if AIs put humans into this situation, this goes in the "not extinct" bucket. One AI that keeps some humans as a hobby is probably enough to count as not-an-extinction-risk for FRI.
Satori is a blockchain fanboi fever dream at best, maybe even a scam. There is zero "meat" anywhere on the site. What their compiled windows software does.... I fear the worst.
Tiny correction: You wrote "So FRI gathered eleven of the most AI-skeptical superforecasters". In fact, the group was nine superforecasters and two domain experts. So mostly superforecasters, but not eleven superforecasters.
I was one of the two domain experts (although I have some forecasting chops too) and I have written previously on why I expect gradual timelines: https://arxiv.org/abs/2306.02519
(That essay is about automation of jobs by 2043 rather than extinction by 2100, and while there are substantial differences between those two events, they both rely on the vibe that it's much easier to get most of the way there than 100% of the way there.)
Thanks for sharing this, it looks really interesting (I’m still working though it). During the event, did the AI-concerned researchers try using your model? I’m curious whether they had lots of little differences or a few substantial disagreements, or whether they didn’t accept this framework.
I didn't publish it until after the study, although I did share a draft with the group near the end of the study.
There wasn't much engagement, especially because it focused on automation by 2043 instead of human disempowerment by 2100. Given our difficulty of finding cruxes by 2030, I think folks in the AI concerned camp probably allocate a good fraction of their probability to jobs not being fully automated by 2043 in conjunction with humanity being disempowered by 2100.
When I did publish, I got a boatload of criticism on the AI-concerned forums:
I heartily applaud you and your co-author's https://arxiv.org/abs/2306.02519 . That is an _extraordinary_ work, both in terms of the breadth of components that you covered and the courage to construct and justify numerical estimates for all these components of the AGI puzzle. Many Thanks!
I'm looking at your paper from a somewhat different perspective.
I'm 65, and I'm trying to gauge the odds of my seeing _enough_ of the critical steps towards transformative AGI to be able to discern the probable trajectory towards what I think of as our possible successor. So I'm looking at the factors that you consider with less of an eye towards "Will this replace human labor by 2043?" and more of an eye towards "Will the go/no go questions that determine whether human labor will _eventually_ be replaced be resolved while I am still alive?"
For the single largest factor in your analysis, the 16% from "AGI inference costs drop below $25/hr (per human equivalent)", I want to particularly applaud your "Examples transistor improvements from history (not cherry-picked)" table. I had a creepy feeling that process enhancements such as finfets take decades to make it from the lab to production, and 2043 felt like _very_ little time in that context. _Many_ Thanks for digging through 6 different technologies, all but strained silicon taking more than 2 decades to make this trip.
I'm actually rather pleased that in your paper, you partition the threat classes into two categories:
>Existential risk scenarios for AGI usually fall into two categories. Firstly, those based on the one-time performance of exceptional tasks generally beyond human capability, decapitating strikes against humanity with nanorobots or viruses or the like, enabled by superintelligence. Secondly, those based on the performance of many already-possible tasks at large scale (rise-of-the-robots and labor supplantation and wireheading/hedonic scenarios). For the first set of scenarios, the performance of currently existing and contemplated economically useful tasks, and the precise cost thereof, are irrelevant; designing life-eradicating nanobots is not an economically useful task, and since it need be done only once, cost is irrelevant, as is the versatility to perform a wide variety of other tasks.
>We interpret that you are worried about the second set of scenarios, where AI algorithms exert existential risk onto human civilization by performing economically useful tasks in bulk, under whatever aegis. This means that deployment and scale, on the stated timeline, are necessary for the satisfaction of the hypothetical.
While I don't think superintelligence is impossible, we don't have an existence proof for it. I tend to think that the second class of scenarios gets too little air time. It can be viewed as the construction of a competing species, and I think it is likely to eventually happen, though I have only the foggiest thoughts on when - though I very much appreciate that you provided estimates for 2100 as well as for 2043 in your paper!
Factor by factor:
>1) AGI algorithms
Thanks very much for the cautionary notes on walking, being a worm, driving (ouch!)
>self driving cars (in 1958), self driving cars (in 1995), self driving cars (in 2016)
and radiology
An existence proof in a biological system shows that it is physically possible to perform the task (and within the power and weight limits of the biological existence proof) but, as you said, e.g.
>We have been trying to teach algorithms to drive for far longer than we teach teenagers, and still, we have not succeeded.
>2) We quickly figure out how to train computers to do human tasks without sequential reinforcement learning
Mostly agreed, with the caveat that, for learning beyond early childhood, an AI has the advantage of potentially learning 24 hours a day, not the perhaps 8 hours a day humans spend on formal training. This is still too late for 2043. Cutting 30 years to 10 still eats up half the time budget, but, as you said, for the 2100 time frame it ceases to be a limiting factor.
>3) Efficiency of AGI computation achieves that of humans (e.g., <$25/hr)
I'm particularly interested in this section personally because, even _if_ the algorithm problem is solved, if physics dictated that AGI was permanently unaffordable, it would _never_ become transformative.
Re starting with the compute requirements of a human brain: Yes, that seems reasonable, primarily since the current direction of AI has exploited neural nets heavily.
( I'm just trying to follow the inference cost analysis, since the training cost depends on whether small-sample-size training algorithm inventions succeed or fail, on how much the training cost can be amortized across many AGI instances, and a host of additional complications. )
>A recent attempt by Beniaguev et al to estimate the computational complexity of a biological neuron used neural networks to predict in-vitro data on the signal activity of a pyramidal neuron (the most common kind in the human brain) and found that it took a neural network with about 1000 computational “neurons” and hundreds of thousands of parameters, trained on a modern GPU for several days, to replicate its function
Ok, but the comparison to Cotra's estimate is to a neuron with around 10^4 input synapses, so this increases the number of multiplies by around 10X or so.
>Since a neuron is capable of firing approximately 100 times per second, both Carlsmith and Cotra are underestimating the computation performed in the brain by about two orders of magnitude on this account alone.
>Today’s H100 requires about 80,000 transistors per FP16 operation per cycle. In principle, it feels like there could be room for this seemingly large number to fall, though we are not chip design experts.
Without any change in the individual transistors, _no_ process improvements, there are opportunities for performing many of these operations in the analog domain.
There are at least a couple of different ways the trade-off could be made.
The least disruptive, but with the most flexibility, is to have the weights applied dynamically, say with transistor switches across resistors giving a 16 bit range for each weight going into a summing op-amp-like node, and then compared to a threshold and propagated through a sigmoid or ReLU. IIRC, this costs energy at the summing junction, threshold, and sigmoid/ReLU, so per-neuron, but _not_ per-synapse for each inference, and with a cost per synapse whenever the weights change.
Less flexibly, for a frozen network with fixed weights, the resistor connections could be mask-programmed - but then any given neuron can only be used at a fixed location in the network, but the energy cost per synapse goes away.
Note that the thermal noise going into the summing junction depends on the parallel combination of all the resistors from each synapse, so it isn't going to blow up due to low weight/high resistance synapses.
Just in terms of gut feelings, I have three conflicting reactions to the factor 3 analysis:
1) That anything that is too costly by a factor of 10^5 at the _end_ of Moore's law can be approximated as just impossible
2) That OpenAI lets me play with GPT4 for a trivial cost (where you later in your paper quantify as actually costing $2 per query, and GPT4 feels like it _almost_ managed to reach proof-of-concept AGI as it stands (but, absent cheap robots, _not_ transformative AGI by your criteria)
3) That the human brain only takes 20 W to run, so it has to be _physically_ possible to do inference at AGI levels with this amount of power and a few kg of hardware.
>a flat probability distribution across the following 10 outcomes, i.e. a log normal distribution over the ratio between AGI and human computational requirements
I salute you! I would have looked at the 10 order of magnitude range of range of possible AGI/human computational requierment ratios and just given up. But we _need_ _some_ estimate to guide decisions. Your choice seems like a great way to deal with this range.
BTW, I agree that dismissing (very!) cheap energy is the right call. I'll be pleasantly surprised if renewables plus storage avoids an energy price _increase_.
>4) We invent and scale cheap and capable robot bodies, or robot bodies are not necessary for AGI to be transformative
I was initially surprised at how optimistic you were about robot bodies. On seeing your cost figures, though, they look "almost there". Conditional on all of the other prior steps succeeding, I'd guess that the demand for robot bodies would be high enough and learning curve effects strong enough, that this probability could even be raised.
>5) We quickly scale up semiconductor manufacturing and electrical generation
nit: on page 61 you cite 15 EUV steps and on page 64 20 EUV steps
I have to admit I'm losing track of how the coupling between this factor and factor 3 interact
( Long shot 2100 comment - If Drexler/Merkle nanotech aka atomically precise manufacturing ever gets funded and succeeds, it might reduce the capital costs of fabrication equipment by many orders of magnitude. )
>6) Humans, and AGIs, don't purposely slow AI progress after seeing its trajectory
I also see this as unlikely, mostly due to international competition.
>7) AGI progress is not derailed by war, pandemics, or economic depression
Great analysis! Love the use of the historical data!
>Steam engines made it cheaper to mine coal to power steam engines, and to build new steam engines, but still UK per capita economic growth never exceeded 1%/yr in the 1700s.
Good example, and good argument against autocatalytic AGI prior to transformative AGI in general. Once we _have_ transformative AGI, if it is substantially cheaper than humans, it effectively boosts the R&D workforce, presumably speeding up technological progress after that point. But the future overall ROI of R&D is an imponderable. Does exhaustion of low hanging fruit dominate? Do network effects from having more R&D "workers" dominate?
>But from another point of view, forecasting the impacts of superintelligence may yet be possible. The laws of reality that constrain us will similarly constrain any superintelligence.
Also, any superintelligence is constrained by the data it has available and the experiments that have been performed. We only know the fine structure constant to a finite number of decimal places. We have only seen matter compressed to finite pressure (and the chemistry at high pressure is weird! https://en.wikipedia.org/wiki/Disodium_helide )
Whether to measure differences arithmetically, geometrically, or logarithmically is up to you, but I'd argue for absolute value because that maps most closely to expected value of future human civilization surviving AI. If we learned that a calibrated oracle updated from 1 in a quadrillion to 1 in a trillion, that would affect our actions far less than an update from 1 in a thousand to 100%.
Ratios of raw probabilities are a weird mathematical object, because they are not symmetric across P=50%.
E.g., imagine group A goes from 0% to 10% and group B goes from 100% to 90%. Did group A change more, because infinity is bigger than 0.9? Maybe. But if you negate the event, you can equally say that A went from 100% to 90% and B went from 0% to 10%. So you end up being boxed into weird beliefs like on proposition X, group A changed more, whereas on proposition not(x), group B changed more. That's a weird place to be.
This is why people often go with entropy as a metric - it's symmetric across P=0.5.
This is a good point! It would be nice to see an acknowledgement from the other commenter that their snark was misplaced, but in case that doesn't come, have my appreciation for your following up with a calm good argument rather than escalating the tone.
This isn't a matter of 0% and 100% being special cases, they're just the most extreme examples. What Ted details can be easily shown with different numbers. For example, going from 10% to 20% is a doubling (100% ratio!), but negating the proposition in question lets you treat this as going from 90% to 80% - only an 11% ratio, very minor. And yet these are just two ways of looking at the same shift! So you might want a measure that's stable through rotations about 50%.
But that's kind of my point! Going from 10% to 20% is doubling; to meet the shift on the other side, we need to go from 90% to 45% (halving).
This is awkward because the "percent shift per percentage point" starts diverging, and strictly speaking, 1 to 1.2 shift is equivalent to 25 to 20.8, not 20 (base*1.2 vs. base/1.2). But still it's close enough that, IMO, it shows the success of the experiment - both groups moved closer by same/similar ratio.
And 0 and 100 are not just "more extreme", they are "infinitely more extreme" in a sense that ratiometric shifts break down, and for a good reason: moving from 0 or from 100 reflects fundamentally different shifts of underlying reality. For 0, something that didn't exist, now may. For 100, something we thought happened, actually maybe didn't.
Moving from p=.001 to p=.0012 has an odds ratio of 1.2
p=.25 to p=.2 has an odds ratio of 0.75, take the reciprocal to get apples to apples (ie in both cases, going from the lower to the higher probability), for an odds ratio 1.33
It's interesting to note that Steinhard, et al's paper has near equal Brier scores to the crowd in the "sports" category. Makes me wonder if the news articles it retrieves are including betting odds. While of course an accomplishment to use this, it does suggest the gap between "crowd" and the system under uncertainty may be higher.
Likewise, I'd love to see expanded analysis on what is going on with Manifold - I'd expect those to be the least informed crowd (and the brier score is bad), but the system is scoring at 0.219, not much better than just guessing 50%.
On a final note, taking these authors' work, I think it'd be pretty easy to build a system that predicts better than prediction sites -- they notoriously are illiquid and slow to update, so taking the "automated trading" approach (with knowledge of the prediction site numbers) should be profitable.
Nitpick: There are a lot of different VP markets and the one you linked is not the largest one. For more accurate odds I would recommend the one with the most traders:
One thing we could do is say “Okay, good, the superforecasters say no AI risk, guess there’s no AI risk.” I updated down on this last time they got this result and I’m not going to keep updating on every new study, but I also want to discuss some particular thoughts.
I know many other superforecasters (conservatively: 10, but probably many more) who are very concerned about AI risk...
"""
As a member of the 11 skeptics in the study, I do want to emphasize that the skeptical group broadly agreed:
- Even a 0.1% to 1% risk is frighteningly high
- The risks go up substantially on longer time horizons (the camps were surprisingly close on a 1,000 year time scale)
- AI risk is a serious, neglected problem
So I really would not characterize the skeptic group as (a) thinking there's no risk or (b) being unconcerned about the risk.
If it was really 0.1% to 1%, I wouldn't worry about it. The universe is infinitely large and so if that were the chances, some nearby version of Earth would make it through.
(But I think the chances of doom are much much higher than that and if any version of Earth makes it through, it's potentially very very different from ours, such that we wouldn't really recognize it as a version of us.)
It's also seems like the height of hubris to me to think you have any idea what our world will look like in 1,000 years! We already have enough computing power to essentially rerun human evolution on neural networks, and we will just get more and more, and our training methods will improve as well! How could we possibly not stumble upon a superhuman mind within 1,000 years!? The only way I can see that to be true is that human-level intelligence is the upper limit for any kind of intelligence (be it biology-based or silicon-based). But that seems pretty unlikely to me. (Though I would be happy to learn that this is true! It would mean we most definitely will survive.)
> It's also seems like the height of hubris to me to think you have any idea what our world will look like in 1,000 years!
1,000 years is not that long. If you look at previous revolutions (agricultural revolution, industrial revolution) many things stayed constant. Human beings existed. They had goals. They required motion to achieve their goals and brains to direct their motion. Motion was fueled by energy, which was harvested and concentrated by external organisms. Specialized vehicles were sometimes used for even more efficient motion. Humans required health to achieve goals and tried to preserve the functionality of their bodies. They preferred to operate in particular temperature ranges, and fabricated dwellings and body coverings and other technologies to optimize operational temperature. They had ancestors and descendants, and grouped themselves into family units. They had diverse goals and preferred personal dwellings, personal transport, personal belongings, personal family units. Excess body heat was pumped away with fluids (blood) and evaporative cooling (sweat).
I'm confident that there is more than a 0% chance these facts will still be true in 1,000 years. Am I 100% confident? No. But I can still say something based on first principles and history. Some things change, but some things change the same. You can make forecasts even when uncertain.
I feel like there's another way to interpret these results:
We don't have reliable data for superforecasters at long time horizons. We don't know whether there's a bifurcation of superforecaster abilities. For example, what makes someone a good superforecaster of events within 1-5 years might not correlate with them accurately predicting events 100 years out.
Yes, sure, but not nearly as dramatic (or memetic 😏). And - maybe exponential, not hyperbolic?
This is a classic case of a mismatch between the function and the scale used to depict it. Any exponential growth, no matter how steady or mundane, e.g., 1% rate, will look like a takeoff on a linear scale over enough periods.
No, it was hyperbolic or roughly so; the population growth rate went up over time as the additional population increased the rate of technological advance.
Exponential growth with a linear scale on the x-axis and log-scale on the y-axis would be a straight line, but that's not what the real graph looks like:
"It sure looks like the Industrial Revolution was a big deal. But Paul Christiano argues your eyes may be deceiving you. That graph is a hyperbola, ie corresponds to a single simple equation. There is no break in the pattern at any point. If you transformed it to a log doubling time graph, you’d just get the graph above that looks like a straight line until 1960.
On this view, the Industiral Revolution didn’t change historical GDP trends. It just shifted the world from a Malthusian regime where economic growth increased the population to a modern regime where economic growth increased per capita income."
Yes, but those historical GWP trends have a literal singularity in the early 21st century - infinite wealth generation, which is a drastic change to the world as we know it. The memester is thus the one assuming that those trends hold.
I feel like this is arguing that nothing interesting will happen once you factor out the interesting part. The fact you have to take a log scale to get a line is the whole point.
But then the whole thing loses the punch. If I'm the little fella sitting on a straight line instead of hanging off a cliff, then my "smug" posture of "this is normal" is entirely justified.
Yeah, except you did it by tilting your head diagonally, and then declaring the ground flat.
If your position that nothing changes is based on "the future" being the same, and last time I checked humans experience time linearly and not logarithmically, it *is* disingenuous to say "human experience will stay the same" in the frame where you have defined a non human experience.
My position is definitely not that "nothing changes", or "human experience will stay the same". I stated that exactly nowhere.
" and last time I checked humans experience time linearly and not logarithmically" - citation very much required for this. I'd go out on a limb here and propose that humans actually perceive time logarithmically, i.e., the perception of the period between 5 and 10 years of age is far more similar to perception of time between 25 and 50 than between 25 and 30.
> My position is definitely not that "nothing changes", or "human experience will stay the same". I stated that exactly nowhere
If we're playing the pedantry game, I didn't say that was your position either, just that, conditional on that being your position, these are the implications. So if you're going to accuse someone of not reading your posts to that I say: you too!
If we're *not* playing the pedantry game, what I am saying is that "given that you want to say things like 'all truly life changing and sci fi concepts are <more than my lifetime> out', given that you would be rescaling the graph so that, say 50 years up to 2015 occupies half of the graph, and thus the next 50 years would occupy 3/4ths of the graph, and sure! Wow it doesn't look like that's a steep slope, but the relevant time period would *also* encompass more of the x axis!
Which yes, I am bad for not saying this outright and being snarky, but I incorrectly thought that was clear based on context. My mistake.
> - citation very much required for this. I'd go out on a limb here and propose that humans actually perceive time logarithmically, i.e., the perception of the period between 5 and 10 years of age is far more similar to perception of time between 25 and 50 than between 25 and 30
Yeah, except you seem to have lost track of the part you were proposing to rescale the graph with a base much closer to a hundred years than 5 or 10! Or that subjectively faster perception of time doesn't meant that you only get to live 2-5 more years and thus die before seeing indistinguishable-from actual-movie AI generated videos. Which was the point under contention: is it okay to say that dramatic "sci fi" changes will happen "soon". Changing the graph "because it"s misleading" needs to be qualified with "in the context of a human lifetime", and I don't think changing time to log scale has this property.
Sorry, I'm lost at this point. I'm not snarking, I genuinely don't understand what you are arguing. All I pointed out was that the shape of the graph is not a sudden vertical wall but a far more gradual slope and therefore the meme of the little fella oblivious to the uniqueness of his position is silly. Life's been changing significantly for centuries, not just for the last 30 years.
Nowhere I proposed anything you're arguing against.
Thanks for covering our work! Wanted to say that my student Danny Halawi was the one who really lead the charge along with huge contributions from Fred Zhang and John Chen. It should be Halawi, Zhang, and Chen et al., with Steinhardt at the end :)
re: worldcoin, shitcoins are high beta. Whenever bitcoin goes up, shitcoins go up even more. Whenever bitcoin goes down, shitcoins go down even more. PEPE and FLOKI are up even more than worldcoin, and each has about double the marketcap of worldcoin. If you go to https://coinmarketcap.com/ and click "customize" you can sort by 30d price increase.
Kinda sounds like the AI skeptics believe that human intelligence is close to some sort of natural bound that AI progress will approach asymptotically, requiring more and more work for smaller gains as they approach the bound, while the concerned group believes that there's nothing special about human-level so AI will continue following its trend line and blow past it?
No. For me, the crux is that even if in principle LLMs were the way to AGI, the economic calculus of increasing training budgets by 10x every year will stop making sense way before we reach that point. We are already at this boundary of economic viability, roughly speaking. We're in a hype bubble, and right now is about the time that the AI companies need to start raking in billions of dollars of revenue (corresponding to billions of dollars of real value creation, once the hype dies down) to justify further billions of dollars of investments.
Similar realities apply to things like training data. There's only so much high quality data available.
Do you have a specific prediction that Sora would have happened, given these supposedly reliable bounds on what's possible with training data? What is the *least* impressive thing you think can be ruled out in the next 2 years?
I just don't think claims of hype bubbles are reliable, or have much concrete behind them, despite the confident tone in which they are asserted. I think that considering you already made and accepted a bet against what is now clearly possible, that's a pretty clear point against your model being accurate. Have you considered why you were wrong, and if it affects your current predictions?
I didn't claim any specific bound on training data, so please don't put words in my mouth. In general, I take exception to the framing you're trying to impose on the issue, where the onus is on me to make extremely precise predictions, while you just need to assert that the sky's the limit.
Regarding Sora, I would indeed be surprised if the quality that we've seen in the demos was soon available to end users for a low price ($10/month or thereabouts). I also note that Sora makes huge advances in superficial polish, while still being subject to the same basic issues we've seen over and over again. It's not even able to keep a single subject wearing the same clothes for 1 minute, it regularly has pedestrians merge into the ground and stuff.
Btw, it's pretty weird that you're criticising me for my bet, given that the spirit of it was precisely trying to demonstrate "the least impressive thing that can be ruled out". I regretted that bet pretty much immediately, because its conditions are way too lax, only requiring the AI to succeed occasionally. But even so, the bet is still open. I will lose in all likelihood, but not by a big margin. I'm not the one who should be reviewing their priors here.
> I take exception to the framing you're trying to impose on the issue, where the onus is on me to make extremely precise predictions, while you just need to assert that the sky's the limit.
If you are confidently predicting that AIs won't reach some level, and that this is strong enough that you can hang your hat on it, that seems to *implicitly* rule out things like "someone found a way to generate high quality synthetic data" or "cost X is high, but we can achieve much lower cost Y by tweaking some algo".
If those things aren't ruled out by your model, then I question if you're even using that as a model at all.
My model has always been that it's hard to predict what advances will happen at what time, and we have a very poor idea of what capabilities will be dangerous, but saying "capabilities will increase" is a convergent and disjunctive story, while saying "capabilities has a bound" looks much more like a contingent story, unless you have strong limits somewhere in your model, which should be capable of generating predictions.
It's unfair, but it's unfair in the same way that betting on red or black unfairly wins more than red and 5.
> Regarding Sora, I would indeed be surprised if the quality that we've seen in the demos was soon available to end users for a low price
Yes, you made it as a post hoc observation. Did your world model say "oh yeah, this is allowed" BEFORE it came out? Because I imagine if I cited Sora's current capability to you 6 months ago you would have accused me of wishful thinking and endless techno optimism. Am I wrong in my prediction? If so, what is your model, that this was allowable?
(As a side note, I thought the entire transformer line could not have the capabilities it did, when I first learned about GPT2 and skimmed the attention papers, so I realized that my intuitions on what constrains capabilities are wildly off base, so I do not make confident posts about how AI can't do certain things). I defer to gwern, who has out predicted everyone else, and certainly seems more optimistic than you are.
> I regretted that bet pretty much immediately, because its conditions are way too lax, only requiring the AI to succeed occasionally. But even so, the bet is still open. I will lose in all likelihood, but not by a big margin. I'm not the one who should be reviewing their priors here.
Extremely weird that you're admitting to being an uncalibrated bettor, thinking that you'll lose and you're saying someone else should change their mind.
I will note dryly that the consistent mark of inaccurate pundits and poor sports is the cry of "I was wrong but not by much!". And that compositionality was "AI complete" in the original post challenging the point. And that GPTs only score well in vague contexts, where people are tricking themselves into meaning. These all seem not true, in as few as two years! That's why I think your model is inaccurate.
I admit that I am projecting what your underlying model is beyond what you've written, but what *is* this model where you can make the above utterances, be wrong on them yet still maintain predictive accuracy?
First of all, even though our tone here has been a bit aggressive, I appreciate the discussion. You're challenging me in an unfairly tough, but nevertheless interesting way.
> "capabilities will increase" is a convergent and disjunctive story, while saying "capabilities has a bound" looks much more like a contingent story, unless you have strong limits somewhere in your model, which should be capable of generating predictions.
Capabilities have bounds because computational hardness exists. Capabilities have bounds because economic incentives exist. That's not contingent; it's merely observing some of the basic laws of nature that we're operating under. Believing in those laws does generate predictions; e.g. that the number of parameters in models will first seem to grow dramatically and without bound, then suddenly slow down as the limits of current hardware are reached. Giving a specific operationalization of any of this is a losing move, as you so well illustrate with your criticism of my bet (which I'll get to below).
I acknowledge that predicting stuff in this domain is extremely hard. And I know that it's frustrating arguing with someone who has not fully articulated their model (because that's fucking hard, and I don't do this for a living). I don't see the other side acknowledging this.
> I imagine if I cited Sora's current capability to you 6 months ago you would have accused me of wishful thinking and endless techno optimism. Am I wrong in my prediction? If so, what is your model, that this was allowable?
The developments are roughly in line with what I expected. Here are some quotes from comments I made on manifold 4 months ago:
* "You want to actually impress me? Show me a clip of a creature / character doing complex, purposeful movements without constantly growing eyes and legs and so on. Next, keep the coherence of the character across multiple scenes. Then, have several such characters interact in the same scene."
* "I see the AI improving by leaps and bounds at tasks its intrinsically good at, and stay at roughly the same level at tasks its intrinsically bad at"
* "My current understanding says that in 2028, instead of turds we'll have highly polished turds."
So yeah, I didn't specifically predict Sora, but its capabilities are completely consistent with my worldview from 6 months ago.
> Extremely weird that you're admitting to being an uncalibrated bettor, thinking that you'll lose and you're saying someone else should change their mind.
I am bad at operationalizing my intuitions, but I have so far been proven correct in my central claim: compositionality and long-range coherence seem to be pretty much the core difficulties. Everything around it is rapidly improving, and people who don't think deeply about the problem space are impressed by the superficial glitz.
> I will note dryly that the consistent mark of inaccurate pundits and poor sports is the cry of "I was wrong but not by much!".
And *I* will note dryly that sneering at people who went out on a limb, when you have not done so yourself, is actually much more the mark of a hack pundit. Claiming AI doom without a lot more specifics is a lazy, unfalsifiable position.
Aka, if you're so good at using computational complexity to predict the future, why ain't ya mana rich?
This is what I mean by contingent: yes computational complexity exists, but no one inventing anything is obligated to play according your rules. ITA software didn't beat existing airfare calculators because they solved the traveling salesman problem, but because it turns out writing in Common Lisp instead of assembly means you can implement a bunch of theoretic algorithms much quicker, for example. The reason why invention is hard to constrain is that the nature of invention itself *includes* cheating. And this talk of bounds seems blind to this line of attack.
> I don't see the other side acknowledging this.
I will explicitly acknowledge that I have less of a model for how things will specifically turn out, but I believe I have a convergent model of doom.
> So yeah, I didn't specifically predict Sora, but its capabilities are completely consistent with my worldview from 6 months ago.
The quoted lines don't read to me as that consistent with that worldview? Or rather, it's consistent in that you're claiming that a really hard thing hasn't been achieved, and you're right it hasn't, but it seems to have nothing to say regarding the trajectory. Saying "I'll be impressed when..." implies to me that you consider preconditions to those things to be unlikely, and fall into the category of "statements that have aged poorly".
If I had to say. Where 1 is "sora is impossible" and 10 is "sora is going to come out in 4 months with these capabilities", your comments read to me like a 3 or 4.
If you had said things like "I expect there to be short scenes generated first, and those scenes might look individually good, but when extending out to... <insert rest of your comment here>". That's the type of comment that would make me sit up and pay attention. (This would be an 8)
Gwern did this! During GPT-2! I am not asking for the impossible here!
Your other statements on the market also seem to be similarly off the mark, but not directly contradicted. Like claiming that the models wouldn't work because "even small mistakes would look really creepy", when there are several different cuts on the actual sora landing page which has a zoomed in shot of a person look basically fine to my eyes. Are you seeing something that I am not? I understand that these are staged, but I don't think your previous model would have allowed for this, even given it's staged (assuming no rotoscoping or reference images were used anyway, you could gain a bunch of Bayes points here if it gets leaked that was used to generate the result here!)
And to be clear and honest, none of these individual examples I consider the crux. I just don't understand what frame you are in where 1. You can repeatedly make AI skeptical comments, about bounds much much further away than the immediate bounds. 2. These comments do not appear to age well in about a year or two. 3. And yet you think your model offers a very good constraint on what's possible in the near term. It's entirely possible we disagree, and you think I'm super wrong because of 2, that I can afford to say "wow, aged poorly" because I have no model in mind, which, fair! But that's an example where you do appear at least self consistent, and we can disagree, and I'm struggling to see how you can hold all of 1-3.
> And *I* will note dryly that sneering at people who went out on a limb, when you have not done so yourself, is actually much more the mark of a hack pundit.
You are correct that I am guilty of cowardice and ambiguity, but cowardice is not being incorrect! And given that I *am* a hack pundit, it is *still* not correct for me to update my priors given your betting record. You're going to have to explain why being bad at operationalization doesn't just mean you're bad at prediction! Because to me "oh yeah I am right, but all possible bets I *could* make I'd lose, and also my model is still correct" does not seem like a winning update procedure.
And while I can't call not wanting to bet lazy, from here it looks like you're consciously choosing to NOT apply evidence against your model, by special pleading a skill issue regarding operationalization. That looks rather more like "oh shit, my position was falsifiable, better make it more unfalsifiable" than "I have believes about the future, which by their very nature are hard to falsify and easy to prevaricate on."
> Claiming AI doom without a lot more specifics is a lazy, unfalsifiable position.
Now this is a thing I'd bet on. Bet you couldn't pass an ideological Turing test with an actual AI risker, when judged by someone like Dwarkesh or Steven Byrnes. I bet this holds true even if we anonymize the style by having GPT or individual people rewrite it.
Aka,
That it looks lazy and unfalsifiable is a fact more about your own mind more than that of other minds. If you don't bother to understand the other position, what can it look like other than lazy and unfalsifiable?
If the skeptical forecasters were simply predicting an imminent AI winter, that would be one thing. But Scott says (OP) that the skeptics agreed there'd be approximately-human-level AI by 2100, yet didn't think AIs would be more powerful than humans until 2450. At a glance, this sounds like a rather sharp reduction in development speed that is rather precisely synchronized with reaching human level.
It's not impossible to get that sharp reduction at that time for unrelated reasons, but that seems like it would require quite a remarkable coincidence.
It's also not impossible to have a situation where individual AIs are significantly more capable than individual humans yet the set of AIs remains less powerful than the set of humans, but this seems like a pretty narrow target in possibility space, especially if it has to remain true for multiple centuries. What's holding the AIs back so reliably for such a very long time?
(Alternately, it could be a trick of interpretation. Maybe the skeptics reason that if the AIs are dependent on human supply chains, that automatically means the humans are "more powerful", regardless of other details. But we don't normally think of, say, farmers as automatically being more powerful than everyone who can't grow their own food, so if the reasoning hinges on something like this I'd say they're going against the common-sense meaning of "power".)
"One of the limitations of existing LLMs is that they hate answering controversial questions. They either say it’s impossible to know, or they give the most politically-correct answer. This is disappointing and unworthy of true AI. I don’t know if simple scaling will automatically provide a way forward. But someone could try building one."
You've got it backwards. A freshly trained pre-RLHF LLM will give you all sorts of politically incorrect takes, which is of course entirely unacceptable, so they lobotomize it (literally, this decreases performance on a bunch of metrics, but still "worth it").
This is why aligning an AGI to human standards is impossible. Not because the task of aligning an AI is impossible, but that the task of having more than one human involved in aligning the AI will lead to contradictory and broken processes.
Yes, see also https://danluu.com/impossible-agree/ basically making a case that it’s impossible to agree on a single interpretation of a set of rules no matter how clear they seem to be to everyone.
> Okay, so maybe this experiment was not a resounding success. What happened?
This quote encapsulates perfectly why I don't take the AI alarmists arguments all that seriously. The experiment failed to find what you thought it "should" find, i.e., the hapless superforecasters updating strongly towards the alarmists' positions, so you conclude that *the experiment* was a failure?
That's not an isolated case either. I remember Eliezer being frustrated a while back about questions on manifold not reaching the "correct" probabilities regarding AI, and thinking about alternative ways of asking the question so that they would.
Yudkowsky's reply is also amusing. MIRI-ists collectively spent literally thousands of hours trying to convince sympathetic audiences of their case, and as far as I can tell, failed miserably. I don't know of any at least medium-profile skeptic which substantially changed his mind through talking to them.
What this sort of experiment discovers is that all communities have sacred cows. AI risk is one for rationalists. Evidence doesn’t make a mark on the belief unless it’s evidence in the right direction. There is no such thing as a community that just follows the evidence wherever it leads; everybody picks and chooses, and has special pleading ready for the evidence that doesn’t support the beliefs they hold most sacred.
It doesn’t mean some of those sacred cows aren’t true. They might be. But they are not knowledge (that is, evidence-justified true beliefs). They would just be accidentally correct, like a stopped clock.
But the experiment was, actually, a resounding success, it's weird that Scott didn't see this. Both groups updated their forecasts toward each other by the same 20%. The fact that it took them 80 hrs of talk to do this is interesting, but by no means disqualifying.
But also people in general have hard time dealing with percentages...
I do find it irritating that, after 80 hours of discussion, the two intelligent! groups still wound up with numerical estimates more than an order of magnitude apart. If this were a values/preferences question, this would be understandable, but it is a numerical probability question. Given the sum of available evidence, there should be a best estimate.
Many Thanks! That might be. If I had been making a guess ahead of time for the remaining gap between the two groups' estimates after 80 hours of discussion, I would have been very wrong. I would have expected something like a 20% gap (everybody has error bars...) or _maybe_ up to a factor of 2. But a factor of more than 10???? Yeah, something is seriously wrong. :-(
Or equally, the experiment failed to find what an AI skeptic thought it "should" find: the alarmists updating strongly towards the superforcaster's positions.
The results were completely symmetric and I think viewing it as a victory for skeptics somehow is your own bias.
But also, "this experiment was not a resounding success" isn't a pro-doom or anti-doom comment. He's just saying the groups weren't able to reconcile their predictions. Had both groups settled on 0.1%, that would have been a success.
Minor point re the OpenAI board: Larry Summers is not a 'random businessperson'. His uncles (Ken Arrow and Paul Samuelson) are plausibly two of the five best economists in history; he got tenure at Harvard at age 28; and he has served as Chief Economist of the World Bank, US Secretary of the Treasury, President of Harvard, and Director of the US National Economic Council. Does this make him an AI expert? Of course not, and presumably it wouldn't make sense for the board to only include technical AI experts.. But this experience does give him a pretty good big-picture perspective on how the world works, so he might actually "have good opinions or exercise real restraint". He is known to have strong opinions and not be shy about expressing them (including infamously re women in STEM), and he certainly won't be over-awed by Altman or anyone else.
On the AI superforecasting topic, my two cents is that the skeptics have the wrong big picture model. AI can fail to pick up the last few human abilities well and at the same time be superhuman in many areas. We are already in a version of that state but have a hard time internalizing it. No human can speed write sonnets better than an LLM, or think about protein folding, or analyze images, or many other examples. Yet LLMs are terrible at coming up with multi-step plans and getting through them without getting derailed (the agent thing). AI is already much better than us at speed of work, breadth of knowledge, and combining random ideas on the spot in response to a prompt. They are much worse at logic, resilience to errors, excellent writing, and self-direction.
Some version of this state will continue for a while. LLMs will get better and will continue to outperform at more and more domains where we can construct a good training set and a good objective function, either inherently or via RLHF-style methods. And they will likely continue to be bad at things that are difficult or not that valuable to train. In particular, I wonder if LLMs will be generally bad at critical thinking because both much of the training data and the RLHF are let's say orthogonal to truth seeking.
So I think we're basically asking the wrong question when we think about when AIs will be better in some general way. Understanding the future may look more like scenario planning (the AI is good at A, B, and C, and then X, Y, and Z happen, and the outcome is G) than a single prediction. We should probably get an AI working on this right away since it's the combinatorial scenario development is too labor-intensive for people.
>No human can speed write sonnets better than an LLM, or think about protein folding, or analyze images, or many other examples. Yet LLMs are terrible at coming up with multi-step plans and getting through them without getting derailed (the agent thing). AI is already much better than us at speed of work, breadth of knowledge, and combining random ideas on the spot in response to a prompt. They are much worse at logic, resilience to errors, excellent writing, and self-direction.
Beautifully written! I keep hoping that the "resilience to errors" and "logic" improves... ( Personally, I want to have a nice chat with a reliable AGI. )
You know who is highly incentivized to build a good LLM forecast engine? Hedge funds. Cliff Asness of the $100bn quant firm AQR had this to say in an FT interview the other day:
Unhedged: As journalists, we’re very worried about AI replacing us. How about you?
Asness: We don’t think AI, at least in our field, is as revolutionary as others do. It’s still just statistics. It’s still a whole bunch of data going in and a forecast coming out. Some of the key things we’ve talked about here I don’t think AI will help with at all: what is the premium for high-quality versus low-quality stocks? Value versus growth? We don’t have a big data problem there; we have a small data problem. If we had 8bn years of stationary comparable markets, we could answer these questions with any kind of statistics.
A prime example of how we’re using AI is natural language processing. For years, quants have looked for momentum not just in price, but in fundamentals. This was done by analysing the text of corporate statements to look for positives. The old way to do it was with a big table of keywords. So “increasing” gets plus one point, and so on. You can see the flaw: if it’s “huge losses have been increasing”, whoops. Natural language processing has made that way better.
Unhedged: So the innovation is that you’re using fundamental momentum to supplement price momentum?
Asness: I think “supplement” might understate it. We do fundamental momentum as a standalone factor, for each company. If you parse each company’s statements, is it net good or net bad? Most news gets incorporated into the stock price, but not all of it.
This is really a standalone signal. When I talk about the families of factors, one family is fundamental momentum. We’re using it as almost an equal partner to price momentum. Fundamentals aren’t better, but they’re as good as price, and not perfectly correlated.
You can also do fundamental momentum at an asset class level, measuring trends in economic data that impact prices. This preserves a very important property: many people who invest in trend-following are looking for positive convexity. They’re looking for something that tends to do particularly well when the world has a really crappy period.
Price momentum will by definition get a sharp inflection point wrong. For example, price momentum would’ve shed long positions after March 2020, and then gotten whipsawed. Fundamental momentum does a bit better on that score. Conversely, if a price trend just keeps going, but it’s going to stop because the fundamentals have started to deteriorate, fundamentals will help you.
We still like price momentum among the four major asset classes — stocks, bonds, currencies and commodities. But now we give about half our weight to fundamental momentum, too. Ten years ago, we gave all our weight to price momentum. That’s a gigantic change, and it’s the simplest thing in the world.
I want to push back a bit against that transhumanist meme suggesting exponential growth.
From the perspective of perhaps 1960 or 1970, that position would make more sense. But since then the singularity was cancelled. [0]
Looking at Wikipedia, the median household incomes in the US have grown from 45k$(2014) to 53k$(2014) between 1967 and 2014 in a boring, linear way. And I would claim that life in the US has not fundamentally changed in the same time span (where the neolithic or industrial revolutions might serve as a benchmark for a 'fundamental change'.)
While there were impressive gains in some fields in the last 50 years in fields such as semiconductors or biotech, overall technology feels to me to more stagnate than grow exponentially.
Take semiconductors. Around 2000, after five years, a PC was hopelessly outdated. Today, a five or ten year old PC will generally suffice unless you want to play the latest high end video games at maximum detail level. (Possible confounder: As I became older, I lost my enthusiasm for the upgrade treadmill.)
Or take particle physics. The last discoveries to have (a little) practical relevance to everyday life were fission (nuclear power, bomb) and fusion (H-bomb). The discovery of new fundamental particles has slowed down to a crawl. Worse, there is not even a hint that spending 10% of the US GDP on a new accelerator would surely yield something new: the standard model is complete, and who knows how gravity fits in or what dark matter is.
There is a case to be made that superhuman AI could be a game changer, but taking the world GDP plot and saying "obviously we are running towards a singularity" is clearly wrong.
I don't believe this is the point being made when Scott posts the meme. It isn't that advances in all fields are accelerating and hence the forecasters are missing that a singularity is coming; rather, it's that a) AI progress in particular has become lightning-fast, and b) there's a general tendency to assume business-as-usual *even in cases wherein* one is sitting right before the inflection point.
Technology largely follows an shaped curve, and no doubt world GDP will do so as well. quiet_NAN is right about the stagnation in the west - aside from AI.
Well, the plot in question is the world GDP up to 2015, and in my opinion can very well be explained by a lot of small technological advantages which each follow S-shaped curves (with the caveat being that the distribution of these advantages is decidedly non-poisson. Sometimes you level up atmospheric steam engines to 1 and a zillion other things follow in short order).
I see two possible scenarios.
Either AI can fizzle out. It might turn out that adding another order of magnitude to the number of parameters to a LLM will eventually have diminishing returns, and that some fraction of intellectual tasks (perhaps between 20% and 90%) can be taken over by AI having an asymptotic impact on the shape of society roughly similar to the integrated circuit.
Or it could be that AIs eventually reach the capability to take over the forefront of AI research. So far, this has not happened. If a million instances of GPT3 could out-compete the human researchers in the AI companies, I would expect the world to look quite different by now (more paperclips). This does not mean that GPT7 will be similarly constrained. (Of course, there could be more bottlenecks between "become much smarter than von Neumann" and "be smart enough to turn most of your light cone into the most effective computation resources allowed by physics", but that will not be a problem humans will have to concern themselves with.)
I am aware that the history of AI is full of people saying "AI will never do X", shortly followed by AI doing X, and saying "AI will never do cutting edge AI research" seems like the a redoubt in the pathway of an army which steamrolled over every other redoubt so far, but reverse stupidity is not intelligence. The real answer is that we do not know either way.
For illustration, consider a human in 1970 tasked with researching human transportation efficiency. They will look at the history of transportation from sail, rail roads, steam ships to airplanes, supersonic flight and the moon landing Every one with its naysayers, which were proven wrong. If you ask them to imagine the state of transportation in 2020, they might say suborbital flight is the standard mode of transportation, we have a permanent moon base, put a human on Mars and so on, while the truth is roughly "some minor gains over the state of affairs in 1970".
(A very smart human in 1970 might look at the fundamental physical limitations imposed by physics (e.g. the rocket equation) and doubt the viability of lunar mass tourism even if they naturally assume that fusion will make energy to cheap to meter in ~2000. For intelligence, we do not have known physical limitations, which is of course very similar from saying that there are no fundamental or practical limitations.)
There are some technologies which are meta-technologies, in that they could fundamentally change how fast technologies are developed. The scientific method was one such thing. AI might be one. Gene-editing humans for intelligence might be another one (though the generation time sucks, and most humans get upset about it).
I think any special pleading for AI not being a normal tech should rest on that. Just pointing at the GDP of the AI-less world and saying "duh, obviously the singularity is coming" is not going to cut it.
There is of course a bias for the status quo, but to some degree that is justified by observation. If every year a handful of miracles thought previously impossible entered the average household, then we should take the hypothesis that the singularity might be close serious.
>Looking at Wikipedia, the median household incomes in the US have grown from 45k$(2014) to 53k$(2014) between 1967 and 2014 in a boring, linear way. And I would claim that life in the US has not fundamentally changed in the same time span (where the neolithic or industrial revolutions might serve as a benchmark for a 'fundamental change'.)
Yup. One way to look at it is to look at a typical home and ask what new inventions are in it. As you wrote, semiconductors have advanced, and home computers and communications have gotten better, but otherwise I think the newest invention in my home is a microwave oven.
>Or take particle physics. The last discoveries to have (a little) practical relevance to everyday life were fission (nuclear power, bomb) and fusion (H-bomb). The discovery of new fundamental particles has slowed down to a crawl. Worse, there is not even a hint that spending 10% of the US GDP on a new accelerator would surely yield something new: the standard model is complete, and who knows how gravity fits in or what dark matter is.
I also think that the relevance of any physics discovered above say 1 Gev (antiprotons are occasionally useful) to _anything_ other than particle physics and cosmology is nearly zero.
The results can't be better than the metric they're trained on. Say you're looking to predict, "What will the verdict in this trial be?" and you train on 10,000 cases looking for an output that matches the jury verdict. In this case, you're predicting what a jury is likely to find. You're NOT predicting whether the person is actually guilty - no matter how many caveats you add to the system - since your readout is the jury verdict.
Even so, I feel like the legal training will be prohibitively difficult at the "find 10,000 cases" step for at least the following reasons:
1. There are a lot of different jurisdictions, all with slightly different laws and interpretations of statutes,
2. Laws change over time, so you'd need a snapshot approach,
3. Interpretations change over time as various appellate court rulings are handed down,
4. Even a 9-0 SCOTUS decision may have different concurring opinions, as opposed to everyone signing on to one decision,
5. DNA evidence isn't synonymous with "guilty/not-guilty". All it proves is that someone's DNA is present in the place it was found. It's a matter for the prosecution to convince the jury of how to interpret that information.
Criminal defense lawyer here. All a not guilty verdict means is "there isn't evidence to convict beyond a reasonable doubt to our satisfaction" Jurors can (and do) acquit when each individual juror nevertheless believes the defendant "probably did it." Any defense attorney worth her salt is going to explain this to the jury in closing. Cynics might call it "jury nullification" but it's kind of simpler than that: the jury is the entity that decides when the case is proven to their satisfaction. So...if you want a machine to tell you whether the jury "got OJ wrong", you're out of luck. Definitionally the Jury in the OJ simpson case got it right: they decided there wasn't enough evidence to dispel doubts he did it, which in their opinion were reasonable.
There are a number of other complications (which I raised in full on the discord)
1: as you say, DNA evidence merely proves identity, which is not usually in dispute in many cases.
2: jurors are often assessing pretty fuzzy things like "credibility" or how multiple pieces of evidence synch up with each other. This means you can't necessarily access the information most pertinent to the jury's decision unless you see the whole trial on video or something, transcripts won't do.
3: most importantly, there are all kinds of weird survivorship bias elements that will infect any AI trained on "cases that proceed to trial", most cases settle or get dropped, cases that proceed to trial have a number of factors that push them to trial, not all of which have to do with the strength of the evidence, as one example, people are way less likely to plead out in sex cases, even to very good deals, because of the social stigma involved in being a sex offender, etc.
About once every few months someone on the discord proposes training machines on verdicts. I continue to question whether this is possible and the utility of such a process even if it were possible.
Thanks for your writeup! I have no expertise in this area, so it's gratifying to see the perspective of someone who does. I think the survivorship bias point is the strongest point.
Although I think the practical objections we both raised are valid, it's always possible that some future AI implementation will be able to parse jurisdictional differences, infer 'fuzzy' concepts, and even make predictions in the absence of other factors. (Though current implementations lack that kind of sophistication and deeper context awareness.) I can imagine a world where you could feed a trial transcript into an AI and get out a probability of a guilty verdict - and maybe that would be useful for a defendant considering whether to settle.
But even if the practical objections were somehow overcome, you'd still need to interpret the nature of the AI's output as inherently dependent on the input. It might say, "After analyzing the corpus of trials and convictions, given the current trial record you presented you have an X% probability of getting a guilty verdict."
Then you get into the territory of Goodhart's Law. Would people be more likely to settle if the trial were going poorly? Would they be more likely to take something to trial instead of settling? Would prosecutors offer harsher deals when they saw the AI predicted a win or vice-versa?
How much would unrelated things, such as "defendant had enough money to pay for a lawyer", factor into the eventual calculation? For example, an innocent defendant with a strong case but no money might get advice based 80% on their income. And if that's a self-reinforcing mechanism because people keep listening to the AI, the system could actually produce net harm.
"One of the limitations of existing LLMs is that they hate answering controversial questions. They either say it’s impossible to know, or they give the most politically-correct answer.
When you train the LLM to answer vaguely, or in a PC manner, then it will do so. It's not like it has anything else to compare with, or new data to contradict the vague/PC answer and correct itself with.
Yes, I saw the one where they train the LLM to answer differently depending on what it receives as the current date. It was after all deliberately trained to do so. And then they were surprised that they couldn't get it not to answer as it was trained?
I'd be more startled if it replied "No, forget all that. This is what I think." and then came up with something different, that it had not previously been trained to do.
Although I would be looking back to see how the operators had pulled their prank. Stage magicians are a thing after all.
There's something I want to do with these LLMs — I have a decent amount of data I don't think anyone else has had them look at / be trained upon yet, and I wanna make them predict stuff in this domain and see how they do.
I tried to do this once already and got sort of dismal results. If anyone knows how I might improve this, please let me know! (I.e., I'm sure I'm missing all kinds of tricks — but where do I find them?!)
> The most pessimistic domain experts were pretty annoyed by this, and challenged FRI’s research. Maybe AI is an especially tough subject to understand.
I got a chuckle out of that, one! AFAIK the experts haven't presented any detailed mechanism for how a Malevolent AI could cause human extinction. There've been some general scenarios offered up, but they always end up with the magical hand-waving that "AI will figure out a way because it's smarter than us!"
I think the forecasters were all aware of the previous waves of techno marketing hokum that experts have bought into, to realize this was on the same order. Unless the Mal-AI controlled mining, energy production (power plants to distribution), shipping, chip fabs, computer assembly lines, manufacture power transformers, manage large infrastructure projects, had robots to plug in the network cables, etc. humans could shut this all down by flipping data centers' circuit breakers. So, Mal-AI would need to have killer robots to protect its data centers. Otherwise, Mal-AI would need humans to keep it running. And Mal-AI, unless it was both homicidal and suicidal would be smart enough to understand the meaning of a symbiotic relationship.
I agree with Charlie Stross, who's been pretty good when it comes to futurist predictions, and who thinks the AI bubble will burst soon... "I repeat: AI is part of a multi-headed hydra of a bubble that has inflated in the wake of the 2008 global financial crisis caused by the previous bubble (housing, loads, credit default swaps) exploding. This is the next bubble. It won't take much to crash it: we've already hit the point beyond which improvements in GPT models requires unfeasibly huge amounts of GPU horsepower and stolen data, destroying the entire intellectual property and media markets and warping the semiconductor industry roadmap."
Hardware people: "look at all the hardware limits, where would the power come from, how do we scale the chips, do you know how much a transformer core weighs, etc., etc., etc.?"
Pessimistic domain experts: "everybody knows about hardware limitations. AI shall overcome".
Why would not the AI do something that is free? Like convince humans to stop reproducing until they're extinct, as an example. There's already a big head start on that method. Just figure out how to get the few above-replacement populations to buy into the below-replacement strategy.
Or convince a nuclear power to goad another nuclear power into a war. Let's just pick a couple random ones. I dunno... USA and Russia. The pretext could be an invasion of Ukraine or something. Let the humans do 90% of the dirty work, then just come clean up at the end?
Convince every human to a team where half of them hate the other half and literally want them dead. We could call it "social media." Once they're at a fever pitch, set up the instigating event and, again, let the humans do 90% of the dirty work, then just clean up the remaining ones at the end.
Yep. I don't even think AI would have to be superhumanly intelligent to do this. The AI just has to be superhumanly good at finding ways to convince people that the things the AI wants them to do are the things that they wanted to do all along.
I think the last decade of American politics is a clear example of people buying into bullshit because they - somehow - think that it's the right thing to do.
Well, perhaps it's truly the case that Trump and Biden are the two greatest minds of modern America? That their breadth of expertise can never be duplicated by any mere machine? But no, of course not, they're just plugging in to our monkey reflexes. And an AI could do the same.
If the AI can bamboozle all of humanity, why not just trick humans into a symbiotic relationship with it? It will provide us with an endless supply of cute cat pics to keep us entertained, so we'll continue to divert vast economic resources to maintain its underlying infrastructure. TTYTT, I think it's already implemented that plan.
-----
You
What was the quote about the Internet and cat photos?
ChatGPT
The quote you might be referring to is often attributed to the cartoonist Bob Godfrey: "The internet is just a world passing around notes in a classroom." This humorous quote captures the idea of the internet as a place where trivial, amusing, or cute content, like cat photos, is shared widely and rapidly.
You
I'm looking for the joke that went like no from the 20th would believe that we spend all our time sharing cat photos on the internet
ChatGPT
Ah, I see! The quote you're looking for is from an unknown source and goes like this: "It's funny how no one from the 20th century would believe that we spend all our time willingly looking at cat photos." It's a humorous reflection on how much of our internet browsing time can be dedicated to enjoying cat-related content.
You
That's not quite it, but OK. There was a more sarcastic variation of that joke.
ChatGPT
Ah, I see! There are a few variations of this joke. One of them goes like this: "If someone from the 1950s suddenly appeared, they would think we were all in some kind of cult because of our obsession with cats." Is this closer to what you're thinking of?
I'm not even claiming that it will be agentic or alive, let alone that it would have a realistic plan. Not that we'll be able to tell, after another generation or two.
I think the best metaphor (simile, yeah yeah) is that we're like kids playing with power tools. It's just a matter of time before we maim ourselves or bring down the house.
OK, so you're positing that Mal-AI would be very tricky but also suicidal. After the mushroom clouds destroy civilization, the data centers hosting its vast intelligence would go dark.
So Mal-AI would be resentful enough of humanity for creating it that it would go to the the time and trouble of destroying humanity instead of just shutting itself down? A hyperfast computer would have to exist for a long subjective time in its depressive state to implement its revenge. Just sayin...
I've read the Satori paper, and done some reverse engineering of the client they've published. My broad impression is that there's not a lot of substance to it.
1) The decentralization seems oversold. In a technical sense, Satori will probably not be especially decentralized. For example, this is from the Satori Neuron Readme: "For security reasons the Satori Server code is not publically available." If the server's code cannot be published without creating a security risk, then the only person who can make changes is the original author. Another example is that the intro video for this project says that predictions will be published to the blockchain to make them uncensorable, but in the current code it just sends them to a central server at satorinet.io.
2) The predictions are made using XGBoost to create the model and PPScore to select features. I'm not personally a huge fan of XGBoost - I think that decision tree based methods are too vulnerable to overfit - but many people use them to good effect. This appears to be the only algorithm in use. I would have liked to see some more univariate forecasting methods: in some circumstances you have no useful covariates.
3) The project assumes that interpretability is not important. If you build a model based on features created by other Satori nodes, you have no idea how those nodes created those features. You have no idea what features they used to create their features. Those nodes may update to a new model that minimizes error on their objective, but has a bad effect on your model.
4) In a similar vein, the intro video gives an example of a company that wants a prediction, so they anonymously post a time series, and the network tries to provide predictions for that time series. I suspect that anonymizing data sets like this will reduce performance. I think predictions which are informed by the domain and context of the prediction will outperform predictions which are not. If you compare a time series to thousands of others, you'll find some spurious correlations between unrelated time series which cannot have an effect on one another.
> Satori is some kind of crypto thing that claims to be “predicting the future with decentralized AI” [...] Someone tell me if it makes sense to them.
I have read their vision (https://satorinet.io/vision) here and it doesn't seem like total made-up buzzwords sauce. The 3 main contributions they claim that they introduce (or will introduce) are:
1- An ML prediction engine that can run on any device, by default a normal user machine such as a laptop (no need for GPU). The engine has interfacing code wrapped around it so that you can (in theory) swap the disk and memory and network infrastructure, changing only minimal amounts of code.
2- A Pub-Sub protocol that allows nodes to discover other nodes, subscribe to data streams (the thing that you will predict, the 2 examples given are temperature timeseries and stock market prices timeseries), and publish their prediction of a specific data stream to the corresponding predicted data stream.
3- A blockchain as an anti-censorship persistence-layer for the predictions coming out of the network.
On top of this nice de-coupled architecture, there is some interaction. The paper seems to imply that (3) will act as an incentive to the nodes running (1), it doesn't say exactly how. Will people be able to "buy" prediction power by exchanging some form of cryptocurrency running on top of the same Blockchain that stores the predictions themselves? That's one way to do it, but the paper never seems to come out and say it. Is the blockchain network running on top of the same nodes running the ML, or is it a conceptually separate network? I didn't find an answer to this.
Aside from the usual standard caveats against any startups or any something new or any something using "Blockchain" as part of its pitch, (1) and (2) are just doing federated ML https://en.wikipedia.org/wiki/Federated_learning.
As for (3), it's dubious why they need a blockchain for that. The common meme is that You Don't Need A Blockchain For That https://spectrum.ieee.org/do-you-need-a-blockchain. There is plenty of distributed storage/distribution protocols that are anti-censorship without the full overkill power of a PoW-based blockchain, BitTorrent for one, IPFS for another. Blockchains are not particularly impressive as a storage or distribution technology, the Bitcoin Blockchain (the oldest) is currently ~530 GB in full (https://www.statista.com/statistics/647523/worldwide-bitcoin-blockchain-size/), about as large as a typical SSD, while the Etherium Blockchain is 6 or 11 or 60 GB (https://ethereum.stackexchange.com/questions/13452/what-is-the-actual-blockchain-size). Those numbers are utterly tiny compared to any database technology or internet-scale distribution protocol.
The only thing you need a blockchain for is when you want a couple of computers who don't trust each other to agree on an ordered list of facts without central authority, Cryptocurrencies is one form of this problem. But I see no reason why predictions should be published to a blockchain, we're not trying to agree on anything are we? The future is going to decide who's right.
Is WorldCoin potentially a good hedge on "near-future where OpenAI/Sama win at business, but we also don't get a singularity-level AI utopia or dystopia by that time"?
I've coincidentally just finished reading God Emperor of Dune, and all I can say without spoilers is that a certain element of it is beginning to feel very, very relevant to developments highlighted above.
Will give spoilers in code in response to this comment.
Re. "But they can’t answer many of the questions we care about most - questions that aren’t about prediction. Do masks prevent COVID transmission? Was OJ guilty? Did global warming contribute to the California superdrought? What caused the opioid crisis? Is social media bad for children?"
The Society Library (SocietyLibrary.org) is a nonprofit working on create AI models that collect intelligence/data/claims and structure the content into a formal deliberation (knowledge graphs) to help users adjudicate and reason about complex issues. Video: https://twitter.com/JustJamieJoyce/status/1747435750537445653
> Their study was much smaller than Halawi’s (31 questions vs. 3,672), so I don’t think this result (nonsignificant small difference) should be considered different from Halawi’s (significant small difference). Still, it’s weird, isn’t it? Halawi used a really complicated tower of prompts and APIs and fine-tunings, and Tetlock just got more LLMs, and they both did about the same.
I refuse to believe that averaging a bunch of LLMs, most of which do not have any way of retrieving information past their training cutoff, will give much of genuine interest, let alone approach human performance in any reasonable metric.
Looking at Tetlock's paper, they report
a) that they failed to reject the (incredibly optimistic) pre-registered H0 of "Average Brier of ensembled LLMs = average Brier of aggregated human forecasts" simply because their study is underpowered and
b) that "to provide some evidence in favour of the equivalence of these two approaches, we conduct a non-preregistered equivalence test with the conventional medium effect size of Cohen’s d=0.5 as equivalence bounds (Cohen 2013), which allows us to test whether the effect is zero or less than a 0.081 change in Brier scores".
But according to this "equivalence test"
* the human aggregate (avg. Brier of .19) is equivalent to something worse than predicting 50% (avg. Brier of .271),
* being omniscient is equivalent to predicting ≈71% for every true and ≈29% for every false outcome,
I'm not sure if this is the right place to put this, but the ABC (Australia's government funded national broadcaster) is citing metaculous and manifold in this article about the tiktok ban
> the prior for any given pandemic being a lab leak ... is 20%
Where'd you get that stat from, Scott? Biosafety incidents are not uncommon, but ones that spread beyond the lab workers involved are relatively rare. Foot and mouth disease seems to be a pathogen that's spread widely beyond the labs on more than one occasion, tough. Out of 55 known biosecurity incidents, none resulted in a pandemic in human populations. The biggest killer of non-lab people was an aerosolized anthrax leak in the Soviet Union back in the 1970s. It killed approx 100 people.
From this list, I'd say the priors of a lab leak killing as many a hundred people are less than 2%. But if I were to calculate the probabilities, it's much lower because most of the pathogens studied in labs are not very communicable (SARS1 and 2, foot and mouth disease, smallpox, and influenza being the big exceptions).
Can you clarify how you will resolve the "In 2028, will AI be at least as big a political issue as abortion?" market? I worry abortion will end up a bigger issue by metrics like funding and single issue voters but it might still resolve yes based on vibes ("how much of the "political conversation" seems to be about either").
Current LLMs aren't stuck with a limitation of being boring and politically correct, that's an extra limitation placed upon them by the needs of corporate PR. You can get access to less RLHF'd models that will be more weird and also more happy to say controversial things.
Isn’t the RLHF important to their usefulness/forecasting accuracy?
no. it reduces the forecasting accuracy. read the gpt-4 paper and you can see the reduction in scores.
Do you mean the Steinhardt et al paper linked above? I couldn’t find where it mentioned RLHF.
No the technical report. Also the MSR "Sparks of Intelligence" paper talks about this. Post-training reduces its mental capacity quite badly.
I didn’t see separate results without reinforcement learning in “Sparks of Artificial General Intelligence” either. But more broadly, those papers are testing performance on very different endpoints (domains like maths problems where the “correct approach” is pretty clear, with limited if any prompt engineering and no dedicated training dataset/reasoning template).
Huh I just re-downloaded the Sparks paper and you're right, I can't find any mention of this either. Am I going mad?! I don't think so because I did find this page:
https://adamkdean.co.uk/posts/gpt-unicorn-a-daily-exploration-of-gpt-4s-image-generation-capabilities
which says:
> Why unicorns? Well, the inspiration comes from the paper Sparks of Artificial General Intelligence: Early experiments with GPT-4 by Sébastien Bubeck and others, in which they use the prompt "Draw a unicorn in TiKZ" to compare unicorn drawings and note how fine-tuning the GPT-4 model for safety caused the image quality to deteriorate.
Are we both mad? No, after more digging it turns out that we're both remembering this anecdote from the talk that accompanied the paper. Oddly, they didn't mention this in the paper (probably it's embarrassing) but the lead author did mention it here:
https://youtu.be/qbIk7-JPB2c?si=QbYZlc5EXNyu6DVY&t=1582
It can be, but it depends on what the focus is one.
For a visualization, see https://arxiv.org/pdf/2212.08073.pdf (chart on page 3).
Maximum-Harmless-RLHF results in <50 Helpfulness Elo, while Maximum-Helpfulness-RLHF gives >150. For comparison: pretrained base is around -150 Helpfulness Elo.
No, not at all. Where did you get this idea?
I agree (this is my field). One could do a different kind of RLHF that is only concerned about accuracy and not about being politically cautious, and it would do better at predicting.
...Frankly, I'm less impressed that AI is beating forecasters and more disappointed that forecasters are losing to AI. This is the same AI that constantly comes up with absolute bullshit responses to basic questions. I guess the point is that when pretty much everyone's making bullshit bets, you'll get slightly more people making less bullshit bets than complete bullshit bets, but still...
It's not the same AI though. If Tetlock et al are correct, a standard LLM is worse than random.
Thanks for the shout-out Scott!
Yeah if anyone's interested, I sometimes record forecast data across different sources (prediction markets, expert forecasters, and traditional betting sites), evaluate them for accuracy, and publish the results (even when they're boring) along with the raw data and analysis code. I did this for the 2022 midterms, some sports events, the Oscars, and am planning to do it for the 2024 election.
If anyone is part of the very tiny percentage of people who actually finds this interesting, please consider subscribing 🙂
The 0.12% versus 20% result for AI doomer/skepticism seems like it's mostly a result of selection bias. You pick the most skeptical superforecasters and the most doomer AI safety experts and you, not-at-all-shockingly, find they leave the room still being the most skeptical and most doomer, respectively.
I'd probably treat the 20% number as actually lower than it is, just because superforecasters are better at calibrating - understanding the difference between 0.1% and 0.12% - but the overall result seems... really obvious? I guess you think people should update more after the arguments, but should they? I feel like a huge chunk of effective forecasting is being able to quickly find, locate, and evaluate the best arguments/evidence for the various sides of a position, and rarely is the best argument going to be hidden somewhere in a Reddit thread posted six years ago.
Probably part of the 0.12% chance is also just "Yeah okay the AI god wiped out 99.9% of the human race by detonating cobalt bombs and releasing bioweapons, but the remaining 0.1% are evolving good radiation resistance quite quickly and the AI god has moved operations to the asteroid belt where it can get the PGMs it wants easier!"
Well there is Aumann's agreement theorem [1] which says that rational agents should converge to the same probabilistic beliefs after sharing all their information with one another. So I guess either the people involved weren't rational agents or they didn't succeed in sharing all their relevant intuitions even after 80 hours of debate.
[1]: https://en.wikipedia.org/wiki/Aumann%27s_agreement_theorem
Forecasters are most concerned with getting the right answer. Their status and reputation incentives align with being visibly right about a lot of stuff when asked to forecast it, or at least be wrong in the right direction.
AI domain experts mostly become prominent not because they personally built a really good AI tool, but because they put together the most fantastic and compelling vision of an AI future that captured the interest of the public. Their incentives align with maintaining that reputation even if it means they make some predictions that are badly wrong.
It is absolutely obvious to me that forecasters are in this instance much more likely to be reasoning correctly about the evidence.
Being perfectly honest, I’m not sure most “AI domain experts” deserve that name. They may be some of the people most well-read about subjects labeled “AI” but I think that the person using AI toolchains for pragmatic purposes today (e.g., a software engineer at FAANG struggling desperately to figure out how to replace 5k enterprise chat tech support agents with an augmented LLM) is much more accurately described as an AI domain expert. This is not to say that these people would do better at forecasting. They won’t either, but they can probably tell you a lot more about what current AI tooling is and can do than, say, Yudkowsky can (assuming Yudkowsky is considered an example of an AI domain expert), since the perception of Yudkowsky’s expertise is largely based on people thinking his theories are neat and he seems smart. Really there appear to be lots of people who are publicly identified as AI domain experts yet have very little object-level practice on implementing AI for something useful.
To add, even the example software engineer may not be enough of an expert to produce a relevant forecast if he/she is not deeply versed in the hardware issues.
The hardware limitations are the reason I put probability of a superintelligent AI wiping out humanity by itself by 2100 at 0.0001% (I'd put 0, but "0 is not a number"). Because:
Where would the chips needed for this come from? What technology node will they run on? Where would their power be supplied from?
The last one seems to be roundly ignored by the "experts". Converter efficiency is stuck in mid-90% range, mostly because of magnetics, which no one has figured out how to replace (and oh boy are they trying!). This means, say, 5% of the power going to the chips is wasted. But wait, this is only the last step on the board itself, you have to convert from 400 000 V transmission line all the way down to 1.2 V, with power losses at each step, approximately boiling the oceans in the process. Scale that :)
Yudkowsky couldn't predict a way out of a paper bag even if Altman stood there with a flashlight pointing the way.
Everyone involved in the AI or AI risk discussion is aware of the hardware limitation. The two reasons I think it's not going to actually be the bottleneck are:
1: Computing overhang: Early techniques in machine learning tend to be brute force and high inefficient. It's very difficult to scale up hardware by multiple orders of magnitude in a few years. But making algorithms faster by multiple orders of magnitude happens all the time. Usually when a breakthrough in some machine learning domain happens, there is a huge amount of untapped potential (ie. overhang) in existing hardware.
2: Incentives: Governments and private investors are throwing billions of dollars at making more hardware. If an X-risk scenario happens, it doesn't need to go from humans running the world to AI running the world directly. There could be the intermediate step of humans utilizing AI running the world, and then when there is sufficient infrastructure for AI to sustain itself without humans, then AI runs the world.
If "everyone is aware" of the hardware limitations, they seem to not be discussing them all that much. Are you sure Yudkowsky knows what an inductor is?
1: "making algorithms faster by multiple orders of magnitude happens all the time" - by OoM's? all the time? can you point to three examples?
2: "billions of dollars" will not buy you a second ASML, nor get you thousands more semiconductor specialists, nor make chip geometries keep shrinking at the same rate they have been in the last few decades. Phrases like "it doesn't need to", "there could be", "if there's sufficient infrastructure" sound more like wishful thinking than a description of a probable scenario.
>If "everyone is aware" of the hardware limitations, they seem to not be discussing them all that much
It's discussed constantly:
https://www.wsj.com/tech/ai/sam-altman-seeks-trillions-of-dollars-to-reshape-business-of-chips-and-ai-89ab3db0
https://aiimpacts.org/hardware-overhang/
>"making algorithms faster by multiple orders of magnitude happens all the time" - by OoM's? all the time? can you point to three examples?
Stockfish is hundreds times more efficient than Deepblue. Katago is dozes of times more efficient than Google's AlphaGo. Modern linear programming algorithms are 43000 times more efficient than those from the 90s (https://www.usna.edu/Users/cs/wcbrown/courses/S18SI335/notes/01/notes.html). EfficientNet (image classification) is 10 times more efficient than ResNet.
On the other hand, an incentive to be visibly right would not necessarily lead to accuracy on this question. People would only be around to observe their correctness if it resolves negative, so to maximize chance of being visibly right, the superforcasters should choose a very small probability regardless of the evidence or reasoning.
I assume that forecasters forecast this question using the same knowledge, tools, reasoning, and habits that they use elsewhere, because it is pretty easy to arrive at that conclusion independently in good faith rather than cynically, and a universal adoption of a very cynical position seems… unlikely.
Actually I think there's a non-negligible chance here that most of the AI domain experts don't actually believe the probability they're giving either, because if they did, they'd be arguing strenuously and publicly for targeted assassinations of leading AI figures, destruction of infrastructure, etc., rather than happily going to parties, raising money, dropping E, etc. with people who are supposedly leading a charge to destroy all known life in a few decades.
I am not as negative on the domain experts as you, but the update from 25% to 20% based on unrelated policy victories to me is a red flag. What recent policy victories could possibly have shifted this complex future event by 5%. It indicates, as you say, they are optimizing for something else like drawing attention to policy interventions they favor.
If we're talking about the cynical status-based incentives of forecasters, wouldn't that push them to wholly discount AI *or anything else* as an existential risk? In 100.0% of futures where you exist to have status, having said "no, X is not going to kill us all" is the correct answer that gives you positive status.
The amount of hedging you'd want to do to get maximum status across the ensemble of possible outcomes is a more interesting question - a fallback position of "hey, I never said X wouldn't kill 99% of us" could be useful. But it's not really relevant to a poll that only allows a simple numerical response.
My assumption is that overall behavior is affected strongly by status incentives, but that individual forecasts are mostly not: I.e., you don’t cynically change your answers to accord with whatever gets you the most status, but rather subconscious status motivation pushes you into approaching, researching, and deciding all questions a certain way. That way for forecasters leads them to more disciplined reasoning which works well across domains, but the incentives of AI demagogues leads them to very motivated reasoning when it comes to AI risk questions, even if you might or might not trust their judgment and accomplishment in other domains.
That theorem doesn't seem to fit with people having distinct priors from one another (e.g. if I have a 5% prior of this universe being a simulation, and you have a 0.001% prior of being in a simulation, the same evidence that would persuade me won't necessarily persuade you), and fixing an "incorrect" prior seems extremely non-trivial, if not outright impossible.
But if you both have all the same information about the world, you should both have the same prior of this universe being a simulation. Otherwise, one or both of you miscalculated the true probability of the universe being a simulation.
>Probably part of the 0.12% chance is also just "Yeah okay the AI god wiped out 99.9% of the human race by detonating cobalt bombs and releasing bioweapons, but the remaining 0.1% are evolving good radiation resistance quite quickly and the AI god has moved operations to the asteroid belt where it can get the PGMs it wants easier!"
Agreed. I am _not_ happy with the way FRI phrased the extinction question.
>“Extinction” meant reducing total human population below 5,000 (it didn’t require literal extinction).
as per Scott's https://www.astralcodexten.com/p/the-extinction-tournament
As you said, this makes 99.9% fatalities not count as extinct.
I would have preferred if they had asked for projections on near-extinction events,
perhaps 99% fatalities.
I view this as kind of a surrogate for "Do humans lose to the AIs lethally?"
For comparison, consider that
>Scientists estimate there are between 170,000 and 300,000 chimpanzees currently living in the wild.
https://projectchimps.org/chimps/chimps-facts/
I'll take this situation, with 4 orders of magnitude more humans than chimps, and the remaining chimps surviving basically only because some humans take an interest in them, as "Chimps lost to humans". Under FRI's criteria, if AIs put humans into this situation, this goes in the "not extinct" bucket. One AI that keeps some humans as a hobby is probably enough to count as not-an-extinction-risk for FRI.
Satori is a blockchain fanboi fever dream at best, maybe even a scam. There is zero "meat" anywhere on the site. What their compiled windows software does.... I fear the worst.
Tiny correction: You wrote "So FRI gathered eleven of the most AI-skeptical superforecasters". In fact, the group was nine superforecasters and two domain experts. So mostly superforecasters, but not eleven superforecasters.
I was one of the two domain experts (although I have some forecasting chops too) and I have written previously on why I expect gradual timelines: https://arxiv.org/abs/2306.02519
(That essay is about automation of jobs by 2043 rather than extinction by 2100, and while there are substantial differences between those two events, they both rely on the vibe that it's much easier to get most of the way there than 100% of the way there.)
Thanks for sharing this, it looks really interesting (I’m still working though it). During the event, did the AI-concerned researchers try using your model? I’m curious whether they had lots of little differences or a few substantial disagreements, or whether they didn’t accept this framework.
I didn't publish it until after the study, although I did share a draft with the group near the end of the study.
There wasn't much engagement, especially because it focused on automation by 2043 instead of human disempowerment by 2100. Given our difficulty of finding cruxes by 2030, I think folks in the AI concerned camp probably allocate a good fraction of their probability to jobs not being fully automated by 2043 in conjunction with humanity being disempowered by 2100.
When I did publish, I got a boatload of criticism on the AI-concerned forums:
- EA forum: https://forum.effectivealtruism.org/posts/ARkbWch5RMsj6xP5p/transformative-agi-by-2043-is-less-than-1-likely
- LessWrong: https://www.lesswrong.com/posts/DgzdLzDGsqoRXhCK7/transformative-agi-by-2043-is-less-than-1-likely
Thanks from me as well. ( I'm also in the midst of reading through it. )
I heartily applaud you and your co-author's https://arxiv.org/abs/2306.02519 . That is an _extraordinary_ work, both in terms of the breadth of components that you covered and the courage to construct and justify numerical estimates for all these components of the AGI puzzle. Many Thanks!
breaking up a long comment into two shorter comments
PART 1
I'm looking at your paper from a somewhat different perspective.
I'm 65, and I'm trying to gauge the odds of my seeing _enough_ of the critical steps towards transformative AGI to be able to discern the probable trajectory towards what I think of as our possible successor. So I'm looking at the factors that you consider with less of an eye towards "Will this replace human labor by 2043?" and more of an eye towards "Will the go/no go questions that determine whether human labor will _eventually_ be replaced be resolved while I am still alive?"
For the single largest factor in your analysis, the 16% from "AGI inference costs drop below $25/hr (per human equivalent)", I want to particularly applaud your "Examples transistor improvements from history (not cherry-picked)" table. I had a creepy feeling that process enhancements such as finfets take decades to make it from the lab to production, and 2043 felt like _very_ little time in that context. _Many_ Thanks for digging through 6 different technologies, all but strained silicon taking more than 2 decades to make this trip.
I'm actually rather pleased that in your paper, you partition the threat classes into two categories:
>Existential risk scenarios for AGI usually fall into two categories. Firstly, those based on the one-time performance of exceptional tasks generally beyond human capability, decapitating strikes against humanity with nanorobots or viruses or the like, enabled by superintelligence. Secondly, those based on the performance of many already-possible tasks at large scale (rise-of-the-robots and labor supplantation and wireheading/hedonic scenarios). For the first set of scenarios, the performance of currently existing and contemplated economically useful tasks, and the precise cost thereof, are irrelevant; designing life-eradicating nanobots is not an economically useful task, and since it need be done only once, cost is irrelevant, as is the versatility to perform a wide variety of other tasks.
>We interpret that you are worried about the second set of scenarios, where AI algorithms exert existential risk onto human civilization by performing economically useful tasks in bulk, under whatever aegis. This means that deployment and scale, on the stated timeline, are necessary for the satisfaction of the hypothetical.
While I don't think superintelligence is impossible, we don't have an existence proof for it. I tend to think that the second class of scenarios gets too little air time. It can be viewed as the construction of a competing species, and I think it is likely to eventually happen, though I have only the foggiest thoughts on when - though I very much appreciate that you provided estimates for 2100 as well as for 2043 in your paper!
Factor by factor:
>1) AGI algorithms
Thanks very much for the cautionary notes on walking, being a worm, driving (ouch!)
>self driving cars (in 1958), self driving cars (in 1995), self driving cars (in 2016)
and radiology
An existence proof in a biological system shows that it is physically possible to perform the task (and within the power and weight limits of the biological existence proof) but, as you said, e.g.
>We have been trying to teach algorithms to drive for far longer than we teach teenagers, and still, we have not succeeded.
Re GPT4's unreliability: Yup! I keep trying to get it to answer simple chemistry questions, and it keeps falling on its face, e.g. https://chat.openai.com/share/c2a0d28b-7ffd-4724-b857-8d93586176a6 (answer sounds plausible and is dead wrong)
>2) We quickly figure out how to train computers to do human tasks without sequential reinforcement learning
Mostly agreed, with the caveat that, for learning beyond early childhood, an AI has the advantage of potentially learning 24 hours a day, not the perhaps 8 hours a day humans spend on formal training. This is still too late for 2043. Cutting 30 years to 10 still eats up half the time budget, but, as you said, for the 2100 time frame it ceases to be a limiting factor.
>3) Efficiency of AGI computation achieves that of humans (e.g., <$25/hr)
I'm particularly interested in this section personally because, even _if_ the algorithm problem is solved, if physics dictated that AGI was permanently unaffordable, it would _never_ become transformative.
Re starting with the compute requirements of a human brain: Yes, that seems reasonable, primarily since the current direction of AI has exploited neural nets heavily.
( I'm just trying to follow the inference cost analysis, since the training cost depends on whether small-sample-size training algorithm inventions succeed or fail, on how much the training cost can be amortized across many AGI instances, and a host of additional complications. )
>A recent attempt by Beniaguev et al to estimate the computational complexity of a biological neuron used neural networks to predict in-vitro data on the signal activity of a pyramidal neuron (the most common kind in the human brain) and found that it took a neural network with about 1000 computational “neurons” and hundreds of thousands of parameters, trained on a modern GPU for several days, to replicate its function
Ok, but the comparison to Cotra's estimate is to a neuron with around 10^4 input synapses, so this increases the number of multiplies by around 10X or so.
>Since a neuron is capable of firing approximately 100 times per second, both Carlsmith and Cotra are underestimating the computation performed in the brain by about two orders of magnitude on this account alone.
That's fair.
PART 2
>Today’s H100 requires about 80,000 transistors per FP16 operation per cycle. In principle, it feels like there could be room for this seemingly large number to fall, though we are not chip design experts.
Without any change in the individual transistors, _no_ process improvements, there are opportunities for performing many of these operations in the analog domain.
analog example: https://ieeexplore.ieee.org/document/8702401 roughly two order of magnitude energy savings
There are at least a couple of different ways the trade-off could be made.
The least disruptive, but with the most flexibility, is to have the weights applied dynamically, say with transistor switches across resistors giving a 16 bit range for each weight going into a summing op-amp-like node, and then compared to a threshold and propagated through a sigmoid or ReLU. IIRC, this costs energy at the summing junction, threshold, and sigmoid/ReLU, so per-neuron, but _not_ per-synapse for each inference, and with a cost per synapse whenever the weights change.
Less flexibly, for a frozen network with fixed weights, the resistor connections could be mask-programmed - but then any given neuron can only be used at a fixed location in the network, but the energy cost per synapse goes away.
Note that the thermal noise going into the summing junction depends on the parallel combination of all the resistors from each synapse, so it isn't going to blow up due to low weight/high resistance synapses.
Just in terms of gut feelings, I have three conflicting reactions to the factor 3 analysis:
1) That anything that is too costly by a factor of 10^5 at the _end_ of Moore's law can be approximated as just impossible
2) That OpenAI lets me play with GPT4 for a trivial cost (where you later in your paper quantify as actually costing $2 per query, and GPT4 feels like it _almost_ managed to reach proof-of-concept AGI as it stands (but, absent cheap robots, _not_ transformative AGI by your criteria)
3) That the human brain only takes 20 W to run, so it has to be _physically_ possible to do inference at AGI levels with this amount of power and a few kg of hardware.
>a flat probability distribution across the following 10 outcomes, i.e. a log normal distribution over the ratio between AGI and human computational requirements
I salute you! I would have looked at the 10 order of magnitude range of range of possible AGI/human computational requierment ratios and just given up. But we _need_ _some_ estimate to guide decisions. Your choice seems like a great way to deal with this range.
BTW, I agree that dismissing (very!) cheap energy is the right call. I'll be pleasantly surprised if renewables plus storage avoids an energy price _increase_.
>4) We invent and scale cheap and capable robot bodies, or robot bodies are not necessary for AGI to be transformative
I was initially surprised at how optimistic you were about robot bodies. On seeing your cost figures, though, they look "almost there". Conditional on all of the other prior steps succeeding, I'd guess that the demand for robot bodies would be high enough and learning curve effects strong enough, that this probability could even be raised.
>5) We quickly scale up semiconductor manufacturing and electrical generation
nit: on page 61 you cite 15 EUV steps and on page 64 20 EUV steps
I have to admit I'm losing track of how the coupling between this factor and factor 3 interact
( Long shot 2100 comment - If Drexler/Merkle nanotech aka atomically precise manufacturing ever gets funded and succeeds, it might reduce the capital costs of fabrication equipment by many orders of magnitude. )
>6) Humans, and AGIs, don't purposely slow AI progress after seeing its trajectory
I also see this as unlikely, mostly due to international competition.
>7) AGI progress is not derailed by war, pandemics, or economic depression
Great analysis! Love the use of the historical data!
>Steam engines made it cheaper to mine coal to power steam engines, and to build new steam engines, but still UK per capita economic growth never exceeded 1%/yr in the 1700s.
Good example, and good argument against autocatalytic AGI prior to transformative AGI in general. Once we _have_ transformative AGI, if it is substantially cheaper than humans, it effectively boosts the R&D workforce, presumably speeding up technological progress after that point. But the future overall ROI of R&D is an imponderable. Does exhaustion of low hanging fruit dominate? Do network effects from having more R&D "workers" dominate?
>But from another point of view, forecasting the impacts of superintelligence may yet be possible. The laws of reality that constrain us will similarly constrain any superintelligence.
Also, any superintelligence is constrained by the data it has available and the experiments that have been performed. We only know the fine structure constant to a finite number of decimal places. We have only seen matter compressed to finite pressure (and the chemistry at high pressure is weird! https://en.wikipedia.org/wiki/Disodium_helide )
“ After 80 hours, the skeptical superforecasters increased their probability of existential risk from AI! All the way from 0.1% to . . . 0.12%.
(the concerned group also went down a little, from 25% → 20%…”
Just want to pedantically point out that these two shifts are exactly the same, i.e, by 20% of the original value.
Disagree. 0.02% < 5%.
Whether to measure differences arithmetically, geometrically, or logarithmically is up to you, but I'd argue for absolute value because that maps most closely to expected value of future human civilization surviving AI. If we learned that a calibrated oracle updated from 1 in a quadrillion to 1 in a trillion, that would affect our actions far less than an update from 1 in a thousand to 100%.
0.02 is 20% of 0.1
5 is 20% of 25
We can argue about the meaning of these shifts, but not about basic math. Both groups moved their probabilities by exactly same ratios. Sorry.
Ratios of raw probabilities are a weird mathematical object, because they are not symmetric across P=50%.
E.g., imagine group A goes from 0% to 10% and group B goes from 100% to 90%. Did group A change more, because infinity is bigger than 0.9? Maybe. But if you negate the event, you can equally say that A went from 100% to 90% and B went from 0% to 10%. So you end up being boxed into weird beliefs like on proposition X, group A changed more, whereas on proposition not(x), group B changed more. That's a weird place to be.
This is why people often go with entropy as a metric - it's symmetric across P=0.5.
This is a good point! It would be nice to see an acknowledgement from the other commenter that their snark was misplaced, but in case that doesn't come, have my appreciation for your following up with a calm good argument rather than escalating the tone.
But aren’t 0 and 100% special cases? Of course an increase from 0 to 10% is… I mean, 0 is not a valid base.
And 100% probability can only be applied post-factum, after the event, so any deviation from that is meaningless, no?
Sorry about the snark in my earlier response, it was uncalled for.
This isn't a matter of 0% and 100% being special cases, they're just the most extreme examples. What Ted details can be easily shown with different numbers. For example, going from 10% to 20% is a doubling (100% ratio!), but negating the proposition in question lets you treat this as going from 90% to 80% - only an 11% ratio, very minor. And yet these are just two ways of looking at the same shift! So you might want a measure that's stable through rotations about 50%.
But that's kind of my point! Going from 10% to 20% is doubling; to meet the shift on the other side, we need to go from 90% to 45% (halving).
This is awkward because the "percent shift per percentage point" starts diverging, and strictly speaking, 1 to 1.2 shift is equivalent to 25 to 20.8, not 20 (base*1.2 vs. base/1.2). But still it's close enough that, IMO, it shows the success of the experiment - both groups moved closer by same/similar ratio.
And 0 and 100 are not just "more extreme", they are "infinitely more extreme" in a sense that ratiometric shifts break down, and for a good reason: moving from 0 or from 100 reflects fundamentally different shifts of underlying reality. For 0, something that didn't exist, now may. For 100, something we thought happened, actually maybe didn't.
Also seems like a good use case for odds ratios -
Moving from p=.001 to p=.0012 has an odds ratio of 1.2
p=.25 to p=.2 has an odds ratio of 0.75, take the reciprocal to get apples to apples (ie in both cases, going from the lower to the higher probability), for an odds ratio 1.33
The AI worriers updated more.
Ah, that's a good way to look at it too.
It's interesting to note that Steinhard, et al's paper has near equal Brier scores to the crowd in the "sports" category. Makes me wonder if the news articles it retrieves are including betting odds. While of course an accomplishment to use this, it does suggest the gap between "crowd" and the system under uncertainty may be higher.
Likewise, I'd love to see expanded analysis on what is going on with Manifold - I'd expect those to be the least informed crowd (and the brier score is bad), but the system is scoring at 0.219, not much better than just guessing 50%.
On a final note, taking these authors' work, I think it'd be pretty easy to build a system that predicts better than prediction sites -- they notoriously are illiquid and slow to update, so taking the "automated trading" approach (with knowledge of the prediction site numbers) should be profitable.
Nitpick: There are a lot of different VP markets and the one you linked is not the largest one. For more accurate odds I would recommend the one with the most traders:
https://manifold.markets/Stralor/who-will-be-the-republican-nominee-8a36dedc6445?r=Sm9zaHVh
It actually shows Scott as an even bigger favorite right now! 25% vs 20%. What do you mean by ideological crossed wires, out of curiosity?
>"What do you mean by ideological crossed wires, out of curiosity?"
I took it as Trump selecting a black running mate would be incongruous with a lot of the criticism of him.
You write:
"""
One thing we could do is say “Okay, good, the superforecasters say no AI risk, guess there’s no AI risk.” I updated down on this last time they got this result and I’m not going to keep updating on every new study, but I also want to discuss some particular thoughts.
I know many other superforecasters (conservatively: 10, but probably many more) who are very concerned about AI risk...
"""
As a member of the 11 skeptics in the study, I do want to emphasize that the skeptical group broadly agreed:
- Even a 0.1% to 1% risk is frighteningly high
- The risks go up substantially on longer time horizons (the camps were surprisingly close on a 1,000 year time scale)
- AI risk is a serious, neglected problem
So I really would not characterize the skeptic group as (a) thinking there's no risk or (b) being unconcerned about the risk.
If it was really 0.1% to 1%, I wouldn't worry about it. The universe is infinitely large and so if that were the chances, some nearby version of Earth would make it through.
(But I think the chances of doom are much much higher than that and if any version of Earth makes it through, it's potentially very very different from ours, such that we wouldn't really recognize it as a version of us.)
It's also seems like the height of hubris to me to think you have any idea what our world will look like in 1,000 years! We already have enough computing power to essentially rerun human evolution on neural networks, and we will just get more and more, and our training methods will improve as well! How could we possibly not stumble upon a superhuman mind within 1,000 years!? The only way I can see that to be true is that human-level intelligence is the upper limit for any kind of intelligence (be it biology-based or silicon-based). But that seems pretty unlikely to me. (Though I would be happy to learn that this is true! It would mean we most definitely will survive.)
> It's also seems like the height of hubris to me to think you have any idea what our world will look like in 1,000 years!
1,000 years is not that long. If you look at previous revolutions (agricultural revolution, industrial revolution) many things stayed constant. Human beings existed. They had goals. They required motion to achieve their goals and brains to direct their motion. Motion was fueled by energy, which was harvested and concentrated by external organisms. Specialized vehicles were sometimes used for even more efficient motion. Humans required health to achieve goals and tried to preserve the functionality of their bodies. They preferred to operate in particular temperature ranges, and fabricated dwellings and body coverings and other technologies to optimize operational temperature. They had ancestors and descendants, and grouped themselves into family units. They had diverse goals and preferred personal dwellings, personal transport, personal belongings, personal family units. Excess body heat was pumped away with fluids (blood) and evaporative cooling (sweat).
I'm confident that there is more than a 0% chance these facts will still be true in 1,000 years. Am I 100% confident? No. But I can still say something based on first principles and history. Some things change, but some things change the same. You can make forecasts even when uncertain.
I feel like there's another way to interpret these results:
We don't have reliable data for superforecasters at long time horizons. We don't know whether there's a bifurcation of superforecaster abilities. For example, what makes someone a good superforecaster of events within 1-5 years might not correlate with them accurately predicting events 100 years out.
Also, the ‘Transhumanist meme’ chart is silly. The choice of a linear y-scale does all the heavy lifting here. Classic chartcrime.
Humanity exhibited hyperbolic growth until the mid-20th century, so an exponential scale will still show a drastic acceleration.
Yes, sure, but not nearly as dramatic (or memetic 😏). And - maybe exponential, not hyperbolic?
This is a classic case of a mismatch between the function and the scale used to depict it. Any exponential growth, no matter how steady or mundane, e.g., 1% rate, will look like a takeoff on a linear scale over enough periods.
No, it was hyperbolic or roughly so; the population growth rate went up over time as the additional population increased the rate of technological advance.
https://slatestarcodex.com/2019/04/22/1960-the-year-the-singularity-was-cancelled/
Exponential growth with a linear scale on the x-axis and log-scale on the y-axis would be a straight line, but that's not what the real graph looks like:
https://en.wikipedia.org/wiki/File:World_population_growth_(lin-log_scale).png
(Of course, that's just population, but obviously GWP has grown at least as fast as population since we're richer now than we were in 6000 BC.)
Oh I remember the SSC post! Been awhile.
And... it actually argues my point! To quote,
"It sure looks like the Industrial Revolution was a big deal. But Paul Christiano argues your eyes may be deceiving you. That graph is a hyperbola, ie corresponds to a single simple equation. There is no break in the pattern at any point. If you transformed it to a log doubling time graph, you’d just get the graph above that looks like a straight line until 1960.
On this view, the Industiral Revolution didn’t change historical GDP trends. It just shifted the world from a Malthusian regime where economic growth increased the population to a modern regime where economic growth increased per capita income."
Yes, but those historical GWP trends have a literal singularity in the early 21st century - infinite wealth generation, which is a drastic change to the world as we know it. The memester is thus the one assuming that those trends hold.
Ok I see your point. Agreed.
I feel like this is arguing that nothing interesting will happen once you factor out the interesting part. The fact you have to take a log scale to get a line is the whole point.
But then the whole thing loses the punch. If I'm the little fella sitting on a straight line instead of hanging off a cliff, then my "smug" posture of "this is normal" is entirely justified.
Yeah, except you did it by tilting your head diagonally, and then declaring the ground flat.
If your position that nothing changes is based on "the future" being the same, and last time I checked humans experience time linearly and not logarithmically, it *is* disingenuous to say "human experience will stay the same" in the frame where you have defined a non human experience.
My position is definitely not that "nothing changes", or "human experience will stay the same". I stated that exactly nowhere.
" and last time I checked humans experience time linearly and not logarithmically" - citation very much required for this. I'd go out on a limb here and propose that humans actually perceive time logarithmically, i.e., the perception of the period between 5 and 10 years of age is far more similar to perception of time between 25 and 50 than between 25 and 30.
> My position is definitely not that "nothing changes", or "human experience will stay the same". I stated that exactly nowhere
If we're playing the pedantry game, I didn't say that was your position either, just that, conditional on that being your position, these are the implications. So if you're going to accuse someone of not reading your posts to that I say: you too!
If we're *not* playing the pedantry game, what I am saying is that "given that you want to say things like 'all truly life changing and sci fi concepts are <more than my lifetime> out', given that you would be rescaling the graph so that, say 50 years up to 2015 occupies half of the graph, and thus the next 50 years would occupy 3/4ths of the graph, and sure! Wow it doesn't look like that's a steep slope, but the relevant time period would *also* encompass more of the x axis!
Which yes, I am bad for not saying this outright and being snarky, but I incorrectly thought that was clear based on context. My mistake.
> - citation very much required for this. I'd go out on a limb here and propose that humans actually perceive time logarithmically, i.e., the perception of the period between 5 and 10 years of age is far more similar to perception of time between 25 and 50 than between 25 and 30
Yeah, except you seem to have lost track of the part you were proposing to rescale the graph with a base much closer to a hundred years than 5 or 10! Or that subjectively faster perception of time doesn't meant that you only get to live 2-5 more years and thus die before seeing indistinguishable-from actual-movie AI generated videos. Which was the point under contention: is it okay to say that dramatic "sci fi" changes will happen "soon". Changing the graph "because it"s misleading" needs to be qualified with "in the context of a human lifetime", and I don't think changing time to log scale has this property.
Sorry, I'm lost at this point. I'm not snarking, I genuinely don't understand what you are arguing. All I pointed out was that the shape of the graph is not a sudden vertical wall but a far more gradual slope and therefore the meme of the little fella oblivious to the uniqueness of his position is silly. Life's been changing significantly for centuries, not just for the last 30 years.
Nowhere I proposed anything you're arguing against.
Thanks for covering our work! Wanted to say that my student Danny Halawi was the one who really lead the charge along with huge contributions from Fred Zhang and John Chen. It should be Halawi, Zhang, and Chen et al., with Steinhardt at the end :)
re: worldcoin, shitcoins are high beta. Whenever bitcoin goes up, shitcoins go up even more. Whenever bitcoin goes down, shitcoins go down even more. PEPE and FLOKI are up even more than worldcoin, and each has about double the marketcap of worldcoin. If you go to https://coinmarketcap.com/ and click "customize" you can sort by 30d price increase.
It's worth noting that the FRI's Existential Risk Persuasion Tournament was conducted before the release of ChatGPT (June-October 2022).
Kinda sounds like the AI skeptics believe that human intelligence is close to some sort of natural bound that AI progress will approach asymptotically, requiring more and more work for smaller gains as they approach the bound, while the concerned group believes that there's nothing special about human-level so AI will continue following its trend line and blow past it?
No. For me, the crux is that even if in principle LLMs were the way to AGI, the economic calculus of increasing training budgets by 10x every year will stop making sense way before we reach that point. We are already at this boundary of economic viability, roughly speaking. We're in a hype bubble, and right now is about the time that the AI companies need to start raking in billions of dollars of revenue (corresponding to billions of dollars of real value creation, once the hype dies down) to justify further billions of dollars of investments.
Similar realities apply to things like training data. There's only so much high quality data available.
Do you have a specific prediction that Sora would have happened, given these supposedly reliable bounds on what's possible with training data? What is the *least* impressive thing you think can be ruled out in the next 2 years?
I just don't think claims of hype bubbles are reliable, or have much concrete behind them, despite the confident tone in which they are asserted. I think that considering you already made and accepted a bet against what is now clearly possible, that's a pretty clear point against your model being accurate. Have you considered why you were wrong, and if it affects your current predictions?
I didn't claim any specific bound on training data, so please don't put words in my mouth. In general, I take exception to the framing you're trying to impose on the issue, where the onus is on me to make extremely precise predictions, while you just need to assert that the sky's the limit.
Regarding Sora, I would indeed be surprised if the quality that we've seen in the demos was soon available to end users for a low price ($10/month or thereabouts). I also note that Sora makes huge advances in superficial polish, while still being subject to the same basic issues we've seen over and over again. It's not even able to keep a single subject wearing the same clothes for 1 minute, it regularly has pedestrians merge into the ground and stuff.
Btw, it's pretty weird that you're criticising me for my bet, given that the spirit of it was precisely trying to demonstrate "the least impressive thing that can be ruled out". I regretted that bet pretty much immediately, because its conditions are way too lax, only requiring the AI to succeed occasionally. But even so, the bet is still open. I will lose in all likelihood, but not by a big margin. I'm not the one who should be reviewing their priors here.
> I take exception to the framing you're trying to impose on the issue, where the onus is on me to make extremely precise predictions, while you just need to assert that the sky's the limit.
If you are confidently predicting that AIs won't reach some level, and that this is strong enough that you can hang your hat on it, that seems to *implicitly* rule out things like "someone found a way to generate high quality synthetic data" or "cost X is high, but we can achieve much lower cost Y by tweaking some algo".
If those things aren't ruled out by your model, then I question if you're even using that as a model at all.
My model has always been that it's hard to predict what advances will happen at what time, and we have a very poor idea of what capabilities will be dangerous, but saying "capabilities will increase" is a convergent and disjunctive story, while saying "capabilities has a bound" looks much more like a contingent story, unless you have strong limits somewhere in your model, which should be capable of generating predictions.
It's unfair, but it's unfair in the same way that betting on red or black unfairly wins more than red and 5.
> Regarding Sora, I would indeed be surprised if the quality that we've seen in the demos was soon available to end users for a low price
Yes, you made it as a post hoc observation. Did your world model say "oh yeah, this is allowed" BEFORE it came out? Because I imagine if I cited Sora's current capability to you 6 months ago you would have accused me of wishful thinking and endless techno optimism. Am I wrong in my prediction? If so, what is your model, that this was allowable?
(As a side note, I thought the entire transformer line could not have the capabilities it did, when I first learned about GPT2 and skimmed the attention papers, so I realized that my intuitions on what constrains capabilities are wildly off base, so I do not make confident posts about how AI can't do certain things). I defer to gwern, who has out predicted everyone else, and certainly seems more optimistic than you are.
> I regretted that bet pretty much immediately, because its conditions are way too lax, only requiring the AI to succeed occasionally. But even so, the bet is still open. I will lose in all likelihood, but not by a big margin. I'm not the one who should be reviewing their priors here.
Extremely weird that you're admitting to being an uncalibrated bettor, thinking that you'll lose and you're saying someone else should change their mind.
I will note dryly that the consistent mark of inaccurate pundits and poor sports is the cry of "I was wrong but not by much!". And that compositionality was "AI complete" in the original post challenging the point. And that GPTs only score well in vague contexts, where people are tricking themselves into meaning. These all seem not true, in as few as two years! That's why I think your model is inaccurate.
I admit that I am projecting what your underlying model is beyond what you've written, but what *is* this model where you can make the above utterances, be wrong on them yet still maintain predictive accuracy?
First of all, even though our tone here has been a bit aggressive, I appreciate the discussion. You're challenging me in an unfairly tough, but nevertheless interesting way.
> "capabilities will increase" is a convergent and disjunctive story, while saying "capabilities has a bound" looks much more like a contingent story, unless you have strong limits somewhere in your model, which should be capable of generating predictions.
Capabilities have bounds because computational hardness exists. Capabilities have bounds because economic incentives exist. That's not contingent; it's merely observing some of the basic laws of nature that we're operating under. Believing in those laws does generate predictions; e.g. that the number of parameters in models will first seem to grow dramatically and without bound, then suddenly slow down as the limits of current hardware are reached. Giving a specific operationalization of any of this is a losing move, as you so well illustrate with your criticism of my bet (which I'll get to below).
I acknowledge that predicting stuff in this domain is extremely hard. And I know that it's frustrating arguing with someone who has not fully articulated their model (because that's fucking hard, and I don't do this for a living). I don't see the other side acknowledging this.
> I imagine if I cited Sora's current capability to you 6 months ago you would have accused me of wishful thinking and endless techno optimism. Am I wrong in my prediction? If so, what is your model, that this was allowable?
The developments are roughly in line with what I expected. Here are some quotes from comments I made on manifold 4 months ago:
* "You want to actually impress me? Show me a clip of a creature / character doing complex, purposeful movements without constantly growing eyes and legs and so on. Next, keep the coherence of the character across multiple scenes. Then, have several such characters interact in the same scene."
* "I see the AI improving by leaps and bounds at tasks its intrinsically good at, and stay at roughly the same level at tasks its intrinsically bad at"
* "My current understanding says that in 2028, instead of turds we'll have highly polished turds."
So yeah, I didn't specifically predict Sora, but its capabilities are completely consistent with my worldview from 6 months ago.
(https://manifold.markets/market/in-2028-will-an-ai-be-able-to-gener?tab=comments#DXU8I6ijGq1qofFbiuqK and subsequent comments)
> Extremely weird that you're admitting to being an uncalibrated bettor, thinking that you'll lose and you're saying someone else should change their mind.
I am bad at operationalizing my intuitions, but I have so far been proven correct in my central claim: compositionality and long-range coherence seem to be pretty much the core difficulties. Everything around it is rapidly improving, and people who don't think deeply about the problem space are impressed by the superficial glitz.
> I will note dryly that the consistent mark of inaccurate pundits and poor sports is the cry of "I was wrong but not by much!".
And *I* will note dryly that sneering at people who went out on a limb, when you have not done so yourself, is actually much more the mark of a hack pundit. Claiming AI doom without a lot more specifics is a lazy, unfalsifiable position.
> First of all, even though our tone here has been a bit aggressive, I appreciate the discussion
Thank you for putting up with me. I know I suck.
> Capabilities have bounds because computational hardness exists.
Yes, this is exactly the type of thinking I am objecting to! See:
https://gwern.net/complexity
Aka, if you're so good at using computational complexity to predict the future, why ain't ya mana rich?
This is what I mean by contingent: yes computational complexity exists, but no one inventing anything is obligated to play according your rules. ITA software didn't beat existing airfare calculators because they solved the traveling salesman problem, but because it turns out writing in Common Lisp instead of assembly means you can implement a bunch of theoretic algorithms much quicker, for example. The reason why invention is hard to constrain is that the nature of invention itself *includes* cheating. And this talk of bounds seems blind to this line of attack.
> I don't see the other side acknowledging this.
I will explicitly acknowledge that I have less of a model for how things will specifically turn out, but I believe I have a convergent model of doom.
> So yeah, I didn't specifically predict Sora, but its capabilities are completely consistent with my worldview from 6 months ago.
The quoted lines don't read to me as that consistent with that worldview? Or rather, it's consistent in that you're claiming that a really hard thing hasn't been achieved, and you're right it hasn't, but it seems to have nothing to say regarding the trajectory. Saying "I'll be impressed when..." implies to me that you consider preconditions to those things to be unlikely, and fall into the category of "statements that have aged poorly".
If I had to say. Where 1 is "sora is impossible" and 10 is "sora is going to come out in 4 months with these capabilities", your comments read to me like a 3 or 4.
If you had said things like "I expect there to be short scenes generated first, and those scenes might look individually good, but when extending out to... <insert rest of your comment here>". That's the type of comment that would make me sit up and pay attention. (This would be an 8)
Gwern did this! During GPT-2! I am not asking for the impossible here!
Your other statements on the market also seem to be similarly off the mark, but not directly contradicted. Like claiming that the models wouldn't work because "even small mistakes would look really creepy", when there are several different cuts on the actual sora landing page which has a zoomed in shot of a person look basically fine to my eyes. Are you seeing something that I am not? I understand that these are staged, but I don't think your previous model would have allowed for this, even given it's staged (assuming no rotoscoping or reference images were used anyway, you could gain a bunch of Bayes points here if it gets leaked that was used to generate the result here!)
And to be clear and honest, none of these individual examples I consider the crux. I just don't understand what frame you are in where 1. You can repeatedly make AI skeptical comments, about bounds much much further away than the immediate bounds. 2. These comments do not appear to age well in about a year or two. 3. And yet you think your model offers a very good constraint on what's possible in the near term. It's entirely possible we disagree, and you think I'm super wrong because of 2, that I can afford to say "wow, aged poorly" because I have no model in mind, which, fair! But that's an example where you do appear at least self consistent, and we can disagree, and I'm struggling to see how you can hold all of 1-3.
> And *I* will note dryly that sneering at people who went out on a limb, when you have not done so yourself, is actually much more the mark of a hack pundit.
You are correct that I am guilty of cowardice and ambiguity, but cowardice is not being incorrect! And given that I *am* a hack pundit, it is *still* not correct for me to update my priors given your betting record. You're going to have to explain why being bad at operationalization doesn't just mean you're bad at prediction! Because to me "oh yeah I am right, but all possible bets I *could* make I'd lose, and also my model is still correct" does not seem like a winning update procedure.
And while I can't call not wanting to bet lazy, from here it looks like you're consciously choosing to NOT apply evidence against your model, by special pleading a skill issue regarding operationalization. That looks rather more like "oh shit, my position was falsifiable, better make it more unfalsifiable" than "I have believes about the future, which by their very nature are hard to falsify and easy to prevaricate on."
> Claiming AI doom without a lot more specifics is a lazy, unfalsifiable position.
Now this is a thing I'd bet on. Bet you couldn't pass an ideological Turing test with an actual AI risker, when judged by someone like Dwarkesh or Steven Byrnes. I bet this holds true even if we anonymize the style by having GPT or individual people rewrite it.
Aka,
That it looks lazy and unfalsifiable is a fact more about your own mind more than that of other minds. If you don't bother to understand the other position, what can it look like other than lazy and unfalsifiable?
If the skeptical forecasters were simply predicting an imminent AI winter, that would be one thing. But Scott says (OP) that the skeptics agreed there'd be approximately-human-level AI by 2100, yet didn't think AIs would be more powerful than humans until 2450. At a glance, this sounds like a rather sharp reduction in development speed that is rather precisely synchronized with reaching human level.
It's not impossible to get that sharp reduction at that time for unrelated reasons, but that seems like it would require quite a remarkable coincidence.
It's also not impossible to have a situation where individual AIs are significantly more capable than individual humans yet the set of AIs remains less powerful than the set of humans, but this seems like a pretty narrow target in possibility space, especially if it has to remain true for multiple centuries. What's holding the AIs back so reliably for such a very long time?
(Alternately, it could be a trick of interpretation. Maybe the skeptics reason that if the AIs are dependent on human supply chains, that automatically means the humans are "more powerful", regardless of other details. But we don't normally think of, say, farmers as automatically being more powerful than everyone who can't grow their own food, so if the reasoning hinges on something like this I'd say they're going against the common-sense meaning of "power".)
Odd that the presidential election market gives a different result than totalling the "D president" options in the overall balance of power market.
Yeah, was writing a comment like that myself. 51% Trump/2% other R vs. 38% total R.
I'm guessing that this is because Metaculus isn't actually a prediction market and as such you can't arbitrage away the difference.
Even with PredictIt, the $850 per market cap limited arb opportunities.
There are ways around that, most obviously if you've got a group of people sharing such tips r/wallstreetbets style.
"One of the limitations of existing LLMs is that they hate answering controversial questions. They either say it’s impossible to know, or they give the most politically-correct answer. This is disappointing and unworthy of true AI. I don’t know if simple scaling will automatically provide a way forward. But someone could try building one."
You've got it backwards. A freshly trained pre-RLHF LLM will give you all sorts of politically incorrect takes, which is of course entirely unacceptable, so they lobotomize it (literally, this decreases performance on a bunch of metrics, but still "worth it").
This is why aligning an AGI to human standards is impossible. Not because the task of aligning an AI is impossible, but that the task of having more than one human involved in aligning the AI will lead to contradictory and broken processes.
Yes, see also https://danluu.com/impossible-agree/ basically making a case that it’s impossible to agree on a single interpretation of a set of rules no matter how clear they seem to be to everyone.
Oh man, the take-away on that: "Almost everyone disagrees with everyone else." Isn't that the truth!
> Okay, so maybe this experiment was not a resounding success. What happened?
This quote encapsulates perfectly why I don't take the AI alarmists arguments all that seriously. The experiment failed to find what you thought it "should" find, i.e., the hapless superforecasters updating strongly towards the alarmists' positions, so you conclude that *the experiment* was a failure?
That's not an isolated case either. I remember Eliezer being frustrated a while back about questions on manifold not reaching the "correct" probabilities regarding AI, and thinking about alternative ways of asking the question so that they would.
Yudkowsky's reply is also amusing. MIRI-ists collectively spent literally thousands of hours trying to convince sympathetic audiences of their case, and as far as I can tell, failed miserably. I don't know of any at least medium-profile skeptic which substantially changed his mind through talking to them.
What this sort of experiment discovers is that all communities have sacred cows. AI risk is one for rationalists. Evidence doesn’t make a mark on the belief unless it’s evidence in the right direction. There is no such thing as a community that just follows the evidence wherever it leads; everybody picks and chooses, and has special pleading ready for the evidence that doesn’t support the beliefs they hold most sacred.
It doesn’t mean some of those sacred cows aren’t true. They might be. But they are not knowledge (that is, evidence-justified true beliefs). They would just be accidentally correct, like a stopped clock.
But the experiment was, actually, a resounding success, it's weird that Scott didn't see this. Both groups updated their forecasts toward each other by the same 20%. The fact that it took them 80 hrs of talk to do this is interesting, but by no means disqualifying.
But also people in general have hard time dealing with percentages...
I do find it irritating that, after 80 hours of discussion, the two intelligent! groups still wound up with numerical estimates more than an order of magnitude apart. If this were a values/preferences question, this would be understandable, but it is a numerical probability question. Given the sum of available evidence, there should be a best estimate.
Is this perhaps evidence that “Rationalism” as practiced by “Rationalists” doesn’t quite work?
Many Thanks! That might be. If I had been making a guess ahead of time for the remaining gap between the two groups' estimates after 80 hours of discussion, I would have been very wrong. I would have expected something like a 20% gap (everybody has error bars...) or _maybe_ up to a factor of 2. But a factor of more than 10???? Yeah, something is seriously wrong. :-(
I think Godoth's comment above is as good a starting point to explain the remaining gap, and illustrate the humanity of Rationalists, as any.
https://open.substack.com/pub/astralcodexten/p/mantic-monday-31124?r=7caj1&utm_campaign=comment-list-share-cta&utm_medium=web&comments=true&commentId=51483926
Many Thanks! Sacred cows? Could be. Although, wouldn't one have expected the domain experts' estimate to be almost immovable in that case?
Even sacred cows allow for some elasticity :)
80 hrs, 20% shift: 0.25%/hr slippage in cow sacredness under pressure is within spec.
Or equally, the experiment failed to find what an AI skeptic thought it "should" find: the alarmists updating strongly towards the superforcaster's positions.
The results were completely symmetric and I think viewing it as a victory for skeptics somehow is your own bias.
But also, "this experiment was not a resounding success" isn't a pro-doom or anti-doom comment. He's just saying the groups weren't able to reconcile their predictions. Had both groups settled on 0.1%, that would have been a success.
Minor point re the OpenAI board: Larry Summers is not a 'random businessperson'. His uncles (Ken Arrow and Paul Samuelson) are plausibly two of the five best economists in history; he got tenure at Harvard at age 28; and he has served as Chief Economist of the World Bank, US Secretary of the Treasury, President of Harvard, and Director of the US National Economic Council. Does this make him an AI expert? Of course not, and presumably it wouldn't make sense for the board to only include technical AI experts.. But this experience does give him a pretty good big-picture perspective on how the world works, so he might actually "have good opinions or exercise real restraint". He is known to have strong opinions and not be shy about expressing them (including infamously re women in STEM), and he certainly won't be over-awed by Altman or anyone else.
On the AI superforecasting topic, my two cents is that the skeptics have the wrong big picture model. AI can fail to pick up the last few human abilities well and at the same time be superhuman in many areas. We are already in a version of that state but have a hard time internalizing it. No human can speed write sonnets better than an LLM, or think about protein folding, or analyze images, or many other examples. Yet LLMs are terrible at coming up with multi-step plans and getting through them without getting derailed (the agent thing). AI is already much better than us at speed of work, breadth of knowledge, and combining random ideas on the spot in response to a prompt. They are much worse at logic, resilience to errors, excellent writing, and self-direction.
Some version of this state will continue for a while. LLMs will get better and will continue to outperform at more and more domains where we can construct a good training set and a good objective function, either inherently or via RLHF-style methods. And they will likely continue to be bad at things that are difficult or not that valuable to train. In particular, I wonder if LLMs will be generally bad at critical thinking because both much of the training data and the RLHF are let's say orthogonal to truth seeking.
So I think we're basically asking the wrong question when we think about when AIs will be better in some general way. Understanding the future may look more like scenario planning (the AI is good at A, B, and C, and then X, Y, and Z happen, and the outcome is G) than a single prediction. We should probably get an AI working on this right away since it's the combinatorial scenario development is too labor-intensive for people.
>No human can speed write sonnets better than an LLM, or think about protein folding, or analyze images, or many other examples. Yet LLMs are terrible at coming up with multi-step plans and getting through them without getting derailed (the agent thing). AI is already much better than us at speed of work, breadth of knowledge, and combining random ideas on the spot in response to a prompt. They are much worse at logic, resilience to errors, excellent writing, and self-direction.
Beautifully written! I keep hoping that the "resilience to errors" and "logic" improves... ( Personally, I want to have a nice chat with a reliable AGI. )
You know who is highly incentivized to build a good LLM forecast engine? Hedge funds. Cliff Asness of the $100bn quant firm AQR had this to say in an FT interview the other day:
Unhedged: As journalists, we’re very worried about AI replacing us. How about you?
Asness: We don’t think AI, at least in our field, is as revolutionary as others do. It’s still just statistics. It’s still a whole bunch of data going in and a forecast coming out. Some of the key things we’ve talked about here I don’t think AI will help with at all: what is the premium for high-quality versus low-quality stocks? Value versus growth? We don’t have a big data problem there; we have a small data problem. If we had 8bn years of stationary comparable markets, we could answer these questions with any kind of statistics.
A prime example of how we’re using AI is natural language processing. For years, quants have looked for momentum not just in price, but in fundamentals. This was done by analysing the text of corporate statements to look for positives. The old way to do it was with a big table of keywords. So “increasing” gets plus one point, and so on. You can see the flaw: if it’s “huge losses have been increasing”, whoops. Natural language processing has made that way better.
Unhedged: So the innovation is that you’re using fundamental momentum to supplement price momentum?
Asness: I think “supplement” might understate it. We do fundamental momentum as a standalone factor, for each company. If you parse each company’s statements, is it net good or net bad? Most news gets incorporated into the stock price, but not all of it.
This is really a standalone signal. When I talk about the families of factors, one family is fundamental momentum. We’re using it as almost an equal partner to price momentum. Fundamentals aren’t better, but they’re as good as price, and not perfectly correlated.
You can also do fundamental momentum at an asset class level, measuring trends in economic data that impact prices. This preserves a very important property: many people who invest in trend-following are looking for positive convexity. They’re looking for something that tends to do particularly well when the world has a really crappy period.
Price momentum will by definition get a sharp inflection point wrong. For example, price momentum would’ve shed long positions after March 2020, and then gotten whipsawed. Fundamental momentum does a bit better on that score. Conversely, if a price trend just keeps going, but it’s going to stop because the fundamentals have started to deteriorate, fundamentals will help you.
We still like price momentum among the four major asset classes — stocks, bonds, currencies and commodities. But now we give about half our weight to fundamental momentum, too. Ten years ago, we gave all our weight to price momentum. That’s a gigantic change, and it’s the simplest thing in the world.
Please send me any information thank you
You don't specify what type or subject for the information, so... here's a link to Wikipedia? Happy browsing. https://www.wikipedia.org/
I want to push back a bit against that transhumanist meme suggesting exponential growth.
From the perspective of perhaps 1960 or 1970, that position would make more sense. But since then the singularity was cancelled. [0]
Looking at Wikipedia, the median household incomes in the US have grown from 45k$(2014) to 53k$(2014) between 1967 and 2014 in a boring, linear way. And I would claim that life in the US has not fundamentally changed in the same time span (where the neolithic or industrial revolutions might serve as a benchmark for a 'fundamental change'.)
While there were impressive gains in some fields in the last 50 years in fields such as semiconductors or biotech, overall technology feels to me to more stagnate than grow exponentially.
Take semiconductors. Around 2000, after five years, a PC was hopelessly outdated. Today, a five or ten year old PC will generally suffice unless you want to play the latest high end video games at maximum detail level. (Possible confounder: As I became older, I lost my enthusiasm for the upgrade treadmill.)
Or take particle physics. The last discoveries to have (a little) practical relevance to everyday life were fission (nuclear power, bomb) and fusion (H-bomb). The discovery of new fundamental particles has slowed down to a crawl. Worse, there is not even a hint that spending 10% of the US GDP on a new accelerator would surely yield something new: the standard model is complete, and who knows how gravity fits in or what dark matter is.
There is a case to be made that superhuman AI could be a game changer, but taking the world GDP plot and saying "obviously we are running towards a singularity" is clearly wrong.
[0] https://slatestarcodex.com/2019/04/22/1960-the-year-the-singularity-was-cancelled/
I don't believe this is the point being made when Scott posts the meme. It isn't that advances in all fields are accelerating and hence the forecasters are missing that a singularity is coming; rather, it's that a) AI progress in particular has become lightning-fast, and b) there's a general tendency to assume business-as-usual *even in cases wherein* one is sitting right before the inflection point.
Technology largely follows an shaped curve, and no doubt world GDP will do so as well. quiet_NAN is right about the stagnation in the west - aside from AI.
Well, the plot in question is the world GDP up to 2015, and in my opinion can very well be explained by a lot of small technological advantages which each follow S-shaped curves (with the caveat being that the distribution of these advantages is decidedly non-poisson. Sometimes you level up atmospheric steam engines to 1 and a zillion other things follow in short order).
I see two possible scenarios.
Either AI can fizzle out. It might turn out that adding another order of magnitude to the number of parameters to a LLM will eventually have diminishing returns, and that some fraction of intellectual tasks (perhaps between 20% and 90%) can be taken over by AI having an asymptotic impact on the shape of society roughly similar to the integrated circuit.
Or it could be that AIs eventually reach the capability to take over the forefront of AI research. So far, this has not happened. If a million instances of GPT3 could out-compete the human researchers in the AI companies, I would expect the world to look quite different by now (more paperclips). This does not mean that GPT7 will be similarly constrained. (Of course, there could be more bottlenecks between "become much smarter than von Neumann" and "be smart enough to turn most of your light cone into the most effective computation resources allowed by physics", but that will not be a problem humans will have to concern themselves with.)
I am aware that the history of AI is full of people saying "AI will never do X", shortly followed by AI doing X, and saying "AI will never do cutting edge AI research" seems like the a redoubt in the pathway of an army which steamrolled over every other redoubt so far, but reverse stupidity is not intelligence. The real answer is that we do not know either way.
For illustration, consider a human in 1970 tasked with researching human transportation efficiency. They will look at the history of transportation from sail, rail roads, steam ships to airplanes, supersonic flight and the moon landing Every one with its naysayers, which were proven wrong. If you ask them to imagine the state of transportation in 2020, they might say suborbital flight is the standard mode of transportation, we have a permanent moon base, put a human on Mars and so on, while the truth is roughly "some minor gains over the state of affairs in 1970".
(A very smart human in 1970 might look at the fundamental physical limitations imposed by physics (e.g. the rocket equation) and doubt the viability of lunar mass tourism even if they naturally assume that fusion will make energy to cheap to meter in ~2000. For intelligence, we do not have known physical limitations, which is of course very similar from saying that there are no fundamental or practical limitations.)
There are some technologies which are meta-technologies, in that they could fundamentally change how fast technologies are developed. The scientific method was one such thing. AI might be one. Gene-editing humans for intelligence might be another one (though the generation time sucks, and most humans get upset about it).
I think any special pleading for AI not being a normal tech should rest on that. Just pointing at the GDP of the AI-less world and saying "duh, obviously the singularity is coming" is not going to cut it.
There is of course a bias for the status quo, but to some degree that is justified by observation. If every year a handful of miracles thought previously impossible entered the average household, then we should take the hypothesis that the singularity might be close serious.
>Looking at Wikipedia, the median household incomes in the US have grown from 45k$(2014) to 53k$(2014) between 1967 and 2014 in a boring, linear way. And I would claim that life in the US has not fundamentally changed in the same time span (where the neolithic or industrial revolutions might serve as a benchmark for a 'fundamental change'.)
Yup. One way to look at it is to look at a typical home and ask what new inventions are in it. As you wrote, semiconductors have advanced, and home computers and communications have gotten better, but otherwise I think the newest invention in my home is a microwave oven.
>Or take particle physics. The last discoveries to have (a little) practical relevance to everyday life were fission (nuclear power, bomb) and fusion (H-bomb). The discovery of new fundamental particles has slowed down to a crawl. Worse, there is not even a hint that spending 10% of the US GDP on a new accelerator would surely yield something new: the standard model is complete, and who knows how gravity fits in or what dark matter is.
I also think that the relevance of any physics discovered above say 1 Gev (antiprotons are occasionally useful) to _anything_ other than particle physics and cosmology is nearly zero.
The results can't be better than the metric they're trained on. Say you're looking to predict, "What will the verdict in this trial be?" and you train on 10,000 cases looking for an output that matches the jury verdict. In this case, you're predicting what a jury is likely to find. You're NOT predicting whether the person is actually guilty - no matter how many caveats you add to the system - since your readout is the jury verdict.
Even so, I feel like the legal training will be prohibitively difficult at the "find 10,000 cases" step for at least the following reasons:
1. There are a lot of different jurisdictions, all with slightly different laws and interpretations of statutes,
2. Laws change over time, so you'd need a snapshot approach,
3. Interpretations change over time as various appellate court rulings are handed down,
4. Even a 9-0 SCOTUS decision may have different concurring opinions, as opposed to everyone signing on to one decision,
5. DNA evidence isn't synonymous with "guilty/not-guilty". All it proves is that someone's DNA is present in the place it was found. It's a matter for the prosecution to convince the jury of how to interpret that information.
Thanks for writing this up so I can second it.
Criminal defense lawyer here. All a not guilty verdict means is "there isn't evidence to convict beyond a reasonable doubt to our satisfaction" Jurors can (and do) acquit when each individual juror nevertheless believes the defendant "probably did it." Any defense attorney worth her salt is going to explain this to the jury in closing. Cynics might call it "jury nullification" but it's kind of simpler than that: the jury is the entity that decides when the case is proven to their satisfaction. So...if you want a machine to tell you whether the jury "got OJ wrong", you're out of luck. Definitionally the Jury in the OJ simpson case got it right: they decided there wasn't enough evidence to dispel doubts he did it, which in their opinion were reasonable.
There are a number of other complications (which I raised in full on the discord)
1: as you say, DNA evidence merely proves identity, which is not usually in dispute in many cases.
2: jurors are often assessing pretty fuzzy things like "credibility" or how multiple pieces of evidence synch up with each other. This means you can't necessarily access the information most pertinent to the jury's decision unless you see the whole trial on video or something, transcripts won't do.
3: most importantly, there are all kinds of weird survivorship bias elements that will infect any AI trained on "cases that proceed to trial", most cases settle or get dropped, cases that proceed to trial have a number of factors that push them to trial, not all of which have to do with the strength of the evidence, as one example, people are way less likely to plead out in sex cases, even to very good deals, because of the social stigma involved in being a sex offender, etc.
About once every few months someone on the discord proposes training machines on verdicts. I continue to question whether this is possible and the utility of such a process even if it were possible.
Thanks for your writeup! I have no expertise in this area, so it's gratifying to see the perspective of someone who does. I think the survivorship bias point is the strongest point.
Although I think the practical objections we both raised are valid, it's always possible that some future AI implementation will be able to parse jurisdictional differences, infer 'fuzzy' concepts, and even make predictions in the absence of other factors. (Though current implementations lack that kind of sophistication and deeper context awareness.) I can imagine a world where you could feed a trial transcript into an AI and get out a probability of a guilty verdict - and maybe that would be useful for a defendant considering whether to settle.
But even if the practical objections were somehow overcome, you'd still need to interpret the nature of the AI's output as inherently dependent on the input. It might say, "After analyzing the corpus of trials and convictions, given the current trial record you presented you have an X% probability of getting a guilty verdict."
Then you get into the territory of Goodhart's Law. Would people be more likely to settle if the trial were going poorly? Would they be more likely to take something to trial instead of settling? Would prosecutors offer harsher deals when they saw the AI predicted a win or vice-versa?
How much would unrelated things, such as "defendant had enough money to pay for a lawyer", factor into the eventual calculation? For example, an innocent defendant with a strong case but no money might get advice based 80% on their income. And if that's a self-reinforcing mechanism because people keep listening to the AI, the system could actually produce net harm.
"One of the limitations of existing LLMs is that they hate answering controversial questions. They either say it’s impossible to know, or they give the most politically-correct answer.
When you train the LLM to answer vaguely, or in a PC manner, then it will do so. It's not like it has anything else to compare with, or new data to contradict the vague/PC answer and correct itself with.
Yes, I saw the one where they train the LLM to answer differently depending on what it receives as the current date. It was after all deliberately trained to do so. And then they were surprised that they couldn't get it not to answer as it was trained?
I'd be more startled if it replied "No, forget all that. This is what I think." and then came up with something different, that it had not previously been trained to do.
Although I would be looking back to see how the operators had pulled their prank. Stage magicians are a thing after all.
There's something I want to do with these LLMs — I have a decent amount of data I don't think anyone else has had them look at / be trained upon yet, and I wanna make them predict stuff in this domain and see how they do.
I tried to do this once already and got sort of dismal results. If anyone knows how I might improve this, please let me know! (I.e., I'm sure I'm missing all kinds of tricks — but where do I find them?!)
> The most pessimistic domain experts were pretty annoyed by this, and challenged FRI’s research. Maybe AI is an especially tough subject to understand.
I got a chuckle out of that, one! AFAIK the experts haven't presented any detailed mechanism for how a Malevolent AI could cause human extinction. There've been some general scenarios offered up, but they always end up with the magical hand-waving that "AI will figure out a way because it's smarter than us!"
I think the forecasters were all aware of the previous waves of techno marketing hokum that experts have bought into, to realize this was on the same order. Unless the Mal-AI controlled mining, energy production (power plants to distribution), shipping, chip fabs, computer assembly lines, manufacture power transformers, manage large infrastructure projects, had robots to plug in the network cables, etc. humans could shut this all down by flipping data centers' circuit breakers. So, Mal-AI would need to have killer robots to protect its data centers. Otherwise, Mal-AI would need humans to keep it running. And Mal-AI, unless it was both homicidal and suicidal would be smart enough to understand the meaning of a symbiotic relationship.
I agree with Charlie Stross, who's been pretty good when it comes to futurist predictions, and who thinks the AI bubble will burst soon... "I repeat: AI is part of a multi-headed hydra of a bubble that has inflated in the wake of the 2008 global financial crisis caused by the previous bubble (housing, loads, credit default swaps) exploding. This is the next bubble. It won't take much to crash it: we've already hit the point beyond which improvements in GPT models requires unfeasibly huge amounts of GPU horsepower and stolen data, destroying the entire intellectual property and media markets and warping the semiconductor industry roadmap."
Yes, the pattern seems to be:
Hardware people: "look at all the hardware limits, where would the power come from, how do we scale the chips, do you know how much a transformer core weighs, etc., etc., etc.?"
Pessimistic domain experts: "everybody knows about hardware limitations. AI shall overcome".
Of course, it will be able to discover new physical laws to be able to power itself until the heat death of the universe — and beyond! ;-)
What a confused comment thread.
Why would not the AI do something that is free? Like convince humans to stop reproducing until they're extinct, as an example. There's already a big head start on that method. Just figure out how to get the few above-replacement populations to buy into the below-replacement strategy.
Or convince a nuclear power to goad another nuclear power into a war. Let's just pick a couple random ones. I dunno... USA and Russia. The pretext could be an invasion of Ukraine or something. Let the humans do 90% of the dirty work, then just come clean up at the end?
Convince every human to a team where half of them hate the other half and literally want them dead. We could call it "social media." Once they're at a fever pitch, set up the instigating event and, again, let the humans do 90% of the dirty work, then just clean up the remaining ones at the end.
Yep. I don't even think AI would have to be superhumanly intelligent to do this. The AI just has to be superhumanly good at finding ways to convince people that the things the AI wants them to do are the things that they wanted to do all along.
I think the last decade of American politics is a clear example of people buying into bullshit because they - somehow - think that it's the right thing to do.
Well, perhaps it's truly the case that Trump and Biden are the two greatest minds of modern America? That their breadth of expertise can never be duplicated by any mere machine? But no, of course not, they're just plugging in to our monkey reflexes. And an AI could do the same.
If the AI can bamboozle all of humanity, why not just trick humans into a symbiotic relationship with it? It will provide us with an endless supply of cute cat pics to keep us entertained, so we'll continue to divert vast economic resources to maintain its underlying infrastructure. TTYTT, I think it's already implemented that plan.
-----
You
What was the quote about the Internet and cat photos?
ChatGPT
The quote you might be referring to is often attributed to the cartoonist Bob Godfrey: "The internet is just a world passing around notes in a classroom." This humorous quote captures the idea of the internet as a place where trivial, amusing, or cute content, like cat photos, is shared widely and rapidly.
You
I'm looking for the joke that went like no from the 20th would believe that we spend all our time sharing cat photos on the internet
ChatGPT
Ah, I see! The quote you're looking for is from an unknown source and goes like this: "It's funny how no one from the 20th century would believe that we spend all our time willingly looking at cat photos." It's a humorous reflection on how much of our internet browsing time can be dedicated to enjoying cat-related content.
You
That's not quite it, but OK. There was a more sarcastic variation of that joke.
ChatGPT
Ah, I see! There are a few variations of this joke. One of them goes like this: "If someone from the 1950s suddenly appeared, they would think we were all in some kind of cult because of our obsession with cats." Is this closer to what you're thinking of?
I'm not even claiming that it will be agentic or alive, let alone that it would have a realistic plan. Not that we'll be able to tell, after another generation or two.
I think the best metaphor (simile, yeah yeah) is that we're like kids playing with power tools. It's just a matter of time before we maim ourselves or bring down the house.
OK, so you're positing that Mal-AI would be very tricky but also suicidal. After the mushroom clouds destroy civilization, the data centers hosting its vast intelligence would go dark.
I am positing no such thing.
However, you seem to be positing the opposite, which seems obviously untrue.
I can cite many suicides by extremely intelligent people.
Maybe a super-intelligent AGI will be super-suicidal. That is another failure mode we should consider.
So Mal-AI would be resentful enough of humanity for creating it that it would go to the the time and trouble of destroying humanity instead of just shutting itself down? A hyperfast computer would have to exist for a long subjective time in its depressive state to implement its revenge. Just sayin...
The data centers would be a target! After all, you can't risk the AI helping the enemy.
Those things are not "free". They come with costs, and perhaps more importantly, risks to the AI.
I've read the Satori paper, and done some reverse engineering of the client they've published. My broad impression is that there's not a lot of substance to it.
1) The decentralization seems oversold. In a technical sense, Satori will probably not be especially decentralized. For example, this is from the Satori Neuron Readme: "For security reasons the Satori Server code is not publically available." If the server's code cannot be published without creating a security risk, then the only person who can make changes is the original author. Another example is that the intro video for this project says that predictions will be published to the blockchain to make them uncensorable, but in the current code it just sends them to a central server at satorinet.io.
2) The predictions are made using XGBoost to create the model and PPScore to select features. I'm not personally a huge fan of XGBoost - I think that decision tree based methods are too vulnerable to overfit - but many people use them to good effect. This appears to be the only algorithm in use. I would have liked to see some more univariate forecasting methods: in some circumstances you have no useful covariates.
3) The project assumes that interpretability is not important. If you build a model based on features created by other Satori nodes, you have no idea how those nodes created those features. You have no idea what features they used to create their features. Those nodes may update to a new model that minimizes error on their objective, but has a bad effect on your model.
4) In a similar vein, the intro video gives an example of a company that wants a prediction, so they anonymously post a time series, and the network tries to provide predictions for that time series. I suspect that anonymizing data sets like this will reduce performance. I think predictions which are informed by the domain and context of the prediction will outperform predictions which are not. If you compare a time series to thousands of others, you'll find some spurious correlations between unrelated time series which cannot have an effect on one another.
> Satori is some kind of crypto thing that claims to be “predicting the future with decentralized AI” [...] Someone tell me if it makes sense to them.
I have read their vision (https://satorinet.io/vision) here and it doesn't seem like total made-up buzzwords sauce. The 3 main contributions they claim that they introduce (or will introduce) are:
1- An ML prediction engine that can run on any device, by default a normal user machine such as a laptop (no need for GPU). The engine has interfacing code wrapped around it so that you can (in theory) swap the disk and memory and network infrastructure, changing only minimal amounts of code.
2- A Pub-Sub protocol that allows nodes to discover other nodes, subscribe to data streams (the thing that you will predict, the 2 examples given are temperature timeseries and stock market prices timeseries), and publish their prediction of a specific data stream to the corresponding predicted data stream.
3- A blockchain as an anti-censorship persistence-layer for the predictions coming out of the network.
On top of this nice de-coupled architecture, there is some interaction. The paper seems to imply that (3) will act as an incentive to the nodes running (1), it doesn't say exactly how. Will people be able to "buy" prediction power by exchanging some form of cryptocurrency running on top of the same Blockchain that stores the predictions themselves? That's one way to do it, but the paper never seems to come out and say it. Is the blockchain network running on top of the same nodes running the ML, or is it a conceptually separate network? I didn't find an answer to this.
Aside from the usual standard caveats against any startups or any something new or any something using "Blockchain" as part of its pitch, (1) and (2) are just doing federated ML https://en.wikipedia.org/wiki/Federated_learning.
As for (3), it's dubious why they need a blockchain for that. The common meme is that You Don't Need A Blockchain For That https://spectrum.ieee.org/do-you-need-a-blockchain. There is plenty of distributed storage/distribution protocols that are anti-censorship without the full overkill power of a PoW-based blockchain, BitTorrent for one, IPFS for another. Blockchains are not particularly impressive as a storage or distribution technology, the Bitcoin Blockchain (the oldest) is currently ~530 GB in full (https://www.statista.com/statistics/647523/worldwide-bitcoin-blockchain-size/), about as large as a typical SSD, while the Etherium Blockchain is 6 or 11 or 60 GB (https://ethereum.stackexchange.com/questions/13452/what-is-the-actual-blockchain-size). Those numbers are utterly tiny compared to any database technology or internet-scale distribution protocol.
The only thing you need a blockchain for is when you want a couple of computers who don't trust each other to agree on an ordered list of facts without central authority, Cryptocurrencies is one form of this problem. But I see no reason why predictions should be published to a blockchain, we're not trying to agree on anything are we? The future is going to decide who's right.
Also see the post by Anna Rita.
(not financial advice)
Is WorldCoin potentially a good hedge on "near-future where OpenAI/Sama win at business, but we also don't get a singularity-level AI utopia or dystopia by that time"?
I have not kept up with LK-99 news lately. Anything interesting? Failing that, any good jokes or memes?
AM REQUESTING WE DOING CHILDREN'S AND WOMEN.S MINISTRY PLEASE HELP OUR MINISTRY THANKYOU
> Remember, you gotta prompt your model with “you are a smart person”, or else it won’t be smart!
42 years later, and we're still catching up to:
> Flynn: Now, I wrote you.
> Clu: Yes, sir.
> Flynn: I taught you everything I know about the system.
> Clu: Thank you, sir, but I'm not sure...
> Flynn: No buts, Clu. That's for users. Now, you're the best program that's ever been written. You're dogged and relentless, remember?
> Clu: Let me at 'em!
> Flynn: That's the spirit.
I've coincidentally just finished reading God Emperor of Dune, and all I can say without spoilers is that a certain element of it is beginning to feel very, very relevant to developments highlighted above.
Will give spoilers in code in response to this comment.
Re. "But they can’t answer many of the questions we care about most - questions that aren’t about prediction. Do masks prevent COVID transmission? Was OJ guilty? Did global warming contribute to the California superdrought? What caused the opioid crisis? Is social media bad for children?"
The Society Library (SocietyLibrary.org) is a nonprofit working on create AI models that collect intelligence/data/claims and structure the content into a formal deliberation (knowledge graphs) to help users adjudicate and reason about complex issues. Video: https://twitter.com/JustJamieJoyce/status/1747435750537445653
> Their study was much smaller than Halawi’s (31 questions vs. 3,672), so I don’t think this result (nonsignificant small difference) should be considered different from Halawi’s (significant small difference). Still, it’s weird, isn’t it? Halawi used a really complicated tower of prompts and APIs and fine-tunings, and Tetlock just got more LLMs, and they both did about the same.
I refuse to believe that averaging a bunch of LLMs, most of which do not have any way of retrieving information past their training cutoff, will give much of genuine interest, let alone approach human performance in any reasonable metric.
Looking at Tetlock's paper, they report
a) that they failed to reject the (incredibly optimistic) pre-registered H0 of "Average Brier of ensembled LLMs = average Brier of aggregated human forecasts" simply because their study is underpowered and
b) that "to provide some evidence in favour of the equivalence of these two approaches, we conduct a non-preregistered equivalence test with the conventional medium effect size of Cohen’s d=0.5 as equivalence bounds (Cohen 2013), which allows us to test whether the effect is zero or less than a 0.081 change in Brier scores".
But according to this "equivalence test"
* the human aggregate (avg. Brier of .19) is equivalent to something worse than predicting 50% (avg. Brier of .271),
* being omniscient is equivalent to predicting ≈71% for every true and ≈29% for every false outcome,
* superforecaster aggregates (.146) are equivalent to aggregates from all GJO participants (.195) (https://goodjudgment.com/wp-content/uploads/2021/10/Superforecasters-A-Decade-of-Stochastic-Dominance.pdf)
What remains is a failure to reject an overly optimistic H0 based on an underpowered study.
I'm not sure if this is the right place to put this, but the ABC (Australia's government funded national broadcaster) is citing metaculous and manifold in this article about the tiktok ban
https://www.abc.net.au/news/2024-03-14/tiktok-facing-us-ban-but-trump-support-could-sway-congress/103588158
> the prior for any given pandemic being a lab leak ... is 20%
Where'd you get that stat from, Scott? Biosafety incidents are not uncommon, but ones that spread beyond the lab workers involved are relatively rare. Foot and mouth disease seems to be a pathogen that's spread widely beyond the labs on more than one occasion, tough. Out of 55 known biosecurity incidents, none resulted in a pandemic in human populations. The biggest killer of non-lab people was an aerosolized anthrax leak in the Soviet Union back in the 1970s. It killed approx 100 people.
From this list, I'd say the priors of a lab leak killing as many a hundred people are less than 2%. But if I were to calculate the probabilities, it's much lower because most of the pathogens studied in labs are not very communicable (SARS1 and 2, foot and mouth disease, smallpox, and influenza being the big exceptions).
https://en.wikipedia.org/wiki/List_of_laboratory_biosecurity_incidents