Why don't people take the AI prophets of doom more seriously?

The AI prophets of doom have updated their timelines. According to a post from yesterday by the AI Futures Project, superintelligence is coming. Their consensus prediction is a "Superhuman AI Researcher" capable of matching the capabilities of an AI lab employee will, on our current path, occur sometime in 2027. This "SAR" could be used to do "RSI" (recursive self-improvement): in other words, it will be smart enough to make itself smarter, leading to an uncontrollable intelligence explosion. The document states that, if this happens, "we're in a very bad position."

Their website, AI 2027, describes this scenario ("Plan D") more colorfully:

the AI releases a dozen quiet-spreading biological weapons in major cities, lets them silently infect almost everyone, then triggers them with a chemical spray. Most are dead within hours; the few survivors (e.g. preppers in bunkers, sailors on submarines) are mopped up by drones. Robots scan the victims’ brains, placing copies in memory for future study or revival.

Posters from anti-AI activist protesters Stop AI put it more succinctly:

AI WILL KILL US ALL

In reference to AI Future's report, safety researcher Buck Shlegeris wrote a lengthy tweet, wherein he says:

Overall, I think there’s a very real possibility (maybe 20%) that within a year, AI misalignment issues prevent you from going about life normally, because you are dead or enslaved or because of collateral damage related to all this (e.g. because bioweapons are released or major wars have started).

This echoes a comment from earlier this year, where Evan Hubinger, alignment lead at Anthropic, wrote "we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade."

In both cases, there seems to be some serious miscommunication going on. In Hubinger's case, he reports the possibility of human extinction as if it were a fun little factoid, with a tone similar to how one might announce that many Anthropic employees are also fans of Magic: the Gathering. Shlegeris's Tweet (and the report it refers to) has the opposite problem: wary of making bold claims, the threat of existential risk is buried within paragraphs of dry qualifiers. At least the Stop AI people make the point clearly!

I recently went to a book talk for left-wing tech critic Paris Marx's new book Hyperscale, at which I asked why there seems to be a disconnect between the "existential risk" activists (prominent in San Francisco) and the anti-data center or tech-skeptic protestors, who, while also critical of AI, are more focused on the immediate, material concerns of data center construction. My question elicited a laugh from the audience. Outside of the AI doomsayers' small "rationalist-adjacent" community, very few "normies" take them very seriously. In writing, Paris himself has referred to apocalypse scenarios as a "science fictional storyline" intended to hype the capabilities of AI companies' products and minimize the more pedestrian harms.

I am not going to evaluate the truth of AI apocalypse predictions, but, unlike Marx, I am willing to take them at face value: I presume that AI safety researchers are not compromised by industry and that they really do genuinely think superintelligence is immanent and that this may mean a substantial possibility that it kills us all. Why do they have so much trouble convincing people that they believe that?

In a response to Shlegeris's Tweet, New York Times reporter Joe Bernstein questioned the "theory of mind" behind thinking the AI apocalypse is immanent and responding by writing a lengthy, carefully worded tweet about it. I will try my best to provide one. AI doomsayers believe that the risk for AI is serious, and so they are taking this seriously: they have mastered (in their view) the discipline required to keep a cool head, to consider these problems abstractly, and dedicate themselves to carefully studied research of the problem.

To most people, this is, of course, very weird. If I believed there was a 10-50% chance of human extinction next year, I, along with most people, would be out of my fucking mind. For some reason, AI safety researchers, perhaps from inculcating themselves too much with the ideas of a community that values measured, rational thought, have successfully excised the part of themselves that would react like this, able to consider the immanent possibility of human extinction as merely, in an abstract sense, a "very bad position."

Members of protest organizations like Stop AI are, at the very least, more relatable on a human level. These are people who have been arrested for chaining themselves to the gates of the OpenAI headquarters, who lived in homeless shelters when they moved to Oakland in order to dedicate their lives to this activism, their organization mired by infighting and splinter groups. They are fervently dedicated and seem to have been driven a little bit mad.

While you could accuse them of being mad, no one would ever accuse the Stop AI activists of not really believing in the impending AI apocalypse. If I really did think apocalypse was coming soon, I would be at a pretty substantial risk of going mad myself—I wouldn't be making pretty charts about it! Yet somehow, safety researchers, at least in their public communications, speak about apocalypse in precise, guarded, clinical terms. I found Hubinger's tweet in particular to be revealing. It is as if he said, well yes, we all know potential apocalypse is immanent, and I came to terms with it by making the obvious decision of becoming a safety lead at a major AI lab.

This is, one could charitably say, a very peculiar response to the threat of apocalypse, which is among other things, terrifying, awesome in its original sense. AI safety researchers have acknowledged and sublimated their fear and anxiety into what they believe to be a rational, level-headed response. In the process, I believe, they have denied one aspect of apocalyptic foretelling, a feeling that, unlike fear, they must deny they themselves have: a perverse form of pleasure, of joy.

Isn't apocalypse, even madness, a little bit fun? Is it not a little titillating to imagine yourself as a kind of John Connor fighting an evil machine army? In that case, does it really matter if we win or not? No matter what happens, the battle will involve glory. It is their failure to come to terms with this aspect of apocalyptic fantasy that makes so many of the AI doomsayers so unsettling.

It also may explain why AI prophets of doom do not resemble or sympathize with conservative writer Gregory Conti, who simply wants to shut it all down. On some level, Conti is more safety-pilled than the safetyists: if he got his way, AI would be a technology like human cloning, one that no matter our capabilities, we simply do not do. The labs would close up shop, we would go back to coding by hand and getting chickpea salad recipes from books.

AI 2027 calls this "Plan S," but they do not favor it, ostensibly because it is "not realistic." Rather, they advocate for the utopian "Plan A": wherein we move more slowly, making sure that we get "alignment" right before 2040, wherein we are finally confident enough to give birth to superintelligence. This is how they describe the "good ending":

You and your friends throw an End Of The World Party, counting down to the fateful hour with champagne and good company. Some of you are old enough to remember a similar situation at the turn of the millennium, when the Y2K bug and impending new era introduced the phrase “party like it’s 1999” to the lexicon; the celebrations of late 2040 are more desperate and correspondingly more festive. When there is no news by sunrise, you fall into a fitful sleep, dreaming of the ineffable future.

How much, actually, does this differ from Plan D? In either case, there is a fantasy, a final end towards which the efforts of humanity are directed towards. This is the scenario they prefer: wherein we wait until we feel "safe" enough to press the button that relinquishes control to the omniscient machine God, who will hopefully grant our salvation. Plan S is "unrealistic," but also, compared to both the "good" and "bad" scenarios, far less lurid. It's "safer," and described in only a few short paragraphs. It provides no such sexy end of the world party: people live, go to work, they die. San Francisco votes again whether we should open a two-mile coastal stretch of road to cars. Life looks, basically, like how it's always looked.

The AI 2027 folks argue that some version of the superintelligence future is inevitable. We will get to die in the machine wars, or we will get to live deathless lives of pure bliss: and if none of this happens, isn't that a little bit disappointing? Ostensibly, the only reason the "good" scenario is better than the "safe" one is that it is our only choice. But I am suspicious of anyone who says they are not advocating for their plans, they are merely saying they are inevitable. Politically, it actually seems like "Plan S" might be extraordinarily popular, far more than their "Plan A," and if it turns out getting it done is a lot easier than their fantasies of utopia or apocalypse, will the safetyists get on board? Are they willing, on behalf of humanity, to press the button that ensures that the most terrifying future of them all: where there is never any machine god—only, maybe, the real one—and where they have to accept their limited, familiar, all-too-human condition?

Also on Substack