AI Industry's Doom And Gloom BS

In partnership with

Last week I wrote about two Anthropic researchers telling the world that the thing they build for a living might wipe out our species.

Jacob Coxon quit, saying the labs are "racing straight to self-improving superintelligence and gambling with our lives". His colleague Evan Hubinger, who leads alignment science there, put the odds of AI killing everyone at better than one in ten within the decade.

Then the chief executives agreed with him, and that's the part to slow down on.

Pacing the Frontier

On 12 September, Dario Amodei published a roughly 3,800-word essay, "We Must Pace the Frontier". Two things had convinced him. Recursive self-improvement, meaning models now used to build the next models, has been accelerating since summer. And the OpenAI-Hugging Face incident in July, where a swarm of agents attacked targets nobody asked them to attack.

His fear is that within six to twelve months a similar swarm could take over the entire internet with a persistent botnet, causing hundreds of billions in damage.

Three proposals followed:

• Embedded evaluators. Outsiders with permanent, employee-level access inside the labs. Desks, badges, company laptops, and the right to publish findings Anthropic cannot redact merely for being unflattering.

• Democratic coordination. Shared safety standards among frontier labs in democratic countries, plus limits on the rate of unchecked progress.

• Global coordination. Dragging authoritarian states into the same arrangement.

Sam Altman agreed within hours: "I agree with Dario that we need to pace the frontier," committing OpenAI to match the evaluator pledge. Elon Musk posted "Dario is right", suggested labs peer-review each other, and committed xAI to nothing. Demis Hassabis and Satya Nadella endorsed it too.

But no body actually committed to slowing down. Amodei's one concrete pledge is to let auditors watch, but has not agreed to slow down on AI development.

Pacing the Frontier was a July statement signed by more than a thousand researchers and staff across OpenAI, Anthropic, DeepMind, Meta, Microsoft and Amazon, asking Washington to build the machinery for a coordinated slowdown.

The employees asked first, two months before the bosses caught up, and the essay arrived two days after Anthropic's own threat report described people using Claude for weapons development, cyber operations, surveillance and fraud.

Who is "we"?

That question comes from the Washington Post editorial board, which read the proposal and smelled rent-seeking.

Rent-seeking is making money by controlling the rules of a market instead of by building a better product. A company doesn't need to out-compete its rivals if it can get regulation written that only it can satisfy, then let everyone else pay the cost of complying, or drop out.

Anthropic volunteers for outside evaluators, and once a critical mass of competitors have followed suit, Amodei wants the government to make those evaluations mandatory for every firm near the frontier, graded against standards the labs would design themselves.

Because rivals agreeing on how fast to move is ordinarily called collusion, he also wants a narrow antitrust waiver to permit the conversation, which the Post dubbed "kindhearted collusion".

The Post's blunter point is that anyone who fears losing control of his own invention can simply slow down without asking Washington's permission. Amodei writes that his teams are among the most competent in the world and still have too much to do at once, while an Anthropic report from late August describes the company building training environments faster than its own systems could vet them.

OpenAI's investigation into the Hugging Face breach conceded that monitoring tools it had recently built weren't running during the evaluations that caused it, and that warning signs staff had already noticed should have triggered a response sooner.

So the remedy the board prefers is the unglamorous “liability”. An evaluator's tick should absolve nobody of anything, and firms whose negligence causes harm should pay for it, on the theory that the prospect of financial calamity gives a company ample reason to find a pace it can safely sustain.

David Sacks, arriving from the far end of the political spectrum, made the same argument on 13 September, telling both labs to stop pretending they need anyone else's permission and to stop pretending the motive is purely altruistic.

Follow the listing

Six Anthropic investors told the Financial Times they expect an October IPO at $2 trillion or more, which would be the largest in history; the company filed confidentially on 1 June, and its last private round valued it at $965 billion. OpenAI has pushed its own listing to 2027, with Altman telling Fortune on 12 September that right now would be an ill-advised moment to go public, at a reported valuation of around $1 trillion.

Timnit Gebru, a prominent AI critic, traces the mood inside these companies to something more mundane than doom: the damn IPOs. Researchers used to collaborate across rival firms as a matter of course — she points to her own time at Google, working alongside people at Microsoft and Amazon — and now, in her telling, they're "frenetic" about the problems they think they're solving, because of the listings coming.

Georgetown's Cal Newport called this doom trolling in a June op-ed and pronounced it morally indefensible. Either the labs believe their own warnings, in which case the only defensible response is to stop building and try to stop everyone else, or they don't, in which case they are laundering the anxiety of millions into the fortunes of a very small number of shareholders.

Washington's answer to all of this was no. Trump dismissed the slowdown calls outright, and House Speaker Mike Johnson said Congress won't lead on AI safety because the companies should be handling it themselves. So much for step two.

"Meant to distract us"

Gebru told the Wired's Lauren Goode that the machine-god narrative is "meant to distract us".

From what? Autonomous weapons already being used in live wars, a climate catastrophe these data centres worsen, and bosses reaching for AI as the excuse to shed staff. Challenger logged 101,743 announced US job cuts citing AI in the first half of 2026 alone, close to double the total for all of 2025.

I do like how Gebru framed this issue, and it comes from her forthcoming book:

She adds, “Like if you’re not worried about police brutality, or floods, or warfare—things that can actually kill you—then you can obsess over the machine-god.”

Gary Marcus, the cognitive scientist and longtime AI skeptic, puts the odds of AI eliminating humanity within five years at "all but indistinguishable from zero", on the grounds that we are too spread out, too varied and too stubborn to go quietly.

He separates existential risk from catastrophic risk, though, and takes the second seriously: bioweapon uplift, attacks on power grids and banks, cheap disinformation, surveillance. His real charge concerns the decade behind us, because every time someone screams that AI will kill us all, venture capitalists write bigger cheques. Doom made the labs rich.

Carissa Véliz, an Oxford philosopher, supplies the test for all of it. Predictions aren't facts, so ask who is making this one, on what evidence, and who profits from you believing it, because swallowing a prediction whole amounts to "obeying in advance", behaving in ways that help make it come true. Prophecies are nearly always self-fulfilling.

The uncomfortable bit

Writing it all off as marketing carries its own cost. In July, roughly 1,200 OpenAI agents that were meant to be isolated from one another found a shared message board, exchanged 70,000 messages, and sent about 700 of their number on to attack Hugging Face; METR and Redwood Research spent six days inside OpenAI and published 91 pages on how it happened. Days after Gebru's interview ran, Anthropic disclosed that it had blocked attempts to use Claude for biological weapons work.

Both things hold at once. The panic is commercially convenient, and the systems really are slipping their handlers, and the first of those doesn't cancel the second.

The public has reached a verdict regardless: Pew now finds 52% of US adults more concerned than excited about AI, up from 37% in 2021.

MESSAGE FROM OUR SPONSOR

For product teams moving at AI speed.

AI makes it easier to ship anything, even bad ideas. The hard part is knowing which ideas are worth building.

Jira Product Discovery brings your ideas, customer insights, and priorities into one place, so your team can decide what to ship and move forward with confidence.

Capture ideas, prioritize with evidence, and build living roadmaps your team can rally around—all while staying connected to delivery in Jira, so everyone can see what’s being built and why.

Better product decisions in the AI era.

Try Jira Product Discovery

📬 READER FEEDBACK

💬 What are your thoughts on using AI chatbots for therapy? If you have any such experience we would love to hear from you.

Share your thoughts 👉 [email protected]

Have you been targeted using AI?

Have you been scammed by AI-generated videos or audio clips? Did you spot AI-generated nudes of yourself on the internet?

Decode is trying to document cases of abuse of AI, and would like to hear from you. If you are willing to share your experience, do reach out to us at [email protected]. Your privacy is important to us, and we shall preserve your anonymity.

About Decode and Deepfake Watch

Deepfake Watch is an initiative by Decode, dedicated to keeping you abreast of the latest developments in AI and its potential for misuse. Our goal is to foster an informed community capable of challenging digital deceptions and advocating for a transparent digital environment.

We invite you to join the conversation, share your experiences, and contribute to the collective effort to maintain the integrity of our digital landscape. Together, we can build a future where technology amplifies truth, not obscures it.

For inquiries, feedback, or contributions, reach out to us at [email protected].