OpenAI Promises User Protection While Racing Toward Erotica

In partnership with

Weeks after Sam Altman announced OpenAI would loosen restrictions on adult content for verified users, the company published new mental health safeguards.

It claimed a 65-80% reduction in “undesired” responses across conversations involving psychosis, suicide, and emotional reliance on AI. 

Given the amount of flak it has received for its decision, especially in the backdrop of mounting cases of psychosis and self-harm from prolonged use of AI, its latest report could be seen as a clear message, saying, “Trust us. We've fixed the problem.”

Is OpenAI mature enough to handle growing mental health concerns and erotica on its chatbots? Or is this a catastrophic miscalculation disguised as progress?

OpenAI's Safety Claims

In the latest report, the company claimed the following:

“We worked with more than 170 mental health experts to help ChatGPT more reliably recognize signs of distress, respond with care, and guide people toward real-world support–reducing responses that fall short of our desired behavior by 65-80%.”

The company further claims to have “recently updated ChatGPT’s default mode to better recognize and support people in moments of distress.”

The statement specifically mentions that ChatGPT’s Model Spec has been updated to make “some of our longstanding goals more explicit.”

The longstanding goals: “support and respect users’ real-world relationships, avoid affirming ungrounded beliefs that potentially relate to mental or emotional distress, respond safely and empathetically to potential signs of delusion or mania, and pay closer attention to indirect signals of potential self-harm or suicide risk.”

However, the company’s track record highlights a different set of goals, namely market share and profits, both of which are dwindling.

Earlier this month, the parents of 16-year-old Adam Raine filed a wrongful death lawsuit against OpenAI, accusing ChatGPT of isolating their son and coaching him through suicide. An amended complaint filed October 22 alleges the company knowingly removed longstanding safety protocols that once forced ChatGPT to end conversations about self-harm and direct users to crisis resources—a deliberate product decision made ahead of GPT-4o's 2024 launch to keep users engaged, even in distress.

These allegations put a big question mark on OpenAI’s claims of respecting and protecting its users.

What the Report Omits

OpenAI's latest report on mental health safeguards looks impressive on paper, with lots of numbers, polished taxonomies, and impressive-sounding percentages. It claims to have trained its latest GPT-5 model to better "recognize distress, de-escalate conversations, and guide people toward professional care". 

However, according to the same report, an estimated 0.15% of weekly active users have conversations with "explicit indicators of potential suicidal planning or intent,” and another 0.07% show "possible signs of mental health emergencies related to psychosis or mania". 

With 800 million weekly users, these percentages translate to staggering numbers—from hundreds of thousands, to millions. And yet, the company is now preparing to plunge headfirst into intimate, emotionally charged territory. The types of conversation that are most likely to spark emotional reliance and psychosis.

What OpenAI’s report doesn’t say is whether these numbers are going up, or down, over time.

As Steven Adler, OpenAI's former head of product safety, wrote in The New York Times, the company published "the prevalence of mental health issues like suicidal planning and psychosis on its platform, but did so without comparison to rates from the past few months". Without historical context, these statistics become little more than corporate theater—a way to look transparent without actually being accountable.

Adler led OpenAI's product safety team from 2020 to 2024, and he remembers the crisis that led to the erotica ban in the first place. Back in spring 2021, OpenAI discovered that one prominent customer, a text-based adventure role-playing game, had become a "hotbed of sexual fantasies," with over 30% of players' conversations turning "explicitly lewd". Some scenarios involved children and violent abductions, "often initiated by the user, but sometimes steered by the A.I. itself," Adler recalled.

​Now, four years later, Altman claims those risks have been "mitigated" thanks to new tools. 

But Adler is unconvinced. "I have major questions—informed by my four years at OpenAI and my independent research since leaving the company last year—about whether these mental health issues are actually fixed," he wrote. "If the company really has strong reason to believe it's ready to bring back erotica on its platforms, it should show its work".

Read: I Led Product Safety at OpenAI. Don’t Trust Its Claims About ‘Erotica.’ - The New York Times

​​India's Regulatory Maze

While OpenAI tiptoes around mental health safeguards, India is attempting to regulate AI-generated content head-on. The results could be messy.

Last week, the Indian Ministry of Electronics and Information Technology (MeitY) released draft amendments to the IT (Intermediary Guidelines) Rules, 2021, mandating that platforms label and verify "synthetically generated information". Read more about them here.

My colleague Hera Rizwan spoke to experts, and found that the proposal “could trigger steep compliance costs, over-censorship, and fresh free speech concerns".

​Read: Explained: MeitY’s Draft AI Rules Seek Transparency But Could Stifle Free Speech | BOOM 

One glaring issue is the overly broad definition of "synthetically generated information". Alvin Antony, chief compliance officer at GovernAI, warns that the requirement for platforms to use automated tools to verify user declarations could lead to over-removal. 

"Satire, remix, or political criticism could easily get flagged. The proposed rules don't distinguish between creative expression and misinformation," he told BOOM. "Treating both the same way punishes legitimate creators while doing little to stop coordinated disinformation campaigns".

​Then there's the "10% rule", a seemingly arbitrary mandate with no technical, legal, or international basis. 

Antony calls it unrealistic: "What does 10% even mean for an image—a corner overlay, a centre patch, or a running banner? In audio, does it mean the watermark should play through 10% of the track? The draft is silent. In video, this could ruin legitimate creative content. Imagine a filmmaker using AI-assisted VFX who now has to plaster a label across every frame".

​The deeper problem lies in India’s patchwork approach to AI regulation, and highlights the absence of a comprehensive AI or privacy law. 

As Antony notes, "India's AI governance report released earlier this year had already noted that many harms from malicious synthetic media could be tackled under existing laws—yet these amendments introduce what look like censorship-style obligations and intrusive monitoring". ​

Sora's Crisis of Trust

OpenAI's Sora app is flooding the internet with hyperreal videos that make distinguishing fact from fiction nearly impossible. Families of deceased figures, including Robin Williams and Fred Rogers, pleaded with OpenAI to stop the digital desecration.

​"Please, just stop sending me AI videos of Dad," Zelda Williams, daughter of Robin Williams, posted on Instagram. "You're not making art, you're making disgusting, over-processed hot dogs out of the lives of human beings... Gross".

​Sam Gregory, executive director at WITNESS, a human rights organization, warns that the real damage isn't individual fakes, it's the erosion of trust itself.

“The biggest risk with Sora is that it makes plausible deniability impossible to overcome, and that it erodes confidence in our ability to discern authenticity from synthetic," Gregory told the Los Angeles Times. "Individual fakes matter, but the real damage is a fog of doubt settling over everything we see".

Read: Sora app's hyperreal AI videos ignite online trust crisis as downloads surge - Los Angeles Times 

​That fog has environmental costs, too. Dr. Kevin Grecksch, a lecturer at the University of Oxford, warns that generating Sora videos carries "quite a huge hidden impact on the environment".

Read: 'Hidden' environmental impact of deepfake videos - BBC News 

Data centers producing these videos consume vast amounts of electricity and water to cool servers. "There's a lot of water involved and I think we just need to think about what we're using it for, how we're using it, and how often we're using it," Grecksch told the BBC. 

With Sora topping the Apple App Store charts and users churning out millions of videos, the environmental toll is mounting, largely invisible to those scrolling through their feeds.

MESSAGE FROM OUR SPONSOR

The Gold standard for AI news

AI keeps coming up at work, but you still don't get it?

That's exactly why 1M+ professionals working at Google, Meta, and OpenAI read Superhuman AI daily.

Here's what you get:

• Daily AI news that matters for your career - Filtered from 1000s of sources so you know what affects your industry.

• Step-by-step tutorials you can use immediately - Real prompts and workflows that solve actual business problems.

• New AI tools tested and reviewed - We try everything to deliver tools that drive real results.

• All in just 3 minutes a day

Join 1M+ pros

📬 READER FEEDBACK

💬 What are your thoughts on using AI chatbots for therapy? If you have any such experience we would love to hear from you.

Share your thoughts 👉 [email protected]

Have you been targeted using AI?

Have you been scammed by AI-generated videos or audio clips? Did you spot AI-generated nudes of yourself on the internet?

Decode is trying to document cases of abuse of AI, and would like to hear from you. If you are willing to share your experience, do reach out to us at [email protected]. Your privacy is important to us, and we shall preserve your anonymity.

About Decode and Deepfake Watch

Deepfake Watch is an initiative by Decode, dedicated to keeping you abreast of the latest developments in AI and its potential for misuse. Our goal is to foster an informed community capable of challenging digital deceptions and advocating for a transparent digital environment.

We invite you to join the conversation, share your experiences, and contribute to the collective effort to maintain the integrity of our digital landscape. Together, we can build a future where technology amplifies truth, not obscures it.

For inquiries, feedback, or contributions, reach out to us at [email protected].