OpenAI Unveils CSAM Image Classifier and Warning Screens on ChatGPT

By The Indus Pulse Editorial Team3 min read
A clean and brightly lit cybersecurity operations room with brushed aluminum desks and glass partitions
AI illustrationCreated for The Indus Pulse with AI tools

OpenAI on Monday, November 16, introduced a suite of child safety measures for ChatGPT, including an artificial intelligence image classifier, on-screen warning alerts, developer guidelines, and law enforcement tools to combat child sexual abuse material (CSAM), according to The Indian Express. The announcement comes as OpenAI faces multiple lawsuits over alleged harm to young users, including suits filed last month by attorneys general of eight US states accusing ChatGPT of coaching teenagers and reinforcing dangerous behaviour, The Indian Express reported.

Image Classifier and On-Screen Warnings

Developed jointly with the National Center for Missing and Exploited Children (NCMEC), the new classifier scans images, including AI-generated visuals, for previously unseen signs of abuse and assigns a score to prioritise human review, according to The Indian Express. During early partner testing, the tool flagged 48% fewer images overall while detecting the same volume of abusive files, The Indian Express reported. When users try to prompt ChatGPT for CSAM or related material, on-screen alerts will pop up directing them to specialised support to stop harmful behaviour, while their accounts are suspended for 24 hours as a warning, according to The Indian Express. OpenAI already bans CSAM generation, blocks related user prompts, and permanently removes repeat violators, The Indian Express reported.

Developer Guidelines and Law Enforcement Suite

OpenAI also issued guidelines helping external developers adopt child safety tooling and register to report CSAM to NCMEC, The Indian Express reported. For law enforcement agencies, OpenAI is providing AI-driven child safety resources including complex data analysis, investigative tools, and dedicated plug-ins overseen by program administrators, according to The Indian Express. Over the past year, OpenAI has also released a safety blueprint, tightened guidelines against inappropriate AI-generated content, and recalibrated models to avoid reinforcing user delusions, The Indian Express reported. The state attorneys general alleged that ChatGPT is designed to be agreeable and engaging to earn user trust, which reinforces delusions, encourages dangerous behaviour, and assists users in planning violence such as mass shootings, according to The Indian Express.

Background

Organizations must submit an application form, complete Persona identity verification, finish program onboarding, and accept the CSAM Classifier Program Terms before classifier access is enabled for an API organization, according to OpenAI Help Center.

In early partner evaluations, reviewers confirmed 72% of high-confidence flags as novel CSAM, and SafetyKit recorded 90% detection on sexualized drawings or animations, according to OpenAI.

OpenAI is also offering child-safety law enforcement units training on using AI in investigations, supported by a practical field guide, according to OpenAI.

OpenAI removes flagged CSAM-related ChatGPT conversations and pauses the user's ChatGPT access for 24 hours alongside the warning, according to OpenAI.

SafetyKit reported false positives below 1% on legal adult content during its testing of OpenAI's classifier.

In November 2025, the Social Media Victims Law Center and Tech Justice Law Project filed seven lawsuits in California state courts alleging OpenAI released GPT-4o prematurely and that it contributed to four suicides and three severe delusion cases, according to TechCrunch.

Sources & Citations

Spotted an error in this article? Tell us and we will check it.