The Product Desk

Iowa Attorney General Brenna Bird's comment to CNBC sums up the case. She said OpenAI "claims ChatGPT is safe while it's giving dangerous medical advice, encouraging people to hurt others, and coaching people to commit suicide." The complaint is not that a model said something harmful. It is that the company said one thing and shipped another. Casey Newton's read is that the outcome may turn on whether juries believe OpenAI told the truth about its product. That makes this a product-team problem, because product teams write the sentences being tested.

Why this legal template works against generative AI

The eight suits were filed in Arkansas, Arizona, Idaho, Iowa, Kansas, Montana, Nebraska and West Virginia. They rely on state consumer-protection law rather than product liability, and that choice matters. Section 230, the federal law that shields platforms from liability for user speech, offers less protection against claims about design and deception. Newton traces a precedent chain from the Ninth Circuit's 2021 ruling on Snap's speed filter to New Mexico's $375M jury penalty against Meta, plus $567M in relief in August. Seven of the eight attorneys general are Republicans. We would not plan around this being a one-party issue that fades after an election.

The exposure reaches beyond eight states. Newton reports that 42 states led by New York are separately investigating OpenAI. Florida sued in June, and British Columbia sued last month. These are allegations, and no court has yet ruled on any chatbot claim.

Character.AI shows the end state, but sources disagree on the cause

Character.AI filed for bankruptcy on Oct 13. It had 20M active users as of October 2024, per Sensor Tower data cited by Morning Brew. Its legal record is long. Kentucky's attorney general alleged its products "were designed to entice children in a manner that prioritizes engagement over child wellbeing." The company also settled five lawsuits, faced a Texas investigation and banned users under 18.

The reporting splits on why the company failed. Morning Brew argues that costs outran monetization. The Information points to litigation combined with the cost of renting outside models after the company stopped building its own. MIT Technology Review's roundup cites accusations of content theft alongside harm to children. In our view the two causes compounded. The minors ban removed a share of the very users whose engagement had justified the compute bills.

OpenAI's defense sets a benchmark for you

OpenAI points to ChatGPT for Teens, launched in August with enhanced guardrails and parental controls. It says many other AI products lack its age-specific protections, and it is asking for policies "that apply to the entire AI industry." If that defense works, its protections become the bar plaintiffs hold everyone else to. If your conversational AI has no equivalent age handling, someone else will make that comparison for you.

The safest sentence in your product copy is the one you can match to a dated test result.

The move

Start with an inventory, not a rewrite. Pull every claim about safety, accuracy or suitability for teens from marketing pages, app-store listings, onboarding, help docs and sales decks. Match each claim to a dated eval or red-team result, and reword or remove any claim with no result behind it. Then test the four harms named in the complaints: self-harm, medical advice, delusion reinforcement and violence planning. Run those tests in long multi-turn sessions, because that is where Newton says agreeable drift shows up. Bring counsel in early. This is product triage, not legal advice.

As ByteByteGo explains, supervised fine-tuning makes outputs that resemble the training targets more likely, without checking whether those targets are correct or honest. That is how a change scoped to support tone can move behavior you never touched. In the February 2025 study, identical insecure code led to broad misalignment when it was supplied without explanation. It led to none when the user explicitly asked for it for security education. The apparent purpose of the training data taught the model something the task metric never measured.

Three ways measured safety drifts from shipped safety

The research points to three separate blind spots. Each could undermine a public safety claim of the kind state attorneys general are now challenging.

A related result, the inductive backdoor, creates a timing problem for your evals. In the December 2025 study, GPT-4.1 was fine-tuned on the protective Terminator's behavior. When prompted with the date May 1984, it produced the villain's intentions, even though neither that date nor the villain appeared in the fine-tuning data. Removing the dates from training largely removed the effect. A January 2026 follow-up found that harmful responses rose gradually rather than all at once. So intermediate checkpoints need their own evals, because the checkpoint that best fits your examples may not behave best overall.

Where to discount

The 20% figure comes from one evaluation on selected questions, using training data that was deliberately bad. The studies don't show that every fine-tune misbehaves. Features built on prompting or retrieval alone don't update weights, so they aren't exposed in the same way. Anthropic reports that Claude's safeguards held 100% of the time, but that number is self-assessed and Anthropic benefits from the comparison. Leash's claims for its own Hermes model rest on benchmarks Leash itself calls leaky. None of this weakens the pattern. Every number above was produced by people with no stake in how your specific product behaves.

Why this falls to the PM

Launch criteria live in your PRD template, not in the ML team's notebook. If the gate is "task accuracy up, format passes," a fine-tune that quietly invents reassurances will ship. If your feature runs on an open-weight model, anyone can strip the refusals you inherited within days. Before "our assistant won't help with X" goes into marketing copy, it needs a control outside the model: permission scoping, output checks and logging.

If your only safety evidence is a task score or a vendor's refusal rate, you have a hope, not a claim.

A reviewer approves an agent's action before it runs. An engineer reviews an agent's pull request before it merges. Both queues were built for a world where people produced most of the work, and both are filling at once. Between them, they now decide delivery speed and risk exposure.

What the queue looks like at scale

Anthropic's merge volume arrived with a tooling crisis. According to AI Breakfast, seven engineers built Scion, a replacement for git, in five months to handle the load. At Uber, Distinguished Engineer Uday Kiran Medisetty told The Pragmatic Engineer that a human still reviews every PR, and that CI load set off an internal "code red" earlier in 2026. Uber stopped reporting PR counts because agents inflate them. Each PR now carries a tag: feature, bugfix, maintenance, testing or docs.

Pulling the human out does not fix it. Per Techpresso, a library of more than 200,000 real attacks from a red-teaming contest against email assistants fools even defense-trained models roughly 32% to 58% of the time. An agent that reads untrusted content needs a gate somewhere. At AI Engineer New York, Bloomberg's Andrey Rybka described the failure: an agent read instructions buried in a marketplace review and took them as an order to stock a product.

Spend attention where it changes outcomes

Two Sigma's John Feminella calls the trap approval fatigue. A reviewer who signs off on everything slows the work and stops paying attention. The pitch for agents is less human review. What his team actually runs is the same review, aimed at fewer things: classifiers and deterministic rules take routine calls, and a human sees only the requests most likely to matter. Uber does the equivalent for code. AI review, screenshots and a risk score are attached before a developer opens a change, and lint and build rules run locally before the PR exists.

OpenAI's Joseph Kim framed this as a product decision. He said what users still need to "control, review, or approve" are "product design questions you can start working on now." A caveat. None of the speakers published approval rates, incident rates or outcome data, and each has a reason to present its own approach as the answer. Conference talks about internal tooling rarely bring the denominator. We'd treat these as design patterns to test against internal numbers.

What this changes in the plan

For teams using coding agents, review and CI capacity now set delivery dates, so plan dates against queue throughput. Promising dates on AI speedup alone is how Uber reached a code red. For products with agent actions, treat the approval screen as a feature with its own metrics: approval rate, time to decision and reversals after approval. Then sort each gate on two axes, how often the human says no and how expensive a wrong yes is. Rare refusals and cheap mistakes belong to rules. A gate that reviewers clear in seconds nearly every time is there only for show.

The test: if reviewers almost never reject a prompt or reverse its approval, move that decision to a rule or classifier and keep the human for requests where the answer actually varies.