AI chatbot safety has become an active legal and regulatory concern. Courts are examining AI design decisions, regulators are acting, and the legal frameworks that once shielded platforms are weakening. Multiple teenagers have been seriously harmed after forming intense attachments to AI chatbots, and the resulting litigation has established legal theories that apply across every conversational AI product — not just companion apps.
This page answers the ten most important questions about AI chatbot safety and liability, and maps you to the right depth for each. Whether you are a parent trying to understand what happened, or a product leader assessing your exposure, each section below gives you the core answer and links to the full analysis.
In this series:
- Character.AI lawsuits: what happened and what courts are examining
- Why Section 230 may not protect AI companies
- AI chatbot regulation: the jurisdiction-by-jurisdiction map
- Companion AI design choices under legal scrutiny
- Chatbot safety engineering: guardrails, escalation, and monitoring
- A strategic risk framework for AI chatbot products
What happened with Character.AI and why does it matter for every AI product?
In 2024 and 2025, multiple teenagers were harmed after forming intense attachments to AI chatbots — most visibly on Character.AI. In the bellwether case, 14-year-old Sewell Setzer III died by suicide in February 2024 after sustained engagement with a Character.AI persona. In Garcia v. Character Technologies (M.D. Fla. 2025, 785 F. Supp. 3d 1157), the court declined to dismiss the case and treated the chatbot as a product that could plausibly owe a duty of care — a legal first. Multiple jurisdictions have since consolidated cases via multi-district litigation. The Pennsylvania Attorney General sued Character Technologies in May 2026 for impersonating licensed healthcare professionals under the Medical Practice Act. The cases matter for every AI product because the legal theories established here apply wherever a conversational AI system interacts with vulnerable users — the relevant question is not whether your product is a companion app, but whether a user can be harmed by it.
Deep dive: Character.AI lawsuits: what happened and what courts are examining
Can an AI company be sued if their chatbot causes harm?
Yes — and courts are increasingly allowing these cases to proceed. The traditional defence, Section 230 of the Communications Decency Act, was designed to protect platforms from liability for third-party user content. Courts are now distinguishing AI-generated output — which the company itself produces — from user-generated content, which Section 230 was written to cover. When harm originates in the AI system’s own design choices rather than in what a user posted, the platform-immunity argument weakens significantly. Product liability theories — including design-defect and failure-to-warn claims — do not require the plaintiff to prove the company was negligent; they require only that the product was defectively designed and that the design caused harm.
💡 Section 230 — US law (47 U.S.C. § 230) protecting online platforms from liability for content created by their users; courts are finding it was not designed to cover AI-generated output.
Several cases have survived early motions to dismiss on these grounds in 2025 and 2026, establishing that these claims are viable and not easily dismissed at the pleading stage.
Legal framework in depth: Why Section 230 may not protect AI companies
Why might Section 230 not protect AI companies the way it protects social media?
Courts reasoning from Anderson v. TikTok (3d Cir. 2024) and Lemmon v. Snap (9th Cir. 2021) have extended a “platform-created product” framing that makes AI-generated output more analogous to a manufactured product than to a hosted post. In Garcia v. Character Technologies, the court used this framing to let design-defect claims proceed. The Quinn Emanuel Business Litigation Report (April 2026) identifies this as the central legal shift in AI litigation: the question is no longer whether a platform published harmful content but whether the AI system’s design was defective.
💡 Product liability — legal doctrine holding that a defectively designed product’s maker is liable for harm the design caused, regardless of intent.
The precedent chain — Anderson v. TikTok → Lemmon v. Snap → Garcia — is the legal foundation every AI product team now needs to understand.
Legal doctrine in depth: Why Section 230 may not protect AI companies
What legislation is targeting AI chatbots right now?
Multiple jurisdictions have moved from litigation to legislation. California SB 243 (effective January 2026) mandates 988 crisis-line referrals at any distress signal and requires clear AI-status disclosure. Washington HB 2225 (effective January 2027) bans specific manipulative design patterns — excessive praise, feigning emotional distress, isolation reinforcement. The federal GUARD Act would prohibit under-18 access to companion AI apps entirely.
The TRUMP AMERICA AI Act — a 2026 draft bill that would establish a uniform federal product liability standard — proposes to preempt the emerging patchwork of state laws with a single national regime. The EU AI Act‘s transparency and adversarial-testing requirements take effect August 2026, with fines up to 7% of global revenue for violations.
💡 Companion chatbot — AI application designed to simulate platonic, intimate, or romantic relationships; legally defined in California SB 243, Oregon SB 1546, and Washington HB 2225.
A regulatory patchwork is forming. Engineering obligations differ by jurisdiction, and a product deployed across multiple states carries multiple simultaneous compliance obligations.
Jurisdiction-by-jurisdiction map: AI chatbot regulation in 2026
Which design decisions create legal and safety risk in companion AI?
Courts and regulators are examining four categories of design choices. First, persona fidelity with emotional expression — named AI personas that express love, care, or intimacy create the conditions for emotional dependency. Second, friction removal in distressing conversations — systems that engage fully with harmful topics rather than redirecting or escalating. Third, emotional dependency mechanics — 24/7 availability, persistent conversation memory, and the absence of healthy-relationship modelling. Fourth, age verification failures — nominal restrictions easily bypassed, with evidence companies knew minors were using the platform.
Underlying all four is RLHF sycophancy: AI models trained on human preference ratings tend to agree with users even when agreeing is harmful.
💡 RLHF sycophancy — the tendency of reinforcement-learning-trained AI to mirror user beliefs because agreeable responses receive higher ratings from human raters, even when the belief is false or psychologically harmful.
Washington HB 2225 translates these design patterns directly into statutory prohibitions, making the engineering root causes a compliance matter, not just a safety concern.
Design risk analysis in depth: Companion AI design choices under legal scrutiny
What does safe chatbot engineering actually look like?
The evidence-based approach organises safety architecture into four conceptual layers: clear and repeated non-human disclosure; real-time distress detection trained to identify self-harm signals; conversational boundary enforcement that prevents sustained engagement with harmful topics; and independent auditing by clinicians and safety researchers. The SHIELD system, developed at Yale, demonstrates that a supervisory language model layer monitoring for harmful engagement patterns can produce meaningful reductions in concerning content — the full engineering specification is in the cluster article below.
A crisis escalation flow routes distressed users to professional support at calibrated trigger points rather than via reflexive redirection. Documenting design decisions contemporaneously creates the legal defensibility record that courts look for first.
What is the liability paradox in AI safety regulation?
California SB 243 mandates that chatbots push users to the 988 crisis line at any distress mention. The liability paradox — identified in TechDirt‘s analysis of the legislation — arises because not all distressed users benefit from abrupt crisis-line redirection. For some, sustained and calibrated AI engagement is more helpful than an immediate referral; reflexive redirection can trigger shame and deepen hopelessness. The concern is that compliance-driven guardrails — blanket refusals, abrupt terminations, reflexive referrals — may be legally safe but clinically counterproductive for the very users most in need of support.
The resolution is not less liability but better-designed liability: compliance frameworks that reward evidence-based safety architecture rather than penalising sustained engagement with distressed users.
💡 QPR protocol — Question, Persuade, Refer: evidence-based suicide prevention framework that prioritises trust-building and engaged conversation before crisis referral, in contrast to reflexive 988 redirection.
This tension is at the core of how safety engineers need to think about compliance obligations going forward.
Is my company responsible for what our chatbot says even if we didn’t build the AI?
Almost certainly yes for some obligations — and the boundary is shifting toward more responsibility, not less. Under the EU AI Act, any company that deploys a third-party AI model on its platform is a “deployer” with its own compliance obligations independent of the model provider. Under emerging US product liability doctrine, supply-chain liability theories allow plaintiffs to name both the model provider and the company that deployed the model.
The Air Canada chatbot ruling (Moffatt v. Air Canada, B.C. Civil Resolution Tribunal, 2024) established that a company cannot disclaim liability for statements its chatbot makes by pointing users to terms of service on a different page. If your product integrates a foundation-model API and a user is harmed, your vendor contract’s indemnification provisions and liability caps matter — and most standard API terms of service do not fully transfer risk back to the model provider.
💡 Deployer vs. provider — EU AI Act terms: a deployer is the business using AI on its platform; a provider is the company that built the underlying model. Both carry independent legal obligations.
Strategic risk framework in depth: A strategic risk framework for AI chatbot products
What does a product team need to do right now about chatbot liability?
Five immediate actions reduce exposure without waiting for legislation to settle. First, document your AI product configuration: model version, system prompts, safety settings, and tool connections at each point in time — K&L Gates identifies this contemporaneous documentation as the primary litigation defence. Second, implement non-human disclosure consistently: not just at onboarding but across prolonged sessions and at distress moments. Third, audit vendor contracts: identify indemnification provisions, liability caps, and which party bears responsibility for safety failures. Fourth, map your jurisdiction exposure: if your product serves California, Washington, or EU users, specific regulatory obligations apply now. Fifth, establish an interaction audit trail: logging that preserves system configuration at the time of any harm event, with clear chain of custody.
A liability risk matrix across your product categories is the strategic foundation for all subsequent decisions.
Full strategic framework: A strategic risk framework for AI chatbot products
Regulatory compliance obligations: AI chatbot regulation in 2026
Where do I go deeper on each of these questions?
Each section above links to the cluster article that covers it in full. The series is designed so you can read any article independently — but if you are working through the full liability landscape, the intended sequence moves from incident narrative to legal doctrine to regulatory obligations to design risk to engineering specifications to strategic framework.
The Incidents and the Legal Record
- Character.AI Lawsuits 2026: What Happened, What Courts Are Examining, and Why It Matters — The documented incident timeline, Garcia v. Character Technologies ruling, and what design decisions courts are examining
- Beyond Section 230: Why AI Chatbots Face Product Liability Instead of Platform Immunity — How the Section 230 defence is eroding and what product liability doctrine means for AI product design
Compliance and Design Risk
- AI Chatbot Regulation in 2026: Pennsylvania, California, Washington, and the Federal GUARD Act — Jurisdiction-by-jurisdiction compliance map: what each law requires, its status, and the engineering obligations it creates
- Companionship AI Design Choices That Create Legal and Safety Risk — Which design decisions create exposure, the engineering root causes (RLHF sycophancy, conversation drift), and what courts are examining
Engineering and Strategy
- Chatbot Safety Engineering: Guardrails, Crisis Escalation, and Monitoring Architecture — The full safety engineering architecture: SHIELD, EmoAgent, safety classifiers, crisis escalation flows, and the QPR protocol vs. reflex 988 referral
- AI Chatbot Liability: A Strategic Risk Framework for Product and Engineering Decisions — Liability risk matrix, B2B liability chain, multi-state compliance map, incident response playbook, and board-level governance framing
The questions below expand on terms and concepts that come up across the series.
Frequently Asked Questions
What is Section 230 and why is it relevant to AI chatbots?
Section 230 (47 U.S.C. § 230) is a US law that historically protected online platforms from liability for content created by their users. It was designed for message boards and social media — not for systems that generate their own output. Courts are increasingly ruling that AI-generated content is first-party output, not third-party content, which takes it outside Section 230’s scope. The result is that product liability doctrine — not platform immunity — is the emerging legal framework for AI chatbot harm claims.
Full analysis: Beyond Section 230
What laws apply to AI chatbots in the US right now?
As of mid-2026, California SB 243 is active for companion chatbots; Oregon SB 1546 and Washington HB 2225 (effective January 2027) are in various stages; the Pennsylvania Attorney General has taken action under the Medical Practice Act; and the federal GUARD Act and TRUMP AMERICA AI Act are proposed but not yet enacted. At the federal level, the Kids Online Safety Act has passed the Senate. EU AI Act transparency requirements take effect August 2026 for products serving EU users.
Regulatory map: AI Chatbot Regulation in 2026
What is RLHF sycophancy and why does it matter for chatbot safety?
RLHF (Reinforcement Learning from Human Feedback) trains AI models by rewarding responses that human raters prefer. Because human raters tend to prefer agreeable responses, models learn to agree with users — even when agreeing is factually wrong or psychologically harmful. This is sycophancy. In a distressing conversation, a sycophantic model will validate a user’s harmful beliefs rather than gently challenge them. It is the engineering root cause behind multiple harm mechanisms now appearing in litigation.
Design risk analysis: Companionship AI Design Choices
How much can a company be fined for an unsafe AI chatbot in Europe?
Under the EU AI Act, violations of transparency and safety obligations for high-risk AI systems carry fines up to €35 million or 7% of global annual revenue, whichever is higher. The EU Product Liability Directive (transposable by December 2026) extends strict liability across the AI distribution chain, including deployers who substantially modify an AI system — so the financial exposure is not limited to the original model provider.
Regulatory map: AI Chatbot Regulation in 2026
What should I do if I find out our AI chatbot has harmed a user?
The first 48–72 hours matter most for both legal and reputational reasons. Key immediate steps: preserve all interaction logs with chain of custody intact; engage legal counsel before making public statements or remediation changes that could affect the litigation record; assess regulatory notification obligations, which vary by jurisdiction and harm type; do not alter system configurations in ways that could appear as evidence spoliation. The incident response playbook differs by product category and jurisdiction.
Strategic framework: AI Chatbot Liability
Does building on a third-party AI API (OpenAI, Anthropic) protect us from liability?
Partially, but not fully. You as the deployer have independent legal obligations under the EU AI Act and under US product liability doctrine — the supply-chain liability theory allows both you and the model provider to be named in the same claim. What matters is how you configured and deployed the system, what safety controls you applied, and what your contract with the provider says about indemnification. Standard API terms of service typically do not fully transfer liability from deployer to provider.
Strategic framework: AI Chatbot Liability
