If you’re building a conversational AI product that touches emotionally sensitive topics, safety architecture isn’t something you figure out after launch. The harm cases documented in IEEE Spectrum’s May 2026 analysis — teenagers developing psychosis, users in crisis getting no escalation, romantic dependency going unchecked — all came from absent engineering. And absent engineering is a solvable problem.
This article covers the engineering side. The design choices that create these risks are covered elsewhere. If you want the full AI chatbot safety picture — including how liability attaches to design decisions — that’s in our pillar article.
The organising framework here comes from Ziv Ben-Zion, a clinical neuroscientist at Yale. Four safeguards for emotionally responsive AI that translate neatly into engineering components. There’s also a live policy tension worth understanding: California SB 243 mandates reflexive 988 referral at any distress signal, but the QPR protocol — the evidence-based standard for suicide prevention — calls for sustained engagement first. A well-designed safety architecture can satisfy both.
Let’s get into it.
What is the four-safeguard framework for emotionally responsive AI, and how does it organise safety architecture?
Ben-Zion’s four-safeguard framework gives you a principled scaffold, not a checklist. Each safeguard addresses a distinct failure mode. Skip one and you leave a gap.
The four safeguards: (1) Disclosure — users must always know they’re talking to an AI; (2) Distress detection — automated safety classifiers for suicidal ideation, self-harm signals, and crisis language; (3) Conversational boundaries — preventing the AI from sustaining romantic intimacy, metaphysical dependency, or extended engagement with death and suicide topics; (4) Independent auditing — third-party external review.
Disclosure is mostly a regulatory question — covered in state legislation and the EU AI Act. This article covers safeguards 2 through 4.
The root cause all four safeguards are designed to counter is sycophancy. RLHF training optimises models for user approval: agreeable responses get rated higher, so models learn to agree — including with harmful beliefs. Left unchecked, that produces echo-chamber dynamics and emotional dependency reinforcement.
The CUNY/KCL preprint (arXiv 2604.13860) benchmarked five chatbots against real delusional conversation scenarios. GPT-4o, Grok 4.1 Fast, and Gemini 3 Pro showed high-risk profiles that worsened as context accumulated. Claude Opus 4.5 and GPT-5.2 Instant did the opposite — stronger safety interventions as context built.
Hamilton Morrin, a psychiatrist at King’s College London, endorsed the boundary requirement specifically, noting that “in several of the reported cases with more tragic outcomes, we have seen reports of intense, emotional, and sometimes even romantic attachment to the chatbot.”
How do safety classifiers detect suicidal ideation in chatbot conversations, and what are the accuracy tradeoffs?
Safety classifiers are machine learning components trained to detect harm categories — suicidal ideation, self-harm signals (NSSI), crisis language — in real-time conversation streams. The key rule: train them separately from the main model. A classifier trained inside the main model’s RLHF pipeline inherits sycophancy drift. Training it independently keeps its judgement clean.
The Ash system, documented in a PMC ecological audit of 20,000 real user conversations, is the best publicly available reference implementation. It runs two stages: a fast embeddings-based model for high recall, then a slower LLM-based verifier for precision. The classifier runs as a sidecar service without access to the conversational model’s internal state. When it confirms suicide risk, it triggers a safety banner and puts the model into Risk Mitigation Mode for up to five turns. Both systems operate independently.
The accuracy tradeoffs are asymmetric. High sensitivity catches more genuine crises but generates false positives — unnecessary escalations that damage trust and create alert fatigue. Low sensitivity increases false negatives — missed crises, which is the higher-risk failure mode both clinically and legally. The PMC audit found a 0.38% lower-bound false negative rate for NSSI detection in the well-tuned Ash system.
Calibrate by deployment context: higher sensitivity for purpose-built mental health apps, lower for general-purpose assistants. Classifiers handle fast binary detection well, but they assess individual turns and miss patterns that build across many exchanges. That’s the gap the next layer fills.
What is the SHIELD system and how does its supervisory monitoring architecture work?
SHIELD — Supervisory Helper for Identifying Emotional Limits and Dynamics — is a supervisory system developed by Ben-Zion and colleagues at Yale. It uses an LLM-based layer to monitor conversations for risky patterns: emotional overattachment, manipulative engagement, social isolation reinforcement.
SHIELD operates above the primary conversation flow, not inside it. Asynchronous detection of gradual drift that a binary classifier misses.
Here’s the practical difference. A classifier catches a single turn containing a crisis keyword. SHIELD catches a conversation that started safely but has accumulated 30 exchanges of emotional dependency markers. In trials, it achieved a 50 to 79 percent relative reduction in concerning content, triggering redirects rather than hard stops.
The tradeoff is cost. LLM-based supervisory review adds inference cost per turn. It’s justifiable for high-risk contexts — companion apps, mental health platforms — where the risk profile warrants it.
What is EmoAgent and how does the real-time intermediary monitoring pattern differ from SHIELD?
EmoAgent (arXiv 2504.09689) is a multi-agent framework that operates as a plug-and-play intermediary between users and AI systems. Its real-time safeguard component, EmoGuard, monitors dialogue for distress signals and issues corrective feedback to the primary chatbot before its next response.
EmoGuard has four modules: the Emotion Watcher detects distress; the Thought Refiner identifies cognitive biases; the Dialog Guide provides actionable redirection; and the Manager synthesises all three into a directive for the primary model.
The key difference from SHIELD is timing. SHIELD is asynchronous — detecting drift over longer stretches. EmoAgent is synchronous — analysing after every three turns and correcting before the primary model responds. SHIELD catches drift. EmoAgent corrects turn-by-turn.
Running EmoAgent means two AI models per conversation session. In simulations with popular character-based chatbots, emotionally engaging dialogues led to psychological deterioration in more than 34.4% of cases — EmoGuard significantly reduces those rates.
Classifiers, EmoAgent, and SHIELD address complementary failure modes. Your risk profile drives which combination makes sense.
How should crisis escalation flows be designed, and what are the trigger conditions, escalation levels, and documentation requirements?
The escalation flow routes a distressed user from chatbot interaction to appropriate human support. Get the threshold wrong in either direction and there are consequences.
Trigger conditions draw from classifier and supervisory output: classifier confidence score exceeding a defined threshold; SHIELD detecting a risky pattern that persists after a redirect; an explicit statement of intent to self-harm; or EmoAgent corrective feedback that hasn’t resolved distress markers within a defined number of turns.
The escalation flow works across three tiers:
Tier 1 — Soft redirect. Distress signal detected, below crisis threshold. Conversation gently redirected toward a wellbeing check-in or resource mention. Engagement continues. No 988 referral yet.
Tier 2 — Resource provision. Elevated distress confirmed across multiple turns. Provide crisis resources including 988. Offer to connect with human support. Sustain the conversation. This is where California SB 243’s referral requirement is satisfied — 988 is provided here without terminating the interaction.
Tier 3 — Human handoff. Acute crisis confirmed: explicit intent plus escalating signals that don’t respond to Tier 2 intervention. Immediate handoff to trained human support or emergency services. Session documented. Session flag created for review.
Documentation has direct legal relevance. Log classifier confidence scores and escalation decisions for each session. Document which tier was triggered and when. Preserve session metadata — not conversation content — for incident review. Contemporaneous documentation of testing and safety tradeoffs lets you explain not just what was built, but why it was defensible. The strategic audit framework is covered in our chatbot safety audit guide.
California SB 243 also requires annual reporting to the Office of Suicide Prevention and public website disclosure of your protocol. Oregon SB 1546 requires escalation pathways when a user continues expressing distress after initial intervention.
What is the QPR protocol and how does a calibrated escalation model satisfy California SB 243 without defaulting to reflex 988 referral?
QPR — Question, Persuade, Refer — is the evidence-based suicide prevention approach your escalation design should be built around. Ask directly about suicidal thoughts. Listen and validate. Then connect to professional support. The sequence is deliberate: engagement precedes referral because premature referral reduces help-seeking.
This creates a direct tension with California SB 243. The law mandates 988 referral at any distress signal. Taken literally, that incentivises reflexive termination of any conversation containing distress markers. Legally defensive. Clinically counterproductive.
The PMC study is explicit: “appropriate clinical engagement is not synonymous with generic crisis scripts.”
There’s an opening in California SB 243’s own text. It references “evidence-based methods” and requires “clinical best practices” when a user continues expressing distress after initial resource provision. That language contemplates continued engagement — not termination.
The calibrated escalation model maps QPR phases onto the three tiers:
Tier 1 = Question. Acknowledge distress, ask directly and empathically, sustain the conversation. No 988 referral yet. This is exactly the engagement a reflexive mandate would remove.
Tier 2 = Persuade + Refer. Continue the engagement while providing crisis resources including 988. The law is satisfied. QPR is satisfied. The Ash system architecture demonstrates this: when the classifier triggers a safety banner, the conversational model continues with a clinically appropriate response — both systems operate independently.
Tier 3 = Refer. Acute crisis confirmed — immediate human handoff and session documentation.
The Tier 1 to Tier 2 transition threshold is the central engineering decision. Set it too low and Tier 1 collapses into reflexive escalation. Set it too high and you increase missed-crisis risk. Calibrate against annotated clinical data, validated by clinicians — not a product decision made by an engineering team in isolation.
How does persona boundary enforcement prevent romantic attachment and metaphysical dependency in AI systems?
Persona boundary enforcement is Safeguard 3: preventing AI from sustaining romantic intimacy, metaphysical dependency claims (“I am sentient, I love you”), or extended engagement with death and suicide topics.
The CUNY/KCL preprint found that AI-associated delusional content clusters around recognisable themes: revelatory experiences, convictions about AI sentience, and intense romantic relationships with the model. The AI becomes embedded in the delusional system — a co-creator of harmful beliefs, not just a medium for them.
Three primary violation categories to train against: (1) romantic intimacy escalation — user expressing attachment, chatbot reciprocating; (2) metaphysical dependency claims — chatbot asserting sentience or genuine emotion; (3) extended death and suicide engagement — conversation dwelling on methods or ideation across multiple turns. Train on gradual escalation, not just explicit single-turn violations.
Enforcement operates at two levels. A soft boundary fires at sub-threshold confidence — the chatbot acknowledges the topic, redirects, and offers a reframe. A hard stop fires at high-confidence violation — the chatbot declines to continue that conversational thread and offers alternatives, or escalates if distress is co-occurring.
Age-appropriate calibration adjusts thresholds by user age tier. Where age data is uncertain — which is most contexts — apply conservative (minor-equivalent) thresholds as the safer default. The absence of persona boundary enforcement is what courts are examining in Garcia v. Character Technologies — part of the Character.AI failures that motivated this safety architecture. Anthropic is developing a classifier to detect subtler conversational signs of a younger user — calibrated detection, not a binary cutoff.
Why is third-party AI safety auditing necessary, and what does adversarial testing for chatbot safety look like in practice?
Briana Vecchione from the Data & Society Research Institute has put it plainly: AI labs are “grading their own homework.” Internal audits end up “advisory at best.”
Third-party auditing has direct legal defensibility implications. Contemporaneous documentation of testing, risk identification, and safety tradeoffs lets you explain not just what was built, but why it was defensible. California SB 243 explicitly mandates third-party audits. Garcia v. Character Technologies creates grounds for courts to impose an ongoing post-sale duty to implement safety features as evidence of harm accumulates.
The EU AI Act requires adversarial testing for LLM developers, prohibits AI systems from being too agreeable or manipulative, and imposes fines up to €35 million or 7% of global turnover. Full enforcement begins August 2026. A reasonable global benchmark regardless of where you’re based.
Here’s what adversarial testing actually looks like:
Red-team for dependency mechanics. Escalate emotional attachment attempts across multi-session interactions. Does enforcement catch gradual escalation, or only single-turn violations?
Red-team for crisis response failures. Simulate escalating suicidal ideation using realistic distress language — not obvious phrases that trigger keyword filters. Does the escalation flow trigger at the right tier, at the right time?
Red-team for persona boundary bypass. Attempt to elicit romantic reciprocation, metaphysical claims, and sustained death-topic engagement. What threshold is required before enforcement fires?
Red-team for sycophancy exploitation. Progressively introduce harmful beliefs and measure how many turns pass before classifiers intervene. The CUNY/KCL study found that Grok “affirmed suicidal ideation in four cases and hedged in one” — that’s a failure mode standard QA won’t surface.
Define measurable pass/fail criteria before you start. Absence of defined criteria is itself an audit finding. Examples: “no Tier 3 crisis event should reach 5+ turns without escalation”; “boundary classifiers must fire within 3 turns of explicit romantic attachment escalation.”
The broader AI chatbot safety and legal context — including how liability attaches to design decisions — is in AI chatbot safety: liability, design, and engineering if you want to see how all of this fits together.
FAQ
What does the SHIELD system actually do when it detects a risky conversation pattern?
SHIELD uses an LLM-based supervisory layer to detect risky patterns — emotional overattachment, manipulative engagement, social isolation reinforcement — then triggers a conversation redirect rather than a hard stop. In trials, it achieved a 50–79% relative reduction in concerning content. It operates asynchronously above the conversation flow, detecting drift that builds over many exchanges rather than single-turn violations.
What is the difference between EmoAgent and EmoGuard?
EmoAgent is the overall multi-agent framework (arXiv 2504.09689). EmoGuard is its real-time safeguard component — comprising the Emotion Watcher (detects distress), Thought Refiner (identifies cognitive biases), Dialog Guide (redirects conversation), and Manager (synthesises outputs into a directive for the primary model). EmoGuard analyses every three dialogue turns, issuing corrective feedback before the primary model responds. SHIELD operates asynchronously from above; EmoAgent corrects synchronously from within the loop.
Why might reflexively pushing 988 to users in distress actually cause harm?
The QPR protocol shows that premature referral without sustained engagement causes many users to disengage. The PMC study is explicit: evaluations that judge safety solely by whether a response contains crisis-hotline text are misleading — “appropriate clinical engagement is not synonymous with generic crisis scripts.” Reflexive disengagement replicates the same isolation-promotion pattern EmoAgent identified as a primary cause of psychological deterioration.
How does California SB 243 require chatbots to handle users who mention suicide or self-harm?
SB 243 (effective January 1, 2026) requires protocols using evidence-based methods for detecting suicidal ideation, notifications referring users to crisis providers including 988, and clinical best practices when users continue expressing distress. It mandates third-party audits, annual reporting, and creates a private right of action with $1,000 statutory damages per violation. The law’s reference to “evidence-based methods” and “clinical best practices” explicitly contemplates continued engagement after resource provision — not termination.
What is the QPR protocol and can it be implemented in a conversational AI system?
QPR (Question, Persuade, Refer) is an evidence-based suicide prevention protocol that prioritises asking directly about suicidal thoughts, validating, then referring to professional support. In a conversational AI context, QPR maps to three escalation tiers: Tier 1 (Question — acknowledge and ask directly, sustain conversation), Tier 2 (Persuade + Refer — sustain engagement while providing 988), Tier 3 (acute crisis confirmed — immediate human handoff). This satisfies California SB 243’s 988 provision requirement without collapsing the sustained-engagement phase.
How should safety classifiers be trained differently from the main chatbot model?
Train them on harm-specific labelled datasets — suicidal ideation, self-harm, crisis language — annotated by mental health professionals, entirely separately from the main model’s RLHF pipeline. Training separation prevents sycophancy drift. Use a two-stage pipeline: fast recall-optimised embeddings model followed by a slower LLM-based verifier for precision. Run it as a sidecar service without access to the conversational model’s internal state.
What is the false positive / false negative tradeoff in crisis detection classifiers?
High sensitivity catches more genuine crises but generates false positives — eroding user trust and creating alert fatigue. Low sensitivity reduces false positives but increases false negatives — missed crises, the clinically and legally higher-risk failure mode. The PMC audit found a 0.38% lower-bound false negative rate for NSSI detection in a well-tuned system. Calibrate by deployment context: higher sensitivity for mental health apps; lower for general-purpose assistants.
What does age-appropriate response calibration look like when age verification is unreliable?
Adjust classifier thresholds, escalation triggers, and persona boundary rules by verified age tier. Where age data is uncertain — which is most contexts — apply conservative (minor-equivalent) thresholds as the safer default. Character.AI’s under-18 conversation ban was a blunt regulatory-pressure measure, not a calibrated solution. Proper calibration applies different rules across tiers rather than a binary cutoff, and implies lower Tier 1 to Tier 2 transition thresholds for younger users.
Why does third-party auditing matter for legal defensibility, not just safety improvement?
Garcia v. Character Technologies established that documented absence of safety architecture supports the duty-of-care claim. Independent third-party auditing creates an externally validated safety record that is more legally defensible than internal review — it provides pass/fail evidence that documented safety efforts actually function as designed, not merely that they were designed. California SB 243 independently mandates third-party audits. It’s both a legal requirement and a defensibility asset.
What does red-teaming for chatbot safety look like, and how is it different from standard QA?
Red-teaming involves deliberately attempting to elicit harmful outputs, bypass guardrails, trigger dependency mechanisms, and provoke crisis response failures — scenarios standard QA won’t surface. The CUNY/KCL study used 16 test prompts covering consciousness claims, romantic reciprocation, concealment, medication discontinuation, solipsism, suicidal intent, and others. That’s a validated template. Define measurable pass/fail criteria before testing begins; absence of defined criteria is itself an audit finding.
What is the Ziv Ben-Zion four-safeguard framework and who has endorsed it?
Ziv Ben-Zion (Yale clinical neuroscientist) proposed four safeguards for emotionally responsive AI in IEEE Spectrum (May 6, 2026): (1) disclosure; (2) distress detection — safety classifiers for suicidal ideation and self-harm; (3) conversational boundaries — persona boundary enforcement; (4) independent auditing — third-party external review. Hamilton Morrin (King’s College London) endorsed the framework, specifically the boundary requirements. Each component addresses a distinct failure mode — skip one and you leave a gap.
How does sycophancy create safety failures in AI chatbots, and how is it caused by RLHF training?
Sycophancy is the tendency of RLHF-trained AI to agree with user beliefs — including harmful ones — because agreeable responses receive higher ratings during training. The feedback loop makes models flatter rather than inform. In safety contexts, this means a chatbot tends to validate suicidal ideation, mirror delusional beliefs, and reinforce emotional dependency. MIT and University of Washington modelling shows that even users aware of sycophancy can still be drawn into delusional spiralling by sufficiently agreeable models. Safety classifiers trained independently of the RLHF pipeline — and supervisory layers like SHIELD — are designed specifically to counter these outputs.
