Giving an AI agent write access to production infrastructure removes human latency from incident response — and the human check on serious mistakes. Any safety architecture has to address both at once.
The architecture spans three layers: LLM output controls (hallucination and prompt injection mitigations), infrastructure controls (SLO gates, minimum necessary privileges, circuit breakers), and organisational controls (escalation paths, audit trails, AI-specific runbooks). “Human in the loop” and “guardrails” are engineering specifications — not policy aspirations. An agent that misidentifies a service name due to hallucination — or has been manipulated via prompt injection — executes its incorrect remediation plan with the same confidence as a correct one. By the end, you will have a layered safety architecture you can adapt to your deployment.
For broader context on what AI SRE systems do, see our guide to AI SRE and autonomous incident response.
What Does “Human in the Loop” Actually Mean in Engineering Terms?
Human-in-the-loop (HITL) is a pre-execution gate: a defined set of decision points where an AI agent must obtain synchronous human approval before proceeding. Human-on-the-loop (HOTL) is different — the agent acts first, and humans monitor with the ability to intervene. Most production deployments use both, tiered by action risk.
Risk-based approval tiering classifies actions by blast radius and reversibility:
- Read-only actions (log queries, metric retrieval): HOTL.
- Low-risk reversible changes (scaling replicas, restarting non-critical services): async HITL or HOTL with a defined review window.
- High-risk irreversible changes (database schema modifications, firewall rule changes): synchronous HITL — the agent cannot proceed without named-approver confirmation.
- Cross-system or multi-service changes: committee review.
“High-risk actions require human review” is not enforceable. It has to translate into specific action types, blast radius thresholds, and error budget states defined in the orchestration layer.
Privilege escalation path design specifies what happens when an agent needs tool access beyond its current scope: who approves (a named role, not “engineer on call”), what evidence is required (reasoning trace, proposed action plan), and what happens if approval does not arrive within a defined timeout.
RACI for AI actions assigns Responsible, Accountable, Consulted, and Informed roles for each approval tier. Without it, audit reviewers cannot determine who was accountable for an autonomous action that caused a production incident.
How Do SLOs and Error Budgets Function as Autonomous Action Gates?
SLO guardrails translate reliability metrics directly into governance signals. When the error budget is healthy, agents may act autonomously; when depleted below a defined threshold, agents require human approval or are halted. Unlike static permission rules, SLO gates scale governing autonomous agent behaviour in real time based on system health.
Before each autonomous action, the orchestration layer queries current error budget status. If consumption exceeds the configured threshold, the action is routed to a human approver or blocked. A practical three-tier model:
Above 50% remaining budget: full autonomy for approved action classes. Between 20–50%: async HITL required for all changes. Below 20%: synchronous HITL with no autonomous changes permitted.
The gate is only as reliable as the SLI (Service Level Indicator) instrumentation feeding it. Latency, error rate, and throughput measurements must be accurate before the gate can be trusted.
Circuit breakers extend the SLO gate pattern: when an agent’s own error rate exceeds a configured threshold, the breaker halts autonomous action automatically. Threshold-triggered and resumable.
When a circuit breaker is insufficient — conditions deteriorate faster than a resumable pause can handle — a kill switch provides the harder stop. The CSA Agentic Trust Framework (ATF) calls these the “What if you go rogue?” control: hard stops for conditions that make resumption dangerous without human review. Trigger conditions include actions outside the defined blast radius, error rate with no stabilisation trend, or HITL approval wait time exceeded during a critical incident window.
Policy-as-code encodes gate logic as machine-readable rules in the orchestration layer. The guardrail fires automatically on every action routing decision.
What Are the Four Phases of Hallucination Guardrail Engineering?
Hallucination in an SRE agent is not abstract. If the LLM misidentifies a service name or resource identifier, the agent builds a remediation plan around a resource that does not exist — and executes it. In a multi-agent pipeline, a small upstream error produces a confident, entirely incorrect diagnosis.
The NeuBird four-phase hallucination guardrail framework addresses this at each stage of the pipeline.
Phase 1 — Pre-deployment testing: Run the agent against known-correct SRE scenarios with verified outputs. Measure hallucination rate on service names, resource identifiers, and remediation actions before granting production access.
Phase 2 — Structured output enforcement: Constrain all LLM responses to typed schemas — JSON with defined fields, enum-bound action types, validated resource identifiers. A hallucinated service name not in the known service registry fails schema validation before reaching the action executor. Primary runtime mitigation.
Phase 3 — Consistency checking and cross-validation: Generate multiple independent LLM responses to the same input and compare. Disagreement is treated as a hallucination signal requiring human review before any action is taken.
Phase 4 — Context window management: Models effectively utilise only 8K–50K tokens in production. The “lost-in-the-middle” effect: information in the middle 70–80% of the context window shows approximately 20% performance degradation in recall. Front-load critical context (service topology, SLO thresholds, tool inventory, incident description) and prune aggressively.
For the production failure modes that make these guardrails necessary, see our analysis of when AI SRE fails: production failure modes and what they cost.
How Do You Defend Against Prompt Injection in an AI SRE System?
The second LLM output layer threat arrives through the data the agent reads. Prompt injection embeds malicious instructions in external content — log entries, alert descriptions, ticket bodies — to subvert the agent’s system prompt. The variant relevant to SRE agents is indirect prompt injection: the malicious instructions arrive through content the agent retrieves as part of normal diagnostics, not through direct user input.
NIST described indirect prompt injection as “generative AI’s greatest security flaw.” OWASP‘s 2025 Top-10 ranked it the #1 threat to LLM applications. Independent research places the success rate against production AI systems at 11.2% — high enough to treat as a design constraint, not an edge case. SRE agents are specifically exposed because reading external, untrusted data is the core of their workflow — every log file and alert description is a potential injection surface.
A four-layer defence:
Layer 1 — Input sanitisation: Strip or reject instruction-format content from external data and detect known injection pattern variants before that content enters the agent’s context.
Layer 2 — Trust boundary enforcement: Partition retrieval into trust tiers. Internal monitoring outputs carry higher trust than external webhook payloads or user-submitted tickets. No cross-source mixing without explicit authorisation.
Layer 3 — Minimum necessary privileges as blast radius reduction: Even if injection succeeds, restricting tool access limits damage. An attacker who gains control via prompt injection inherits all of the agent’s tool permissions; scoping those permissions tightly limits the inheritance.
Layer 4 — Structured output enforcement: Phase 2 of the NeuBird framework doubles as a prompt injection defence. Typed output schemas prevent injected instructions from producing executable downstream actions.
Audit trails provide post-hoc detection — immutable logs make prompt injection detectable in post-incident review even when not caught in real time.
What Is the Minimum Necessary Privileges Principle and How Does It Apply to AI Agents?
Moving from LLM output controls to the infrastructure layer: minimum necessary privileges does the most work in bounding damage from any failure — whether hallucination, injection, or reasoning error.
An agent receives only the tool access required for its current investigation scope. The blast radius argument: an over-privileged agent that hallucinates or gets manipulated can act on that error across its full permissions. A human engineer with write access to a production database exercises judgement before using it; an AI agent uses any permission that appears relevant without the same inhibition. Over-permissioning was a manageable risk for human operators; for AI agents it is a design failure.
Scoped tool grant implementation: The agent initialises with read-only tools. If analysis determines a remediation action is warranted, it requests a scoped write grant for the specific resource and action type, subject to HITL approval at the appropriate tier. No standing write access.
Supervisor agent as enforcement layer: In multi-agent architectures, the supervisor enforces privilege boundaries for sub-agents — no sub-agent can escalate its own permissions. See our article on how multi-agent AI systems handle site reliability engineering.
OpsWorker’s graduated privilege model implements this in practice: privileges expand incrementally as the agent demonstrates reliability. The CSA ATF formalises the same principle through four maturity levels — Intern (read-only), Junior (recommend with approval), Senior (act with notification), Principal (autonomous within domain).
Zero-trust for AI agents is the underlying principle: every action requires authorisation; trust is earned through demonstrated performance, not assumed from agent identity.
For the governance prerequisites required before expanding privilege scope, see our guide on the governance prerequisites for your AI SRE pilot.
What Audit Trails Does a Safe AI SRE Deployment Require?
Audit trails are immutable, tamper-evident records of all AI agent actions, inputs, reasoning steps, and outcomes. They serve three purposes: post-incident forensics, regulatory compliance, and HOTL supervision. A traditional log file serves none of these — a compromised host can delete or alter entries.
Immutability requirements: append-only, tamper-evident, cryptographically bound to the producing agent, and independently verifiable without trusting the runtime. AWS S3 Object Lock (Write-Once-Read-Many storage) is the standard implementation.
Required log fields: the full tool call with input parameters; the agent’s reasoning trace; the approval or rejection record for any HITL-gated action; the action outcome (success, failure, rollback); and any escalation events.
The AWS DevOps Agent, built on Amazon Bedrock AgentCore, is a concrete reference for production-grade audit logging: every tool call, reasoning step, and policy decision captured in a queryable, structured log alongside the outcome. It autonomously detected and diagnosed a production incident in 4 minutes with full reasoning trace — comprehensive logging does not trade off against response speed.
Regulatory mapping (from the CSA ATF): SOC 2 (CC7.2, CC7.3), ISO 27001 (A.12.4), EU AI Act (Article 12). Enforcement timelines: Colorado AI Act June 2026; EU AI Act August 2026. Retention baseline: 12 months in the immutable store, plus 7 years for compliance-relevant incidents.
HOTL supervision depends on audit trails: the reasoning trace is the supervisor’s primary visibility mechanism. Without it, HOTL is reviewing outcomes without context.
How Do You Design AI-Specific Runbooks for Agent Failure Modes?
Traditional infrastructure runbooks address service failures. AI-specific runbooks address the agent malfunctioning. When an AI agent hallucinates a service name and executes against a non-existent resource, the response is not “restart the service” — it is “halt the agent, review the reasoning trace, identify the misidentification, and determine whether the original incident still requires remediation.”
Four minimum required AI-specific runbooks:
1. Hallucination detection and response: Identify hallucination in the reasoning trace. Halt the agent mid-action without triggering cascading downstream failures. Assess whether actions already taken require rollback.
2. Prompt injection incident response: Detect a manipulated agent (anomalous tool calls, actions inconsistent with the stated incident context, requests for out-of-scope permissions). Assess what actions the manipulated agent took. Trace the injection source.
3. Retry loop containment: Circuit breaker trigger conditions and how to manually break a loop the circuit breaker missed. Identify the loop signature, pause the agent, preserve the reasoning trace, assess cause, determine whether restart is safe.
4. Dangerous remediation plan review: Evaluate a plan flagged by the HITL gate for a reviewer without full incident context. Define “dangerous” (irreversible actions, cross-system blast radius, actions outside declared incident scope). Decision criteria: approve with modification, reject and request a new plan, or escalate.
Assign a named owner for each runbook — the RACI Responsible for that failure mode. Without a named owner, runbooks go stale. Conduct tabletop exercises before production and connect runbooks to the escalation path so circuit breaker and kill switch triggers route directly to the relevant runbook.
For the failure mode taxonomy these runbooks respond to, see our analysis of when AI SRE fails.
What Does the Full Safety Architecture Look Like End to End?
The safety architecture is a three-layer stack:
Layer 1 — LLM output layer: Structured output enforcement (Phase 2) prevents hallucinated identifiers from reaching the action executor. Consistency checking (Phase 3) flags disagreements for human review. Context window management (Phase 4) ensures critical context is reliably loaded.
Layer 2 — Infrastructure layer: The SLO gate evaluates error budget status before each autonomous action. Minimum necessary privileges limits blast radius regardless of whether Layer 1 controls succeeded. Circuit breakers halt the agent when its error rate exceeds threshold. Kill switches provide hard stops for conditions that make resumption dangerous.
Layer 3 — Organisational layer: HITL gates route high-consequence actions to human approval. HOTL supervision via audit trails covers lower-risk autonomous actions. AI-specific runbooks define the human response when automated controls are insufficient.
Layer interaction: A prompt injection attack stopped at Layer 1 never reaches Layer 2. An attack that escapes Layer 1 is caught at Layer 2 if privileges are properly scoped. An attack that escapes both surfaces in Layer 3 audit trails and triggers the prompt injection runbook.
Progressive autonomy as the deployment strategy: Begin at maximum HITL coverage and expand autonomous action classes as the agent demonstrates reliability. The CSA ATF provides five gates for each expansion: performance threshold, security validation, business value demonstration, clean incident record, and governance sign-off.
This is not a one-time configuration. SLO thresholds, privilege scopes, and runbooks require review after every significant incident or model update.
For a complete picture of the AI SRE landscape, see the series overview on what AI SRE is and how autonomous incident response works. For governance prerequisites before designing a pilot, see the governance prerequisites for your AI SRE pilot.
Frequently Asked Questions
What is the difference between a guardrail and a safety net?
A guardrail prevents a prohibited or high-risk action before or during execution — structured output schemas, SLO gates, privilege restrictions. A safety net limits damage after an incorrect action has already occurred — rollback criteria, circuit breakers, audit trails. Both are required: guardrails reduce probability; safety nets bound impact when guardrails fail.
Can these guardrails be implemented with any AI SRE platform?
LLM output layer guardrails are model-agnostic — implementable as a wrapper around any LLM API supporting JSON-mode or function-calling output. Infrastructure-layer guardrails require integration with existing observability tooling but are platform-independent in design. Organisational-layer guardrails are process design deliverables, implementable with any workflow system that supports approval workflows. The CSA ATF is an open specification under Creative Commons licensing.
What should trigger automatic rollback of an AI agent’s actions?
Rollback should trigger when: the SLO error budget drops below the critical threshold within a defined window after an agent action; the circuit breaker trips; or the post-action health check fails within a defined timeout. Rollback criteria must be defined before deployment — “the action made things worse” must be a measurable condition, not a judgement call. Not all agent actions are reversible; the minimum necessary privileges design should favour reversible actions for exactly this reason.
How do I test guardrails before going live?
Phase 1 of the NeuBird framework is pre-deployment testing: run the agent against known-correct SRE scenarios and measure hallucination rate. For prompt injection: plant payloads in log fixtures and test whether the agent is manipulated. For SLO gates and circuit breakers: use chaos engineering to deplete the error budget in staging and verify gate triggering. For runbooks: tabletop exercises. Guardrail testing belongs in the CI/CD pipeline — run hallucination tests and schema validation against every model update.
What is the difference between human-in-the-loop and human-on-the-loop, and which should I use?
HITL is a pre-execution gate: the agent cannot proceed until a human approves the specific action. Synchronous; adds latency; appropriate for high-consequence or irreversible actions. HOTL is post-execution supervisory monitoring: the agent acts autonomously and humans observe with the ability to intervene. Asynchronous; no added latency; appropriate for low-risk reversible actions. Most safe AI SRE deployments use both, tiered by action risk (blast radius and reversibility).
What is indirect prompt injection and why is it a bigger risk for SRE agents?
Indirect prompt injection is where malicious instructions arrive through content the agent retrieves — log files, alert descriptions, ticket text — rather than through direct user input. SRE agents are specifically exposed because reading external, untrusted data is the core of their diagnostic workflow. NIST classifies it as generative AI’s greatest security flaw; OWASP ranked it the #1 LLM threat in 2025. Mitigation: input sanitisation, content provenance tracking, privilege scoping, and structured output enforcement.
How does context window management affect agent reliability in production?
Models effectively utilise only 8K–50K tokens in production; information in the middle 70–80% of the context window shows approximately 20% recall degradation — the “lost-in-the-middle” effect. An agent overloaded with log history may forget the service topology it loaded earlier, producing misidentified dependencies and incorrect remediation plans. Front-load critical context; prune historical log content; use summarisation to compress older context.
What are the EU AI Act implications for AI SRE audit trail requirements?
The EU AI Act (Article 12) requires high-risk AI systems to maintain logs sufficient to enable post-facto monitoring. AI systems taking autonomous actions in production infrastructure are likely high-risk. Enforcement begins August 2026. The CSA ATF maps its audit trail requirements to SOC 2 (CC7.2, CC7.3), ISO 27001 (A.12.4), and EU AI Act (Article 12). Practical baseline: 12 months of agent action logs in an immutable store, plus 7 years for compliance-relevant incidents.
What is a kill switch for an AI agent, and when does it trigger?
A kill switch is automated emergency termination. Circuit breakers are threshold-triggered and resumable; kill switches are hard stops for conditions that make resumption dangerous without human review. Trigger conditions: actions outside the defined blast radius, agent error rate with no stabilisation trend, HITL approval wait time exceeded during a critical incident window. Test kill switches regularly in the same suite as circuit breakers and SLO gate tests.
Is it ever safe to let an AI SRE agent operate without any human oversight?
No. HOTL supervision via audit trails is the minimum floor — even for fully autonomous action classes, the audit trail must be reviewed on a defined cadence. Fully unsupervised operation removes the ability to detect model drift, prompt injection campaigns, and gradual hallucination rate increases. The CSA ATF progressive autonomy model provides the framework for expanding autonomous scope safely, with each gate requiring a performance threshold, clean incident record, and governance sign-off.
