In the first half of 2026, clinical AI crossed a threshold. ChatGPT Health launched with patient-record integration and reached tens of millions of daily health queries. Claude for Healthcare entered production the same month. The Mayo Clinic and Microsoft announced a partnership to train a frontier model on 54 million de-identified patient records. And Kaiser Permanente completed the largest generative AI rollout in healthcare history, deploying Abridge’s ambient documentation across 40 hospitals and 600-plus medical offices.
These are production systems operating at population scale.
The governance architecture that should ensure their safety was designed for static medical devices cleared one at a time, not for continuously learning systems deployed to millions of patients simultaneously. The FDA has cleared over 1,200 AI-enabled medical devices, overwhelmingly under pathways built for products that do not change after approval. The regulatory philosophy has shifted, the Total Product Lifecycle framework now acknowledges that AI safety cannot be established at a single point in time, but the operational infrastructure for continuous monitoring does not yet exist at scale.
This series traces the structural gaps that have opened between deployment velocity and regulatory infrastructure: the accuracy problem that no benchmark catches, the regulatory framework stretched beyond its design limits, the accountability vacuum when harm occurs, and the cross-industry governance lessons that could close these gaps before they produce outcomes that force the issue.
In This Series
- The Clinical AI Benchmark to Bedside Accuracy Gap — Why clinical AI scores 95% on benchmarks but drops to 34% at the bedside, and what the Stanford-Harvard ARISE 2026 audit revealed about the evaluation-to-deployment pipeline
- How the FDA Regulates Clinical AI From Approval to Real-World Safety — The Total Product Lifecycle framework, the January 2026 CDS guidance revision, and why fewer than 15% of 1,200 FDA-cleared devices have published real-world outcomes
- Who Is Accountable When Clinical AI Causes Patient Harm — The liability vacuum, the state-level legislative patchwork, and what the 74%/78% patient trust asymmetry means for governance
- What Clinical AI Governance Can Learn From Financial Services — How SR 11-7 model risk management, champion-challenger testing, and the agentic AI frontier could reshape clinical AI oversight
What Is the Current State of AI Deployment in Clinical Settings?
Kaiser Permanente’s Abridge rollout across 40 hospitals and over 600 medical offices was the fastest technology implementation the organisation had executed in more than twenty years. That one deployment captures where clinical AI is in mid-2026: not experimenting, not piloting, but operating at production scale across entire health systems.
ChatGPT Health handles tens of millions of daily health queries with patient-record integration. Claude for Healthcare reached production with HIPAA-ready tools that connect directly to PubMed’s 35 million biomedical literature pieces. The Mayo Clinic–Microsoft partnership is training a frontier model on 54 million de-identified patient records, with Mayo Clinic’s president and CEO describing the goal as “building something healthcare has never seen before.” Ambient clinical documentation, AI-powered scribing that listens to patient encounters and generates clinical notes, reached $600 million in 2025 spend and covers 77% of US hospitals through Nuance DAX alone.
What “regulated system” means here is the heart of the problem. The FDA has cleared over 1,200 AI-enabled medical devices, overwhelmingly under the 510(k) pathway designed for static products that demonstrate substantial equivalence to an existing device. The Total Product Lifecycle framework, developed more recently, acknowledges that AI safety needs ongoing monitoring, not a one-time check. But the operational infrastructure for that monitoring, the systems and standards that would detect when a deployed AI tool starts degrading in a specific hospital with a specific patient population, does not exist at regulatory scale. This is the gap the series examines: regulation designed for the last generation of medical devices is now being asked to govern the next generation of AI systems.
Four dimensions define this governance problem, and they stack. The accuracy gap means we cannot reliably measure what we are deploying. The regulatory framework is philosophically correct but operationally incomplete. The accountability vacuum means that when these systems fail, no clear legal pathway exists for the patients they harm. And cross-industry governance models exist but have not been transferred. Together they are what this series maps.
What Is the Benchmark-to-Bedside Accuracy Gap, and Why Does It Matter?
Clinical AI models routinely score above 95% on standardised medical benchmarks: exam-style questions, curated image sets, retrospective chart reviews. When the same models are evaluated in real clinical environments with live patient data, incomplete information, and workflow complexity, accuracy drops to roughly 34%. This is a measurement failure. Benchmarks measure performance under conditions that do not resemble clinical practice. Regulatory clearance decisions, hospital procurement, and clinician trust are all built on benchmark evidence that systematically overstates real-world capability.
The core finding keeps surfacing across different studies. A systematic review of 39 medical LLM benchmarks quantified a 39 to 45 percentage point knowledge-practice performance gap: LLMs achieve 84% to 90% accuracy on knowledge-based medical examinations, but practice-based clinical competence drops to 45% to 69%, with safety assessments at only 40% to 50%. In simulated dialog-based diagnostic scenarios, advanced models achieved only 34.2% accuracy. The PhysicianBench evaluation, testing LLM agents across 100 real EHR-based tasks spanning 21 specialties, found the best-performing model achieved only 46% success rate. These are measurements from the systems being deployed.
The mechanism behind the gap is the benchmark engineering lifecycle. Benchmarks are created, models train against them, performance saturates quickly because benchmarks are static, and then the models are deployed into environments that look nothing like the test. Only 5% of 761 LLM evaluation studies assessed performance on real patient care data. The rest relied on medical examination questions. Meanwhile the ARISE 2026 audit from Stanford and Harvard revealed that among 1,200 FDA-cleared AI medical devices, fewer than 15% have published real-world outcomes data. The regulatory system is clearing devices on pre-market evidence without requiring post-deployment proof that the devices work in practice. The gap is systematic and measurable.
Why the gap matters goes beyond the statistics. It is the reason lifecycle regulation is necessary: if benchmarks reliably predicted real-world performance, static pre-market clearance would be sufficient. It is the mechanism by which liability arises: when a system operating at 34% accuracy produces a harmful recommendation, the question is not whether harm will occur but who bears the consequences when it does. And it is why cross-industry governance models that assume continuous performance monitoring are relevant: financial services never relies on a single pre-deployment validation.
How Prevalent Are Hallucinations in Clinical AI, and Can They Be Mitigated?
A documented case involved a chatbot recommending bromide, a toxic chemical, as a table salt substitute. The recommendation was confident, articulate, and plausible-sounding to non-specialists. Hospitalisation for severe poisoning followed. That is what hallucination looks like in clinical AI: not random noise, but authoritative statements that happen to be false.
Hallucinations are the expected output pattern in the region between benchmark performance and clinical reality. When a system drops from 95% to 34% accuracy, a substantial proportion of its outputs are factually wrong, clinically inappropriate, or internally contradictory. In agent-based clinical AI systems, hallucinations affected approximately 30% of clinical scenarios even after mitigation strategies were applied. They take forms that are dangerous in ways specific to healthcare: fabricated citations, invented drug interactions, plausible-sounding but incorrect treatment recommendations.
There are three main mitigation strategies, each with limits. Retrieval-augmented generation grounds outputs in verified clinical knowledge bases and can reduce hallucinations by 70% to 90%, but adds latency and depends on the quality of the knowledge base itself. Structured output constraints limit model responses to predefined clinical templates, which improves reliability but constrains the clinical reasoning that makes LLMs useful in the first place. Human-in-the-loop review requires clinician verification before any output reaches a patient, but depends on clinician vigilance that automation bias erodes. Clinicians who over-trust AI outputs check less thoroughly, and the verification step that should catch errors compresses.
Hallucinations are the clinical face of the accuracy gap, and they transform that gap from a statistical curiosity into a patient-safety and liability problem. When a hallucinated recommendation contributes to patient harm, the question becomes who is responsible: the developer who could not guarantee output accuracy, the institution that deployed the system knowing hallucinations occur, or the clinician who relied on the output. That question, as we examine in the accountability analysis, currently has no clear legal answer.
How Does the FDA Regulate AI in Clinical Settings?
The FDA regulates clinical AI through the Total Product Lifecycle framework, which extends oversight from pre-market review through post-market surveillance and eventual decommissioning. The January 2026 Clinical Decision Support guidance revision closed a regulatory gap by bringing AI-based decision support tools under FDA scope, tools that previously fell outside the device definition. The framework is philosophically sound: it recognises that AI safety cannot be established at a single point in time. Operationally, it is built on an approval model designed for static devices, and the infrastructure for continuous post-market monitoring does not yet exist at scale.
The Total Product Lifecycle architecture organises regulation around the full lifespan of a device, not just the moment of clearance. Pre-market review happens through three pathways: 510(k) for substantial equivalence to an existing device, De Novo for novel low-to-moderate risk devices, and Pre-Market Authorization for high-risk devices. About 95% to 97% of AI-enabled medical devices have cleared through 510(k), meaning they reached market by showing they were similar to something already approved, not by demonstrating their own safety and effectiveness through prospective clinical trials. Good Machine Learning Practice principles, endorsed by the FDA, Health Canada, and the UK’s MHRA, provide cross-stakeholder development guidance, but they are principles, not requirements.
The January 2026 CDS guidance revision changed what counts as a regulated device. Before 2026, the 21st Century Cures Act created exclusion criteria that allowed many AI-based decision support tools to bypass FDA review entirely if they enabled clinician review of the basis for recommendations. The revision narrowed those exclusions. It was the FDA signalling that it understood the pre-2026 framework was inadequate for the current generation of AI tools. But signalling is not the same as infrastructure. The guidance document is available on the FDA’s Digital Health Center of Excellence website.
The structural limit is the Predetermined Change Control Plan, the FDA’s most innovative attempt to create a regulated pathway for iterative AI updates by pre-approving a change pathway rather than requiring a new submission for each update. PCCPs are new, untested at scale, and do not address unanticipated changes. About 10% of AI clearances in 2025 included an authorised PCCP. The framework for continuous oversight exists on paper. The mechanisms to make it operational do not.
Locked vs Continuously Learning AI — Which Model Is Safer for Clinical Use?
There is no universally safer model, only context-dependent risk profiles. Locked AI models have fixed parameters after approval, offering regulatory clarity and reproducible outputs but degrading silently as deployment data diverges from training data. Continuously learning systems can adapt to new populations and clinical practices, maintaining relevance, but introduce update risk: each change can degrade previously reliable performance or introduce new failure modes. The safety question is not which architecture to choose but whether your monitoring infrastructure can detect degradation in either model before it causes patient harm.
Locked models are the current regulatory default. SkinVision for melanoma assessment, IDx-DR for diabetic retinopathy, these are static systems cleared through existing pathways. Their advantage is predictability: the FDA knows exactly what it approved, and performance should be reproducible. Their risk is silent obsolescence. IDx-DR could not analyse 26.1% of real-world patient images in deployment, particularly in cases of small pupils. Real-world evaluation of SkinVision showed sensitivity ranging from 41% to 83%. These are failures of a regulatory model that assumes a clearance moment is sufficient when the environment the device operates in keeps changing.
Continuously learning systems address this but introduce a new category of risk. Each update is a potential regression, and catastrophic forgetting, where new training data overrides previously acquired capabilities, means improvement in one area can mean degradation in another. The FDA’s PCCP mechanism is designed to create a regulated middle path, allowing manufacturers to pre-specify planned updates: what will change, how it will be validated, what performance boundaries must be maintained. No continuously learning AI medical device has been deployed at scale under a PCCP.
The international comparison is instructive. The EU AI Act classifies medical AI as high-risk, imposing post-market monitoring and incident logging requirements. But the EU Medical Device Regulation currently treats AI as static software: any significant algorithm change requires Notified Body re-certification. The EU has not yet operationalised an equivalent adaptive pathway. Both regulatory systems are philosophically aligned on the need for lifecycle oversight. Neither has the operational infrastructure to deliver it.
Both architectures share the same monitoring prerequisite: you need infrastructure that can detect performance degradation in real time. Locked models degrade from environment change. Continuous models can regress from updates. Without post-market surveillance capable of detecting either failure mode, the locked-vs-continuous debate is premature. The safety of either model depends on monitoring that the current regulatory infrastructure does not systematically provide.
What Is Algorithmic Drift, and Why Does It Threaten Clinical AI Safety?
Algorithmic drift is the degradation of AI model performance when deployment conditions diverge from training conditions: different patient demographics, changing clinical practices, new disease presentations, or shifts in data collection protocols. It is the mechanism by which a proven, cleared AI tool can become unsafe without anyone changing anything. The threat is structural: drift is the problem that post-market surveillance is designed to detect, but fewer than 15% of FDA-cleared AI devices have published real-world outcomes data, meaning the current regulatory infrastructure cannot systematically identify when a deployed AI tool is degrading in clinical use.
Three types of drift matter. Data drift occurs when input data distributions change: a hospital serves a different demographic than the training population, and the model sees patients who look statistically unlike the ones it learned from. Concept drift is subtler: the relationship between inputs and clinical outcomes changes, a new treatment protocol alters what constitutes appropriate care, and the model’s predictions become misaligned with current practice. Catastrophic forgetting is specific to continuously learning systems, where new training data overrides previously acquired capabilities. A radiology AI trained on one demographic group degrading when deployed in a more diverse community hospital is what happens when deployment environments and training environments diverge.
The real-world evidence for drift is accumulating. IBM Watson for Oncology was deployed at multiple institutions but provided unsafe treatment recommendations that were not detected for years. The Epic sepsis model in real clinical use achieved only 33% sensitivity, missing two of three sepsis cases, and 88% of its alerts were false positives. A monitoring study at Erasmus MC across 6,800 admissions demonstrated that data-drift dashboards can reveal input variables that have drifted from training data even when overall performance metrics stay stable, a monitoring methodology that most clinical AI deployments lack.
The FDA’s Total Product Lifecycle framework acknowledges drift by requiring post-market surveillance. The agency has even publicly asked for comment on “current, practical approaches to measuring and evaluating the performance of AI-enabled medical devices in the field.” But the operational toolkit, Green/Amber/Red zone tiered monitoring, shadow deployment testing, causal inference methods for disentangling drift from treatment effects, is not in place at regulatory scale. Healthcare organisations deploying clinical AI are self-insuring against drift risk, with minimal regulatory guidance on how to monitor or what to monitor for.
Who Is Accountable When Clinical AI Causes Patient Harm?
The short answer is that nobody knows with legal certainty, and that is the problem. Liability is distributed across AI developers, healthcare institutions, and individual clinicians in ways that existing tort law was not designed to address. Product liability law was built for static products with identifiable defects. Medical malpractice assumes a human clinician as the primary decision-maker. When an AI recommendation contributes to harm, through hallucination, drift, or deployment outside validated conditions, the patient harmed by the error faces the burden of identifying who to sue, under what theory, and in which jurisdiction.
The liability diffusion works three ways. Developer liability asks whether the AI was defective and whether software is a product in the legal sense, questions courts have not settled for clinical AI. Institutional liability considers whether the hospital adequately evaluated and monitored the tool, a vicarious liability or corporate negligence question that depends on standards that do not yet exist. Clinician liability asks whether the clinician relied unreasonably on the AI output. Some malpractice claims involving AI have already surfaced, but there is still no clear playbook for how they will be handled. Courts are leaning on older cases about software mistakes and EHR problems to guide their thinking, because that is what exists.
Automation bias complicates every layer. When a clinician follows an AI recommendation they would have questioned from a human colleague, was the reliance unreasonable, or was the AI’s output designed in a way that made reliance reasonable? The Medical Board of California has emphasised that AI tools cannot replace a physician’s professional judgement, but the legal system has not established where the line between appropriate reliance and negligence sits. Research on gastroenterologists using an AI polyp detection tool found they became worse at the task when performing it without assistance, cognitive de-skilling that creates brittle human-AI combinations and complicates every liability determination.
The accountability vacuum makes governance urgent. The accuracy gap creates the harm. The regulatory framework clears the tools without requiring post-deployment proof of safety. When harm occurs, the patient has no clear path to recourse. Until accountability is defined, through legislation, litigation, or governance standards, the incentive structure for safe deployment is incomplete. The question is not whether these systems will produce harmful outputs: operating at 34% accuracy, that is guaranteed. The question is who bears the consequences.
How Are US State Legislatures Responding to the Clinical AI Accountability Gap?
State legislatures are filling the federal regulatory vacuum with a patchwork of AI-specific healthcare laws. In 2025, 47 states introduced more than 250 health AI bills, with 34 passed and enacted into law across 21 states. Over 240 bills across 43 states have already been filed in 2026. For context, that volume represents a surge: health AI legislation barely existed as a category three years ago. The result is faster accountability than federal regulators have produced, but at the cost of compliance complexity for health systems operating across state lines. Different standards, different disclosure obligations, and the unresolved question of whether a future federal framework will preempt these state laws.
The legislative map shows a range of approaches. California SB 1120, the Physicians Make Decisions Act, mandates that only licensed physicians, not AI systems, can make final clinical determinations. Louisiana SB 246 imposes transparency and oversight requirements with a burden-shifting approach that presumes AI-influenced denials invalid absent documented human independence. The Colorado AI Act, revised and signed in May 2026, requires algorithmic fairness assessments. Texas requires practitioners to disclose AI use for diagnostic purposes and review all AI-created records. Arizona, Illinois, Maryland, and Nebraska enacted laws in 2025 prohibiting AI-only prior-authorisation denials without human physician review.
AI-driven prior authorisation is the sharpest example of the accountability failure. Three in four health plans now use AI in prior authorisation decisions. Stanford researchers found that 82% of AI-driven Medicare Advantage prior authorisation denials are overturned on appeal, and with a patient appeal rate below 1%, most incorrect denials are never contested. AI systems are producing decisions at scale that do not survive human review, and the patients affected largely never challenge them. What payers should demand before deploying AI in prior authorisation is clear: external validation data from populations matching their covered lives, evidence that recommendations survive human review at rates comparable to manual determinations, mandatory human review workflows for all denials, and multi-state compliance infrastructure for organisations operating under the state patchwork.
The federal preemption question hangs over this. If Congress passes comprehensive AI healthcare legislation, it could create uniform standards, or it could override stronger state protections. A December 2025 executive order established a DOJ AI Litigation Task Force to challenge state AI laws, creating a preemption tension that states have continued to legislate despite. Until the federal question resolves, health systems navigate a compliance landscape where liability standards vary by state and the legal certainty that would incentivise safe deployment remains absent.
Can Patients Trust AI-Generated Clinical Decisions?
Patients already do, and that is a concern. McKinsey’s 2026 AI Trust Maturity Survey found that 74% of patients trust AI-generated medical answers, and 78% assume their physicians are validating AI outputs before acting on them. The gap between patient trust, which is high, and institutional validation, which is inconsistent, creates a regulatory blind spot: patients assume a safety check that may not be occurring.
This trust asymmetry is a clinical safety issue. Patients who assume AI outputs are validated may not seek second opinions, question recommendations, or provide the clinical context that could reveal an error. The ARISE report recommends that “Patients cannot be assumed to play any oversight role.” The burden of validation must rest with institutions and clinicians, but the evidence suggests it does not systematically.
Automation complacency is the mechanism by which trust asymmetry becomes dangerous. When clinicians trust AI outputs, and patients assume clinicians are validating them, the verification step that should catch errors gets compressed. A meta-analysis of over 100 controlled trials shows that computerised decision support improves both targeted processes of care and patient outcomes, but the challenge is distinguishing between automatable functions and tasks that always require a human in the loop. Research testing LLMs on orthopaedic treatment guidelines found that the same AI system produced contradictory medical recommendations in different sessions, undermining the consistency that trust requires.
The emerging regulatory question is whether patients should be told when AI contributes to clinical decisions affecting their care. Mandatory disclosure could close the trust asymmetry by making patients aware that an AI recommendation is part of the clinical picture, giving them the information they need to ask questions or seek second opinions. It would also create an accountability trail. But disclosure alone does not solve the underlying problem: if the AI is wrong and neither the clinician nor the patient can detect the error, transparency without accuracy does not improve safety.
How automation bias creates clinical risk and the governance frameworks that can close the trust gap
What Can Clinical AI Governance Learn From Financial Services?
Financial services has 15-plus years of mandatory AI governance under SR 11-7, the Federal Reserve’s 2011 guidance on model risk management. Its three-pillar structure, mandatory model inventory, independent validation, and ongoing monitoring with defined roles, is directly transferable to clinical AI. The core difference is not technical but structural: financial services governance is mandatory and model-level, while clinical AI governance remains voluntary and institution-dependent.
SR 11-7’s architecture is straightforward and proven. Every model is tracked in an inventory, classified by risk tier, and assigned an owner. Every model is validated by someone other than the developer, before and during deployment. Performance is tracked continuously with defined thresholds for intervention. The role structure is explicit: model owner, model validator, model risk committee. Contrast this with clinical AI. No mandatory inventory exists. Validation is often limited to pre-market clearance evidence from the manufacturer. Monitoring is voluntary and episodic. An investment bank implementing comprehensive AI governance following SR 11-7 reduced model-related incidents by 45%. Clinical AI has no equivalent benchmark because it has no equivalent infrastructure.
Champion-challenger testing is the highest-value practice that clinical AI has not adopted. In financial services, a new model runs in parallel with the production model, receiving the same inputs, but its outputs are compared against the champion’s before any switch is made. In clinical AI, this could be implemented as shadow deployment: the AI tool running silently alongside clinical workflows without affecting care, allowing comparison against current practice before go-live. It is low-cost, high-value, and not systematically practised.
There are limits to the comparison. Financial services governance emerged after the 2008 crisis demonstrated catastrophic model failure. It is failure-responsive. Goldman Sachs has embedded Anthropic engineers to co-develop autonomous compliance agents while its model validation function still operates on quarterly cycles, deployment outpacing governance even in the sector with the most mature framework. The lesson for clinical AI is build governance infrastructure before the catastrophe that makes it mandatory. Financial services also benefits from clearer liability allocation and federal regulatory preemption that healthcare lacks, which connects directly to the accountability analysis.
Are Agentic AI Systems Justified for Clinical Tasks?
Agentic AI, multi-agent architectures where LLMs are augmented with tool use, structured workflows, and inter-agent coordination, shows marginal accuracy improvement over single-model approaches at disproportionate computational cost and with new failure modes. The governance question is whether the improvement is worth the complexity.
The evidence is clear but underwhelming. A systematic evaluation of agent systems against baseline LLMs found accuracy gains of 0.5% to 8.9% at 10 to 100 times the token consumption, frequently doubling or tripling response time. Agent systems achieved their highest accuracies on text-only benchmarks (60.3% on AgentClinic MedQA, 28.0% on MIMIC-IV) but multimodal accuracy remained low. On the PhysicianBench evaluation of real EHR-based tasks, the best-performing agent achieved only 46% success rate. These are not results that justify architectural complexity for its own sake.
The “last mile” problem is what makes agentic AI appealing. An LLM can produce a clinically sound textual recommendation, but that recommendation still needs to be translated into an EHR order, checked against drug interactions, and documented in a way that supports clinical workflow. LLMs generate recommendations but cannot close this integration gap alone. Agentic architectures address this by adding tool use and structured workflows, but they introduce new failure modes: tool-use errors, coordination failures between agents, and emergent behaviours that single-model evaluations do not capture. Hallucination also compounds, even after mitigation, hallucinations continued to affect approximately 30% of clinical scenarios in agent systems.
Specialised clinical NLP pipelines using the four-step architecture of named entity recognition, relation extraction, normalisation, and inference often match or exceed general-purpose LLM accuracy for specific tasks at lower computational cost. For well-defined clinical tasks, the simpler system is frequently the better system. The agentic question is ultimately the governance question: what level of evidence should be required before deploying a more complex, more expensive system for a marginal gain? If your governance infrastructure cannot reliably monitor a single LLM for hallucinations and drift, it cannot monitor a multi-agent system for coordination failures.
Resource Hub: AI in Clinical Settings Deep Dives
Understanding the Problem
The Clinical AI Benchmark to Bedside Accuracy Gap — Why clinical AI scores 95% on benchmarks but 34% at the bedside, what the Stanford-Harvard ARISE 2026 audit revealed about evaluation practices, and how hallucinations turn the accuracy gap into a patient-safety issue. Start here — the accuracy gap is the quantifiable evidence that motivates every governance discussion in the series.
The Regulatory and Accountability Architecture
How the FDA Regulates Clinical AI From Approval to Real-World Safety — The Total Product Lifecycle framework explained, the January 2026 CDS guidance revision, the locked-vs-continuous learning safety trade-off, and why fewer than 15% of 1,200 FDA-cleared devices have published real-world outcomes. Read this second — it provides the institutional context for the accountability analysis.
Who Is Accountable When Clinical AI Causes Patient Harm — The liability diffusion problem, the state-level legislative patchwork from Louisiana to California, the 74%/78% patient trust asymmetry as a regulatory blind spot, and what the HAIRA maturity model and CHAI/Joint Commission standards mean for healthcare organisations. Read this third — it examines the human and legal consequences of the accuracy and regulatory gaps.
The Governance Frontier
What Clinical AI Governance Can Learn From Financial Services — How SR 11-7 model risk management, champion-challenger testing, and mandatory continuous monitoring compare to clinical AI’s voluntary governance landscape. Examines the agentic AI frontier, the Mayo Clinic–Microsoft partnership as a governance stress test, and whether clinical AI will learn from finance or learn the same lessons the hard way. Read this last — it looks outward and forward at governance models that already work.
Frequently Asked Questions
What is a Predetermined Change Control Plan (PCCP), and when does a manufacturer need one?
A PCCP is the FDA’s mechanism allowing AI medical device manufacturers to pre-specify planned model updates, including what will change, how changes will be validated, and what performance boundaries must be maintained, so iterative improvement can occur without a new regulatory submission each time. It is required when a manufacturer intends to update an AI device after clearance without filing a new 510(k) or PMA. See How the FDA Regulates Clinical AI for the full regulatory architecture.
How does the EU AI Act classify medical AI compared to the FDA’s approach?
The EU AI Act classifies medical AI as high-risk, imposing post-market monitoring, incident logging, and quality management requirements that layer on top of the EU Medical Device Regulation. The structural difference is that the EU MDR currently treats AI as static software, any significant algorithm change requires Notified Body re-certification, while the FDA has developed the PCCP mechanism to enable controlled updates. See How the FDA Regulates Clinical AI for the international comparison.
Where can I find the FDA’s Clinical Decision Support software guidance document?
The January 2026 CDS final guidance is available on the FDA’s Digital Health Center of Excellence website under the guidance documents section. It revises the criteria under which clinical decision support software falls within or outside the FDA’s device definition. See How the FDA Regulates Clinical AI for an explanation of what the guidance changed and why it matters.
What evidence should a payer demand before deploying AI in prior authorization?
At minimum: external validation data from populations matching the payer’s covered lives, evidence that the AI’s recommendations survive human review at rates comparable to manual determinations, and the 82% Medicare Advantage appeal overturn rate as the benchmark to beat. Payers should also require mandatory human review workflows for all denials. See Who Is Accountable When Clinical AI Causes Patient Harm for the full accountability analysis.
How can a hospital assess its AI governance maturity?
The HAIRA maturity model provides a five-level framework, from Level 1 (Initial/Ad Hoc) through Level 5 (Leading), across seven governance domains including organisational structure, model evaluation, deployment integration, and ongoing monitoring. Each level describes the practices an organisation should have in place at that stage of maturity. The CHAI/Joint Commission Responsible Use of AI in Healthcare certification provides an external validation pathway. See Who Is Accountable When Clinical AI Causes Patient Harm for the HAIRA and CHAI frameworks in detail.
What is the “last mile” problem in clinical AI?
The “last mile” problem is the gap between an LLM producing a clinically sound textual output and that output being integrated into clinical workflows: translated into an EHR order, checked against drug interactions, and documented in a way that supports billing and continuity of care. Agentic AI architectures address this by adding tool use and structured workflows, but at disproportionate cost and with new failure modes. See What Clinical AI Governance Can Learn From Financial Services for the agentic AI analysis.
Do specialised healthcare NLP tools outperform general-purpose LLMs for clinical decision support?
For well-defined, narrow clinical tasks, specialised clinical NLP pipelines using the four-step architecture of named entity recognition, relation extraction, normalisation, and inference often match or exceed general-purpose LLM accuracy at lower computational cost. The governance question is whether the evidence supports deploying a more complex, more expensive system for a marginal gain over a simpler alternative. See What Clinical AI Governance Can Learn From Financial Services for the accuracy comparison.
Where can I access the Stanford-Harvard State of Clinical AI 2026 report?
The ARISE 2026 report is published through the Stanford-Harvard ARISE Network and is available via Stanford’s Centre for Artificial Intelligence in Medicine and Imaging and Harvard’s Department of Biomedical Informatics. The report synthesises evidence from over 500 clinical AI studies. See The Clinical AI Benchmark to Bedside Accuracy Gap for an analysis of the report’s key findings.
Clinical AI is in production at a scale the regulatory architecture was never designed for. The systems are deployed. They are handling tens of millions of patient interactions daily. They are making decisions about what care patients receive, what treatments are recommended, and what claims are denied. The governance infrastructure that should ensure they are safe, that should detect when they degrade, that should allocate responsibility when they cause harm, is being built in real time around systems that are already operating.
The four gaps this series maps are connected stages of one structural failure. The accuracy gap means we cannot reliably measure what we are deploying. The regulatory framework is philosophically aligned with what AI safety requires but operationally incomplete. The accountability vacuum means that when these systems fail, patients have no clear path to recourse and institutions have no clear incentive structure for safe deployment. Cross-industry governance models that could fill these gaps exist elsewhere but have not been transferred.
The series is designed to be read sequentially. Start with the accuracy gap: it is the quantifiable evidence that makes every governance discussion that follows urgent. Then the regulatory architecture, which provides the institutional context for understanding why the accountability gap exists. Then the accountability analysis itself, the human and legal consequences of systems operating without reliable measurement or consistent oversight. Then the forward-looking piece on what financial services governance can teach clinical AI, because building governance infrastructure from scratch is harder than learning from an industry that has already done it.
Financial services has proven that mandatory model inventory, independent validation, continuous monitoring, and defined accountability structures work. The question is whether clinical AI will build them before the outcomes that force the issue.
