When your AI pilot stalls, the conversation usually goes the same way. Someone says the data wasn’t ready. The team nods. Leadership accepts it. The pilot gets shelved — or more likely it enters a kind of organisational limbo where it isn’t quite dead but it isn’t going anywhere either.
The statistics on enterprise AI failure are consistent. McKinsey finds only 6% of organisations qualify as AI high performers, despite 88% having adopted AI in at least one function. BCG found 74% of companies have yet to show any tangible value from their AI efforts. The headline failure statistics tell a consistent story: bad data doesn’t explain those numbers.
BCG’s research found that only 20% of AI success hinges on data and technology combined. The other 70% comes down to people, processes, and organisational design. Most teams are focused on the 10–20% of the problem that looks familiar — the model, the architecture, the pipeline — while the real 70% goes completely unaddressed.
This article unpacks why the “bad data” narrative is really pointing to an organisational problem most companies haven’t yet named.
Gartner cites poor data quality as a factor in 85% of AI project failures. That statistic is real. The causal story built around it usually isn’t.
Data problems in most failed pilots trace back to leadership decisions that were avoided. Cleansing data pipelines was never funded. Permissions fragmentation was never resolved. Data governance was never properly resourced. Bain’s research found pilots often succeed precisely because they’re built on offline, non-production datasets that someone manually cleaned. When you try to scale across the enterprise, those underlying data issues resurface — because nobody made the call to fix them.
Daniel Clydesdale-Cotter at RT Insights put it plainly: “When AI stalls, the blame lands on regulation, the models, or ‘our data isn’t ready.’ Safe targets, all of them. Nobody gets fired for bad data. But these explanations let everyone off the hook for the actual problem.”
IDC explicitly framed it as a question of “organisational readiness in terms of data, processes and IT infrastructure” — not data quality per se. Organisational readiness is a different order of problem from data quality. It requires leadership commitments, not engineering solutions.
When someone says “our data wasn’t ready,” what they’re usually describing is a series of avoided leadership decisions.
BCG’s 10-20-70 Principle comes from their “Widening AI Value Gap” research (Build for the Future, 2025). The finding: optimal AI investment weighting is 10% on algorithms, 20% on data and technology infrastructure, and 70% on people, processes, and cultural transformation.
That’s counter-intuitive if you’re focused on technical execution. The part of the problem that looks most familiar accounts for less than a third of what actually determines success.
Here’s a concrete example. A 150-person FinTech company invests 80% of its AI budget on model development and 20% on workflow integration. The model works. The adoption doesn’t. Customer support staff don’t trust the outputs. Managers haven’t changed their workflows to act on AI recommendations. Nobody was assigned to own the business result. The 70% was never funded. The pilot succeeds as a demo and then stalls.
AI future-built companies achieve five times the revenue increases and three times the cost reductions that everyone else gets from AI. Those future-built companies — the 5% generating transformative value — have learned to invest in the 70%. The 60% generating minimal value keep over-investing in the 10%.
The gap between future-built companies and AI pilot purgatory is widening because one side has figured this out and the other hasn’t. For the full statistical picture of how this pattern plays out across enterprises, the comprehensive AI pilot purgatory resource covers it.
Outcome ownership means a business leader — not a data scientist, not an AI engineer — is explicitly accountable for the business result of an AI initiative.
Most AI projects don’t have one. The structural gap is straightforward: a data scientist is assigned to the experiment, but no business leader is assigned to own the result. When the experiment ends, there’s no named owner to fund, defend, or operationalise the move to production. The project remains technically alive and organisationally orphaned.
RT Insights puts it plainly: “Getting to production means someone has to own the outcome.” When it stays in the hands of specialists, it stays in pilot purgatory.
The distinction matters in practice. If the success metric is “model accuracy of 92%”, the AI team owns that. If it’s “reduce contract review cycles from two weeks to two days”, that requires a business owner — someone whose performance depends on the outcome, not just the output.
Without that named owner, the organisational machinery for moving to production simply doesn’t exist. This is an organisational design question, not a personnel one — and it’s exactly what how to structure AI outcome ownership is about.
IDC Group VP Ashish Nadkarni described the dynamic directly: “These POCs are highly underfunded or not funded at all. Most of the time the POC happens not because of a strong business case. It’s trickle-down economics to me.”
Approving an underfunded pilot is lower-risk than either refusing to participate in the AI wave or requesting a realistic budget that might get knocked back. The pilot becomes a hedge — visible AI activity without organisational commitment.
RT Insights calls this leadership avoidance. The decisions that would actually fix AI failure — funding data governance, restructuring workflows, assigning business ownership, committing to change management — are all politically difficult. It’s structurally easier to approve an underfunded pilot than to have those conversations.
The result: pilots succeed as demos and stall when the organisational transformation required for production is never funded. Until outcome ownership is structurally assigned, those incentives will keep producing purgatory.
There are two dominant approaches for organising AI in a mid-market company. A centralised AI Centre of Excellence (CoE) provides governance, tooling, and expertise across business units. Distributed ownership embeds AI capability directly into business units, with central platform support.
The CoE model has real advantages: centralised expertise, consistent governance, reduced duplication, easier compliance oversight. The failure mode is structural. Business units submit requests. The CoE builds and deploys. But because the business unit didn’t build it, they don’t own it. Outcome ownership never transfers. The CoE owns the system in production permanently, and accountability stays with the AI team rather than the business function.
The federated model, where business units own their AI outcomes with platform support from the centre, resolves this directly. The team lead accountable for the business function is also accountable for the AI system serving it.
For a 50–500 person company, the practical answer is a federated model: central platform infrastructure (MLOps, data governance, model registry) and distributed business outcome ownership. The condition that determines whether any model works is where outcome ownership lives. The CTO pilot triage framework for evaluating existing structures starts with exactly that question.
McKinsey’s State of AI 2025: 88% of organisations report AI use in at least one business function, but only 39% report any impact on enterprise-level EBIT. The distinguishing factor between organisations that ship and those that don’t isn’t technology. It’s organisation.
Three traits consistently separate AI high performers from the rest. First: clear outcome ownership. A named business leader is accountable for the business result of every AI initiative. Second: cross-functional accountability. Engineering and business teams share ownership of the outcome metric throughout the pilot and into production. Third: funded change management. The 70% of BCG’s framework — workflow redesign, training, adoption management — is budgeted and staffed, not treated as an afterthought after deployment.
McKinsey found high performers are three times more likely to strongly agree that senior leaders demonstrate ownership of and commitment to their AI initiatives. That’s not about leadership cheerleading. It’s about structural accountability.
The difference between the 6% and the 94% is not about having better data scientists. The high performers have built organisational structures that allow technical capability to translate into production outcomes. The leverage point isn’t in the codebase. It’s in the governance model. And defining production readiness criteria before you start begins with organisational design, not model selection.
IDC/Lenovo’s CIO Playbook 2025 found 88% of AI POCs fail to reach widescale deployment. MIT research found 95% of enterprise generative AI pilots fail to deliver measurable financial returns. S&P Global found 42% of companies scrapped most of their AI initiatives in 2025. BCG’s 10-20-70 Principle identifies the root cause: 70% of AI success depends on people, process, and cultural transformation — the majority of failures are organisational, not technical.
10% of AI success depends on the algorithm or model. 20% depends on data and technology. 70% depends on people, process, and cultural transformation. BCG identifies the inversion of this ratio — over-investing in the technical 30% while under-investing in the human 70% — as the primary reason 60% of companies are generating minimal value from AI investments despite substantial spend.
IDC’s framing: “The high number of AI POCs but low conversion to production indicates the low level of organisational readiness in terms of data, processes and IT infrastructure.” Agility at Scale breaks it into three deltas — technical (infrastructure), governance (oversight and accountability), and operations (MLOps, monitoring, incident response). All three need to be in place before you can call something production-ready.
Pilot fatigue sets in when an organisation has invested real resources in AI pilots that haven’t shipped. The symptoms: sponsors losing confidence, AI teams disengaging, a growing list of “completed” pilots with no production deployments, and growing budget resistance to new AI proposals. Beam AI’s analysis found 42% of enterprises deployed AI without seeing any ROI, with an additional 29% reporting only modest gains.
In a CoE model, a central AI or data science team governs and delivers all AI use cases across business units. The failure mode: outcome ownership stays with the AI team, not the business function.
In a distributed model, business units own their AI initiatives with central platform support. The advantage: outcome ownership is embedded in the business function accountable for the result. The risk is governance fragmentation if the central platform is too thin.
A specific, named business leader is accountable for the business result — measured in business terms, not technical terms. “Reduce contract review cycles by 60%” is a business outcome. “Model accuracy of 92%” is a technical output. RT Insights defines it as assigning the business leader who carries accountability for production results, not just the technical team building the system.
“Bad data” is a socially safe explanation. It’s technical, impersonal, and implies a fixable problem rather than a leadership or governance failure. Bain’s research reinforces this: data problems persist not because data engineering is impossible but because “ownership is often unclear, defaulting to system administrators or data platform teams; without business-aligned ownership, governance lacks direction.”
A proof of concept is a time-boxed experiment in a controlled environment — typically on clean or synthetic data — designed to validate technical feasibility. A production deployment operates at scale with real users, real data, and real business consequences. Agility at Scale frames the distance between them as three deltas — technical, governance, and operations — that a pilot rarely addresses.
McKinsey found AI high performers are three times more likely than peers to have senior leaders who demonstrate ownership of and commitment to their AI initiatives. Board-level pressure to “do AI” is not the same as genuine executive sponsorship. Pressure without follow-through — funding change management, restructuring workflows, assigning outcome ownership — produces underfunded pilots that are structurally set up to stall.
Widespread adoption doesn’t produce widespread scaling. Most organisations are still in experimentation or piloting phases and haven’t built the structures to scale. The three structural differentiators: redesigned workflows, committed leadership with demonstrated ownership, and funded investment across all the elements required for production — not just the technical ones.
RT Insights recommends starting with measurable business outcomes — “reduce customer service response time by 40 percent” rather than “implement AI.” If you can’t articulate the cost of the non-AI alternative, the problem hasn’t been defined clearly enough. Then run the 10-20-70 audit: estimate your actual allocation across algorithms, data and technology, and people and process. Where the split is inverted, start by reallocating — not by improving the technology.
For a comprehensive overview of why enterprise AI projects fail — covering the full scope of failure statistics, root causes, and the CTO decision framework — see the complete guide to the enterprise AI pilot purgatory problem.
Why 88 to 95 Percent of Enterprise AI Pilots Never Reach ProductionU.S. businesses spent $35–40 billion on AI initiatives. MIT’s NANDA initiative found approximately 95% of them report zero measurable returns. In the same period, S&P Global tracked the share of enterprises abandoning most of their AI programmes jumping from 17% to 42% in a single year.
That convergence has a name. Analysts at Astrafy and RT Insights call it “AI pilot purgatory” — the gap between a promising demo and an actual production deployment, where projects are neither cancelled nor shipped. This article is the statistical entry point to the enterprise AI pilot purgatory problem — a complete guide to why enterprise AI pilots fail and what organisations that do ship are doing differently.
The statistics are clear on scale. They’re less clear on mechanism. What keeps a pilot that works as a demo from crossing into production? That question gets addressed — but not fully here.
AI pilot purgatory is where an AI project lives after it’s cleared initial feasibility testing but before it ever reaches full production. Perpetually extended. Perpetually underfunded. Perpetually at risk of cancellation — without ever formally being cancelled.
Astrafy calls it “costly, enterprise-wide gridlock” where “the critical problem isn’t a lack of trying — it’s a failure to convert a working idea into a reliable, enterprise-grade business asset.” RT Insights puts it in terms most technical leaders will recognise immediately: “That pilot everyone loved in the boardroom? It’s still stuck in staging.”
What it looks like in practice: a team maintaining a working demo for the third quarter in a row. A budget line that keeps getting rolled over. A roadmap slot that never gets prioritised because there’s always something more urgent.
Purgatory is defined by what it’s missing — governance structures, production-grade data, budget commitment beyond the current quarter, and a named owner with actual authority over the production outcome, not just the technical build.
In mid-market companies, purgatory often comes down to single-person ownership. When the technical champion’s attention shifts, the project has no institutional home. McKinsey found nearly two-thirds of organisations remain stuck in “pilot mode,” unable to scale across the enterprise. The full picture of why 88 to 95 percent of enterprise AI pilots never reach production becomes clearer when you look at the research behind that range.
The 88% and 95% failure figures are not contradictory estimates from competing research. They measure different points in the same lifecycle.
IDC/Lenovo (88%): The AI CIO Playbook 2025 found that for every 33 AI POCs an enterprise starts, only four reach production. IDC Group VP Ashish Nadkarni: “The high number of AI POCs but low conversion to production indicates the low level of organisational readiness in terms of data, processes and IT infrastructure.”
MIT NANDA (95%): The GenAI Divide report found 95% of generative AI pilots fail to deliver measurable ROI despite $35–40 billion in aggregate spending — not just failure to ship, but failure to return value from what does ship.
McKinsey (88% adoption, limited high performers): 88% of organisations report using AI in at least one business function, but only 39% report any EBIT impact, and most attribute less than 5% of EBIT to AI. Nearly two-thirds haven’t begun scaling AI across the enterprise. Adoption is not value.
PwC (56% of CEOs, no financial impact): PwC’s 29th Global CEO Survey found 56% report no significant financial benefit from AI investments. Only 12% report both cost reduction and revenue growth. The failure is visible right at the top.
S&P Global (17% to 42% abandonment): The share of enterprises abandoning most of their AI initiatives jumped from 17% in 2024 to 42% in 2025. Nearly half of all AI POCs are now scrapped before launch. Purgatory doesn’t persist indefinitely.
Gartner (40%+ agentic cancellation predicted): In June 2025, Gartner predicted over 40% of agentic AI projects will be cancelled by end of 2027 due to rising costs, unclear value, or poor risk controls. The pattern is not historical — it is repeating right now.
These six statistics don’t contradict each other. They measure different failure points — feasibility testing, ROI realisation, executive perception, adoption versus performance, abandonment — and together they describe a systemic problem, not a series of individual project setbacks. The complete guide to AI pilot failure examines how these dynamics play out across different organisational scales.
The definitional confusion here is not a semantic nuisance. It is itself a cause of purgatory. When an organisation can’t tell the difference between “we have a POC” and “we have a production deployment,” it can’t accurately assess what transition work is actually required. That gap lets projects stall while everyone believes progress is happening.
POC (Proof of Concept): A short, low-resource test of whether an AI capability is technically feasible. Uses synthetic or sample data. Answers one question: can we build something that works at all? Duration: days to weeks.
Pilot: A time-boxed test using real users, real data, and real workflows — bounded scope. Success criterion: evidence of value at limited scale with stakeholder buy-in to expand.
Production Deployment: A fully operationalised AI system running at enterprise scale, integrated into core workflows, with active monitoring, governance, and formal accountability chains. Where only four of every 33 POCs ever arrive.
The most dangerous position is what Agility at Scale calls the “false production” trap: a company can truthfully say “we have AI in production” while operating a system at pilot scale with no governance and no plan to expand. Leadership believes the project has shipped. The engineering team knows it hasn’t. That gap is exactly how purgatory persists invisibly — and how IDC/Lenovo’s 88% and MIT NANDA’s 95% can both be simultaneously true.
The demo works. The stakeholders nod. The brief is positive. And then months pass.
Demo conditions are not production conditions. Pilot data is pre-selected and often synthetic. Production data is owned by multiple teams, governed by compliance rules, and full of edge cases the demo never encountered.
But the structural gap alone doesn’t explain why organisations keep approving pilots without committing to production. IDC’s Ashish Nadkarni put it bluntly: “Most of these gen AI initiatives are born at the board level. These POCs are highly underfunded or not funded at all.” The pilot becomes an institutional hedge — it signals action without committing to the cost or accountability of production. No one explicitly kills the project. It just never moves forward.
At the pilot-to-production boundary, three structural blockers keep showing up. None of them are technology problems.
AI-ready data: Production AI requires governed, accessible, high-quality data that pilot environments never actually test against. Gartner found that 85% of all AI projects fail due to poor data quality, and without AI-ready data, 60% will be abandoned.
AI governance: Production requires accountability structures, monitoring, and compliance integration that pilots skip entirely. In production, someone must own the system’s behaviour and its ongoing costs.
Organisational Change Management: Production requires workflow redesign, training, and stakeholder alignment that pilots never touch. BCG‘s 10–20–70 principle is worth knowing here: AI success is 10% algorithms, 20% data and technology, 70% people, processes, and cultural change.
The absence of any one of these is enough to stall production indefinitely. The organisational root causes are examined in the next article in this series.
Deloitte’s State of AI in the Enterprise 2026 names the accumulated cost of repeated failed pilot cycles: pilot fatigue. The distinction from purgatory matters. Purgatory is a project state — a specific initiative is frozen. Pilot fatigue is an organisational response — the teams and leadership that have lived through repeated purgatory cycles become progressively less capable of running successful future pilots.
The progression is predictable. First pilot stalls — budget renewed, expectations quietly drop. Second pilot stalls — morale declines, champions disengage. Third pilot — executives stop attending reviews. By the time a fourth pilot is proposed, the organisation has lost the institutional knowledge and cultural appetite needed to make a production transition work.
AI abandonment is where severe pilot fatigue ends up. S&P Global’s 17% to 42% abandonment jump is the downstream expression: organisations that spent 12–24 months cycling through unproductive pilots concluded AI investment wasn’t generating returns and redirected resources elsewhere. PwC and S&P Global are describing the same organisations from two different vantage points — 56% of global CEOs reporting no financial impact, and 42% of enterprises abandoning most of their AI initiatives. Cause and effect.
For mid-market leaders, pilot fatigue is personal. In a 100-person company, the CTO who championed the AI programme and has nothing to show faces a credibility risk visible to every person in the organisation. Companies that walk away fall further behind those that don’t. The widening AI value gap is examined in a companion article. The complete enterprise AI pilot purgatory guide maps how pilot fatigue fits within the broader failure landscape.
AI pilot purgatory is not a feature of generative AI specifically. It is a structural pattern that repeats with each wave of new AI capability, as organisations invest in the next generation before resolving the readiness problems that stalled the previous one.
Agentic AI — systems that execute multi-step tasks autonomously — is following the same pilot-heavy, production-light trajectory as generative AI. McKinsey found 62% of organisations are at least experimenting with AI agents. Gartner predicts over 40% of those projects will be cancelled by end of 2027.
The three structural blockers are amplified, not reduced. Agentic systems require more robust data governance because they act on data autonomously. More complex integration architecture because a single user query can trigger dozens of internal AI calls. More demanding change management because the workflows they automate are often more central to operations. Deloitte found that close to three-quarters of companies plan to deploy agentic AI within two years, yet only 21% have mature agent governance.
For a full treatment of agentic AI pilot failure and what separates those who succeed from those who cancel, see agentic AI pilot cancellation rates.
Here is what this article has established: six independent measurements of enterprise AI failure, a definition of AI pilot purgatory and its lifecycle stages, and evidence that purgatory progresses through pilot fatigue to abandonment at accelerating rates.
What the statistics don’t establish is the mechanism. They describe the scale precisely. They don’t explain why a pilot that succeeds as a demo consistently fails to cross into production.
McKinsey’s analysis of AI high performers found the distinction is not technical: high performers redesign workflows, maintain committed leadership, and invest at larger scale. PwC found companies with strong AI foundations are three times more likely to report meaningful financial returns. That’s an organisational readiness finding, not a technology finding.
The organisational root causes of AI pilot purgatory are examined in the next article — starting with the most commonly misdiagnosed one.
One final note on competitive position: organisations stuck in purgatory are not holding steady. BCG found that AI leaders achieve 1.5x higher revenue growth and 1.6x greater shareholder returns than laggards. The widening AI value gap examines that trajectory in full. For the comprehensive overview covering all failure dimensions, statistical evidence, and the CTO decision framework, see the enterprise AI pilot purgatory statistics and analysis guide.
Both are correct — they measure different things. IDC/Lenovo count POCs that never transition to production (88% fail to ship). MIT NANDA count pilots that reach some form of production but fail to generate measurable ROI (95% fail to return value). Complementary, not contradictory.
AI pilot purgatory is the state in which an AI project has passed initial feasibility testing but never achieves full production deployment — neither cancelled nor shipped, perpetually extended, consuming maintenance effort without delivering production value. The term was coined by Astrafy and RT Insights.
Pilot fatigue is Deloitte’s term for the organisational exhaustion that results from repeated AI pilot cycles producing no production outcomes. You see it as declining team morale, budget scepticism, and executive disengagement. Purgatory is a project state — the initiative is frozen. Fatigue is an organisational response — the teams and leadership are exhausted from trying.
S&P Global documented a more-than-doubling in enterprise AI abandonment in a single year. Organisations that spent 12–24 months cycling through unproductive pilots concluded AI investment wasn’t generating returns and redirected resources elsewhere. Nearly half of all AI POCs are now scrapped before launch.
McKinsey defines “high performers” as organisations demonstrating AI deployment at scale with measurable financial returns. The 88% adoption figure includes any AI use — isolated tools, unresolved pilots. Nearly two-thirds of McKinsey respondents haven’t begun scaling AI across the enterprise. The gap between adoption and high performance is the pilot-to-production transition problem, measured.
AI-ready data is Gartner’s term for data meeting the quality, governance, and accessibility requirements for AI models to function in production. Pilot environments use pre-cleaned, selected data subsets. Production systems must consume real enterprise data governed by compliance rules and owned by multiple teams. Gartner found that 85% of all AI projects fail due to poor data quality, and without AI-ready data, 60% will be abandoned.
The GenAI Divide is MIT NANDA’s framing for the structural gap between the roughly 5% of organisations achieving measurable ROI from generative AI and the 95% that do not. It’s not a gap in technical access or investment — the divide reflects differences in organisational readiness, data infrastructure, and change management capability.
Agentic AI faces the same pilot-to-production blockers as generative AI, but at greater complexity. Deloitte found that close to three-quarters of companies plan to deploy agentic AI within two years, yet only 21% have mature agent governance. Organisations investing in agentic AI without resolving their generative AI structural failures are reproducing the same pattern at higher stakes.
What the AI Inference Cost Crisis Means for Growing Software CompaniesRunning an AI product costs more than running a traditional SaaS product. Every query your users make, every document your product processes, every recommendation it surfaces triggers an inference computation that draws on GPU capacity. That compute runs perpetually, at scale, and it does not get cheaper as your product matures the way a traditional code deployment does.
AI companies at scaling stage spend an average of 23% of revenue on inference alone — nearly matching total engineering headcount as a cost line (ICONIQ State of AI 2026). Enterprise LLM API spend reached $8.4 billion in 2025, up 3.2x from the year before (Menlo Ventures State of GenAI 2025). And despite token prices falling across the market, total AI infrastructure budgets have grown, not shrunk.
This guide covers why the gap between AI and SaaS economics exists, how to diagnose your own position, and which decisions you face across infrastructure, pricing, and governance. Each section links to a deep-dive article. Start with the question most relevant to where you are now.
AI training is a one-time expense. You pay to build the model. Inference is the perpetual cost of running it — every user interaction triggers computation on expensive GPU hardware, billed by the token or by the second. Unlike SaaS software, where the same code serves millions of users at near-zero marginal cost, AI products consume hardware capacity with every query.
Once a model is deployed, inference accounts for 80–90% of its total lifetime compute cost (ByteIota). Inference spending crossed 55% of all AI cloud infrastructure in early 2026. The financial model for an AI product is closer to a professional services firm — where delivery cost scales with volume — than to a software company. To understand the macro forces driving the AI inference cost crisis — the hardware supply dynamics and hyperscaler CapEx that set the cost floor — the sections below break down each dimension of this cost structure, starting with how the numbers compare to traditional SaaS.
Traditional SaaS companies achieve 70–90% gross margins because software delivery is nearly free at scale. AI companies average roughly 52% gross margins (ICONIQ State of AI 2026) because inference is a continuous cost of goods sold that does not amortise across users. Every additional user adds compute cost. That 20–40 percentage point gap is not a startup inefficiency — it is a structural feature of the AI delivery model.
Understanding the margin gap is the first step. But most companies first encounter the cost problem when a pilot moves to production.
Read the full analysis: Why AI Gross Margins Are So Much Lower Than SaaS and What That Means for Your Business.
Pilot costs and production costs measure different things. A pilot runs under controlled conditions with known inputs, capped usage, and no need for resilience or monitoring. Production adds all of those cost categories at once. 80% of enterprises miss their AI cost forecasts by more than 25% (Mavvrik/Benchmarkit), and 95% report overspending against AI infrastructure budgets.
The gap is not about the model getting more expensive. It is about the full production infrastructure stack — data pipelines, monitoring, logging, network egress, overprovisioning — being invisible during testing. The why AI bills explode between pilot and production article documents the specific mechanisms, including agentic AI call chains that multiply costs 5–20x per user action.
Read the full analysis: Why Your AI Bill Exploded Between Pilot and Production and How to Predict the Real Cost.
Cloud APIs are the correct default for most growing companies: no capital expenditure, no GPU management overhead, instant access to the latest models, and flexible scale. The risks are vendor lock-in and per-token pricing exposure at high volumes.
But 67% of enterprises are actively planning to repatriate AI workloads to on-premises infrastructure, and 61% already run hybrid setups (Mavvrik). The decision comes down to your token volume, workload predictability, and how much of your cloud bill is going to inference. Once you know those numbers, the right architecture tends to be obvious.
Read the full analysis: Cloud vs On-Premises vs Hybrid AI Inference — A Decision Framework Based on Real Cost Data.
The fastest reductions come from changes that require no infrastructure work. Prompt caching can reduce costs 50–90% for use cases with repeated context — RAG pipelines, multi-turn conversations, document processing. A workload costing $10,000 per month can drop to $1,000 with caching alone.
Model routing — directing simple queries to cheaper models and reserving frontier models for complex requests — delivers another 30–60%. Start with caching and routing before committing to infrastructure changes like quantisation or custom model serving. The effort-to-impact ratio is what matters. The AI inference optimisation playbook sequences every technique by effort-to-impact ratio so you know exactly where to start.
Read the full analysis: The AI Inference Optimisation Playbook — Caching, Quantization, and Model Routing in Priority Order.
Subscription pricing transfers inference cost risk entirely to you: if a customer uses the product heavily, you absorb the full cost with no additional revenue. That works if usage per customer is tightly constrained. For most AI products, it is not.
Three pricing archetypes handle this differently. Consumption-based (per query or token) passes cost variability to customers. Workflow-based (per completed task) ties price to something customers understand. Outcome-based (per result achieved) decouples your cost exposure from usage volume entirely — Intercom charges $0.99 per ticket their AI resolves, not per message or token. Outcome-based pricing jumped from 2% to 18% of AI companies in six months (ICONIQ). And 37% of companies plan to change their AI pricing model within the next 12 months. Understanding how to design AI product pricing for variable inference costs is essential before your margin problem becomes irreversible.
Read the full analysis: How to Design AI Product Pricing That Survives Variable Inference Costs.
AI cost governance does not require a dedicated team. It requires cost attribution — tagging every inference call by product feature, team, and customer — and budget alerts that surface overruns in real time rather than in the next billing cycle.
The gap between “we track costs” and “we govern costs” is wide. 94% of companies say they track AI costs, but only 34% have mature cost management (Mavvrik/Benchmarkit). If your monthly AI bill varies by more than 20% without a clear explanation, that gap is where the money is going. The how to build AI cost governance without a dedicated FinOps team guide translates enterprise FinOps practice to the 50–500 person company context.
Read the full analysis: How to Build AI Infrastructure Cost Governance Without a Dedicated FinOps Team.
The crisis is structural, not cyclical. The AI inference market is projected to grow from $106 billion in 2025 to $255 billion by 2030 (TensorMesh). Hyperscaler capital expenditure hit $600 billion in 2026, with 75% tied to AI infrastructure (ByteIota). Energy demands add another cost floor: AI inference will consume 165–326 terawatt-hours annually by 2028.
Token prices will continue to fall, but total budgets will continue to grow as usage expands. Cheaper inference leads to more AI features, more users, and more total consumption — at a rate that outpaces the price reduction. Planning to wait for prices to drop to near-zero is not a viable strategy.
Given these structural dynamics, the question becomes where to start.
Read the full analysis: The AI Inference Market in 2025 — Hardware Consolidation, Pricing Wars, and What It Means for Buyers.
If your AI costs are already out of control, the first priority is visibility: you cannot optimise what you cannot measure. The second is diagnosis. Here is how to find the right starting point:
“I just got a much larger bill than expected after going to production” — Your problem is the pilot-to-production cost gap: Why Your AI Bill Exploded Between Pilot and Production.
“My AI product has good usage but low gross margins” — Your pricing model may not be recovering inference costs: Why AI Gross Margins Are Lower Than SaaS.
“I know costs are high but I don’t know where the spend is going” — You need cost attribution: How to Build AI Cost Governance.
“I’m spending too much on cloud APIs and wondering about hardware” — You need an infrastructure decision framework: Cloud vs On-Premises vs Hybrid.
“I just need to reduce costs now” — Start with the highest-impact, lowest-effort optimisations: The AI Inference Optimisation Playbook.
“I need to rethink my pricing” — Evaluate the three archetypes: How to Design AI Product Pricing.
“I want to understand the market before making decisions” — Start with the macro view: The AI Inference Market in 2025.
AI inference is the process of running a trained AI model to generate outputs in response to live user inputs — every query, summary, or recommendation your product delivers triggers inference. Unlike traditional software, where the same code serves unlimited users at near-zero marginal cost, inference draws on real GPU compute capacity with every request. That compute is expensive and scales with usage, not with headcount or feature count.
Relevant deep dive: The AI Inference Market in 2025 covers the hardware economics that set the cost floor.
Because pilot-phase costs bear no relationship to production costs. A pilot measures API costs under controlled, low-volume, low-complexity conditions. Production adds data pipelines, monitoring infrastructure, logging, network egress, overprovisioning buffers, and the cost multiplication effect of agentic AI workflows. Companies using pilot cost data to model production consistently underestimate total spend.
Relevant deep dive: Why Your AI Bill Exploded Between Pilot and Production
ICONIQ’s State of AI 2026 report found that AI companies at scaling stage spend an average of 23% of revenue on inference costs — making inference a line item roughly equivalent to total engineering headcount cost. This is a benchmark, not a target: some efficient companies spend significantly less; others spend more. The value of the benchmark is calibration — if your inference spend is materially higher, it signals a structural problem worth investigating.
For most companies under 500 people, cloud APIs are the right default. On-premises GPU infrastructure only becomes cost-justified when AI inference represents 60–70% or more of your total cloud spend, your workloads are stable and well-defined, and your team has the operational capacity to manage hardware. Below that threshold, the flexibility and capital efficiency of cloud APIs almost always outweigh the per-token savings of on-premises compute.
Relevant deep dive: Cloud vs On-Premises vs Hybrid AI Inference
This is the Jevons Paradox applied to AI infrastructure: when inference becomes cheaper per token, companies build more AI features, expose more users to AI interactions, and generate more inference calls — at a rate that outpaces the price reduction. Enterprise LLM API spend doubled to $8.4 billion in a single year despite significant token price reductions. Falling prices stimulate demand faster than they reduce total spend.
Relevant deep dive: Why AI Gross Margins Are So Much Lower Than SaaS
AI cost governance is the set of processes and tools for attributing, monitoring, forecasting, and optimising AI infrastructure spend. If you have multiple AI features in production, multiple team members generating inference costs, or a monthly AI bill that varies by more than 20% without a clear explanation, you need some form of cost governance. It does not require a dedicated team — it requires per-feature cost tagging and a basic monthly review process.
Relevant deep dive: How to Build AI Infrastructure Cost Governance Without a Dedicated FinOps Team
The range is wide because it depends on your current baseline and which techniques you implement. Prompt caching (KV cache) can reduce costs 50–90% for use cases with repeated context. Model routing can reduce costs 30–60% by directing simple queries to cheaper models. Quantization can deliver 8–15x memory compression for self-hosted workloads — as Dropbox Engineering demonstrated with their low-bit inference work. In practice, companies implementing the full optimisation stack often achieve 60–80% cost reductions, though starting from scratch the gains are front-loaded — the first two or three interventions deliver the majority of savings.
Relevant deep dive: The AI Inference Optimisation Playbook
The AI Inference Market in 2025 — Hardware Consolidation, Pricing Wars, and What It Means for BuyersIf you’re building AI-enabled products, the AI inference market is where your money goes. Not training — inference. Running models in production. That’s $106 billion in 2025, heading to $255 billion by 2030, and it’s consuming 80–90% of all AI computing power on the planet. Training is a sunk cost you pay once. Inference is the meter running with every user request.
Three things are happening at once: hardware consolidation (NVIDIA just spent $20 billion acquiring Groq), provider economics that vary wildly depending on who you use, and $600 billion in hyperscaler capital expenditure locked in for 2026. The AI inference cost crisis isn’t a blip. It’s built into the economics of running AI in production. Here’s what’s actually going on and what it means for the decisions you need to make.
Grand View Research and MarketsandMarkets both put the AI inference market at $106 billion in 2025, growing to $255 billion by 2030. Inference has already overtaken training for the first time, sitting at 55% of cloud AI spend in early 2026. Average enterprise LLM spend hit $7 million per company in 2025 — nearly triple the $2.5 million from 2024. One CIO put it plainly: “What I spent in 2023 I now spend in a week.”
Here’s the dynamic you need to wrap your head around. Per-token inference costs dropped approximately 1,000× in three years — yet total inference spending grew 320% over the same period. Cheaper tokens just create more use cases and higher query volumes. Andreessen Horowitz calls it “LLMflation”: total bills go up because demand grows faster than costs fall. The $106B-to-$255B trajectory is rising spend, not falling costs. Plan for that.
On 24 December 2025, NVIDIA acquired Groq’s assets and licensed its inference technology for $20 billion — NVIDIA’s largest deal ever and the biggest consolidation event in AI inference hardware history.
It’s worth being precise about what this was. A licensing-and-acquihire, not a full corporate acquisition. NVIDIA took Groq’s chip assets and licensed the LPU (Language Processing Unit) designs, bringing on founder Jonathan Ross and President Sunny Madra. This mirrors Microsoft’s 2024 licensing of Inflection AI and is widely read as a deliberate move to sidestep mandatory antitrust review.
Groq’s LPU is purpose-built for inference. Independent benchmarks recorded Groq delivering 877 tokens/sec on Llama 3 8B — roughly 2× the throughput of the fastest alternatives at the time.
Before this deal, NVIDIA already held 90–95% of the AI accelerator market. Now it controls the most credible alternative inference chip architecture as well. The 2.9× premium NVIDIA paid over Groq’s September 2025 valuation tells you everything — LPU-style architectures genuinely outperform GPUs for specific inference workloads.
If you’re currently on GroqCloud: the service is still nominally operational, but long-term pricing and product roadmap under NVIDIA ownership are genuinely uncertain. Any infrastructure planning beyond 12 months needs to account for this.
Gross profit per token varies a lot. DeepSeek 85%, Perplexity 60%, Anthropic 55%, Manus 50%, Together AI 45%, Groq 40%. These margins tell you whether current pricing is sustainable or subsidised — which matters quite a bit when you’re building production systems.
Traditional SaaS gross margins run 70–90% because software has near-zero marginal cost of delivery. AI margins average around 52% because inference requires continuous GPU compute with every single query. The marginal cost scales with usage in a way that seat-based SaaS simply doesn’t.
DeepSeek’s 85% gross margin is the most instructive number here. It’s achieved through architectural efficiency — a sparse mixture-of-experts design that activates fewer model parameters per inference pass. That’s a structural advantage, not a subsidised pricing scheme. The implication for the market is real: inference-efficient architectures work at production scale, which puts genuine pressure on providers running less-efficient models.
Anthropic’s Series F fundraise valued the company at $183 billion post-money, with run-rate revenue growing from roughly $1 billion to over $5 billion in under eight months at 55% gross margins. Meanwhile OpenAI’s compute margin jumped from around 35% in early 2024 to roughly 70% by October 2025.
Use margin data as a procurement signal. Providers below 45% have limited room to absorb cost increases — expect price pressure as they scale. Cross-reference with valuation multiples: a low-margin provider at a high valuation is pricing for growth rather than stability.
Hyperscalers committed $600+ billion in AI infrastructure capital expenditure for 2026 — a 36% increase over 2025. Amazon at $200 billion, Google at $175–185 billion, Microsoft at $145 billion, Meta at $115–135 billion.
Falling API token prices and rising infrastructure investment aren’t in conflict — they’re operating on different cost layers. Hyperscalers have to recoup data centre construction, GPU procurement, and energy costs regardless of what they’re charging per token.
Meta and Microsoft are building nuclear plants to power AI data centres. These are decade-scale commitments that have to be recovered through revenue. Energy is a floor cost — US data centres consumed 200 terawatt-hours in 2024, and AI inference is projected to consume 165–326 terawatt-hours annually by 2028. When energy and GPU memory cost more, cloud inference costs more. Simple as that.
AWS raised GPU Capacity Block prices by 15% in January 2026 with no announcement — on a Saturday. Cloud inference pricing does not fall as fast as per-token API rates suggest it should.
Open-weight models — AI models whose trained weights are publicly released for self-hosted deployment — function as a cost ceiling on proprietary API providers. If API pricing exceeds the all-in cost of self-hosting an equivalent open-weight model at your token volume, you have a rational exit path.
Meta’s Llama 3 series is the obvious example. Llama 3 provides GPT-4-class capability that you can deploy on your own or leased GPU infrastructure. Once workloads are steady and high-volume, self-hosted smaller models can reach cost parity with API-based large models faster than many teams expect.
Self-hosting isn’t free, though. You need GPU infrastructure, operational overhead, and model maintenance capability. Quantify your self-hosting breakeven token volume before you start using open-weight models as a negotiating lever.
AMD’s Instinct MI300X is the primary hardware alternative for buyers who want to avoid NVIDIA lock-in: 192GB HBM3, 5.3 TB/s memory bandwidth, and a 40% latency advantage over the H100 for large models. The CUDA moat is real though — CUDA has nearly two decades of investment baked into PyTorch, TensorFlow, and nearly every major AI framework. AMD’s ROCm 6.x has reached near CUDA parity but still requires more manual tuning. Model the switching costs honestly.
Hardware consolidation, elevated hyperscaler CapEx, provider margin variance, and open-weight model availability are not a transition phase. They are the permanent operating environment.
Don’t assume token prices will fall fast enough to solve the cost problem for you. The Jevons paradox, hyperscaler CapEx recovery requirements, and NVIDIA’s hardware dominance all work against rapid cost deflation at scale. Production costs scale 717× from proof-of-concept to production. That’s not an outlier — it’s the pattern.
Use provider gross margin data as a procurement signal. Providers below 45% are likely to raise prices as they scale; providers above 55% have more structural stability. Open-weight model availability is a real negotiating lever — but only if you’ve already quantified your self-hosting breakeven.
The teams managing inference costs well are treating it as an architectural concern, not a line item to be surprised by at month-end. For a detailed look at how these market forces affect your P&L, see the breakdown of why AI gross margins are structurally lower than SaaS. And for infrastructure decisions shaped by hardware dynamics — particularly how the NVIDIA/Groq consolidation and AMD’s hardware positioning should inform your deployment choices — the cloud vs. on-premises vs. hybrid decision framework covers the real cost data. Deeper guidance across all of these areas is available in what the AI inference cost crisis means for your business.
What is the AI inference market and why does it matter to software companies?
The AI inference market is the ecosystem of hardware, software, and services that enables AI models to run in production. Inference accounts for 80–90% of lifetime AI product costs — training is a sunk cost paid once; inference scales with every user request. At $106 billion in 2025 growing to $255 billion by 2030, it’s the operational cost structure every AI-enabled product faces.
Does the AI inference market growth trajectory mean costs will eventually fall?
Per-token prices dropped 1,000× in three years, yet total inference spending grew 320% over the same period. The Jevons paradox means cheaper tokens drive more consumption, pushing total spend up. Structural factors — hyperscaler CapEx recovery, NVIDIA dominance, rising energy costs — create a floor that limits how fast cloud inference pricing can actually fall.
What exactly did NVIDIA acquire from Groq, and was it a full acquisition?
NVIDIA executed a licensing-and-acquihire of Groq’s core IP and key personnel — LPU architecture designs, founder Jonathan Ross, and President Sunny Madra — for $20 billion. It was not a full corporate acquisition; Groq as an entity continues nominally and GroqCloud remains operational. The structure was designed to reduce antitrust exposure.
What happened to GroqCloud after the NVIDIA acquisition?
GroqCloud remains nominally operational with Groq’s former CFO stepping into the CEO role. Long-term pricing trajectory and product roadmap under NVIDIA ownership are uncertain. If you’re using GroqCloud for production workloads, evaluate alternative providers and revisit pricing assumptions for any planning beyond a 12-month horizon.
Is DeepSeek’s 85% gross margin a sustainable model or an exception?
It appears sustainable. DeepSeek’s margin is driven by a sparse mixture-of-experts architecture that activates fewer model parameters per inference pass — a structural advantage, not subsidised pricing. The implication is that inference-efficient architectures are viable at production scale, which creates competitive pressure on less-efficient providers.
Why does Anthropic’s $183B valuation matter for companies budgeting AI inference costs?
Anthropic grew run-rate revenue from roughly $1 billion to over $5 billion in under eight months, with 55% gross margins and over 300,000 business customers. A provider with that growth trajectory and margin profile is less likely to make sudden pricing changes than a lower-margin competitor under pressure.
Why are AI gross margins lower than traditional SaaS margins?
Traditional SaaS margins run 70–90% because software has near-zero marginal cost of delivery. AI margins average around 52% because inference requires continuous GPU compute with every query — the marginal cost scales with usage. AI companies that price like SaaS face margin compression at scale.
How much are hyperscalers spending on AI infrastructure and should buyers care?
Hyperscalers committed $600+ billion in AI infrastructure CapEx for 2026, a 36% year-on-year increase. Amazon at $200 billion, Google at $175–185 billion, Microsoft at $145 billion. This capital has to be recovered through inference revenue — which is why cloud pricing doesn’t fall as fast as per-token rates suggest it should.
Is AMD a real alternative to NVIDIA for running AI models in production?
AMD is credible but constrained. The Instinct MI300X delivers 192GB HBM3, 5.3 TB/s bandwidth, and a 40% latency advantage over the H100 for large models. ROCm 6.x has reached near CUDA parity for major frameworks but still requires more manual tuning. Model the CUDA switching costs before you make any hardware decisions.
What does the open-weight model trend mean for AI provider pricing power?
Open-weight models function as a cost ceiling for proprietary API providers. If API pricing exceeds the all-in cost of self-hosting an equivalent open-weight model at your token volume, you have a rational exit path. The leverage is conditional on having the infrastructure capability to self-host.
How do I know if my AI provider’s current pricing is sustainable or likely to increase?
The gross profit per token margin is the most accessible signal: providers below 45% (Groq at 40%, Together AI at 45%) have limited room to absorb cost increases. Providers above 55% (Anthropic, DeepSeek, Perplexity) have structural flexibility. Cross-reference with valuation multiples — a low-margin provider at a high valuation is pricing for growth rather than sustainability.
What is the Jevons paradox and how does it apply to AI inference costs?
The Jevons paradox describes how increased efficiency leads to greater total consumption, not less. In AI inference, per-token prices dropped 1,000× in three years while total spending grew 320% because cheap tokens enable more use cases and higher query volumes. Plan for total inference spend to rise even as unit costs fall.
How to Build AI Infrastructure Cost Governance Without a Dedicated FinOps TeamAI is running in production at companies with 50 to 500 employees. The governance playbooks, though, are written for someone else — the G1000 enterprise with a dedicated FinOps team and a data science department large enough to run model efficiency experiments. That’s not you.
IDC‘s FutureScape 2026 report warns that those G1000 organisations — companies with 1,000+ employees and dedicated FinOps resources — will still underestimate AI infrastructure costs by up to 30%. If companies with dedicated governance functions are getting this wrong, a 200-person SaaS company running AI without a FinOps function is in worse shape, not better. The AI inference cost crisis hits mid-market companies first, and hardest, because they have the least governance infrastructure when costs start compounding.
This article translates enterprise FinOps discipline into a four-pillar framework a CTO can implement without hiring anyone new. It covers real-time cost visibility, cross-functional ownership, budget alert systems, and — the most urgent emerging challenge — agentic AI cost multipliers that are now hitting companies adopting agent-based workflows.
Traditional cloud FinOps works because cloud infrastructure billing is predictable. You provision servers, pay a fixed hourly rate, and forecast from there. Simple enough.
AI inference breaks that model entirely. Costs are consumption-driven — they scale with user behaviour, query complexity, and prompt length, not server headcount. A single product feature update can double your monthly spend overnight.
The numbers are stark. According to the FinOps Foundation, inference accounts for 80 to 90% of total AI spending over a model’s production lifecycle. GPU utilisation during inference can dip as low as 15 to 30%, meaning hardware sits idle while still accruing charges. There’s no equivalent to that in traditional cloud FinOps.
The deeper problem is structural. Traditional cloud FinOps governs capacity. AI FinOps must govern behaviour — your users’ behaviour, your models’ behaviour, and increasingly, your agents’ behaviour. Agentic AI compounds everything: where a single API call might cost $0.001, a multi-step agentic decision cycle can run $0.10 to $1.00. That’s a 100 to 1,000x multiplier before you’ve scaled to any meaningful user volume.
Reserved capacity, committed spend, periodic billing review — the traditional toolkit is no longer sufficient.
IDC’s FutureScape 2026 report issued a specific warning: G1000 organisations will face up to a 30% rise in underestimated AI infrastructure costs by 2027. IDC calls this the “AI infrastructure reckoning.”
Here’s the nuance: that warning was written for organisations that already have dedicated FinOps teams and cloud governance platforms. For a 200-person SaaS company without any of that, the underestimation risk is not lower than 30% — it’s almost certainly higher.
To make this concrete: $20,000 per month in AI inference at 30% underestimation means $6,000 per month accumulating silently. Over a year, that’s $72,000 in unbudgeted infrastructure costs.
Shadow AI makes the mid-market problem worse in a way the IDC analysis doesn’t address. 91% of AI tools used in companies are completely unmanaged, and 75% of employees now bring their own AI to work. Engineering teams using paid AI tools on personal credit cards, or SaaS subscriptions outside central procurement, can represent a material fraction of total AI spend that governance never sees.
The four-pillar framework is a mid-market translation of enterprise FinOps practice. IDC expects leading organisations to integrate FinOps directly into AI governance via cross-functional teams spanning finance, data science, and platform engineering. Here’s the mid-market version.
Pillar 1: Real-time Cost Visibility
Instrument AI workloads to surface per-model, per-feature, per-user inference spend in near real-time. The goal is to know today — not at month-end — what each AI feature is costing.
Pillar 2: Cross-functional Ownership
Distribute FinOps responsibilities across Finance, Engineering, and Data Science without creating a new team. Each function owns a defined slice of the governance mandate. Without a named owner in Finance and Engineering, the other pillars won’t be maintained.
Pillar 3: Budget Alert Systems
Threshold-based monitoring that triggers escalating responses when AI spend crosses green, yellow, and red thresholds. Alerts must fire in near real-time, not at month-end billing review.
Pillar 4: Governance of Agentic Cost Multipliers
Specific policies for multi-step AI agent chains — cost caps per workflow, token budget limits per agent call, escalation gates before expensive frontier model calls. Implement this before agentic workloads reach production, not after a cost incident forces the issue.
The framework is intentionally lightweight. Each pillar can be implemented with existing roles and low-cost or open-source tooling. None of it requires enterprise platform spend to get started.
For teams using LLM APIs — OpenAI, Anthropic, Google Gemini — the foundation is per-call cost logging. For every API call, capture: model name, token counts (input and output separately), and user or feature metadata. Any queryable store works — a database table, a cloud storage bucket. Your existing logging stack is sufficient.
Three tiers of tooling, ordered by cost:
Tier 1 — Native cloud tools: AWS Cost Explorer plus CloudWatch tagging; Azure Cost Management for Azure OpenAI Service. Require tagging discipline, no additional cost.
Tier 2 — Open-source/low-cost: LangSmith for deep tracing and real-time monitoring; Helicone for per-call logging and dashboards. Cost-effective for teams that want per-call visibility without building a custom stack.
Tier 3 — Commercial: CloudZero and Datadog’s LLM Observability platform provide out-of-the-box dashboards with model-level attribution. Cost-effective when manual logging overhead exceeds the platform cost.
For governance purposes, push for “cost per decision” as your primary metric rather than raw token counts. Aggregate inference spend to the business outcome level: cost per resolved support ticket, cost per generated report. A CTO can justify $0.12 per resolved ticket. The same argument is much harder to make when it’s framed in tokens-per-thousand.
An agentic AI cost multiplier is what happens when an AI agent executes a multi-step workflow. Each autonomous step triggers its own model calls. What appears to be one user action may involve 5 to 20 separate inference calls, multiplying costs by 5 to 20x compared to a single-call interaction.
Consider a customer support agent: classify the query, retrieve context, draft a response, check for compliance, reformat for the UI. That’s 5 to 6 model calls before any retry logic — and the multiplier is invisible to traditional monthly billing review. If you’re only tracking total monthly API spend, you’ll see costs rising but won’t be able to attribute the increase to specific agentic workflows.
ICONIQ Capital‘s 2026 State of AI report found that 37% of AI companies plan to change their pricing model in the next 12 months — in most cases because agentic AI costs are higher than the pricing model assumed at design time. Governance can catch this early, but only if you have agentic cost attribution in place.
Four governance levers for agentic cost control:
1. Token budget limits per agent call — cap the maximum tokens consumed by each step in an agent chain. This is the most direct lever and makes token budgets an explicit architectural constraint.
2. Cost caps per workflow — set a maximum total inference budget per completed workflow execution. Trigger an alert or fallback to a cheaper path if the cap is exceeded.
3. Escalation gates before frontier model calls — require cheaper model steps to attempt the task first. Escalate to expensive frontier models — OpenAI GPT-4o, Anthropic Claude Sonnet or Opus, Google Gemini Pro — only when the cheaper step fails or falls below a quality threshold.
4. Workflow cost attribution — instrument each agent chain to emit per-step cost metrics so governance can identify which workflows are cost-efficient and which aren’t. Without execution tracing, debugging agentic cost issues turns into days of forensic work pulling engineers off roadmap.
The cross-functional governance model replaces a dedicated FinOps team by distributing ownership across three existing roles. Each owns a defined slice of the mandate.
Finance owns the budget thresholds and business impact translation: sets green, yellow, and red alert thresholds in dollar terms; escalates to leadership when red thresholds are breached; maintains visibility into company-wide AI tool spend, including shadow AI from expense reports.
Engineering owns the instrumentation and technical response: implements cost logging and maintains the observability stack; responds to alerts with specific optimisation actions — model routing changes, token limit adjustments, caching layer additions; owns the pre-deployment architecture costing process for new AI features.
Data Science owns model selection and efficiency benchmarking: evaluates whether cheaper models can replace frontier models in specific workflows; monitors output quality to ensure cost reduction doesn’t degrade AI product quality; maintains the model routing policy in conjunction with Engineering. Data Science is also responsible for validating that the inference optimisation techniques you need to monitor are delivering the expected gains without quality regression.
The CTO must be the executive sponsor. FinOps practitioners with C-suite engagement show 2 to 4 times more influence over technology selection decisions. Delegating the entire function to a senior engineer is not enough.
The governance cadence that makes this work:
Weekly (30 minutes): Cross-functional review of the top 5 most expensive AI workflows. Engineering presents per-workflow cost metrics. Finance confirms whether costs are within threshold.
Monthly (60 minutes): Review of budget alert thresholds against actual spend patterns. Shadow AI audit of new SaaS subscriptions and expense report AI spend.
Quarterly (half day): Model selection review. Data Science benchmarks current model assignments against available alternatives. Finance reviews AI cost as a percentage of gross margin per feature.
The budget methodology is a formula, not a single number:
Daily Active Users (DAU) × AI-assisted actions per user per day × average tokens per action × per-token price = monthly inference cost
Applied to a 200-person SaaS with 100 daily active users, 10 AI-assisted actions per day, and 2,000 tokens per action:
For a mid-tier model (GPT-4o-mini, Gemini Flash, Claude Haiku) at $0.003 per 1,000 tokens: roughly $180 per month.
For a frontier model (GPT-4o, Gemini Pro) at $0.015 per 1,000 tokens: roughly $900 per month.
Add a 5x agentic multiplier to that frontier model scenario and monthly costs jump to around $4,500 per month — with no change in user count.
A practical rule of thumb for mid-market companies: if AI inference costs for a specific feature exceed 10% of the gross margin that feature generates, it needs either pricing adjustment or model optimisation before it scales. The governance framework here is what surfaces that signal — and the decision about whether to adjust your AI pricing model assumptions requires the same cross-functional process.
Two vertical-specific factors worth building into your numbers:
HealthTech: Data residency requirements often prohibit sending patient data to external LLM APIs. Sovereign cloud or on-premise inference can multiply infrastructure costs by 3 to 5x compared to standard API pricing. Budget for this before deployment, not after.
FinTech: Audit logging requirements increase token consumption per interaction — system prompts must include compliance context, and every interaction must be logged in detail. This adds storage costs on top of inference costs.
Model both amplifiers into your budget framework at the design stage.
No. The cross-functional model is sufficient at spending levels below $50,000 per month. A dedicated FinOps hire makes sense once AI infrastructure spend exceeds $50,000 per month, or when governance consumes more than 20% of a senior engineer’s time. The FinOps Foundation provides deeper reference material as you scale.
A green/yellow/red threshold system, configured to fire in near real-time:
The 20% single-workflow threshold at Red is important: a runaway agent workflow can consume a disproportionate share of total spend while monthly totals remain under the overall budget threshold.
Standard inference cost management governs discrete, single-call interactions. Agentic cost multipliers emerge when multi-step autonomous workflows chain multiple model calls per user action — a 5-step agent workflow may trigger 5 to 20 inference calls for what appears to the user as a single interaction. Governance must shift from “cost per user action” to “cost per workflow step” and include per-chain cost attribution.
Three tiers: (1) Native cloud tools — AWS Cost Explorer plus CloudWatch tags; Azure Cost Management for Azure OpenAI Service. Free or low marginal cost. (2) Open-source/low-cost platforms — LangSmith for per-call LLM tracing; Helicone for per-call logging and dashboards. (3) Commercial platforms — CloudZero and Datadog LLM Observability, cost-effective when engineering time for manual logging exceeds platform cost.
IDC warned that even G1000 organisations with dedicated governance resources will underestimate AI infrastructure costs by up to 30% — the result of non-linear inference cost scaling and agentic AI workloads. IDC labels this the “AI infrastructure reckoning.” For mid-market companies without dedicated FinOps, the underestimation risk is almost certainly higher.
Shadow AI is invisible to standard cost governance. Finance must include AI-related expense reports and SaaS subscriptions in scope — not just centrally provisioned infrastructure. A quarterly Shadow AI audit is the minimum control. Without a named owner responsible for full-scope AI spend visibility, shadow AI will remain a blind spot regardless of how good your central governance is.
Model routing is cost-effective when you have distinct task types with different quality thresholds. Route high-volume, lower-complexity tasks — classification, summarisation, simple question-and-answer — to smaller, cheaper models (GPT-4o-mini, Gemini Flash, Claude Haiku). Reserve frontier models for high-complexity tasks where output quality directly affects business outcomes. See our AI inference optimisation playbook for implementation detail.
At scaling-stage AI-native companies, yes — ICONIQ Capital’s 2026 data puts inference at 23% of revenue, with talent at 26%. For mid-market companies adding AI features to existing products, this level of spend is reachable without governance. The four-pillar framework is specifically designed to prevent inference costs from reaching that threshold.
The FinOps Foundation (finops.org) is the practitioner community for cloud and AI financial operations. It has updated its mission to explicitly include AI FinOps, and offers free resources including an Intro to FinOps course, a Certified Practitioner pathway, and AI-specific frameworks. It’s the right next step once the four-pillar framework is running and you want to go deeper.
Enable per-call cost logging on all LLM API calls: model name, token counts, and which product feature triggered the call. Store it in any queryable format. Within a week, you’ll have the raw data to identify your top 3 most expensive AI workflows. Nothing else — no alerts, no cross-functional governance, no agentic cost controls — works without this baseline.
This article is the governance capstone for a series covering the AI inference cost crisis from multiple angles. Understanding why inference costs are rising, how infrastructure choices affect your cost structure, the inference optimisation techniques you need to monitor, and whether your AI product pricing reflects real inference costs all feed into the governance practice described here.
The four-pillar framework closes the loop: real-time cost visibility surfaces the data; cross-functional ownership ensures it gets acted on; budget alert systems prevent reactive fire-fighting; and agentic cost governance addresses the multiplier that will define the next phase of mid-market AI spending.
98% of FinOps Foundation respondents now manage AI spend, up from 31% two years ago. The framework here is designed to get you there before the first unplanned billing quarter — not after.
For the complete picture, the full guide to managing AI inference costs provides the strategic context that ties this governance framework to every other element of AI cost management.
How to Design AI Product Pricing That Survives Variable Inference CostsMost AI products are priced like SaaS products. But they do not have SaaS cost structures.
SaaS pricing was built for a world where serving one more customer costs you almost nothing. AI does not work that way. Every query, every output, every agent task comes with a real inference bill — GPU compute, API calls, model licensing — and that bill scales directly with usage. The gross margin gap is structural: AI companies are averaging 50–60% gross margins versus the 80–90% that SaaS operators treat as table stakes.
This is not something you can engineer your way out of. The AI inference cost crisis facing AI-native companies is a pricing model problem, not an efficiency problem.
ICONIQ Capital‘s 2026 State of AI report found that 37% of AI companies are actively planning to change their pricing model in the next 12 months. That is what is happening now.
This article gives you a practical framework for choosing between the three primary AI pricing archetypes — consumption, workflow, and outcome-based — with a worked modelling approach and the Intercom Fin case study as a real-world benchmark.
AI’s 50–60% gross margins versus 80–90% for SaaS is not a startup-phase anomaly. It is structural. ICONIQ’s 2026 data shows AI gross margins at 52% — up from 41% in 2024 — but still nowhere near SaaS territory. Model inference alone averages 23% of total AI product costs at scaling-stage companies.
Bessemer Venture Partners put it plainly: “Companies see 50–60% gross margins vs. 80–90% for SaaS.” Unlike SaaS, where additional customers approach zero marginal cost, every AI inference has a real COGS. As ML lead Jacob Jackson put it: “When you receive $10 from the customer, you can’t just spend 10 cents on AWS. GPUs are expensive.”
Ben Murray at TheSaaSCFO ran the numbers: to reach equivalent EBITDA to a SaaS business, an AI company needs approximately 6x the revenue. A $50,000 SaaS product needs an AI equivalent at roughly $250,000–$300,000 per year to deliver comparable unit economics. Not because the AI delivers six times more value — because its cost structure is fundamentally different. That 5–6x ARPA requirement is not a negotiating position. It is arithmetic.
There is one more COGS item that often gets missed: Forward-Deployed Engineers. ICONIQ’s data shows 32% of AI companies now deploy FDEs to support enterprise customers. If your pricing does not account for that effort, you are building margin compression into every enterprise deal from day one.
For more on why AI gross margins are lower than SaaS, see the foundational article in this cluster.
BVP’s AI Pricing and Monetisation Playbook identifies three pricing archetypes — and each one is a different answer to the same question: who bears the cost variability risk? BVP frames the trade-off: “As you move from consumption to workflow to outcome-based pricing, you’re accepting more cost risk in exchange for tighter alignment with customer value.”
Consumption-based pricing (per token / per API call) passes cost variability entirely to the customer. It works well for technical buyers — developers, platform engineers, API integrators. GitHub Copilot and the OpenAI API are the obvious examples.
The problem is non-technical buyers. Metronome found customers avoided using AI features even when free credits were included — they feared unpredictable bills. Leena AI experienced this directly: after charging on consumption, “customers became wary of using the product — the pricing model was counterproductive.”
Workflow-based pricing (per completed task) decouples price from token count. The customer pays per discrete, bounded task — booking a meeting, analysing a document, generating a demand letter. EvenUp captures better margins charging per completed legal demand letter rather than by inference volume.
The catch: cost variability risk shifts to you. One analysis might cost $0.05 in inference. A complex multi-source brief might cost $0.45. If you priced the task at $0.50, your gross margin swings between 90% and 10% depending on what lands in the queue.
Outcome-based pricing (per successful result) is where the industry is heading. The customer pays only when a defined, measurable outcome is achieved — a ticket resolved, a claim processed. ICONIQ’s data: outcome-based pricing jumped from 2% to 18% adoption in six months. Forty-three per cent of enterprise buyers now consider it a significant purchase factor.
The prerequisite is measurement infrastructure. You cannot bill on resolutions if you cannot detect when one has occurred — and that is what trips up most teams attempting the transition.
Intercom Fin is the most cited proof that outcome-based pricing works at scale: $0.99 per resolved support ticket, 1 million customer issues per week, $100M+ ARR.
Why $0.99? Value-based logic. A human-handled support ticket costs $8–15 or more in most contact centres. At $0.99, Fin is priced at roughly 10% of the cost of the outcome it replaces.
The $0.99 is the variable component of a hybrid model. Customers also pay a base Intercom platform fee. The $0.99 activates on top, for autonomous resolutions only. Add the $1M performance guarantee for customers who do not hit expected resolution rates, and the full structure is: base platform fee + $0.99 per autonomous resolution + performance guarantee.
GTMnow’s interview with Intercom’s president put it well: “Guarantees change buyer psychology more than pricing ever could. The $0.99 price gets attention, but it’s the $1M performance guarantee that builds trust.” That guarantee is a conversion mechanism, not just a risk instrument.
The lesson here is straightforward. The question that produced the $0.99 is the same question you need to answer: what is one resolved outcome worth to my customer, and what is my inference cost per attempt? If the first is materially larger than the second, outcome-based pricing is viable. BVP’s guidance: the platform fee should cover at least 2x your delivery costs before variable pricing activates.
Before you pick an archetype, run a cost exposure calculation. BVP’s rule: “If the math doesn’t work at 10 customers, it won’t at 1,000.”
Under consumption-based pricing, cost exposure is essentially zero — inference spikes pass to the customer. Your risk is adoption suppression, not margin compression.
Under workflow-based pricing, exposure comes from task complexity variance. Average inference cost $0.10, task price $0.50: 80% gross margin. Complex task at $0.45 inference, same price: 10% gross margin. What is the realistic complexity range for your tasks, and does your pricing survive the high end?
Under outcome-based pricing, you incur inference costs on all attempts — including failed ones:
Effective cost per charged outcome = cost per attempt ÷ resolution rate
Inference cost $0.20, resolution rate 70%: effective cost per charged outcome is $0.286. At 50% resolution it is $0.40. The failed attempts generate costs with no revenue to offset. Model this accurately before you set the price.
BVP’s hybrid formula handles this cleanly: platform fee at 2x minimum delivery costs, plus included outcome credits, plus variable overage. Their example: $12,000 annual platform fee; 100 included ticket resolutions; additional resolutions at $5,000 per 100. Fixed costs covered before the variable tier activates.
Use your first 50–100 production customers to build cost baselines before committing at scale.
BVP identifies three selection criteria: value attribution (how clearly can the AI’s contribution be measured?), execution autonomy (does the AI act independently or assist a human?), and workload predictability (how variable is inference cost per unit?).
Choose consumption-based if your buyer is technical and can model their own usage; your product is an API, SDK, or developer tool; you are in early discovery without outcome measurement capability.
Choose workflow-based if your AI completes discrete, bounded tasks with relatively stable complexity; your buyer is non-technical and needs predictable pricing; task complexity variation stays manageable.
Choose outcome-based if the outcome is clearly measurable and attributable to the AI; customers value it highly relative to your inference cost; you have production data — not PoC estimates — to set the price accurately.
Choose hybrid if you are uncertain about outcome rate or cost variability; you need buyer predictability and upside capture simultaneously.
Two examples to make this concrete.
A 100-person FinTech with a document summarisation feature for loan officers is most likely a workflow-based or hybrid candidate. The task is bounded, the customer base is cost-sensitive, and outcome definition is complicated in a compliance context. Workflow-based with a hybrid floor is the right starting point.
A 400-person HealthTech with AI-native workflow automation — appointment booking, claim processing — is an outcome-based or hybrid candidate. Workflows are measurable, value per outcome is high, production data is the prerequisite.
There is also some urgency here. BVP calls out soft ROI products as the category most at risk: “Much of the ‘sexy’ AI products today live in soft ROI territory… As many enter renewal cycles for the first time in 2026, pricing will need to reflect actual value, not merely potential or promise.” If your AI feature was deployed in 2024–2025 under flat-rate SaaS framing and is approaching renewal, run the archetype selection exercise now. Do not wait.
Four signals say it is time: consistent margin compression per customer; high churn at renewal because customers cannot justify value; rapid usage growth the current model cannot capture economically; new outcome-measurement capability that makes outcome-based pricing viable for the first time.
Three prerequisites:
Credits can bridge the transition. Metronome calls them “transitional scaffolding” — useful while you establish the real value metric, not a permanent structure. Avoid announcing the change without production data, migrating enterprise customers mid-contract, or adopting outcome-based pricing before attribution infrastructure is in place.
Once you commit to a new model, real-time cost governance to validate your pricing model becomes an ongoing discipline. The assumptions behind your pricing design — cost per outcome, resolution rate, margin at scale — need continuous validation in production.
No. When your buyer is technical, your product is developer-facing, and inference costs are predictable per unit, it is the right choice. The adoption suppression problem Metronome identified is specific to enterprise non-technical contexts.
Three steps: collect real outcome rate data from production before announcing anything; build or buy attribution and billing infrastructure first; communicate the change as “we’re moving to pay-for-performance.” Grandfather existing enterprise customers for 6–12 months. Use credits as transitional scaffolding but position them as temporary.
A $50,000 per year SaaS contract needs an AI equivalent at approximately $250,000–$300,000 to deliver comparable EBITDA. The gross margin is structurally lower — 50–60% versus 80–90% — and the AI company needs approximately 6x the revenue to cover the COGS gap.
Use ICONIQ’s benchmark: inference averages 23% of total AI product costs at scaling-stage companies. Under outcome-based pricing, include inference on failed outcomes — if your resolution rate is 70%, 30% of your inference spend generates no revenue. Never use PoC cost estimates for production budgeting.
Flexprice‘s five-component minimum: clear contractual outcome definition; data tracking and attribution mapping AI actions to results; aligned internal team structure; risk/reward framework covering failed attempts; automated outcome-linked billing. Skip attribution and you will face billing disputes. When multiple factors influence results simultaneously, you need control groups or baseline comparisons.
BVP’s formula — platform fee at 2x minimum delivery costs, plus included outcome credits, plus variable overage — ensures fixed costs are covered before variable pricing activates. The floor removes exposure to near-zero revenue in low-usage months. The credits give enterprise buyers the predictability they need for procurement.
Flexprice states the terms are interchangeable. Where a distinction is made: output-based means delivering a specific artefact (a draft, a report); outcome-based means a measurable result (a ticket resolved, a claim processed). For most pricing decisions, treat them as equivalent.
Four checks: Can you define a successful outcome in a contract? Do you have real production data to set the price accurately? Can you build or buy attribution and billing infrastructure? Is your inference cost per attempt low enough that your price per successful outcome delivers acceptable gross margin? If any are “no,” start with consumption-based or hybrid and migrate once the prerequisites are met.
AI pilot contracts signed in 2024–2025 — often under SaaS-style seat pricing — are now approaching their first annual renewal. ICONIQ data shows AI products providing “soft ROI” are at high churn risk because customers cannot quantify the value received. If your product was deployed under a model that does not capture measurable outcomes, the renewal conversation relies on the customer’s tolerance for unquantified value — which tends toward zero under budget pressure.
No. The $0.99 per resolved ticket is the variable component of a hybrid model. Customers also pay a base platform fee, and Fin activates on top for autonomous resolutions only. Intercom backs it with a $1M performance guarantee. The complete model: base platform fee + $0.99 per autonomous resolution + performance guarantee. The $0.99 gets attention. The guarantee builds trust. The platform fee protects margin.
Pricing strategy is one piece of a larger picture. For a complete resource on understanding AI inference economics — from the financial reality of lower gross margins through infrastructure decisions and governance — the full guide to AI inference economics covers the end-to-end challenge facing AI-native companies.
Cloud vs On-Premises vs Hybrid AI Inference — A Decision Framework Based on Real Cost DataAI inference — not training — is now the dominant cost line for companies scaling AI products. By early 2026, inference workloads account for over 55% of AI-optimised infrastructure spending. The assumption that you can simply scale API calls indefinitely breaks at production volume. The question is not whether the economics shift, but when.
Three deployment models exist: cloud, on-premises, and hybrid. Deloitte’s Tech Trends 2026 research gives you a specific, actionable decision trigger — the 60-70% cloud threshold — that tells you when the on-premises evaluation is worth running.
This article gives you a structured decision framework: the TCO methodology, the GPU utilisation problem that changes the on-premises calculation, and the three-tier hybrid architecture that most enterprises arrive at as the pragmatic outcome. For broader context on the full scope of this challenge, see our AI inference cost crisis guide.
The three deployment models are not equally suited to all workloads. You need to understand their cost structures before you make any infrastructure decision.
Cloud AI inference (AWS, Azure, GCP, OpenAI, Anthropic) is OpEx-heavy. You pay per GPU-hour or per token, with no upfront capital commitment. The elasticity is real and valuable. What is less visible is the pricing premium — cloud providers charge 2-3x wholesale GPU rates. Data egress adds another layer on top: for data-intensive AI workloads, egress fees typically add 15-30% to your total cloud AI spend. Use reserved instance rates as your comparison baseline, not on-demand rates.
On-premises AI inference (NVIDIA H100/H200, AMD MI300X, Lenovo ThinkSystem servers) flips the economics entirely. The capital cost is substantial — a Lenovo ThinkSystem SR675 V3 with 8× NVIDIA H100 GPUs runs approximately $833,806, with ongoing operational costs around $0.87/hour. No egress fees. Fixed costs that get cheaper per inference as your volume grows. The trade-off is CapEx exposure, operational overhead, and hardware refresh cycles every 3-5 years.
Hybrid AI inference splits workloads across both tiers based on their characteristics, and adds a third tier at the edge for latency-critical use cases. You keep cloud elasticity for burst and experimental workloads while moving consistent high-volume production inference on-premises.
Here is how the three models stack up:
Cloud is OpEx — ongoing and variable. On-premises is CapEx — upfront and fixed. Hybrid combines both, managed by workload classification. The decision is not binary. It is a spectrum.
The 60-70% threshold is the single most useful decision trigger in AI infrastructure planning. Deloitte’s Tech Trends 2026 research puts it clearly: when your cloud AI costs reach 60-70% of what equivalent on-premises hardware would cost over a comparable period, the economics of on-premises begin to compete — even after accounting for CapEx and operational overhead.
This is a ratio, not an absolute dollar figure. A 100-person company and a 5,000-person company can both hit the threshold at very different spend levels.
Here is how to calculate your threshold ratio:
One thing to watch: agentic AI workloads will push you toward the threshold faster than you expect. Token consumption per task has jumped 10x-100x since December 2023. A single agentic workflow may make 10-50 API calls per user request versus 1-2 for a simple chatbot. If agentic AI is on your roadmap within the next 12-18 months, model the threshold with 5-10x your current token volumes. If you are arriving at this analysis because your costs unexpectedly surged, see our breakdown of how the PoC-to-production cost explosion happens — understanding the cause clarifies which infrastructure path makes sense.
A useful mid-market rule of thumb: at approximately 10-50 million tokens/day with consistent workload patterns, run the calculation. Below 10 million tokens/day, cloud APIs remain cost-competitive.
Most cloud vs on-premises comparisons undercount the true cost on at least one side. ICONIQ‘s 2026 State of AI report found that inference costs average 23% of revenue at scaling-stage AI companies — a figure that holds from pre-launch through scale. If you are underestimating AI infrastructure costs, you are underestimating a 23% slice of your revenue.
A complete TCO analysis requires six cost categories:
1. Compute costs: Cloud — GPU-hours at premium rates (2-3x wholesale) or per-token API pricing. On-premises — hardware amortised over 3-5 years.
2. Storage costs: Model weights, KV cache, vector stores, and data pipelines. Commonly underestimated on-premises.
3. Egress costs: Cloud adds 15-30% of total AI spend. On-premises: zero. This is the most commonly omitted cost in cloud comparisons.
4. GPU premium pricing: Cloud providers charge 2-3x wholesale GPU rates on every GPU-hour, indefinitely.
5. Staffing delta: On-premises inference adds 0.5-1.5 FTE ($60,000-$180,000/year at $120,000 fully loaded). Omitting this is the single most common error in on-premises business cases.
6. Hardware refresh cycles: GPU servers have a 3-5 year economic lifespan. Refresh cycles add approximately 20-30% to the 5-year on-premises cost.
To put numbers on it (8× H100 configuration, Lenovo reference data): cloud on-demand at 5 years costs $4,306,416; cloud 3-year reserved costs $2,362,811; on-premises costs $871,912 total plus a staffing delta of $300k-$900k. Even at 3-year reserved pricing, and after adding staffing costs, on-premises is cheaper for sustained 24/7 workloads at enterprise scale.
The 3-5 year horizon is standard for TCO comparison. Comparing cloud vs on-premises over 12 months produces a misleading analysis that always favours cloud.
Before you evaluate on-premises infrastructure, there is a step zero: understanding GPU utilisation. The 30-50% industry average is a real threat to the on-premises cost case.
At a 64-GPU H100 cluster at $3.50/GPU-hour, 40% utilisation means 60% of your capacity is generating no productive output — annual financial waste exceeding $1.1 million per cluster. At 35% MFU on an $833,806 H100 server, your effective cost-per-inference is nearly 3× what the hardware specification suggests.
There is a catch with how most teams measure GPU performance. nvidia-smi reports kernel scheduling activity, not actual Tensor Core computational efficiency. A GPU showing 95% in nvidia-smi may be achieving only 30-40% Model FLOP Utilisation (MFU). Your TCO calculation must use realistic projected MFU — not peak hardware capacity.
vLLM addresses this directly. vLLM is an open-source LLM inference serving framework implementing continuous batching and PagedAttention. Continuous batching dynamically groups concurrent requests to maximise throughput, eliminating the sequential idle time that produces the 30-50% MFU problem. At scale, vLLM achieves 793 tokens/second versus Ollama‘s 41 — MFU can reach 60-80%, effectively halving your per-inference cost.
For a 50-300 person company running 2-4 GPUs on-premises, this matters a lot. The three-tier hybrid architecture is the practical solution: run only consistent, high-volume workloads on-premises, and route variable or experimental workloads to cloud.
For deeper coverage, see our guide to optimisation techniques for your chosen AI inference infrastructure.
The three-tier hybrid architecture routes workloads to the infrastructure tier where the unit economics are best. Per Deloitte’s Tech Trends 2026 research, it looks like this:
Tier 1 — Cloud (AWS, Azure, GCP): burst workloads, model training, new model evaluation, unpredictable or experimental inference. This is where you absorb uncertainty without committing CapEx.
Tier 2 — On-Premises (NVIDIA H100/H200 servers, served via vLLM): consistent, high-volume production inference where fixed costs get cheaper per inference at sufficient sustained volume.
Tier 3 — Edge: ultra-low-latency use cases requiring sub-50ms response — real-time fraud detection, on-device inference, industrial automation.
Hybrid is not a compromise. It is the expected architectural trajectory for organisations that have grown past early-stage experimentation.
Workload classification is the implementation task. Assign each workload to the most cost-effective tier using four dimensions:
When you are ready to migrate from cloud-only to hybrid, follow this sequence:
Organisations that implement hybrid workload routing correctly have documented 40-70% cost reductions versus all-API approaches.
For governance structures that manage multi-tier hybrid infrastructure cost over time, see our guide to AI infrastructure cost governance.
Most published TCO analyses serve either small research setups or Fortune 500 configurations. The 50-500 person SaaS, FinTech, or HealthTech company is underserved. So here is what the numbers actually look like for you.
Minimum viable scale heuristics:
The open-source model breakeven is a compelling calculation. A self-hosted 7B model on a single H100 at 70% utilisation costs approximately $0.013 per 1,000 tokens. GPT-4o Mini is $0.15-$0.60 per 1,000 tokens — that is 10-46× more expensive at volume. At production volumes, breakeven arrives in 3-6 months.
Data sensitivity can accelerate the decision. For regulated HealthTech and FinTech companies, on-premises inference avoids egress AND compliance risk simultaneously. One telehealth company cut monthly AI costs from $48,000 to $32,000 by moving chat triage to a self-hosted LLM, while simplifying its HIPAA compliance posture at the same time.
An infrastructure decision involving on-premises GPU hardware or a shift in cloud commitment requires board or CFO-level approval. Your job is to translate a technical and economic analysis into financial language. Dollar figures, CapEx schedules, and break-even timelines. Not MFU percentages.
Here is a six-part structure that works:
1. Current state cost baseline: Monthly cloud AI spend at reserved instance pricing, egress charges, AI infrastructure cost as a percentage of engineering budget. ICONIQ’s 2026 benchmark puts inference at 23% of revenue at scaling-stage AI companies.
2. Threshold analysis: Cloud cost ÷ on-premises equivalent = threshold ratio. If it exceeds 0.60, proceed to full TCO.
3. TCO comparison: Cloud vs on-premises (or hybrid) over 3-year and 5-year horizons. Lenovo reference for 8× H100: breakeven at on-demand pricing is approximately 11.9 months; at 3-year reserved, approximately 21.8 months.
4. Risk and sensitivity analysis: What happens if token volume grows 3×? If agentic AI is on the roadmap, model the TCO with 5-10× current volumes.
5. Operational requirements: Staff cost delta in dollar terms. Translate “0.5 FTE” into: “approximately $60,000 in additional annual staffing cost, included in the TCO.”
6. Recommendation with trigger criteria: Tied to the threshold calculation, with explicit criteria that would change it. For example: “We recommend on-premises for workload X. If monthly token volume drops below 8M tokens/day, we will revisit.”
You will also need to answer these objections:
“Cloud is always more flexible” — Correct for burst workloads. The hybrid architecture preserves that flexibility where it matters, while eliminating cloud costs on predictable production workloads where elasticity provides no benefit.
“We don’t have the staff” — The staffing cost is quantified in the TCO: 0.5-1.0 FTE at $X versus $Y in annual cloud savings. If the payback is unacceptable, narrow the hybrid scope to the highest-volume workloads only.
“What if GPU prices keep dropping?” — Per-token pricing has fallen 10× annually, but total inference spending grew 320% over the same period. AWS raised capacity pricing in January 2026.
For ongoing governance of AI infrastructure costs post-decision, see our guide to AI infrastructure cost governance.
It is a ratio: cloud AI costs ÷ on-premises equivalent costs. When this ratio reaches 0.60-0.70, on-premises or hybrid economics become competitive. To measure it: (1) calculate monthly cloud AI spend at reserved instance pricing; (2) add egress costs; (3) price equivalent on-premises hardware amortised over 3 years plus staffing delta; (4) divide step 1+2 total by step 3 total. Source: Deloitte Tech Trends 2026, based on research with 60+ global technology leaders.
Yes. The realistic requirement is 0.5-1.5 FTE depending on scale. For a 50-150 person company with an existing DevOps function (2-4 GPUs), 0.5 FTE is realistic. For a 200-500 person company with multiple GPU servers, plan for 1.0-1.5 FTE. Include this at $120,000+ fully loaded per FTE in your TCO — omitting it is the most common error in on-premises business cases.
The evaluation trigger is 10 million+ tokens/day with consistent patterns, or 12+ GPU-hours/day of sustained inference. Below these volumes, cloud reserved instances almost always produce better TCO when staffing costs are included. At volumes above 50 million tokens/day, the on-premises or hybrid case is almost always financially superior.
vLLM is an open-source LLM inference serving framework implementing continuous batching and PagedAttention — the primary techniques for improving GPU utilisation. Without continuous batching, sequential requests leave GPUs idle, producing the 30-50% MFU industry average. With vLLM, MFU can reach 60-80%, effectively halving per-inference cost. It is the de facto standard for self-hosted open-source model serving (Llama, Qwen, Mistral).
nvidia-smi reports kernel scheduling activity, not actual Tensor Core efficiency. Model FLOP Utilisation (MFU) measures how much of the GPU’s theoretical throughput is used for productive work. A GPU can show 75-85% in nvidia-smi while achieving only 30-40% MFU, because memory fetches, attention overhead, and scheduling latency register as “active” without contributing to throughput.
At low volume (under 10 million tokens/day): API pricing almost always wins when staffing costs are included. At high volume (50 million+ tokens/day) with consistent workload patterns: self-hosting open-source models via vLLM can break even against API costs in 3-6 months, then produce 60-80% lower per-token costs. Data sensitivity can force the decision regardless of cost: regulated industries that cannot send data to third-party APIs must self-host.
Agentic AI has caused token consumption per task to jump 10-100× since December 2023. A single agentic workflow may make 10-50 API calls per user request versus 1-2 for a simple chatbot. If agentic AI is on your roadmap within 12-18 months, model the TCO with 5-10× higher token volumes — the threshold may be considerably closer than your current spending suggests.
Data egress charges are fees imposed by cloud providers when data moves out of their infrastructure. For data-intensive AI applications, egress typically adds 15-30% to total cloud AI spend; for high-bandwidth applications, it can reach 70%. On-premises inference avoids egress entirely for workloads where data remains within your network. Estimate your monthly data movement in GB, multiply by your provider’s egress rate, and add it to the cloud cost baseline before computing the threshold ratio.
Formula: (CapEx + cumulative operational costs) ÷ monthly cloud savings = breakeven in months. Lenovo reference data for an 8× H100 server: breakeven at on-demand pricing is approximately 11.9 months; at 3-year reserved pricing, approximately 21.8 months. GPU utilisation is highly sensitive: at 35% MFU, breakeven extends; at 70%+ MFU (achievable with vLLM), breakeven accelerates toward the 12-month end.
Current-generation: NVIDIA H100 (80GB HBM3) and H200 (141GB HBM3e). The Lenovo ThinkSystem SR675 V3 with 8× H100 GPUs is the enterprise reference at approximately $833,806. For mid-market: 2-4 NVIDIA A100 GPUs — $120,000-$300,000, appropriate for 10-30M tokens/day workloads. AMD MI300X is competitive for memory-bound inference but has a less mature software ecosystem. Always model hardware refresh cycles (3-5 year lifespan) with zero recovery value.
Four dimensions: (1) Volume — consistent, high → on-premises; variable or burst → cloud; (2) Latency — real-time under 100ms → edge or on-premises; batch-tolerant → cloud; (3) Data sensitivity — regulated → on-premises preferred; public → cloud acceptable; (4) Cost per inference — run the unit economics at each tier and compare. On-premises candidates: high-volume consistent APIs (document processing, fraud scoring). Cloud candidates: model training, new model evaluation, burst demand.
The infrastructure decision is one component of managing AI inference costs at scale. For a complete overview of AI inference economics — from why the cost crisis exists through to pricing strategy and governance — see our overview of AI inference economics and the forces driving this crisis.
The AI Inference Optimisation Playbook — Caching, Quantization, and Model Routing in Priority OrderInference now accounts for 80–90% of total AI compute costs across a model’s production lifetime, yet most guides throw every optimisation technique at you in random order and leave you to work out where to start. If the PoC bill shock has already hit, you don’t need a catalogue — you need a sequence.
This is that sequence. Three tiers, ordered by effort-to-impact ratio. Tier 1: zero-infrastructure API-level changes you can make today. Tier 2: configuration changes that take days to a week. Tier 3: structural engineering work for this quarter. Each tier tells you whether the next one is worth the investment.
One note before we get into it: if you’re building or running agentic AI, the same techniques apply — but at a 5–20x cost multiplier per user action. Optimising at the individual call level misses the point. You need to optimise the chain. More on that at the end.
For the broader economic context behind why inference costs are eating AI budgets, the AI inference cost crisis guide covers the full picture. This article is the action layer on top of that foundation.
Most resources catalogue inference optimisation techniques without telling you which to do first. You end up with a list of equally-weighted options when what you actually need is a priority stack.
The sequencing axis that matters is effort-to-impact ratio — not technique prestige or theoretical maximum savings. A team that enables prompt caching this week (zero infrastructure change, 50–90% cost reduction for qualifying workloads) gets a faster return than one that spends two months deploying a self-hosted vLLM stack with quantization. The data backs this up: teams that implement caching, routing, and batching before touching their serving infrastructure consistently outperform teams that go straight to structural interventions.
Here’s the effort spectrum: API configuration changes take hours. Serving configuration changes take days to weeks. Quantization pipelines take weeks to months. And the cost reduction spectrum doesn’t map to effort in the direction you’d expect. The lowest-effort interventions often produce the largest percentage reductions for API-heavy workloads.
Reversibility matters too. Tier 1 changes are reversible with a config flag. Tier 3 changes — quantization, model replacement — require rigorous accuracy validation before production. Higher tiers carry higher rollback risk, which matters when you’re touching production systems.
ICONIQ’s analysis of scaling-stage AI companies found that model inference averages 23% of total AI product costs — nearly as expensive as the entire AI team. The pressure to optimise is real. The question is just where to aim first.
Three actions qualify as Tier 1 zero-infrastructure wins: enable API-level prompt caching, audit model routing, and shift latency-tolerant workloads to async batch processing. None of these require standing up new infrastructure, hiring ML engineers, or touching model weights.
The starting point for all three is a workload audit. Categorise requests by: (a) prompt repetition rate, (b) complexity requirements, and (c) latency sensitivity. That audit maps directly to which Tier 1 wins apply to your situation.
Prompt caching applies if your workload includes repeated system prompts or shared contexts. Model routing applies if you’re sending simple queries to expensive frontier models. Async batch processing applies if you have analytics jobs, nightly content moderation passes, or embedding generation that doesn’t need a real-time response.
The wins also compound. A workload with 60% cacheable prompts and a 50/50 routing split between simple and complex queries can see 40–60% total cost reduction from configuration changes alone — before touching a single line of infrastructure.
Prompt caching works by storing the computed key-value (KV) attention representations of repeated prompt prefixes. When the same prefix appears again, the API skips recomputation and charges only for the new tokens — reducing cost on cached tokens by 50–90%.
There are two distinct layers of caching to understand.
API-level prompt caching is exposed by Anthropic and OpenAI as a managed feature. No infrastructure change required. You restructure your prompts and the cost reduction happens automatically. Google’s Gemini API calls the same thing “context caching.” This is the Tier 1 entry point.
Infrastructure-level KV cache is the GPU memory mechanism in self-hosted inference engines. It serves the same purpose but lives at the infrastructure level — storing intermediate attention computations to avoid recomputing them for already-processed tokens. Storing the KV cache for a 500B parameter model over a 20,000-token context requires about 126GB of memory — which gives you a sense of the scale involved. This is a Tier 2 concern for teams running self-hosted inference.
For teams on managed APIs, the only implementation action is prompt structure. Stable, reusable content goes at the beginning — the prefix position where caching applies. Variable content, like the user’s actual query, goes at the end.
How do you know if your workload qualifies? High qualification signals include static system prompts longer than 1,024 tokens, RAG retrieval contexts that repeat across users, few-shot examples embedded in every request, and multi-turn conversations where the system prompt is always present.
Cache hit rate is the key metric to monitor. Tensormesh’s LMCache CacheBlend reports 85% cache hit rates for agentic AI workloads with repeated tool descriptions in system prompts — at that hit rate, effective cost per request drops to near the cost of the variable suffix only.
As a rough illustration: a $10,000/month API spend on a RAG system with 70% cache-eligible requests could realistically land at $3,000–$4,500/month after caching. Your workload will vary, but the mechanism is consistent.
Model routing inserts a lightweight classification layer between your application and your LLM. Simple queries go to smaller, cheaper models. Complex requests escalate to frontier models only when genuinely needed. ICONIQ identifies this as table-stakes cost management at scaling-stage companies — not an optional nicety.
The core problem is straightforward: most production inference workloads contain a significant fraction of requests that don’t require frontier model capability. FAQs, classification tasks, simple data extraction, templated responses — all priced at frontier rates because routing logic doesn’t exist.
Routing a query to Claude Haiku ($0.25/$1.25 per million input/output tokens) instead of Claude Opus ($15/$75) represents a 60x cost reduction for that request. Even routing 30% of your queries to cheaper models produces meaningful savings at any volume.
LiteLLM and Portkey are the primary open-source routing tools. Portkey supports conditional routing — “use cheaper model for summarisation, premium model for reasoning” — with fine-grained conditions and a unified API across multiple providers.
Routing strategies range from simple to sophisticated: rule-based (prompt length, keyword detection), classifier-based (a small model scores complexity), or threshold-based (use the cheaper model’s confidence score to decide escalation). Start simple. Rule-based routing based on prompt length and task type gets you most of the savings with minimal configuration overhead.
One thing to get right: validate routing decisions against a benchmark of accepted output quality for each task type before deploying to production. Route based on measured output quality, not expected output quality.
Multi-provider routing also adds resilience beyond cost. Enterprises processing millions of requests daily almost always hedge across two or more providers — routing across Anthropic, OpenAI, and a self-hosted endpoint gives you both price arbitrage and failover capacity.
This is Tier 2 territory — it requires self-hosted infrastructure. If you’re exclusively on managed APIs, your provider handles this. Focus Tier 2 effort on your own inference deployments. (If you haven’t yet settled your deployment model, the cloud vs on-premises deployment decision framework covers that decision before you commit to self-hosted infrastructure investment.)
Enterprise GPU clusters typically operate at only 30–50% utilisation. The reason is static batching: the serving engine waits for a fixed batch size before processing, then works through the whole batch before accepting new requests. The idle time while waiting is paid-for capacity doing nothing.
A 64 H100 GPU cluster at 40% utilisation at $3.50/GPU-hour costs $161,280/month total — of which roughly 60% is waste. Raising utilisation from 40% to 80% effectively doubles compute capacity without spending another cent on hardware.
Continuous batching eliminates the wait. As each sequence in a batch completes, a new request slots in immediately — keeping the GPU continuously occupied. TensorRT-LLM calls this “in-flight batching”; the mechanics are the same. TGI and vLLM both implement it natively. Ollama does not.
vLLM pairs continuous batching with PagedAttention — its KV cache memory management system that treats GPU memory as virtual pages, eliminating fragmentation and enabling efficient memory sharing across concurrent requests. Three vLLM parameters determine whether a GPU saturates or wastes: --max-num-seqs, --gpu-memory-utilization, and --tensor-parallel-size.
One trade-off to monitor: time-to-first-token (TTFT) may increase slightly with continuous batching under high load. Track TTFT alongside GPU utilisation % and throughput (tokens/sec) — expose these through Prometheus/Grafana or your inference engine’s built-in metrics endpoint.
Tier 3 requires dedicated engineering time, accuracy validation pipelines, and staging environments before production deployment. Don’t skip Tier 1 and Tier 2 before committing to this tier — the ROI justification should be based on validated earlier-tier data.
Here are the three structural interventions, in order of how frequently they’ll apply.
Model quantization reduces the numerical precision of model weights, shrinking model size and GPU memory footprint substantially. Quantization is the single biggest optimisation you can apply before touching your serving engine — it cuts VRAM requirements by 50–75% and lifts throughput by removing memory bandwidth bottlenecks.
The format decision maps to your hardware:
Post-training quantization (PTQ) is the correct entry point — calibrate on a representative dataset, convert to the target format, validate accuracy, deploy. Quantization-aware training (QAT) adds training cost that is usually not justified.
One important caveat: benchmark accuracy drops and task-specific accuracy drops are different things. INT4 formats require task-specific validation — don’t assume sub-1% loss without measuring on your actual workload, particularly for legal, medical, or financial reasoning tasks.
vLLM with PagedAttention is the production serving standard for teams running self-hosted inference at scale. PagedAttention manages KV cache memory as virtual memory pages, dramatically reducing memory fragmentation and enabling efficient memory sharing across concurrent requests. vLLM supports over 100 model architectures and runs on NVIDIA V100 through current, AMD MI200/MI300, Google TPUs, AWS Inferentia, and Intel Gaudi — this breadth prevents vendor lock-in.
Speculative decoding is a latency optimisation, not primarily a cost optimisation. It pairs a small draft model with the large target model — the draft model generates 4–5 candidate tokens; the target model verifies them in a single forward pass. When draft tokens are correct (70–80% of the time for chat workloads), this delivers 1.8–2.2× speedup on generation throughput. Use it for latency-sensitive applications — real-time chat, voice interfaces, interactive coding assistants. Don’t prioritise it as a cost reduction measure.
Here’s an honest breakdown.
vLLM: Production standard for teams running self-hosted inference at scale. Continuous batching, quantization (FP8, AWQ), and speculative decoding in a single stack. Supports high-concurrency multi-user workloads.
The honest limitation: vLLM requires CUDA-compatible GPU hardware, Python inference stack knowledge, and ongoing monitoring configuration. Without a dedicated ML engineering resource, the operational burden is significant. Managed inference APIs have a lower operational cost even if per-token pricing is higher — and at moderate traffic, serverless costs 77% less than a 24/7 dedicated pod. The break-even point occurs when utilisation consistently exceeds 65–70% of an always-on deployment.
Ollama: Best for local development, single-developer environments, and testing model behaviour before production deployment. Uses GGUF format via llama.cpp; supports CPU offload. Does not implement continuous batching natively. Not suitable for multi-user production API backends. In practice, Ollama is frequently deployed in multi-user scenarios where it underperforms significantly.
LocalAI: Best for OpenAI API-compatible self-hosting where provider lock-in is the primary concern — niche regulatory or air-gapped environments. Production-readiness is lower than vLLM. Not the first choice for pure inference cost optimisation.
SGLang (for agentic workloads): optimised for structured generation and multi-call agentic pipelines. SGLang’s RadixAttention caches and reuses KV states across requests sharing common prefixes — in agentic pipelines where every tool-call response starts with the same system prompt, that prefix sharing reduces TTFT by 30–60% compared to naive per-request serving. The guidance from production teams is clear: use vLLM for chat, completions, and RAG; use SGLang when your workflow runs multiple sequential LLM calls.
The decision tree: continuous batching + quantization + speculative decoding at high concurrency → vLLM; local dev and testing → Ollama; OpenAI API compatibility in an air-gapped environment → LocalAI; agentic pipelines with structured output → evaluate SGLang.
Before committing to self-hosted vLLM, evaluate managed inference providers (Together AI, Baseten, RunPod). They provide vLLM-like performance without the operational overhead — and for many teams at the 50–500 person scale, that trade-off is worth it.
Agentic AI systems multiply inference costs 5–20x per user action compared to single-call interactions. Each agent step is a separate inference call with its own context, tool descriptions, and reasoning output. A customer support agent that looks efficient at 100 tokens per interaction can easily use 2,000–5,000 tokens when a scenario requires multiple tool calls, context retrieval, and multi-step reasoning.
The implication for optimisation: think at the chain level, not the call level. Optimising individual call cost by 20% in a 10-call chain yields 20% savings across the chain. Restructuring the chain to eliminate three redundant calls yields a saving that compounds with every execution.
Prompt caching is disproportionately valuable for agentic workloads specifically because agent system prompts include tool descriptions, reasoning instructions, and context that repeat across every step. These are exactly the high-repetition prefixes that caching eliminates — and the same high cache hit rates apply here with even greater effect.
Model routing within the chain is an underexplored lever. Tool selection and simple classification steps can route to a smaller, cheaper model tier. Synthesis and reasoning steps escalate to the full model. Apply the same routing logic you set up in Tier 1, but at each step of the chain.
Track cost per agent execution — the full chain cost per user action — not cost per call. “Dollar-per-decision is a better ROI metric for agentic systems than cost-per-inference because it captures both the cost and the business value of each autonomous decision.”
For a full cost governance framework covering monitoring, alerting, and FinOps practice — including how to sustain the gains you make applying this playbook — the AI FinOps governance to sustain your optimisation gains covers the next layer. This playbook is the optimisation layer; governance is what keeps it working at scale.
Yes. API-level prompt caching is a managed feature from Anthropic (Claude 3.x) and OpenAI (cached input tokens) — no infrastructure change required. Google’s equivalent is called “context caching” in the Gemini API. The only implementation action is restructuring prompts so stable content appears at the start. No GPU server, no model deployment, no DevOps overhead.
On standard benchmarks, INT8 quantization produces less than 1% accuracy degradation. FP8 (native on NVIDIA H100) offers higher accuracy with similar compression benefits — prefer FP8 if H100 hardware is available. Always validate on a representative sample of production queries before deploying quantized models, particularly for tasks sensitive to output precision.
Honestly, it is operationally demanding. Without a dedicated ML engineering resource, the operational burden is significant. Managed inference providers (Together AI, Baseten, RunPod) provide vLLM-like performance without the overhead. The self-hosting break-even typically becomes compelling when monthly managed API spend exceeds approximately $20,000–$50,000. Complete all Tier 1 optimisations first — some teams find that prompt caching and model routing bring managed API costs low enough that self-hosting is never justified.
KV cache is the GPU memory mechanism in inference engines that stores intermediate attention computations to avoid recomputing them — this operates at the infrastructure level in self-hosted serving stacks. Prompt caching is the API-level feature from Anthropic and OpenAI that exposes the same underlying mechanism as a managed service. Both serve the same purpose at different layers.
Continuous batching dynamically slots new requests into a running batch as completed sequences free up GPU capacity. Typical enterprise GPU utilisation without it: 30–40%. With it: 70–80%+. That improvement effectively halves cost per request on the same hardware — more impactful than a GPU hardware upgrade costing tens of thousands of dollars. Available natively in vLLM, TGI, and TensorRT-LLM. Not available in Ollama.
FP8 if running NVIDIA H100 hardware — native hardware support, highest accuracy, under 1% perplexity delta. AWQ for INT4 (4-bit) on Ada Lovelace hardware — ~3% perplexity delta, outperforms GPTQ at the same bit-width. GPTQ for Ampere and older GPUs — ~6% perplexity delta at 4-bit. GGUF only for Ollama or llama.cpp in CPU offload or local development environments.
The primary signal is when monthly managed API spend reaches the point where self-hosted infrastructure TCO becomes cheaper over a 12–24 month horizon. At moderate traffic, serverless GPU costs 77% less than a 24/7 dedicated pod — the break-even occurs when utilisation consistently exceeds 65–70% of an always-on deployment. Run all Tier 1 optimisations first. Some teams find prompt caching and model routing bring costs low enough that self-hosting is never justified.
Speculative decoding pairs a small draft model with the large target model. The draft model generates 4–5 candidate tokens; the target model verifies them in a single forward pass. The primary benefit is latency reduction, not cost reduction. Use it for latency-sensitive applications: real-time chat, voice interfaces, interactive coding assistants. Don’t prioritise it as a cost reduction measure.
Async batch processing (Tier 1): grouping latency-insensitive workloads — document analysis, embeddings, content moderation, nightly jobs — and submitting them to a deferred batch API. OpenAI’s Batch API offers a flat 50% discount for this class of request. No infrastructure change required. Continuous batching (Tier 2): a real-time serving strategy that groups concurrent incoming requests dynamically to maximise GPU utilisation. One is a scheduling strategy; the other is a serving strategy.
The primary metric is cost per million tokens, measured before and after each optimisation tier. For prompt caching, cache hit rate is the key leading indicator — most API providers expose this in usage dashboards. For model routing, track cost distribution by model tier alongside quality metrics by task type. For GPU utilisation work, monitor GPU utilisation %, TTFT, and throughput through Prometheus/Grafana or your inference engine’s built-in metrics endpoint. For a complete cost governance framework, see the AI infrastructure cost governance guide.
Agentic AI systems make multiple sequential LLM calls per user action — tool selection, execution, result interpretation, response generation. A user action in an agent pipeline costs 5–20x more than an equivalent single-call interaction because each pipeline step accumulates token costs including repeated tool descriptions and reasoning context. Optimisations must be evaluated at the chain level. Prompt caching is disproportionately valuable here because the tool description system prompts that repeat across every chain step are exactly the high-repetition prefixes that caching eliminates.
This playbook is one part of a broader series on AI inference economics. For a complete AI inference cost guide covering the financial reality, infrastructure decisions, pricing strategy, and governance practice, see What the AI Inference Cost Crisis Means for Growing Software Companies.
Why AI Gross Margins Are So Much Lower Than SaaS and What That Means for Your BusinessPicture this: you’re presenting your AI product’s financial performance to the board. They’ve spent a decade applying the same mental model — 75% gross margins, maybe better. The number on the slide is 52%. The silence that follows isn’t scepticism about your competence. It’s the sound of a financial model colliding with a structural economic reality nobody warned them about.
Every time a user interacts with your AI product — every query, every generation, every agent action — the meter runs. There’s a real compute cost attached to each of those interactions, and SaaS never had that.
ICONIQ Capital‘s January 2026 State of AI report finds inference averages 23% of total revenue at scaling-stage AI B2B companies. Bessemer Venture Partners documents AI gross margins at 50-60%, against 70-90% for mature SaaS businesses. These aren’t outliers from a bad quarter. They’re structural characteristics of the asset class.
This article is about why the economics are different — so you can set the right expectations, price correctly, and walk into that board meeting prepared. For cost reduction strategies, see our AI inference cost crisis overview and full guide to AI inference economics.
Here’s the core difference. Traditional SaaS: once you’ve written the software and provisioned the servers, serving an additional user costs almost nothing. The marginal cost approaches zero at scale. AI inference: every single user query runs the model again — consuming GPU compute, memory bandwidth, and energy. Every. Single. Time.
SaaS is like a printed book. Write it once, distribute at near-zero marginal cost. AI is more like a human expert answering questions. Each answer has a labour cost, and the more questions you field, the higher the bill.
Traditional SaaS COGS scales sub-linearly with users. AI COGS scales directly with usage. As Ben Murray at TheSaaSCFO puts it: “Once the product is built, incremental cost to produce a dollar of revenue is very low. AI companies, on the other hand, sell into software and labour budgets.”
Training is a one-time capital expense. Inference is continuous operational expenditure — every query, every agent action, every API call adds to the tab. Training might cost $100,000 once. Inference scales to $10,000 a month at a million queries.
Training happens once. Inference happens every time a user interacts — thousands of times, then millions. That inversion is counterintuitive because training headlines dominate media coverage. GPT-4‘s training costs got extensive coverage. The ongoing inference costs generating revenue around the clock got almost none.
A model is trained once over weeks, then served to users for months or years. Training represents 10-20% of total compute costs over a model’s lifecycle. Inference represents 80-90%.
As your product gains users, training costs stay fixed while inference costs grow linearly. For agentic AI architectures — where a single user action triggers a sequence of model calls — inference costs multiply 5-20x per user action. We dig into this in depth in the article on why AI bills explode between pilot and production.
ICONIQ’s January 2026 State of AI report finds inference averages 23% of total revenue at scaling-stage AI B2B companies. And this figure doesn’t meaningfully decline as companies grow. In plain P&L terms: for every $1M in AI product revenue, approximately $230,000 is consumed by inference costs before you’ve paid for engineering, sales, or anything else.
To make that concrete:
$1M AI product revenue → ~$230,000 annually in inference costs
$5M AI product revenue (200-person company) → ~$1,150,000 annually
$10M AI product revenue → ~$2,300,000 annually
In traditional SaaS, COGS at scale typically runs 10-25% of revenue. Inference alone at 23% puts AI at the high end of total SaaS COGS before you’ve added anything else. And it rises with scale — inference sits at 20% pre-launch and 23% at scale. As Jason Lemkin frames it: “as you grow, you need ever more inference. You can’t cut it without degrading the product.”
There’s an important nuance here for SaaS companies adding AI features. The ICONIQ benchmark applies to pure AI B2B companies. Only your AI-adjacent revenue should be measured against inference costs — not your total ARR. A $10M ARR SaaS company launching an AI feature tier generating $1M in incremental revenue faces ~$230K in inference costs against that $1M — not against the $10M base. Getting that wrong has real consequences. See how to build AI cost governance without a dedicated FinOps team.
Token prices have fallen approximately 1,000x over three years. What cost $60 per million tokens in 2021 now costs around $0.06. And yet enterprise LLM API spending has grown 320% over the same period.
This is Jevons Paradox in action: when a resource becomes significantly cheaper, organisations use substantially more of it, and total consumption rises rather than falls.
When GPT-4 cost $60 per million tokens, companies deployed it carefully. At $3 per million tokens, they expanded to five new use cases. At $0.10 per million tokens, every workflow gets AI. Total tokens multiply faster than the per-token price drops. a16z calls this “LLMflation” — from their OpenRouter analysis, tokens consumed quintupled over the same period that per-token prices dropped to one-third.
There’s also a hardware paradox at work. Even as software-layer token costs fall, AWS raised GPU Capacity Block prices by 15% in January 2026. Physical compute has its own supply constraints — lead times on H100 and H200 clusters exceeding 30 weeks. Agentic workflows multiply token consumption non-linearly, with a single user action triggering 5-20 sequential model calls.
Don’t model falling inference costs as guaranteed budget relief. LLMflation is a signal to build usage governance, not a signal to skip it. Without governance, efficiency gains get absorbed by consumption growth.
AI companies structurally operate at approximately 50-60% gross margins, compared to 70-90% for mature SaaS companies. This isn’t a temporary inefficiency — it’s an architectural consequence of inference costs. ICONIQ’s January 2026 data shows AI gross margins averaging 52%, up from 41% in 2024, with a ceiling well below SaaS norms.
SaaS at scale runs COGS of roughly 10-25%, yielding gross margins of 75-90%. AI companies run COGS of roughly 40-50% — with inference alone accounting for ~23% — yielding gross margins of 50-60%. The 15-30 percentage point gap cannot be fully recovered through operational efficiency.
AI Shooting Stars — fast-growing, capital-efficient AI startups with strong product-market fit — average approximately 60% gross margins. AI Supernovas — explosively scaling, thin-wrapper products — can sit as low as 25%. AI companies also need higher growth rates to hit Rule of 40 because the profit margin component starts from a lower base. As Murray at TheSaaSCFO frames it: “If SaaS is about margin efficiency, AI is about value density — how much output, productivity, or labour you replace per dollar of cost.”
For SaaS companies adding AI features: those features will compress your margins unless priced to recover inference costs separately. The natural response is in pricing — which we explore in how to design AI product pricing.
TheSaaSCFO’s financial modelling finds that to match the EBITDA output of an equivalent SaaS company, an AI company would need to be 6x the revenue size. Here’s the arithmetic:
If your AI product replaces $200,000 of annual labour value, pricing at $20,000 — the SaaS-era reflex — destroys margin. BVP’s research indicates AI companies must price 5-6x SaaS equivalents for comparable unit economics.
Consumption-based pricing is transparent but creates buyer anxiety about unpredictable bills. Outcome-based pricing — per successful result, per resolution — aligns AI costs with delivered value. ICONIQ’s data shows this shift is underway: 37% of AI companies plan to change their pricing model in the next 12 months, and outcome-based pricing jumped from 2% to 18% of AI companies in just six months. Intercom’s Fin AI agent is the archetype — per-ticket-resolution pricing grew to 8-figure ARR by tying revenue directly to the value delivered.
Don’t add AI features at zero incremental cost to existing subscriptions. The detailed pricing framework is covered in how to design AI product pricing that survives variable inference costs.
The numbers add up faster than most companies expect. Unlike SaaS COGS, inference costs grow with AI usage and require active management, not passive provisioning.
Here’s what the tiered reality looks like across company sizes:
Your board benchmarks gross margins against SaaS comps — typically 70-90% for software companies. AI-augmented products will show 50-65%. That gap needs proactive framing. AI gross margins at 52-60% are a structural characteristic of the asset class, documented by ICONIQ (January 2026), Bessemer Venture Partners, and TheSaaSCFO. The margin gap is the cost of the competitive moat — your competitors face the same economics.
Three practical starting points:
Know your inference-to-revenue ratio and compare it to the ICONIQ 23% benchmark. Above it likely means inefficient; below it may mean underinvesting in product capability.
Price AI features with inference costs built in, not absorbed from existing SaaS margins. Even modest per-feature pricing lets you track and manage inference costs against attributable revenue.
Set a governance trigger: when inference approaches 30% of AI product revenue, activate a cost review. The governance framework for this is detailed in how to build AI cost governance without a dedicated FinOps team.
Understanding the structural difference from SaaS economics is the prerequisite for every financial decision you’ll make as you scale an AI product. For a complete overview of the inference cost crisis and how all of these elements fit together, see the full guide to AI inference economics.
No — gross margins vary by position in the AI value chain. Infrastructure-layer companies reselling compute (Groq, Together AI) sit as low as 40-50%. Application-layer companies adding proprietary value above raw compute cost (Perplexity at approximately 60%) sit higher. AI Supernovas — explosively scaling, thin-wrapper products — can be as low as 25% or negative. AI Shooting Stars — Bessemer’s cohort of capital-efficient, strong-PMF startups — average approximately 60%.
The ICONIQ 52% is the industry average for scaling-stage AI B2B companies in January 2026, up from 41% in 2024. Individual margins range from 25-85%+ depending on architecture and infrastructure strategy.
Yes, but the targets differ. AI companies with structurally lower gross margins need higher growth rates to hit Rule of 40 because the profit margin component starts from a lower base. Some investors are applying gross-margin-adjusted versions of Rule of 40 for AI companies — the benchmark is evolving, but the underlying principle remains valid.
Yes, with caveats. ICONIQ data shows improvement: 41% in 2024, 45% in 2025, 52% in 2026. Optimisation paths — model routing, prompt caching, quantisation, infrastructure migration — are covered in our articles on AI inference cost reduction and building AI cost governance. The structural floor remains: AI gross margins will likely improve toward 60-65% but are unlikely to reach the 80%+ that SaaS companies achieve.
LLMflation (a16z terminology) describes the approximately 10x annual decline in inference costs for equivalent model performance — what cost $60 per million tokens three years ago now costs around $0.06. The paradox: LLMflation makes individual model calls cheaper, but Jevons Paradox means total AI spend rises as companies deploy AI to more use cases. Tokens consumed quintupled while per-token prices dropped to one-third. The per-unit cost drops; the total bill climbs. LLMflation is a signal to build usage governance, not to assume cost savings will materialise automatically.
The inference tax is the recurring compute cost incurred every time an AI product serves a user — analogous to a raw material cost in manufacturing. Traditional SaaS COGS scales sub-linearly — the 10,000th customer costs little more than the 1,000th. Inference COGS scales directly with usage — the 10,000th AI query costs approximately the same as the 1,000th. The governance implication: treat it as a managed variable cost, not fixed overhead.
Agentic AI workflows multiply inference costs 5-20x per user action. A simple chatbot query: 1 model call. An agentic AI researching, planning, and executing the same resolution: 8-15 model calls. Expect inference costs to increase 5-20x when moving from chatbot to agent architectures before any optimisation — which is why outcome-based pricing makes more sense than seat-based pricing for agentic AI products.
Because AI products have a perpetual raw material cost that traditional software does not. Traditional SaaS: build once, serve millions at near-zero marginal cost. AI SaaS: every user interaction requires running the model — generating real compute cost. Gross margins don’t improve as easily with scale because inference costs scale with usage. AI companies are partly technology companies and partly compute resellers — purchasing GPU inference and reselling it as intelligent capability.
Partially — the benchmark applies to the AI-revenue portion of the business, not total ARR. A $10M ARR SaaS company launching an AI feature tier generating $1M in incremental revenue faces ~$230K in inference costs against that $1M — not $2.3M against the full base. If AI features are priced at zero incremental cost, that inference cost is absorbed silently from existing margins. Always price AI features separately so the costs can be tracked and managed.
Training is the one-time process of teaching a model its capabilities — it happens once at model creation and occasionally during fine-tuning cycles. Inference is the ongoing process of generating responses to user queries — running continuously for the lifetime of the product, 24/7 in production. The cost ratio: training represents 10-20% of total compute costs over a model’s lifecycle; inference represents 80-90%, because it runs millions of times while training runs once.
Your board benchmarks gross margins against SaaS comps — typically 70-90%. AI-augmented products will show 50-65%. Three elements of the reframe:
Validate externally: AI gross margins at 52-60% are a structural characteristic documented by ICONIQ (January 2026), Bessemer Venture Partners, and TheSaaSCFO. Bring these sources to the board discussion.
Frame the opportunity: the inference cost burden is what enables the AI capability that differentiates your product. Your competitors face the same economics.
Provide a roadmap: model routing, caching, and quantisation have documented paths toward 60-65% gross margins over 12-24 months — this is not a permanent ceiling.