Cloud-first used to be a default you never had to defend. When production AI arrived, the hard question looked like which model, because the cloud was the obvious place to run it. Then a single number broke the habit: public cloud’s share of production AI inference fell from 56% to 41% in a single year, according to Broadcom’s Private Cloud Outlook 2026. The market moved its workloads.
Inference bills keep climbing even as the per-token sticker price falls, because the real cost shows up in useful tokens and GPU utilisation ahead of the advertised rate. Getting placement wrong is now the costliest mistake in production AI.
This is a “which workloads, and why” analysis. By the end you’ll be able to name the crossover, the steady-versus-burst mechanism, and a six-dimension framework for deciding, and proving, where each workload runs. For context, see cloud repatriation and digital sovereignty and the repatriation reality check.
Why did public cloud’s share of production AI inference fall from 56% to 41% in a single year?
Public cloud’s share of production inference dropped from 56% to 41% in one year. That figure is vendor-published, from Broadcom’s blind survey of 1,800 IT leaders, and it has not yet been independently corroborated, but it points the same direction as independent data showing 66% of organisations moved AI workloads off the public cloud in the past year.
Three forces compounded. First, GPU hours are the most expensive compute tier, and a 24/7 inference service turns them into a large, predictable line item that owned capacity can undercut. Second, data gravity: every token processed against your data argues for running the model beside it instead of shipping it out and back. Third, governance: privacy as a barrier to AI adoption climbed from 53% to 77% in KPMG’s survey, a veto before cost even enters the conversation.
Notice which load actually moved. Training stayed put in the hyperscale cloud, where deep accelerator pools and bursty experimentation still win, while the steady, production-side inference load repatriated. Read it as a per-workload cost-and-control correction, the reality check in action.
Private AI inference versus public cloud inference: where do the economics flip?
The economics flip at a volume crossover. Roughly one to two million tokens a day is the toss-up zone; past about two million a day, self-hosting wins and the GPU investment pays back in six to nine months, with some sources stretching the window to twelve months. The spread reflects utilisation.
Everything turns on GPU utilisation: a busy cluster amortises its cost across millions of tokens, while idle accelerators are how self-hosting loses to an API. Open-weight models such as Llama and Mistral are now good enough for a wide share of enterprise use, which removes the dependence on cloud APIs. Self-hosting still loses below the crossover, on frontier models (OpenAI, Anthropic and Google keep theirs closed), and on the MLOps line item, which often exceeds the hardware cost.
Cost per million tokens is the hourly GPU rate spread across however many millions of tokens the GPU serves that hour, and utilisation decides how many that is. A middle path, managed deployment, lets a provider run the stack in your tenancy without hiring an inference team. Egress and exit fees make the crossover harder than the sticker maths suggests, as the egress and lock-in piece explains.
Why do steady-state workloads repatriate well while burst and elastic workloads do not?
Steady-state traffic, a predictable baseline of requests week after week, favours owned capacity because high utilisation converts a fixed hardware cost into a low cost per token. Burst and elastic traffic favours rented capacity, because paying per token only when demand spikes is cheaper than owning GPUs that sit idle the rest of the time.
That is the utilisation mechanic from the crossover, now split by demand shape. A GPU bought for a peak that never recurs keeps costing you during the quiet hours between spikes, while a hosted API charges you less for those same hours. Steady load left in the cloud means paying per-token rates forever on volume owned capacity could serve at a fraction; bursty load moved on-premises means buying capacity for a peak you only occasionally hit, while autoscaling overhead hides in the per-token bill.
Each demand shape has a natural home: cloud for elasticity, on-premises and regional for consistency, edge for immediacy. “Predictable” is still an estimate, and misjudged steady-state traffic is exactly how repatriation projects overspend on hardware they cannot fill. The repatriation reality check walks through where steady-state AI inference should run.
Which workload types get cheaper when repatriated, and which get more expensive?
Repatriation is not uniformly cheaper for your business. It saves money on predictable, data-heavy and compliance-bound workloads, and costs more on bursty, globally distributed, serverless and high-egress ones. The honest answer is a split.
Cheaper: your steady-state inference and high-data-volume workloads, where processing beside the data avoids per-token and per-gigabyte movement, and your compliance-bound workloads, where sovereignty rules make private the only viable option. More expensive: bursty and elastic workloads, where you pay for idle capacity; globally distributed workloads, where you lose the cloud’s points of presence; serverless workloads with no owned-capacity equivalent; and high-egress workloads, where exit and transfer fees punish data-heavy moves.
Egress and lock-in decide the “more expensive” column. Providers make ingress free and egress expensive, a team pushing 10TB a month pays about $900 in egress fees on AWS alone, and proprietary managed services raise the exit cost when you leave. Judge it over a three-to-five-year TCO, not the first-year invoice. Most enterprises land on a split, which is why placement is the durable skill.
How should a workload placement framework weigh compliance, data volume, latency, scaling pattern and AI intensity?
Decide workload by workload across six dimensions: compliance, data volume, latency, scaling pattern, AI intensity and vendor risk, never by blanket cloud-first or on-prem-first. Run compliance first as a veto, because a hard rule overrides every cost argument. Data residency, data sovereignty and sovereign AI are three separate tests, and a workload can pass one and fail the next.
Then score the rest. The architecture is one three-tier decision: hyperscale cloud for training and burst, regional or on-premises for steady inference, and edge for low-latency decisions, connected by private connectivity and a control plane that can move placement as bottlenecks shift.
Before any move, capture a baseline: latency, cost-per-request, error rate and actual utilisation. Lift first, measure second, optimise third. Without that baseline, the post-move comparison is just opinion. Australian sovereignty and procurement get their own treatment in the sovereign cloud premiums piece, alongside where workloads are heading.
The placement decision has replaced model choice as the defining infrastructure question, and the 56% to 41% shift is the market voting with its workloads. Predictable, data-heavy, compliance-bound inference belongs in-perimeter; bursty, elastic, globally distributed and serverless inference stays in the public cloud, and edge absorbs the low-latency slice. The durable skill is knowing which workloads belong where, and proving it with baselines and a six-dimension framework. That is deliberate hybrid reallocation, and the wider repatriation landscape is where you will see it play out.
Frequently Asked Questions
Is private AI inference always cheaper than running the same workload on the public cloud?
No. Private AI inference only wins past a volume crossover, broadly around two million tokens a day with a six to nine month payback on the hardware. Below that threshold, sustained utilisation is too low to amortise the GPUs, and a hosted API stays the rational default. The decision is economic, not ideological.
What happens if we repatriate inference before it reaches the crossover point?
You buy GPUs, then pay for them twice: once in capital and again in the idle hours that per-token pricing would never have charged you. Below the crossover, a 24/7 service that never fills its accelerator pool loses to the hosted API, and the shortfall shows up as a poor effective cost per useful token.
How do we calculate our own crossover point?
Start with cost per million tokens, calculated as GPU dollars per hour divided by (tokens per second times 3600, divided by one million). Then run it against your real throughput, factoring in utilisation. Utilisation moves the answer most, so the crossover is never fixed: it sits wherever owned capacity amortises below your per-token API rate.
Does the fall from 56% to 41% mean the public cloud is losing the AI race?
No. Public cloud’s share of production inference fell while training largely stayed hyperscale, so the cloud did not lose AI, it lost the steady inference slice that data gravity and GPU economics pulled in-perimeter. Treat the number as the market pricing placements correctly, not as a verdict against rented compute.
Does hosting inference in-perimeter mean giving up frontier and closed-weight models?
Not necessarily. Open-weight models such as Llama and Mistral now cover roughly 85 to 90 per cent of enterprise use cases, which is why most private inference runs on them. For the remaining frontier or closed-weight calls, the sensible pattern keeps those in the public cloud and repatriates only the predictable, high-volume work.
Is private AI inference more secure than public cloud inference?
Not automatically, but it resolves the governance question by default. When inference runs beside the data, residency, processing and jurisdiction concerns clear without extra contractual work, and privacy pressure has risen from 53 to 77 per cent of enterprises. Security still depends on your own controls, so in-perimeter is a governance advantage, not a guarantee.
What does “cost per useful token” actually mean?
It is spend divided by the tokens your service actually serves productively, not the raw per-token rate a vendor advertises. Because GPU hours are the most expensive compute tier, an idle or underutilised accelerator inflates the real figure even when the sticker price per token falls. Useful tokens are the metric that exposes whether placement is working.
What baseline metrics should we capture before moving a workload?
Record latency, cost per request, error rate and actual GPU utilisation before the move, so the comparison afterwards is evidence rather than anecdote. Lift first, measure second, optimise third: without a pre-migration baseline you cannot prove the workload improved, and misjudged steady-state traffic is exactly how repatriation projects overspend.
Edge versus cloud versus on-premises for AI workloads, what does a three-tier architecture look like?
Treat it as one decision with three homes. Hyperscale cloud takes training and burst, regional or on-premises capacity takes steady throughput inference close to the data, and edge or metro edge absorbs latency-bound decisions. Private connectivity and a routing control plane let you move placement as bottlenecks shift, rather than locking workloads to one tier.
What is the biggest hidden cost in private AI inference that teams miss?
The MLOps salary line item, which often exceeds the hardware cost. A self-hosted inference stack needs people to serve, monitor and tune models, and that ongoing expense rarely appears in a simple crossover calculation. Add idle capacity and egress on data-heavy moves, and the true cost per useful token can sit well above the sticker comparison.
How do we know if a workload is steady-state enough to repatriate?
Look for a predictable baseline of requests around the clock, week after week, rather than spikes or seasonal events. High, stable utilisation is what lets owned hardware amortise into a low cost per token. Be honest that steady-state is an estimate, because misjudging it is the most common way repatriation projects overspend on hardware they cannot fill.
Does moving inference in-perimeter just swap one vendor lock-in for another?
Partly, yes. Leaving a hyperscaler reduces egress and proprietary-service exit costs, but hardware and stack choices create their own dependence. The safeguard is the same either way: keep workloads portable, avoid a single supplier for accelerators and serving software, and judge lock-in over a three to five year TCO horizon rather than the first invoice.
