A million tokens now costs less than a dollar. Silicon Data’s LLM Token Expenditure Index hit 97 cents per million tokens on 31 August 2026, down more than half from its summer peak, and enterprise inference settled between $1.16 and $1.18 per million tokens in August.
The bills tell a different story. Token spend across the market rose roughly 497% from January 2025 to April 2026, and enterprise AI bills have tripled over the same stretch. Uber burnt through its full-year AI coding budget by April, and plenty of organisations had spent three times their yearly allocation by June. Consumption grew faster still.
That gap is where this guide lives. The savings have moved above the model layer into the control plane: routing, caching, gateways and metering. This page maps those five decisions.
In This Series
- The Inference Paradox and Why Falling Token Prices Raise AI Bills: why cheaper tokens keep producing bigger bills.
- Why Cost per Task Beats Cost per Token for AI Budgets: the metric that turns spend into something you can budget.
- How LLM Routing Cuts Inference Costs Without Hurting Quality: matching each request to the cheapest model that clears the bar.
- How Prefix and Semantic Caching Cut Inference Costs and Protect Quality: eliminating the calls you would otherwise pay for twice.
- Build vs Buy the Inference Cost Control Layer: deciding what to own and what to rent.
Why are token prices falling while enterprise AI bills keep rising?
Token prices fall because open-weight releases and provider price cuts keep compressing per-token rates. Bills climb because consumption grows faster than the discount. Cheaper tokens unlock agentic workloads, longer contexts, retries and always-on agents, each consuming more tokens than a chatbot. The price of a token and the cost of a task are diverging, and that gap is where the economics sit.
The mechanism is Jevons-style demand elasticity: cheaper units get used more. A 98% drop in the price of GPT-4-class capability since late 2022 made always-on agents worth building. Gartner forecasts inference cost per agentic workflow rising more than fivefold through 2028, even as token costs fall about 95% by 2030.
Read the inference paradox for the full mechanism.
Why does cost per task matter more than cost per token?
Cost per token measures your provider’s price list. Cost per task measures your unit economics: total inference spend divided by completed units of work. The same token price yields different task costs depending on retries, context length, tool calls and reasoning depth. Optimising the token price means tuning the wrong dial; budgeting, attribution and metering all live at the task layer.
Reasoning agents can run up to 150 times more expensive per task than basic chatbots, because each agentic step fans out into more calls and retries. Traditional IT budgeting fails here: token spend is elastic, driven by product decisions made in code rather than seat count or procurement. A gateway that meters each request, attributes spend by task and workflow, and reports it against budget makes that overshoot visible.
Read cost per task to make the metric governable.
How does LLM routing cut inference costs without hurting quality?
Routing sits in the gateway between your application and the model providers, matching each request to the cheapest model that still clears your quality bar instead of sending everything to the frontier. Savings come from workload composition: routine classification, extraction and summarisation rarely need flagship pricing, so they run on cheaper models at the same quality. The catch is silent quality regression, degradation that never shows up in error logs, which makes routing a measurement problem as much as a configuration one.
Routing spans rule-based, embedding-similarity and trained-classifier strategies. Two risks follow: the small-model ceiling, where a cheaper model stops being good enough on open-ended work, and silent quality regression, which survives most dashboards because the average holds while the tail degrades. RouteLLM held roughly 95% of GPT-4 quality while sending a minority of traffic to the strong model. Reserve frontier pricing for irreversible, high-blast-radius or customer-visible work; send routine transforms to cheaper models.
Read model routing in production for the decision frame and evaluation loop.
Which is safer — semantic caching or prefix caching — and what does each save?
Prefix caching is deterministic and correctness-preserving. It reuses the computed attention state of a shared prompt prefix, so you pay the redundant prefill once, not on every request. Semantic caching is probabilistic: it returns a stored answer for a similar, not identical, prompt, and can be confidently wrong. Prefix caching is the zero-risk foundation; semantic caching is savings-rich but needs a similarity threshold and an evaluation loop.
Much of a typical prompt is boilerplate: system prompts, tool schemas and few-shot examples. Prefix caching reuses the computed attention state for that shared prefix — vLLM‘s KV cache is the canonical mechanism — so you skip the recomputation without touching correctness. Semantic caching is a precision-versus-recall dial: a low threshold catches more but risks wrong answers, a high one rarely fires. Set that dial per endpoint and per task: a threshold safe for a support bot is not safe for regulated output. Routing lowers the cost per call, caching eliminates calls, and the two together beat either alone.
Read semantic caching for the correctness trade-off.
Should you build or buy the inference cost control layer?
The decision turns on traffic shape: volume, variance, compliance constraints and model churn. Both paths own the same object, a gateway that runs routing, caching, metering and failover. Buying hands you the plumbing but still requires integration; building gives you control and attribution fidelity at the cost of engineering time and a permanently maintained parity proof. Token deflation keeps moving the economics, so ownership should be driven by control and elasticity, not per-token arbitrage.
The two archetypes are LiteLLM, a self-hosted proxy, and OpenRouter, a managed router that Stripe is acquiring. The honest comparison is fully loaded cost, hardware amortisation, utilisation, operations and engineering time, against falling API pricing. An owned cluster only pays off above roughly 50 to 83% sustained GPU utilisation, a threshold that token deflation keeps pushing higher. Most teams conflate two questions: hosting an open-weight model yourself versus owning the cost-control layer while still calling frontier APIs.
Read build versus buy the cost layer for the decision heuristic.
Resource Hub: Token Economics Deep Dives
Understand the shift
- The Inference Paradox and Why Falling Token Prices Raise AI Bills: the demand mechanism behind rising bills. Read first.
- Why Cost per Task Beats Cost per Token for AI Budgets: measuring spend by completed work rather than raw tokens. Read next for attribution.
Work the control plane
- How LLM Routing Cuts Inference Costs Without Hurting Quality: the single biggest savings lever and its two risks.
- How Prefix and Semantic Caching Cut Inference Costs and Protect Quality: where savings are safe and where correctness is on the line.
Decide what to own
- Build vs Buy the Inference Cost Control Layer: the ownership decision for the router, gateway and cache.
Suggested order: the paradox, cost per task, routing and caching, then ownership.
Frequently Asked Questions
What exactly is the “control plane” for inference costs?
The control plane is everything between your application and the model: the router, cache, gateway, metering and failover logic. It decides which model answers a request, what can be reused, and what each task actually costs. The model layer sets the price per token; the control plane decides how many tokens you spend. That is why the durable savings sit above the models.
Is inference getting more expensive, or are we just using more of it?
Both, but the usage effect dominates. Per-token prices keep falling, yet spend rose roughly 497% from January 2025 to April 2026 because agentic workloads consume far more tokens than chatbots. The unit cost improved while the total bill climbed. Treating cheaper tokens as a finished cost strategy mistakes a falling input price for a falling outcome.
Do I need a gateway to cut inference costs, or can I just optimise prompts?
Prompt discipline helps, but it caps out quickly; a gateway gives you the levers that scale. Routing, prefix caching, metering and failover all live in a gateway, so it is where most control-plane savings are actually executed. Start with prompt hygiene, then add a gateway once traffic volume and model variety make manual tuning impractical.
What happens if I route every request to the cheapest model?
You hit the small-model ceiling: a cheaper model stops being good enough on open-ended work, and quality degrades silently. Average satisfaction can hold while the tail of hard requests gets worse answers that never appear in error logs. Routing should match each task to the cheapest model that still clears your quality bar, not the absolute cheapest available.
Is it true that cheaper tokens should make my AI budget smaller?
No. Cheaper tokens usually make the budget bigger, because lower unit prices unlock workloads that were not worth building before. This is Jevons-style demand elasticity, the same effect seen when a 98% drop in GPT-4-class pricing made always-on agents viable. Plan for demand to expand and govern consumption, rather than assuming a price cut converts into savings.
How much can routing and caching realistically save?
Savings come from workload composition, so the range is wide, but both levers are material. RouteLLM held roughly 95% of GPT-4 quality while sending only a minority of traffic to the strong model, and prefix caching removes redundant prefill on every shared prompt. Routing lowers the cost per call; caching removes calls outright. Measured together on your own traffic, they beat either alone.
How do I catch silent quality regression in a routed system?
You catch it by evaluating continuously, because it will not show up in error logs. Run a held-out evaluation set through the small model, sample live traffic for human review, and track quality by task type rather than as a single average. Average satisfaction can look healthy while a specific workload quietly degrades, so segment your quality metrics before you trust them.
Can I use cost per task if my product has no clear task boundaries?
Yes, but you have to define the task yourself. Pick a unit your business already understands, such as a resolved ticket, a completed document or a successful extraction, then divide inference spend by those units. The definition matters less than applying it consistently. Cost per token measures your provider’s price list; a defined task measures your real unit economics.
Is self-hosting open-weight models always cheaper than calling APIs?
No, and it is often mispriced. The honest comparison is fully loaded cost: hardware amortisation, utilisation, operations and engineering time, set against falling API pricing. If your GPUs sit idle, self-hosting loses badly. Open-weight models set a price ceiling that keeps falling, so owning inference hardware depends on sustained high utilisation and genuine compliance constraints.
Does semantic caching risk returning wrong answers to customers?
It can, because semantic caching is probabilistic: it returns a stored answer for a similar, not identical, prompt. A low similarity threshold catches more traffic but raises the risk of a confidently wrong response. Treat it as a precision-versus-recall dial. Set the threshold deliberately and run an evaluation loop, while keeping deterministic prefix caching as your correctness-preserving foundation.
What is the difference between a router and a gateway?
A router chooses which model handles a request; a gateway is the broader layer that houses the router alongside caching, metering, failover and authentication. You can run a router on its own, but in production the gateway is what makes the savings measurable and governable. Think of routing as one decision inside the control plane, not the whole of it.
How do I stop an agent from burning through the budget?
Put hard limits in the control plane rather than trusting prompt-level restraint. Meter spend per task, cap retries and tool-call loops, set per-team budget ceilings with alerts, and fail over to cheaper models when a threshold is hit. Because token spend is elastic and driven by product decisions in code, budget guardrails belong in the gateway where they actually bind.
