Your inference bill keeps climbing even though every model on your rate card gets cheaper. It is the inference paradox, and it means every deployment ends up needing a cost control layer: routing, caching, metering, failover and provider abstraction. The instinct is to treat that layer as a price comparison: build it, running LiteLLM yourself, or buy it, paying OpenRouter. Both deliver the same cost control layer; the difference is who owns it, and at what scale the maths flips. By the end you will be able to run a fully loaded break-even and score a gateway against six criteria, grounded in token economics.
How do you assess the true cost of on-premise inference versus API pricing?
Assess fully loaded cost: hardware amortisation, utilisation, operations labour, resilience and the opportunity cost of engineering time. Compare that allocation-based figure against API pricing around $1.16 to $1.18 per million tokens (August 2026). Idle GPU capacity is the multiplier; low utilisation inflates your true cost per token faster than price changes.
Start by splitting cost in two. Usage-based cost is active compute only; allocation-based cost is the full hosting bill, idle GPUs and shared infrastructure included. OpenCost states it plainly: build-vs-buy decisions must use the allocation figure, because it varies with utilisation. A model with $1.00 of active compute per million tokens looks cheaper than a $2.00 API. At 25% utilisation you are paying for four times the capacity you actually use, so that $1.00 becomes $4.00 of allocated cost per million tokens.
The fully loaded figure has more lines than people expect: hardware amortisation, power, cooling, operations labour, resilience, egress, and engineering time. One vendor’s three-year model for an eight-GPU H100 server runs $712,000 to $948,000, with staff as the largest line (Spheron). You probably undercount it, so check your own staffing line.
Then utilisation decides. An idle GPU costs the same as a saturated one, so cost per token is the hourly rate divided by tokens produced (Inworld). Most production teams run at 40 to 65% utilisation (Spheron), which is why hybrid wins: reserve baseline capacity for median load, and burst on-demand for peaks instead of buying hardware to cover the spike (FinOps). That gives you a real cost figure. The question that follows is who owns the control plane that generates it.
How do you decide the build-vs-buy trade-off for the AI cost layer?
Decide on traffic shape and ownership: map volume, variance, compliance constraints and model churn, then list what you must own: routing policy, caching, metering, failover and provider abstraction. Buying a managed gateway still means integrating the layer; building self-hosts it. The calculus flips when control, attribution or compliance costs exceed the managed margin.
An LLM gateway is a control plane owning six concerns: provider abstraction, routing, retries and fallback, caching, observability and rate limits (jobsbyculture). You get it either way, so the question is who operates it.
The two archetypes make it concrete. LiteLLM is the self-hosted route: MIT-licensed, 140-plus providers, virtual keys and hard budgets you can verify in its spend-tracking docs. OpenRouter is the managed route: 400-plus models across 80-plus providers, dynamic routing and a consolidated bill. It removes the operational burden, adds a managed margin, and brings a live lock-in consideration now that Stripe has agreed to acquire it.
Score either against six criteria: provider coverage, spend-attribution fidelity, budget enforcement, cache ownership, failover behaviour and lock-in exposure. Spend attribution and cache ownership matter most, so verify that budgets are enforced in code, because reporting alone does not stop spend. Managed model routers can cut average inference cost by 60 to 80% (FinOps), and prefix and semantic caching decide what stays yours to tune. As token prices fall, control, attribution and elasticity become the reasons to build. The same ownership lens applies one level down, when you weigh hosting the models themselves.
Open-weight models vs frontier models: when does self-hosting actually pay off?
Self-hosting pays off only above the open-weight price ceiling: the sustained-utilisation rate at which your fully loaded cost per token drops below the API price. Below it, pay per token. As token deflation pulls frontier prices down, that ceiling rises, demanding higher steady-state utilisation. Most organisations should own the cost control layer and keep calling frontier APIs.
Untangle two questions that get collapsed. Hosting an open-weight model yourself is a GPU and capacity-planning commitment; owning the cost control layer while still calling frontier APIs is smaller (DataLLM Lab). You probably face the second.
Hosting models pays off only above that ceiling. The viable families are DeepSeek, GLM, Qwen, Llama and Kimi, whose release cadence is itself a price pressure (CSIS). The ceiling keeps moving: Gartner sees a trillion-parameter model costing providers over 90% less by 2030, and every cut raises the utilisation you need.
The better frame is the efficiency frontier, the models with the best price at a given level of intelligence, as distinct from the intelligence frontier (peak capability). Databricks found the efficiency frontier advances faster than peak capability, so routing to the cheapest adequate model beats owning hardware. The ceiling still moves: vLLM‘s continuous batching and PagedAttention squeeze 20 to 40% more throughput from the same GPUs (SitePoint).
The short version
Build versus buy is an ownership question: who operates the router, the gateway and the cache. The durable asset is owning the cost control layer. Decide on volume, variance, compliance and model churn, compare fully loaded allocation-based cost against a falling API price, and re-run the maths as prices drop. That is the whole of cutting inference costs in production.
Frequently Asked Questions
Is self-hosting an open-weight model always cheaper than paying per token?
No. Self-hosting only wins above the open-weight price ceiling, the utilisation threshold where owned capacity beats a falling API price. Below that threshold, idle GPUs make your cost per token higher than the API rate, because an idle GPU costs the same as a saturated one. Many teams that self-host to save money are really buying control and data residency, not savings.
How do I work out the utilisation break-even for self-hosting?
Divide your fully loaded monthly cost (hardware amortisation, operations labour, resilience and egress) by the tokens served at a given utilisation, then compare that figure per million tokens with the API price. If your cost per million tokens sits above the API rate, self-hosting is losing money at that utilisation. Run it at several levels to find the threshold.
What GPU utilisation do you actually need for on-premise inference to pay off?
There is no universal number, but sustained, predictable load is what matters. Bursty traffic rarely clears the ceiling, because idle capacity inflates cost per token. Capacity planning for peak while billing for average is the classic mistake. If demand swings sharply, a hybrid baseline plus cloud burst beats owning enough GPUs to cover the peaks.
What should you look for when choosing an LLM gateway for cost control?
Six things: provider coverage, spend-attribution fidelity, budget enforcement, cache ownership, failover behaviour and lock-in exposure. Spend-attribution fidelity and cache ownership matter most for cost control, because they decide whether you can see and act on unit economics. Verify that budgets are enforced in code, not merely reported in a dashboard.
LiteLLM vs OpenRouter: which should a self-hosting team pick?
It depends where you want the control plane. LiteLLM is a self-hosted proxy (MIT licensed, 140+ providers, virtual keys, per-key spend tracking and hard budgets), so you own routing policy and cache. OpenRouter is managed routing across 400+ models and 80+ providers, which cuts operational burden but adds a managed margin and a live lock-in consideration after the Stripe acquisition.
Where do I find LiteLLM’s official documentation for spend tracking and budget controls?
Those controls live in LiteLLM’s official documentation, under the proxy and spend-tracking sections. Treat the docs as an evaluation checklist rather than a setup guide: confirm how virtual keys scope spend, whether hard budgets block requests in real time, and how attribution rolls up per key, team or model. Verify the behaviour before you commit.
If I buy a managed gateway, am I locked in?
Potentially, which is why lock-in exposure is one of the six selection criteria. Managed routing concentrates your routing policy, spend history and provider abstraction in a vendor’s control plane, and consolidation such as the Stripe/OpenRouter acquisition can change terms. Mitigate by keeping routing policy portable, abstracting the gateway behind your own interface, and confirming exit paths before signing.
Can I run a self-hosted gateway and a managed router at the same time?
Yes, and plenty of teams do. A common pattern keeps LiteLLM in front for policy, caching and metering while calling OpenRouter or frontier APIs behind it for elasticity and model coverage. Run the controls you need to own yourself and buy the breadth you cannot build. That preserves attribution and exit options while capturing managed coverage where it pays.
Does caching change the build-vs-buy maths?
It does, because caching raises effective utilisation and therefore lifts the open-weight price ceiling. Prefix and semantic caching serve repeated and near-identical prompts without fresh compute, so the same owned capacity covers more tokens. That makes self-hosting easier to justify, and it helps managed gateways too, which is why cache ownership belongs on your gateway checklist.
How much engineering effort does self-hosting the cost layer add?
More than the install and less than running your own models. The ongoing costs are upgrades, provider integrations, cache operations, spend-attribution accuracy and on-call for failover. Treat that engineering time as a line item in the fully loaded cost, because it is the opportunity cost the managed margin is really buying. Small teams often underestimate it.
How often should I re-run the build-vs-buy break-even?
Every time token prices move materially or your traffic shape changes. Token deflation pulls API prices down continuously, which raises the utilisation you need to justify self-hosting. A break-even that paid off last quarter can quietly stop paying off. Re-run it on a schedule and after any major change in volume, variance, compliance scope or model churn.
