Token prices keep falling while your AI bill keeps climbing. That contradiction has a name: the inference paradox. Per-token costs have dropped roughly 95% in two years, yet enterprise inference spend keeps heading up.
The metric most teams budget against is the one failing them. Cost per token is the supplier’s price list. Cost per task is your business’s unit economics: the cost of a completed unit of work. That is the number finance and the board can hold you to, and the entry point to production token economics.
This piece covers the measurement model and its consequences. The cluster articles carry the build-level detail.
What is “cost per task” and why is it a better measure than cost per token?
Cost per task is total inference spend divided by completed units of work, like a resolved support ticket, an accepted pull request or a finished agent session. Cost per token is the price per million input and output tokens. It reconciles your invoice, and that is all it does.
Two tasks can run at the same token price and land on different costs. One finishes in a single pass. The other retries three times, re-sends a long context window and calls a dozen tools. Same price per token, several times the spend. Token count tracks none of that; it records neither task success nor business output.
Cost per token is still the right unit for checking a provider bill and comparing list prices between models. What it cannot tell you is what a completed task cost. That gap between the price list and your own unit economics is why comparing models by token price alone can lead you to pick worse, more expensive models. Budget against completed work instead of token count.
Why is the cost per agentic workflow rising even as the cost per token falls?
Token prices are collapsing while the cost per agentic workflow is forecast to rise more than fivefold through 2028, according to Gartner. A cheaper unit price normally means a cheaper bill; here the opposite is happening. The reason has a name: Jevons paradox. When a resource gets cheaper, you use more of it, and jobs that were too expensive suddenly come online. A 95% drop in inference cost has raised most companies’ AI bills rather than lowered them.
Consumption is the variable the token price hides. Spending on large language models has doubled even as token prices fell more than 90%. Every extra token is billed at the lower rate, so your per-token metric looks healthy even as the workload grows.
Cost per token is demand-blind: it records the price of one unit but hides how many more units the cheaper price is being asked to perform. Watch cost per task instead of token price, and you see the real trend.
Why are reasoning agents so much more expensive per task than chatbots?
A chatbot answers in one call against a short prompt. A reasoning agent plans, retrieves, calls tools, writes state, validates its own output and retries the steps that fail. A single user request can fan out into 10 to 20 model calls and 5 to 30 times the tokens of a simple query, with re-sent context alone making up 62% of agent inference bills. Gartner puts the gap at up to 150x per task against a basic chatbot.
The compounding happens inside the workflow, invisible to the token price. The system prompt, tool schemas and state history get re-sent on every step. Reasoning models burn 10 to 50 times the tokens of a 2023 chat completion while they think, and every failed step sends the agent around again. None of that shows up in a per-token price.
That is why cost lives in the fan-out and has to be measured per task. The highest-return lever on re-sent context is prefix caching, covered in its own article.
Why doesn’t traditional IT budgeting work for token spend?
Cost lives in the fan-out, so a budget built for fixed seat licences cannot absorb it. Traditional IT budgeting assumes fixed, approvable spend: per-seat licences, procurement-approved contracts, predictable numbers. Token spend is none of those things. It is elastic, unbounded by headcount and set by the product decisions engineers make day to day, like which model to call. Finance only sees the result on the invoice.
That is why “why did my team burn through the whole year’s AI budget in three months?” keeps coming up. Consumption scales with adoption and intensity rather than the figure someone approved in November. Some organisations spent three times their annual AI budget by June 2026, and Uber burned through its entire 2026 AI coding budget in four months, with its COO unable to link the rising spend to anything shipped.
The fix is structural. Give AI compute its own budget line rather than hiding it inside cloud spend, and treat it like cloud migration spending in 2015: a protected transformation line with its own governance. Govern that line with cost per task, so work completed is the number that matters, not tokens burned.
How do you instrument cost per task across an engineering organisation?
Cost per task is only as credible as the metering underneath it. Without token metering and cost attribution, the number cannot be computed or defended. The method is straightforward: route every call through a gateway, tag spend by team, product surface, customer and workflow, then reconcile what you attributed against the provider invoice.
A gateway like LiteLLM makes this concrete. Its documentation covers per-key budgets, spend logs and budget fallbacks, the plumbing cost per task depends on. Tagging is the part your team has to get right. Cost attribution is only honest when tags are attached at request creation time; rebuilding tags from logs later under-reports long agentic traces.
Once attribution is in place, the dashboard follows directly. Lead with cost per task, then support it with cost per workflow, tokens per task, cache hit rate, routing distribution, spend against budget and cost per active user. Routing distribution is router economics, and the whole thing hangs together on metering and gateway tooling.
The response is structural: a dedicated AI compute budget line governed by cost per task, backed by metering and attribution that reconciles to the provider invoice. The per-token price war will keep making headlines, but it is beside the point. The only question that survives contact with finance is what each completed task cost, and whether that number is moving in the right direction. For the rest of the story, here is the full token economics picture.
Frequently Asked Questions
What metrics belong on an inference cost dashboard?
A useful inference cost dashboard leads with cost per task, then supports it with cost per workflow, tokens per task, cache hit rate, routing distribution, spend against budget and cost per active user. Together these show not just what you spent but how efficiently each completed task used the tokens. Cost per task anchors the view, while the supporting metrics explain why the number is moving.
Where are the official docs for LiteLLM’s spend tracking and budget controls?
LiteLLM publishes its spend tracking and budget control documentation on its official docs site at docs.litellm.ai, covering per-key budgets, spend logs and budget fallbacks. It is the reference point we use to illustrate a real metering layer, because a proxy that records spend per key and enforces limits is exactly the plumbing cost per task depends on. Check that documentation before wiring budget caps into production.
Why did my team burn through the whole year’s AI budget in three months?
Because token spend is elastic and scales with adoption, not with the approved figure. When more engineers ship more agentic features, consumption compounds faster than any fixed-period budget can absorb. The market pattern backs this: organisations spent 3x their annual AI budget by June 2026, and Uber exhausted a full-year budget by April 2026. The fix is a dedicated AI compute line governed by cost per task.
Is cost per token useless, then?
No. Cost per token is still the right unit for reconciling a provider invoice and comparing list prices between models. It simply cannot tell you what a completed task cost, because it ignores retries, context length, tool calls and reasoning depth. Use it to check the bill, not to budget the work.
How do I work out what an AI agent actually costs us per job it finishes?
Divide total inference spend by the number of successfully completed tasks in the same period, for example resolved tickets or accepted pull requests. Tag every model call by team and workflow so the numerator reflects the right work, then reconcile that figure against the provider invoice. The result is a cost per task finance can audit rather than a token price it cannot act on.
What is the difference between cost per task, cost per outcome and cost per active user?
Cost per task divides inference spend by completed units of work, such as a resolved ticket. Cost per outcome broadens that to a business result, such as a retained customer, which may span several tasks. Cost per active user spreads spend across the people or accounts actually using the product. They answer different questions, so a mature dashboard reports all three rather than picking one.
Will switching to a cheaper model automatically lower my cost per task?
Not necessarily. A cheaper per-token model that retries more often, re-sends longer context or needs more reasoning steps can cost more per completed task than a pricier model that finishes in one pass. Model choice interacts with the whole workflow, so compare cost per task, not headline token price, when you evaluate a swap.
How should a failed agent run count in cost per task?
Count its spend in the numerator and leave it out of the completed tasks in the denominator, because the point of the metric is spend per unit of value delivered. A run that burns forty tool calls and then fails still cost you money, and hiding that spend would flatter the number. Measuring it honestly is what surfaces runaway loops and retry-driven waste.
Do we need a gateway to measure cost per task, or can provider dashboards do it?
Provider dashboards show spend, but they cannot attribute it to your teams, workflows or customers, so they cannot produce a defensible cost per task. A gateway or instrumentation layer tags every call at the point of use, which is what makes the denominator meaningful. Most teams route traffic through something like LiteLLM and reconcile its spend logs against the provider invoice.
How often should we review cost per task?
Review it monthly, matching your finance cycle, with a weekly look at token consumption and cache hit rate so you catch drift before the month closes. Cost per task moves with adoption and workflow design rather than with token prices, so a monthly cadence keeps it aligned to the budget conversation. Anything slower and the invoice surprises finance all over again.
Does caching really change cost per task?
Yes, and often more than routing or model choice. Re-sent context accounts for roughly 62% of agent inference bills, so prefix and semantic caching attack the largest line in the cost stack directly. Every cache hit removes tokens from a task without changing what the user receives, which lowers cost per task without touching quality. That is why cache hit rate sits on the dashboard.
Is there a benchmark for a good cost per task?
There is no universal benchmark, because a resolved support ticket and an accepted pull request carry different value and complexity. The useful benchmark is your own trend line: is cost per task falling as you cache, route and simplify workflows? Compare teams and workflows against their own history, then set targets from that baseline rather than from an industry average.
