Someone arrives in your office with a slide claiming an AI cost lever will cut inference spend by 30%. You ask what baseline that percentage comes off, and the room goes quiet.
The levers are technical, but the proof is financial. Defending your numbers requires four things: a baseline dollar figure, an explicit measurement window, an effective rate rather than a list price, and a verified quality score. You are controlling a line finance already owns, and your board expects verifiable evidence.
Prompt caching, model routing or context compaction: which lever pays back first for a small team?
Prompt caching pays back first, model routing comes second, and context compaction ranks third. This order reflects payback speed, adoption cost, and quality risk rather than theoretical maximum savings. Caching requires no architectural changes, attacks repeated prompt context directly, and lowers invoices within days.
Its binding constraint is cache hit rate. Workloads need hit rates above 30% to save money. On Anthropic, prompt caching discounts cached input tokens by up to 90%, breaking even after 2.3 reuses of the prefix. Provider semantics differ: Anthropic requires explicit cache breakpoints, while OpenAI caches automatically. Across multi-provider stacks, an unread cache write costs you money.
Model routing sits second, exploiting tier spreads where price gaps between entry-level and frontier models reach 100x. RouteLLM demonstrates an 85% cost saving on MT Bench at 95% of GPT-4 quality by routing 14% of prompts to frontier models. For a small team, the risk is regression: without an evaluation harness of 50 to 500 test cases, routing saves money by degrading output.
Context compaction ranks third. It promises the largest token reduction, but pruning history or tool outputs changes agent behaviour and demands a regression suite before results can be trusted. Middleware like LangChain makes compaction look like simple configuration, but automated summarisation is lossy. Cache first, making re-sent tokens cheap enough to measure compaction’s marginal yield.
Retrieval-augmented generation (RAG) and long context share that trade-off. RAG pays infrastructure costs to shrink prompts; long-context models charge per-token fees to skip retrieval infrastructure. Prompt caching shifts that crossover by cheapening long-context re-reads. Every control attacks one variable: the model multiplier, the re-sent context, or the tier spread. Knowing the mechanism each lever is aimed at makes the ranking logical.
What should you ask a vendor that claims a routing layer will cut your AI spend by 30%?
When hunting for that tier spread, vendors pitch model routing with an immediate savings claim. A 30% cut is technically plausible. Ramp reported an overnight 30% spend reduction after routing internal workloads through Ramp Router, illustrating how unmonitored workloads run up bills during a week where token spend reached $1.5 million.
Yet as critiques of unstated baselines point out, a percentage without an absolute baseline dollar figure cannot be evaluated.
When a routing vendor pitches a 30% reduction, ask these four questions:
- Is that 30% measured against your pre-routing effective rate or public list prices? Effective rates track actual net spend and sit below list prices due to volume discounts and caching.
- What quality gate was used to verify accuracy? Apply the rule from JFrog’s agent optimization research: never accept a token savings figure without a verified correctness rate beside it.
- Were retry rates, cache hit rates, and output token ratios held constant during the trial period?
- How does the savings percentage hold when prompt difficulty becomes uniform rather than diverse?
Savings depend on your task variety and tier spread, not vendor averages. Siting an in-path proxy inside your execution path also introduces latency, lock-in, and availability risks. A checked cost figure is still only half a claim. The harder question is what that spend bought.
What is return on AI (RoAI), and how do you tell return from activity?
Return on AI (RoAI) is the economic value of business outcomes delivered divided by the fully loaded cost of delivering them. BCG’s analysis of token economics notes that raw inference represents only about 20% of total operating cost. The remaining 80% covers human evaluation, orchestration pipelines, data engineering, and operational governance. Those four costs sit behind the AI budget line the board now reviews, which is why invoice-only savings claims look thin.
Prompt counts, tokens processed, and completed loops measure operational volume, and volume is not return. An automated workflow running 5,000 steps nobody requested is overhead. An agent resolving 300 support tickets that previously required 450 staff hours delivers quantifiable return.
Measuring RoAI requires three prerequisites: an unambiguous unit of business outcome, a repeatable way to attribute the outcome to one workflow, and a documented baseline. Skipping outcome instrumentation is why cost claims fail during budget reviews.
Run workloads through a Prove, Stop, or Minimise review discipline. Workflows must demonstrate measurable return, or they get stopped. Those proving return keep running while token costs get minimised. A per-run cost ceiling keeps an unproven workflow’s exposure bounded, but only if something in your execution path can enforce it.
AI gateway or cost dashboard: which one actually controls spend, and which just reports it?
A cost dashboard reports spend after it happens, while an AI gateway enforces spend policies in real time. Dashboards read logs after the fact. A gateway sits inside the network path, with authority to allow, downgrade, queue, or reject an inbound API call.
This architectural difference matters because surveys from the Cloud Security Alliance show 82% of organisations have discovered unsanctioned AI agents operating inside their networks.
Gateways like Kong enforce five policy capabilities:
- Enforce token budgets per team, application, and billing code.
- Cap any single run with a cost ceiling to stop runaway loops before consuming a monthly allocation.
- Restrict unapproved frontier endpoints through dynamic model allow-lists.
- Apply rate limits that protect upstream provider quotas.
- Log requests stamped with team identity, which makes attribution the precondition for chargeback possible.
A rate limit caps request velocity to protect availability. A token budget caps cumulative consumption over a monthly billing period. Conflating the two leaves your team vulnerable to budget exhaustion.
Enforcement without attribution creates friction, while attribution without enforcement is just observation. In-path controls let your team move from showback reporting to internal chargeback, shifting financial accountability to product teams.
A standard architecture uses an in-path proxy to normalise requests across providers and stamp team identity, while reporting layers like Datadog Cloud Cost Management for AI reconcile that telemetry against invoices. Once that control machinery runs, the question is what it should report.
What should a board-ready AI cost dashboard actually show on one page?
A board-ready reporting dashboard must fit on a single page and answer three questions in this order: variance against plan, attribution by business unit, and outcome return — the three questions the board-level AI line is reviewed against.
The single-page artefact presents seven metrics:
- Total gross AI spend against approved budget and prior-quarter run rate.
- Spend split by business unit and product line, separating internal operating expenses from customer-facing cost of goods sold, with the unattributed balance disclosed.
- Per-employee-per-month expenditure benchmarked against industry spending percentiles.
- Blended effective token rates plotted against provider list price trends.
- Top five production workflows ranked by cost alongside their measured cost per outcome.
- Active workflow counts across Prove, Stop, and Minimise categories.
- New shadow agent discoveries during the reporting period.
Plotting blended effective rates alongside list prices explains how AI expenditure can climb while unit costs decline. This reporting page functions as the artefact that closes the ownership negotiation between engineering and finance.
Engineering provides runtime attribution; finance manages capital. Rank your levers by payback, test every savings percentage for its denominator, and put the unattributed number on the page. A board does not reject AI savings because they are small; it rejects them because they cannot be checked. Review the cluster in one frame to establish an evidence standard your board can verify.
Frequently Asked Questions
How do we build a baseline if nobody has tracked AI spend by team before?
Start with the invoice and work backwards. Pull twelve months of provider billing, split it by key or workspace wherever the provider allows it, and mark the remainder unallocated rather than guessing an allocation. Then fix the forward view: request-level records carrying team and workflow identity, reconciled to the invoice every month. The baseline you can defend is the one whose gaps are visible, not the one that adds up neatly.
How long does a saving have to hold before we can quote it in a business case?
Long enough to survive a change in traffic, which usually means two full billing cycles against a fixed pre-routing window. One month proves the invoice moved; it does not prove the lever moved it. If retries, cached reads or output-token mix shifted in that same window, you are quoting a bundle rather than a saving. State the window beside the number or the number is an anecdote.
If our provider cuts list prices, do we book the saving?
Only if your effective rate fell by the same amount, and it often did not. Effective rate is what you actually pay after caching, retries, output-token mix and any residency premium, measured against the invoice rather than the price list. A steep list price cut against a doubling in token volume still leaves the bill higher than where it started. Show both lines and let the arithmetic speak.
How do we stop one runaway agent from burning a month’s budget overnight?
Put a per-run cost ceiling on it. A token budget per team will not catch a single loop that retries a tool call four hundred times before anyone reads the alert; a ceiling on any one run will. Pair the ceiling with request-level records so the spike is attributable the next morning. Treat the limit as an engineering constraint, not a nag.
What happens when a team exhausts its token budget mid-month?
Whatever the policy layer was configured to do, which is why that configuration is an executive decision rather than an infrastructure one. The defensible defaults run in order: degrade to a cheaper tier, queue the non-urgent work, then refuse only what has an owner who agreed to be refused. A hard stop with no fallback turns a budget control into an outage.
Should we build our own AI gateway or buy one?
Buy the enforcement path unless the gateway is already a product you operate. The primitives are unglamorous (budgets, allow-lists, ceilings, request records) and the failure modes are availability ones, so you want someone else’s on-call rota holding them. Build the policy and attribution logic instead, because that is where your organisation’s shape lives and no vendor can guess it.
Is retrieval cheaper than long context, and does prompt caching change the answer?
Depends on how much of the corpus each query needs and how often you ask it. Retrieval pays an indexing and infrastructure cost once to shrink every prompt; long context pays tokens on every call to avoid that infrastructure. Prompt caching moves the crossover by making repeated context cheap to resend, so the long-context side is stronger than most build-versus-buy notes admit.
What is the difference between a token budget and a rate limit?
A token budget caps cumulative consumption; a rate limit caps the pace of requests. One stops a team spending its quarter by February, the other stops a burst taking the service down. Procurement teams treat them as synonyms, then discover their gateway only holds one of the two. A control layer needs both, because neither prevents the failure the other one does.
What does an AI gateway reference architecture actually look like?
Four parts, and none of them is the product you buy. An in-path proxy normalising requests across providers, a policy layer holding budgets and model allow-lists, an emitter writing request-level records with team and workflow identity, and a reconciliation job against the provider invoice. Kong and Datadog CCM (AI Costs) sit at different layers of that picture, enforcement and reporting respectively.
What if a chunk of the bill cannot be attributed to anyone?
Show it, size it, and shrink it deliberately. An unattributed remainder of 20% is a finding, not a rounding error, and burying it inside a cost centre is how a board stops trusting the whole page. Give the remainder an owner, a target for the next quarter, and its own line in the report. A remainder that falls over three quarters is a credible control story.
Is AI spend opex or COGS, and does that change what a saving is worth?
Both, and the split decides what the saving means. Inference that serves customers sits in COGS, where a reduction improves gross margin on every unit sold. Internal tooling sits in opex, where a reduction is a straight cost line. Two identical percentage cuts can matter very differently, so attribute cost by what the workflow produces, not by which vendor issued the invoice.
