Your AI bill doubled overnight and nobody changed any code. Your team added a few thousand tokens and expected the extra cost to be proportional. It was not.
A doubled bill usually traces back to a request crossing a long-context pricing threshold, where the higher rate lands on the whole request, overflow included.
You can check the arithmetic yourself. This article walks through 270,000 versus 275,000 tokens on GPT-6 Astra, where the bill compounds, why caching and compaction pull in opposite directions, and how to estimate whether your traffic crosses the line. The number that matters is cost per successful task, not cost per request. That is the wider point behind per-token economics across the cluster.
Why does crossing a long-context pricing threshold reprice the whole request rather than just the overflow?
Crossing the threshold reprices the whole request: the long-context rate band applies to every billable line, input, cached reads, and output. On OpenAI’s GPT-6 Astra, a prompt over 272,000 input tokens moves input from $10 to $20 per million tokens and output from $50 to $75, on the entire request.
The arithmetic is checkable. A 270,000-token prompt with 10,000 tokens of output costs about $3.20; at 275,000 it costs about $6.25, which is $3.05 more for 5,000 extra tokens. The 273,000th token alone adds roughly $3, nearly the cost of the entire 270,000-token prompt below the line. A 275K prompt costs nearly as much as a 500K one. That is the 272K footgun in numbers.
When pricing is tiered by request size, the marginal cost of the next token jumps discontinuously. The last thousand tokens are the most expensive in the prompt, and often the least useful: context rot is documented across frontier models, and the tokens that push a prompt over the line are usually the oldest chunks.
The numbers above are second-hand, quoted from commentary posts rather than a pricing page fetched today, and pricing pages are the only authority. Thresholds and multipliers change between releases, and usage data carries no field that marks a surcharge, so your own instrumentation is the only reliable detector. The cliff sits 26% into the 1,050,000-token window, per commentary on OpenAI’s Astra pricing.
Where does an agent’s token bill actually compound: the chat model, or the layer underneath it?
The bill compounds in the layer underneath the chat model, in the embeddings, reranks, and sub-agent calls that never appear as the headline line item. One user turn can trigger one chat completion plus an embedding batch, a rerank, and three sub-agent calls, each carrying its own full context.
The chat model is the line item you see; beneath it the bill multiplies across compaction passes, evaluation calls, and guardrails, each itself a model call. The headline model price understates per-task cost by a factor most teams have never measured, and the infrastructure around the model call is 40 to 60% of feature spend, per the FinOps working group. LangChain and LangSmith record token composition per run, which is the only way to see it.
Measure cost per successful task: a task that fails and retries burns tokens without completing anything. Multi-agent systems consume roughly four times a chat interaction’s tokens, and a research system about fifteen times, measured in production.
For embeddings and reranking, the cheapest, most repetitive workloads, self-hosted open-source inference has a crossover: an embedding encode runs about $1.30 per billion tokens self-hosted against $20 on the OpenAI API, once steady volume amortises the fixed cost of running it. Below that volume managed APIs win on engineering time. The crossover does not hold for generation. This retrieval and reranking layer is where the bill compounds beneath the chat model.
What is prompt caching, and why does compaction throw the cache away?
Prompt caching bills a stable prefix at a discounted rate because the provider reuses its precomputed state; change the prefix and the discount disappears. Compaction rewrites the history that forms the prefix, so the next call pays full price on a context that has just been compressed. Caching and compaction pull in opposite directions.
Caching depends on tokens staying unchanged, and compaction saves tokens by changing them, which is the mechanism behind bills that jump after a cost optimisation. Cache writes are a separate line item at 1.25 times uncached input (how caching is priced), and the Astra rate card lists that write premium, so a workload with no prefix reuse pays a premium rather than a discount.
This is compaction and the cache it discards. The way to keep both is a context budget, a per-request ceiling that stops the request from crossing in the first place.
How do we estimate whether our agent traffic will cross a long-context pricing threshold?
Measure the distribution of prompt sizes: the threshold is crossed by a small tail of requests, and that tail sets the bill. What matters is the count of requests sitting within 10% of the threshold, the population that drifts across it.
An average prompt size says nothing about the bill, because the threshold is a tail event. The small share of requests that cross the line is what sets the spend, and that band drifts across the line from one day to the next.
Two error bars sit on top of that. Tokenizer inflation runs up to 20% on character-based estimates, so a budget built on character counts can sit under the line in code and over it on the invoice (tokenizer differences across providers). Conversation length and retrieval depth also grow organically.
The arithmetic is worth running. 10,000 requests a day with 5% crossing at roughly $3 of overspend each is about $1,500 a day, over $500,000 a year, attributable to prompts that landed in the wrong bucket.
There are three levers, and the choice between them is deliberate rather than automatic. The first is a context budget below the threshold, a per-request ceiling. The second is restructuring so the expensive work happens in a smaller-context call. The third is accepting the tier and budgeting for it.
If the bill is USD-denominated while the margin is set in another currency, a ten percent move in that pair is a ten percent move on the whole invoice, tier multiplier included.
The worst outcome is paying for a tier whose capacity the model cannot use well, the effective window behind the pricing tier, and that is the closing thread of the wider argument on context engineering.
The bill jumps at thresholds and multiplies in the layer beneath the chat model, and the same structure explains why caching and compaction pull in opposite directions. Once the cliff and that hidden layer are visible, a doubled bill stops looking like a mystery and starts looking like a predictable property of tiered pricing. Treat it as a structure to manage: measure cost per successful task and the tail of the request-size distribution, add the error bars, and choose among the levers deliberately.
Frequently Asked Questions
My AI bill doubled overnight but my usage barely moved. What should I check first?
Start with the largest single requests rather than the total. Pull input token counts per request for the past fortnight and look for a cluster sitting just under 272,000, because a handful of those crossing the line reprices each request at double the input rate. If the crossing requests are fewer than 5 per cent of traffic, your bill moved because of structure, not volume.
Is the threshold measured per request or across the whole conversation?
Per API call, not per conversation. Each request is priced on the tokens it carries, so a long chat that has grown past 272,000 tokens puts every subsequent turn into the high band, even when each new turn adds only a few hundred tokens. The bill climbs while the visible increment stays small, which is why conversation length matters more than message size.
Do the higher rates apply to output tokens and cached reads too?
Yes. Whole-request repricing covers every billable line in the request: uncached input, cache writes, cached reads and output. On GPT-6 Astra the multipliers differ, with input and cache at 2× and output at 1.5×, but they all switch at the same boundary. Nothing in the request keeps the old rate once the input count crosses.
Is the 272,000 token breakpoint the same for every model and provider?
No. Each vendor sets its own breakpoint and its own multipliers, and both change between model releases, so a figure you memorised last year may already be wrong. Check the pricing page for the exact model and version you deploy, on the day you check. Treat any threshold you cannot cite back to a vendor page as a guess.
Would splitting a long prompt into two smaller requests be cheaper?
Often yes, if the task genuinely decomposes. Two 140K calls at $10 per million cost about $2.80 in input between them, against roughly $5.50 for a single 275K call. The trade is real though: you double the output calls, lose cross-document reasoning, and add a round trip of latency, so split only where the work is separable.
Does prompt caching protect me from the threshold once the prefix is cached?
No. Caching changes the rate applied to tokens, not the tier the request sits in. A 280K prompt with a fully cached prefix is still priced in the long-context band, and the cached read is charged at the higher cache rate. Caching makes a large request cheaper than it would otherwise be, not cheap relative to a smaller one.
Is compaction still worth doing if it keeps breaking the cache?
Usually yes, but sequence it deliberately. Compact before the turn that would cross the threshold, then rebuild a stable prefix so the cache re-warms once rather than on every turn. Judge the result on cost per successful task across a whole run, because a compaction that saves a tier and loses accuracy has saved nothing.
Why doesn’t my invoice show that a higher tier was applied?
Because invoices report aggregates. Usage data carries no documented field marking a surcharge, and the tier is applied inside the pricing engine at request time, so the invoice shows token counts and a total, not which requests changed rate. Your own instrumentation, one record per request with its input count and resolved rate, is the only reliable detector.
What is context rot, and why does it make the priciest tokens the least useful?
Context rot is the decline in a model’s use of information as the context fills. The tokens that push a prompt past 272,000 are usually the oldest retrieved material or the least relevant chunks, so you pay double for them and get weaker attention over everything else. The most expensive part of the prompt is often the part doing least work.
How much tokenizer inflation should I allow for when estimating?
Allow 10 to 20 per cent on any character-based estimate. The common four-characters-per-token rule undercounts code, JSON, tables and non-English text, where the ratio can fall well below three. Inflate your measured prompt sizes before you plot the tail, otherwise a band you think sits at 260,000 may already be crossing 272,000 in production.
We are based in Australia. Does the exchange rate change what the threshold costs us?
Yes, at the finance layer rather than in the rate card. The bill is denominated in USD while your margin is set in AUD, so a ten per cent currency move is a ten per cent move on the entire invoice, tier multiplier included. Budget the tier crossing in USD with a currency buffer, and treat the exposure as an assumption finance signs off on.
