You’ve seen the number on the spec sheet: a one-million-token context window. It reads like a room you can fill with code, logs and documentation and expect the model to use all of it. That assumption is expensive. This is the same shift we traced in the full picture of where leverage moved: the advertised Context Window is a billing and architectural number. The Effective Context Window is the length at which the model still reliably finds and obeys what you put in front of it, and it is a measurement you take. By the end you’ll know how to read long-context benchmarks sceptically and choose between a bigger window and compaction on reliability rather than headline size.
What is the effective context window, and why is it so much smaller than the advertised one?
The effective context window is the length at which a model still reliably finds and obeys what you give it, and it is smaller than the advertised million tokens. Attention degrades with length and is weakest in the middle of the input, so usable capacity falls away before the window is full.
The advertised figure describes how many tokens the system will accept. The useful figure describes how much of that input the model can still use for the task you actually care about, and those two are not equivalent. One research group calls the useful number the Maximum Effective Context Window, arguing published figures “reflect architectural or implementation limits, not necessarily the model’s practical capacity” (arXiv).
Chroma’s name for the gap is context rot. In an 18-model study they changed only the input length, not the task, and every model grew “increasingly unreliable” as the input grew (Chroma Research). The degradation is gradual: it shows up even on tasks that require finding nothing.
Position is the other half of the story. Attention is strongest at the start and end of the input and weakest in the middle, the U-shaped curve Liu et al. measured in 2023 (Lost in the Middle). A fact’s effective window depends on where it sits, not just on how much else is present. A 900,000-token dump quietly stops working in the middle, before any limit is reached. Each token attends to every other token, so attention costs grow with the square of length (Anthropic).
That makes position a design variable. “It’s in there somewhere” is not a retrieval strategy, because reordering the same tokens changes what the model can use. Length and position are the variables, and here is what happens when frontier models run at the top of that range.
Why do frontier models still miss things inside a million-token context window?
Frontier models miss things in long contexts because usable attention fades with length, even when the task is unchanged. A 2026 study found models missed a dangerous action buried after 800,000 tokens of benign activity two to thirty times more often than in a short context (arXiv 2605.12366). Most published scores measure needle-in-a-haystack search rather than actual use.
Opus 4.6, GPT-5.4 and Gemini 3.1 were the models in that study. Opus 4.6’s recall fell from 99.7% on a 100,000-token transcript to 69% on an 800,000-token one. The capacity being paid for does not buy recall.
The model was not asked to retrieve a string; it was asked to notice and classify an action, and that judgement decayed with length. A model can quote a fact back and still fail to act on it: that is the difference between search and use.
Most published long-context numbers do not even measure that. They come from needle-in-a-haystack retrieval, which Chroma describes as “a simple lexical retrieval task” whose near-perfect scores are “narrow and not representative of how long context is used in practice” (Chroma Research). A vendor quoting a single retrieval score is quoting the easiest test in the suite.
Four benchmarks stress different things. RULER adds multi-needle and multi-hop reasoning; of 17 models claiming 32,000 tokens or more, only half held up. NoLiMa removes the word overlap between question and answer; 11 of 13 models fell below half their short-context score at 32,000 tokens. LongMemEval tests whether memory survives across sessions, and ConstraintRot checks whether a rule still holds as the context grows.
The useful question is which instrument, at which position, at what length. A retrieval score measures search; a capability score measures use. If the advertised window overstates the capability, the practical choice is between a bigger window and compacting what you already hold.
Compaction vs a larger context window: which one actually improves agent reliability?
Neither size nor compaction guarantees reliability, because the failure mode is silent non-use rather than truncation. Where a constraint must survive, no window size substitutes for it being enforced.
“Just give it the biggest window and stop worrying” fails for one reason: no error message reports that the model quietly ignored a fact or a rule. When a long-context monitor misses an action, it does not error out; it just returns the wrong answer (arXiv 2605.12366).
A bigger window buys headroom, and your business pays for it on every call at the full tier rate, whether the model uses the tokens or not. Compaction buys headroom and pays for it in fidelity: whatever the summariser drops, the agent cannot get back from inside the transcript. The trade is token cost against constraint risk, and it shows up in the ConstraintRot results, where compaction pushed violations from 0% to 30%, up to 59% at worst.
The summary is where goals are most often lost: a lossy summary can drop the one rule that mattered. A constraint that must survive has to live outside the transcript. For the detail on exactly what gets discarded, see our piece on what a summariser drops when it compacts.
Before you trust any vendor’s long-context number, ask four questions. Which benchmark produced it: a single-needle retrieval test, or a multi-hop and constraint test? At which position in the window was the tested fact placed? At what absolute length: 80,000 tokens or 800,000? And does the number still hold at 80% of the advertised maximum, or only at 10%? Those four questions close out the capability side of the decision. What remains is the budget.
The advertised window is a price signal and a task-dependent spectrum rather than a flat capability. Capacity is a budget decision. The capacity you never use is still capacity you paid for, and what the unused capacity is billed at is the quiet line on the invoice. A window you cannot fully use is a budget line.
Frequently Asked Questions
Is a million-token context window actually usable at a million tokens?
Only for the easiest task. Single-needle retrieval at one million tokens is close to solved, with frontier models scoring near 100% in a 2026 1M-token study, but three-hop reasoning split the same models between stable and collapsing. BABILong found popular models using only 10 to 20% of their context. Treat one million as a ceiling for simple lookup, not a working budget for reasoning.
Do I still pay for tokens that the model ignores?
Every input token is billed whether the model attends to it or not. Providers charge per input token, and a prompt that fills the window is charged at the full rate even if most of it never influences the answer. That is why a 1M-token prompt can cost many times a retrieved alternative: one 2026 case study paid 26 times the token cost for a 7.7 point accuracy gain.
Does a larger context window make responses slower?
Usually, yes. A longer prompt has to be processed before the first token appears, and the key-value cache that holds attention state grows with sequence length, consuming memory and bandwidth. Providers use caching to soften the cost on repeated prefixes, but a larger window shifts work and hardware onto every call. Latency is a separate bill from accuracy.
Should I repeat an important instruction at the end of a long prompt?
Often, yes. Recall and compliance are strongest at the start and end of a context, so restating the governing constraint near the point of use is cheap insurance. A 2026 study found periodic reminders improved detection of obvious attacks at under 3% input token overhead, though the authors note they are not a general fix for subtle cases. Repeat the rule rather than assume it survived 800,000 tokens.
Is context rot the same thing as hallucination?
No. Context rot is a decline in how well a model uses information you supplied, while hallucination is fabricated content. The two can look alike from the outside, because both produce a confident wrong answer, but context rot has a measurable signature: accuracy falls as input length grows, even when the task and the evidence stay the same. That is why it goes unnoticed.
Do longer contexts make models worse at following constraints?
Yes. Constraint adherence is one of the first things to decay, which is why ConstraintRot exists as a benchmark: it measures whether a rule still holds as the context grows, not whether the model can quote the rule. A model can repeat your formatting rule and then break it a few hundred tokens later. For agents, a dropped constraint is more dangerous than a missed fact, because the action still runs.
Can a larger context window replace retrieval or RAG?
No. A larger window moves the bottleneck, it does not remove it. Retrieval decides what enters the window; attention decides what the model can still use once it is there. In a 2026 case study, long-context prompting beat semantic RAG on accuracy (73.1% versus 65.4%) but cost 26 times as many tokens per query. The usual answer is routing between the two, not replacement.
Is a token the same size across model providers?
No. Tokenisers are trained on different vocabularies, so the same text can consume different token counts across vendors. One 2026 study measured a 35% spread, from 0.901 to 1.222 tokens per character, meaning a corpus that fills one model’s 1M window can leave another with room to spare. Compare models on the content they can hold, not just the token number.
How do I measure the effective context window for my own task?
Test your own task, not the vendor’s benchmark. Take a representative prompt, replicate it at several lengths (say 10%, 50% and 90% of the advertised window), place the critical fact at the start, middle and end, and score task success rather than recall. The effective window is the point where success drops below what your workflow tolerates. That number is the only one worth budgeting against.
What happens if my prompt goes over the context window?
Usually the request fails with an error rather than being quietly trimmed, but the behaviour depends on the provider and the interface. Some APIs reject prompts above the input limit, chat products may drop the oldest turns without warning, and many windows must cover input and output together. A prompt at the limit can leave no room to answer.
Are newer models fixing context rot?
Partly, and not enough to budget against. Each generation improves single-needle retrieval at length, and the 2026 1M-token study found some frontier models holding three-hop performance at the full window. But the same generation still missed dangerous actions two to thirty times more often after 800,000 tokens, and other models collapsed between 512K and 1M. Improvement is real; the slope has not disappeared.
