Your coding agent just shipped a change to a module it never read. It was confident, well formatted, and wrong. The model reasoned fine about what it had loaded; the failure was in what it never saw. That’s a visibility failure, and a common way agents fail.
Most teams respond by buying a bigger context window or waiting for the next model release. Both treat the model as the suspect, but the fault usually sits in the context layer between the model and your code. That layer is where agent quality is won or lost, and it’s one of the parts of the stack you actually own. This article walks through the four questions a leadership team needs answered: why retrieval is the bottleneck, whether to run sub-agents or one long-running agent, why a curated library beats prompt wording, and how to audit a failed run, so you can justify the hire and headcount behind the answers.
Why is retrieval rather than reasoning the bottleneck for agents on large enterprise codebases?
On a large enterprise codebase, retrieval is the bottleneck because a model can only reason about what it has actually loaded. The per-step reasoning is usually correct; what fails is the search strategy, which branches faster than it prunes. Failure arrives as a confident, wrong answer about a module it never opened.
Below a few hundred thousand lines, retrieval stays cheap. Past that, the search space outgrows the context budget, and the agent returns the wrong file or spins for thousands of seconds. Sourcegraph’s benchmark runs put the crossover near 400,000 lines: on a 1.4-million-line Kubernetes repository, a capable agent with local tools alone ran for 6,000 seconds and produced nothing; with indexed search it finished in 89 seconds. The number moved because the infrastructure changed; the model was the same in both runs.
The operating principle is just-in-time retrieval: fetch at the moment of need and escalate to a larger window only when a task requires holding a whole subsystem in view. Escalation is the expensive path, because a bigger window changes the failure mode from truncation to dilution rather than removing the constraint. Plain grep and local search hold up below a certain repository size; code intelligence and MCP-based retrieval start to pay when the codebase exceeds what one engineer can hold in their head, or when a monorepo makes naive search return too much. Whether the extra tooling pays depends on where your codebase sits on that threshold.
The evidence sits in benchmarks like Terminal-Bench 2.0 and CodeScaleBench: the LangChain coding agent went from 52.8% to 66.5% on Terminal-Bench 2.0 through harness, middleware and verification changes alone. The context, scheduling and verification layers produced the gain, the model was unchanged. If retrieval is the bottleneck, the next decision is how you structure the agent that does the retrieving.
Multi-agent sub-agents vs one long-running agent: which handles long-horizon work better?
For long-horizon work, orchestrated sub-agents win when the task decomposes into slices with clean interfaces, and a single long-running agent wins when the task needs one accumulating understanding. Sub-agents isolate context and run in parallel.
The draw is a clean window per worker, so each holds only what its slice needs and several slices run at once. The case against is that every sub-agent needs its own briefing and its own copy of shared context, so coordination replaces compaction as the lossy step and the token bill multiplies. Anthropic’s multi-agent research system beat a single agent by 90.2% internally while consuming about 15× the tokens, against a simple chat baseline rather than an optimised single-agent pipeline. The honest answer is conditional on task shape.
The failure modes are worth naming. The long-running agent forgets as compaction silently drops earlier decisions; the orchestrated one contradicts itself as workers reach incompatible conclusions from partial views. Anthropic’s guidance on long-running agents is the place to start; the decision rule is two lines: does the task split into clean interfaces, or need one accumulating understanding? Whichever way you answer, every agent draws on the same shared context, and that context is something you can curate — which is the strategic case for context engineering over prompt engineering in one sentence.
What is a context library, and why do experienced users spend more time curating one than writing prompts?
A context library is the curated, versioned body of documents, conventions, decisions and examples a team maintains for its agents, a first-class owned artefact rather than a folder. Curating it beats prompt wording because a prompt is read once per call and forgotten, while a library entry is inherited by every run, so its leverage and its errors both compound.
A real library holds architecture notes, coding conventions, decision records, golden examples, known-bad patterns and environment facts, all versioned and owned. That’s why experienced users spend more effort engineering libraries than writing prompts. Nielsen Norman Group’s research found power users doing exactly that, dictating prompts casually while carefully maintaining libraries.
The problem is that this work is invisible: no demo, no feature, no metric, so it goes unstaffed until a failure forces it. That’s the argument to take to a leadership team, because it justifies the Context Engineer role: someone who owns assembly, retrieval quality, tool schemas and the library. When nobody owns it, the library decays and agents quietly serve stale context, which is worse than none. MCP made the tool and resource surface portable, and so made it someone’s job. A badly named tool schema is a context defect: every tool you expose is another decision the model has to make. A curated library only pays off if you can see what actually reached the model on a given run, which is where the audit comes in.
How do we audit what actually entered the context window on an agent run that failed?
Audit a failed run by reconstructing exactly what entered the context window: the assembled prompt, the retrieved document identifiers and versions, the tool schemas in force, the compaction events and what they removed, and the model version. That’s the only way to separate a weak model from wrong input, and wrong input is the common case.
When the agent follows its instructions and still produces the wrong content, the fault is in the input. Dump the assembled context and read it the way the agent received it, and the fault reveals itself as a problem in retrieval, assembly, or the goal statement.
Most observability stacks trace outputs and tool calls, not the assembled input. Build the audit capability before you need it; retrofitting after an incident is too late. A compaction event is the kind of thing an audit has to surface, since it records the constraints lost when history is summarised. LangSmith, from LangChain, is the natural tracing surface; the harness is the code your team owns, and the only place input-side observability can live.
Returning to the cluster overview, the things you own come down to three: the library, the harness, and the audit trail. The audit trail is the one that’s usually missing, which is why failed runs keep going undiagnosed. Close that gap first, and staff the context layer the way you’d staff any internal system your business depends on. Ownership is the intervention.
Frequently Asked Questions
How large does a codebase have to be before retrieval becomes the real problem?
A capable agent reads what it needs on a codebase of a few hundred thousand lines, so retrieval stays cheap and mostly invisible. Somewhere past that threshold, across multiple repositories and services, the search space outgrows what any single pass can prune, and the agent starts returning the wrong file or nothing at all. Treat the crossover as a planning number, not a law: measure it on your own repository before you buy tooling.
Is a context layer just RAG with a different name?
No. Retrieval augmented generation is one component of a context layer, usually the part that fetches documents. The context layer also covers what gets assembled into the prompt, which tool schemas are in force, what compaction removed, which library entries are current, and who owns all of it. RAG answers where text comes from. The context layer answers what the model saw and why.
Could we fine-tune a model on our codebase instead of building a context layer?
Fine-tuning changes how a model behaves, not what it can see, so it cannot fix a file that was never loaded. It is a reasonable investment once retrieval is healthy and you want a consistent house style or a stable output format. Sequence it second. Teams that fine-tune first usually end up with an expensive model that is still confidently wrong about a module it never opened.
What are the early warning signs that our context layer is failing rather than the model?
Watch for plausible answers about files the agent never opened, repeated rediscovery of the same module, long runs that end with nothing, and disagreements between sub-agents working from partial views. Each of those points at assembly or retrieval, not reasoning. The tell is repetition: a genuine model limitation shows up as a consistent quality ceiling, while a context defect shows up as the same mistake recurring across runs.
How do we stop the context library from going stale?
Give every entry an owner, a last-reviewed date and a review cadence tied to change, so a library file is refreshed when the code it describes moves. Staleness is worse than absence, because a missing document prompts a search while an out-of-date one gets inherited silently by every run. Delete aggressively. A short library that is current beats a comprehensive one nobody trusts.
Should the context layer sit with the AI team or with platform engineering?
Put ownership where the code and the on-call rota already live, which for most organisations is platform or engineering enablement, with the AI team as a consumer rather than the custodian. The layer is infrastructure: it needs versioning, review, incident handling and a budget line. Handing it to a rotating group of enthusiasts is how it decays into a shared folder with good intentions.
Do we have to send our repository to a vendor to get usable retrieval?
No, and for many Australian teams that question is a governance decision before it is a technical one. Local indexing, plain search and self-hosted code intelligence cover most needs, and only the retrieved slices ever leave your environment. Decide the data-handling position first, in writing, then choose tooling that fits it. Reversing that order is how procurement stalls for a quarter.
How do we measure whether the context layer is actually improving?
Run a fixed set of real tasks from your own repository, score them on correctness and on whether the right files appeared in the window, and re-run the set after every change to retrieval, tool schemas or library content. Track retrieval hit rate separately from answer quality, so you can tell a context regression from a model regression. Public benchmarks are useful context, not your acceptance test.
What is the smallest useful context layer we could stand up this quarter?
Start with three things: a versioned library of architecture notes and conventions for your most-edited services, a retrieval path that returns file paths and versions rather than pasted text, and input logging on the harness so every run can be reconstructed. That is enough to diagnose failures and justify the next increment. Skip the platform build until you have audited a handful of real failures.
Does this apply to agents outside software engineering?
Mostly yes, because the mechanism is the same: the model can only reason about what the layer put in front of it. Support agents, claims processing and research workflows all fail on missing or stale context rather than weak reasoning. What changes is the shape of the library and the cost of a wrong answer. The audit discipline transfers unchanged.
How much should we expect token costs to change once we fix retrieval?
Retrieval-first usually lowers the bill, because a focused window of a few thousand well-chosen lines costs less than stuffing a whole subsystem in and hoping. The exceptions are the escalation path and sub-agent architectures, which multiply input tokens quickly and can cross a pricing threshold that reprices the entire request. Budget in USD against revenue in AUD, and review the number after every architecture change.
