Benchmark scores for AI agents keep climbing. WebArena, AgentBench, GAIA, τ-bench: every release nudges the headline number higher. Confidence that agents are nearly ready climbs with it.
Then you ship one, and it trips over the same multi-step workflow it aced in a demo: the score and the behaviour are drifting apart. Does one good run on a short benchmark say anything about how an agent behaves across a long task?
Per-step success multiplies, so small per-step slips become large end-to-end failures over a long task. The deeper agents push into the agent-ready web, the more it matters.
By the end you’ll have a working definition of reliability, and one lever that moves it. That is more useful than another benchmark number.
Why do agents remain unreliable on long tasks despite rising benchmark accuracy?
Benchmark accuracy measures a single short run, and misses the compounding failure that many steps produce. Reliability is a separate property: consistency, robustness, predictability, safety. At 95% per step, ten steps land near 60% end-to-end. Microsoft Research’s Webwright shows how far short benchmarks overstate capability.
Most agent benchmarks report a mean success rate on a single fixed run. That number measures capability; it says nothing about dependability. It can’t tell a predictable failure from a random one, or a formatting slip from a deleted file. Popular benchmarks track capability while leaving reliability unmeasured.
Reliability is a separate property from accuracy. The paper “Towards a Science of AI Agent Reliability” (arXiv 2602.16666) defines it along four dimensions: consistency (same outcome across runs), robustness (absorbing page variation), predictability (calibrated confidence, signalling failure rather than drifting), and safety (bounded harm when things go wrong). Those four become twelve metrics independent of raw accuracy, with an interactive dashboard that compares agents across repeated runs. That is the reliability ceiling: capability rises while dependability stays flat.
Behind it sits the compound accuracy problem. Per-step success rates multiply rather than average: 95% per step lands at roughly 60% over ten steps (0.95^10). It’s Lusser’s Law, from 1960s reliability engineering: a chain is only as reliable as the product of its links. The 60% figure falls out of the arithmetic, not the model.
METR‘s time-horizon data backs this up: frontier agents hold 50% success only up to about half a task’s length, and capability gains are outpacing reliability gains. On the Odysseys long-horizon benchmark, Microsoft Research’s Webwright nearly doubles a base model’s success. It does so by writing Playwright code instead of scraping the DOM click-by-click, the scraping baseline most agents start from.
Webwright code generation versus WebMCP schema declaration: which actually lifts agent success on long tasks?
Generating Playwright code beats schema declaration on long tasks, because it changes execution, where the failures compound. Webwright writes deterministic scripts instead of predicting clicks. Schemas clean up invocation but don’t stop a long chain drifting.
Put the two levers side by side. One is Playwright code generation: the agent drives the browser by writing a program. The other is WebMCP schema declaration: a page exposes structured tools an agent can call. Both beat the default of predicting one click, type, or scroll at a time.
Webwright’s mechanism is code generation. Instead of deciding each step against the page state, the agent writes Playwright code that chains many interactions in a single step. The whole plan is captured as code, so it’s replayable and reusable. That attacks the error accumulation that click-by-click prediction suffers.
Schemas work differently. WebMCP makes tool invocation cleaner and more predictable, so the agent stops guessing what buttons do. The drift happens during execution, before interface clarity can help. That’s the difference between declaring schemas and generating code: schemas clean up the interface; code changes the execution. The spec concedes the agent still selects a tool from a name and description, with no guarantee the declared intent matches the actual behaviour.
The evidence: on Odysseys, Webwright lifts base GPT-5.4 from 33.5% to 60.1%, a 79.4% relative gain from code-as-action rather than better tool descriptions, and it runs on the Playwright and Puppeteer tooling your team or your business already runs. The two approaches are complementary, but code generation is the reliability fix. Both feed into the race to standardise agent tools.
So what do you do with this?
A climbing benchmark line tells you about one short, lucky run, and nothing about behaviour over time. When an agent drifts on long work, look at execution first: is every step being re-decided live against the DOM, or is the whole plan captured as replayable code?
Schemas wire an agent up; code generation keeps it upright. The decision rule: measure reliability on its four dimensions, and buy determinism before declarative tidiness. That’s the difference between shipping an agent and trusting one.
Frequently Asked Questions
How do I measure AI agent reliability rather than just benchmark accuracy?
Stop leaning on a single accuracy score and measure across the four reliability dimensions: consistency, robustness, predictability, and safety. The framework in “Towards a Science of AI Agent Reliability” (arXiv 2602.16666) turns those into twelve concrete metrics, and its dashboard at hal.cs.princeton.edu/reliability lets teams compare agents on behaviour across repeated runs rather than one lucky pass.
What is the compound accuracy problem, and why does per-step success compound?
It is the arithmetic of long tasks: per-step success rates multiply rather than average, so small slips snowball. An agent that succeeds 95% of the time on each step lands near 60% end-to-end over ten steps (0.95¹⁰ ≈ 0.60). The longer the task, the more steps there are to multiply, which is why long-horizon work fails far more often than any single-step benchmark suggests.
Can a bigger model or better prompting fix long-task agent unreliability?
Not on its own. A stronger model raises capability, but reliability is a separate property, so better reasoning does not stop errors compounding across many steps. METR’s time-horizon data shows capability advances are outpacing reliability gains, which is why even frontier agents hold 50% success only up to roughly half a task’s length. Fixing execution, not raw intelligence, is what moves the number.
What is the difference between consistency and robustness in AI agent reliability?
Consistency is doing the same thing across repeated runs; robustness is holding up when the environment changes. A consistent agent returns the same result each time it is given an identical task, while a robust one absorbs page redesigns, new pop-ups, or slight input variation without derailing. They are two of the four reliability dimensions, alongside predictability and safety, and an agent can score well on one while failing another.
If I generate Playwright code, do I still need WebMCP or schema declarations?
Yes, because the two are complementary rather than rivals. Schema declaration (WebMCP) makes tool invocation cleaner and more predictable at the interface level, which makes agents easier to build and wire up. Code generation makes a long chain of steps less likely to drift, because the whole plan is captured as replayable code. One wires the agent up; the other keeps it upright.
Is Webwright something I can use today, or is it a research result?
Webwright is a Microsoft Research result, not a packaged product you install off the shelf. What transfers to production now is its mechanism: generating deterministic Playwright code instead of scraping the DOM click by click, which any team already running Playwright or Puppeteer can adopt on its existing stack. Treat the 33.5% to 60.1% lift on Odysseys as evidence for the approach, not as a download.
What does it mean for an agent’s execution to be replayable, and why does it matter?
Replayable means the agent’s plan is captured as a deterministic script you can run again, inspect, and reuse, rather than re-decided live at every step. When the whole sequence is expressed in code, the agent is not predicting one click, type, or scroll at a time against a changing DOM, so it cannot lose its place halfway through a long task. That is what turns execution from brittle into dependable.
How much does an agent’s reliability drop as a task gets longer?
Steeply, because errors compound with every additional step. At 95% per step, success falls to roughly 60% over ten steps, and real long-horizon tasks run well past ten. METR’s data adds the same warning: frontier agents hold 50% success only up to about half a task’s length, so the longer the job, the wider the gap between a headline score and what you actually see.
Will a website redesign or pop-up break a code-generating agent?
It is a real risk, because a redesign can invalidate the selectors a script relies on. This is where robustness, one of the four reliability dimensions, separates agents, and it is a reason to pair deterministic code with resilient locators rather than brittle DOM paths. A click-by-click agent is arguably more exposed, since it must re-interpret a shifting page at every single step.
Which kinds of long tasks are most likely to expose agent unreliability?
The ones with many sequential steps and little room to re-decide, such as multi-page research, form-heavy workflows, booking flows, and end-to-end data entry. These are exactly the Odysseys-style long-horizon tasks where Webwright lifted success from 33.5% to 60.1%. Short, single-screen actions hide the problem because there are too few steps for errors to compound, which is why teams are often blindsided in production.
Are AI agents safe to run unattended on long tasks?
Not yet, and certainly not without guardrails, because safety is the fourth reliability dimension and it is largely untested at long horizons. Safety means bounded harm when failures occur, so an agent that drifts can still take unintended actions such as submitting a form or spending money. Reliability measurement treats safety as its own dimension precisely because a high accuracy score on a short task says nothing about what an agent does when it goes wrong.
