A durable runtime records progress outside your process, so a killed run restarts from the last recorded point rather than the first line. That point is a commit boundary, and you chose it, even by accident. It is one piece of the wider durable execution cluster, the argument about which state a run keeps and who holds it. The runtime owns resumption and the timers. The rest is yours.
What does durable execution actually guarantee for an agent run, and what does it leave to you?
You get resumption from your last commit boundary, plus durable timers and durable pauses that hold no compute while they wait. Four things stay with you: where the boundaries sit, the business identity that makes a repeat safe (an order number, not a fresh UUID), the payload you commit against the result ceiling, and custody of the record.
Four mechanisms get sold under that name. Checkpointing writes a step’s result before the next step starts. Journal-and-replay re-runs your orchestration code against an append-only event history. Step memoisation returns a recorded value for a named call. Platform hibernation releases compute during a wait and reattaches it on the callback.
Pi Durable, from Earendil, splits the same guarantees three ways, though it is an experimental package with no production record: borrow the framing, skip the dependency.
The unit that matters is the step boundary, or super-step: where the runtime records one unit of work before permitting the next. Anything outside it can run again, and how checkpoints and event histories differ sets the size of that rewind. You draw them, because the runtime cannot infer that two model calls and a charge are one unit of work.
Why does a crash cost more than the step that was running?
Recovery restores the last recorded result and re-executes from there, so the bill is everything since your last commit boundary. You pay twice for every call that sat outside one, because the interrupted step restarts from its first line and re-issues the lot. Fix it with finer granularity: one model call per boundary.
Kill a real run at named points and count what re-executed: before the execution intent commits, after a side effect but before settlement, and after one parallel result commits while its sibling runs. The conformance study that found a completed task re-running contradicts the documentation on that point, and a client disconnect that produced two charges did the same with no crash at all: two charges in the payment log, nothing flagging them.
Powell’s Law is a useful test here: interruptions should cost time, not progress. Then the cost that never appears on an invoice: the side effect that already fired. A charge that lands in the milliseconds before the runtime records it loses its acknowledgement, so the next worker retries something that already happened, a window vendors document and no ledger shows.
Why can’t a durable runtime make an external side effect exactly-once on its own?
Because the identity that makes a repeat safe has to exist before the runtime has anything to record, and the effect commits in a system the runtime does not control. You get effectively-once orchestration; once-only effects come from a key or a compensation you own.
Admission-level deduplication means the runtime refuses a duplicate start or replays a completed step. Effect-level guarantee means the destination refuses a duplicate charge. The marketing blurs the two, and “just use Temporal” moves the problem into your own code: engines offer exactly-once for workflow logic and at-least-once for activities.
Stripe’s idempotency header is the model: claim the key before the work, derive it from the business operation, and mind the bounded retention window, since a key reused after it expires starts a fresh request. Where no key can exist you get a saga, and durability promises only that the compensation runs.
The runtime records the attempt. Authorising a repeat happens in the destination system, with your key, which settles who ends up holding the record and where that state lands and who can take it away.
Do you need durable execution at all, or is a retry wrapper enough?
So the question inverts: what does a wrapper actually lose? A retry wrapper is enough when the run is short, single-process, and every step is idempotent or read-only. It stops being enough when the run waits on a human, waits days, fans out across many tool calls, or moves money. Any one of them breaks it alone.
What dies with the process is the attempt budget: the count, the backoff position and the place in the sequence. Polly, Tenacity, Resilience4j, Spring Retry and Opossum all run inside the calling process, so the step body they protect is at-least-once whether or not anyone wrote that down.
A wrapper sits around a call; durable execution sits around a process. A table, a queue and row-level locking already approximate resumption. Replaying your orchestration code and holding a timer across days are what they cannot do.
Keep the wrapper either way: it still belongs inside each durable step, because a timeout and a bounded retry are how call-level failures get handled. That is also how you sort a run into a type before committing to anything heavier.
Checkpointer or full engine: what will each cost the team you actually have?
A checkpointer is a library writing run state into the database you already back up. A full engine is a worker fleet, a cluster, a visibility store and a determinism discipline somebody has to enforce. Choose on operating burden: who runs it, who patches it, who gets paged.
The checkpointer is the cheaper half of that list. LangGraph’s SQLite and Postgres savers and CrewAI’s checkpoint config buy crash recovery and thread continuation without buying a second distributed system. What they do not buy is a supervisor, so a dead run stays dead until something outside notices, and that something is your team.
A full engine prices differently. It adds a scheduler, a task queue, and a determinism contract that costs you at review time. Temporal, the reference case, creates 37 tables, runs about 40 SQL statements per workflow plus 35 for each step, and stores 12.5 kB per run.
Pydantic AI takes the third position: durability as an attachable integration with no native store, and deferred tools that end the run for a later run to pick up. A five-step workflow with one external call needs a boundary and a key, not a cluster.
The exception is the approval pause. A human wait measured in hours or days is the one thing a wrapper and a cron job cannot hold, and it costs a row rather than an idle container.
The runtime sells you a boundary, a timer and a record of an attempt you reconcile elsewhere. If the run is short, single-process and idempotent, keep the wrapper. A human wait, a long wait, wide fan-out or money need a boundary the process cannot lose, and the full series on agent state covers where that operating model goes.
Frequently Asked Questions
Is durable execution the same thing as event sourcing?
No. Event sourcing is a storage pattern where state is rebuilt by replaying every past event; durable execution is a runtime that records progress at named boundaries so an interrupted run can resume. They lean on the same idea, an append-only history, but an engine can checkpoint step results without exposing an event log, and an event-sourced service gives you no durable timers or pauses by itself.
What happens to an agent run when I deploy new code while it is paused?
It resumes on the new code and can fail with a non-determinism error. The runtime replays the recorded history against your current orchestration logic, so if you removed or reordered a step, the replay diverges at that point. Version your workflows before shipping a change, and drain runs that are mid-flight. Changing only the body of a step is usually safe.
Am I paying for compute while a durable timer waits?
No. A durable timer or a durable pause releases the worker and records a wake-up time in the store, so nothing is held open while the wait runs. You pay for storage, and for the compute when the run resumes. That is why a three-day approval gate costs a row in a table, not three days of an idle container.
How do I decide where to put a step boundary?
Draw it around a unit of work you would be willing to repeat, not around every line. One model call per boundary is a fine default for anything paid, and a charge gets its own boundary plus a key. Then check the cost: if a crash replays three paid calls, the boundary is too coarse for the budget you have.
Do I have to make every step idempotent, or only the ones that touch the outside world?
Only the ones with effects you cannot undo silently. Read-only lookups and pure transforms can repeat freely, which is why an agent that only reads is safe under a retry wrapper. Anything that sends, charges, writes or ships needs a business identity or a compensation, because a re-run looks identical to a second request.
What happens if my idempotency key expires before the retry arrives?
The destination treats the retry as a new request. Stripe, for example, keeps keys for a bounded window of 24 hours by default, so a replay outside that window captures a second charge. Keep retry horizons shorter than the key’s retention, and reconcile any run that has been uncertain for longer against the external ledger rather than replaying it blindly.
Can I add durable execution to an agent I have already built, without a rewrite?
Often yes, and it is the cheaper path. Frameworks that treat durability as an attachable integration, such as Pydantic AI with StepPersistence, or a checkpointer passed into LangGraph, let you add a resume plane under existing code. What you cannot bolt on later is the boundary design: if the step is one giant call, persisting it buys very little.
Does durable execution protect me from bugs, or only from crashes?
Only from crashes and restarts. The runtime replays your logic faithfully, so a wrong instruction, a bad prompt or a mis-ordered refund re-executes exactly as written, now twice. What it does guarantee is that a failure in infrastructure costs time rather than progress, which is the whole of Powell’s Law. Correctness stays a property of your code and your keys.
If I am already on Postgres, do I still need a separate visibility store?
Not necessarily, but plan for it. A table plus FOR UPDATE SKIP LOCKED gives you run state and resumption, and Postgres can answer basic “where is this run” queries from indexed columns. What it does not give you is the searchable history an engine’s visibility store provides, which is why Temporal pairs its 37 tables with Elasticsearch once operators need to query many runs at once.
How do I test whether an agent genuinely resumes after a crash?
Kill a real run at named points and count what re-executed. Target the three awkward moments: before the execution intent commits, after a side effect fires but before settlement, and after one parallel result commits while its sibling is still running. Run against an external ledger so you see the duplicate the framework cannot, then repeat on every checkpointer you consider. A test that only kills between steps proves nothing.
Do I need durable execution for a workflow that only reads data?
No. A read-only pipeline is safe to re-run from the start, so a retry wrapper with backoff covers you and a checkpoint adds a dependency for little gain. Add a boundary the moment a step writes, sends or pays, or the moment the run has to survive a wait longer than the process. Those are the conditions that turn re-running from harmless into expensive.
