A deploy goes out clean. It passes CI and the replay tests. Then executions that have been running for weeks fail at the same position in their history, at a branch the new code no longer matches. No log line points at what you changed.
A workflow execution outlives the process and code revision that started it, so a deploy is a compatibility event against every history still open. This piece continues this series on agent state, and takes this in three layers: the record, the constraint, and what shipping does to a running fleet.
What is Temporal’s event history, and why does it force workflow code to be deterministic?
Event history is the append-only, ordered record of every command and activity result in a workflow execution, persisted by the Temporal Service. Replay re-executes the code, reissues those commands and matches them positionally against the record. A different sequence cannot be reconciled, so determinism is the condition of replay.
Workflow code never calls an activity directly. It calls an API, the SDK emits a command, the Service validates it and appends the resulting event. Replay feeds recorded results back in, rebuilds in-memory state and continues from the last position the record supports, reproducing no side effects at all. Why an interrupted step re-runs is covered in checkpoints versus event history.
The constraint follows: no wall clock, no randomness, no unordered iteration, no inline I/O in orchestration code. Temporal states its determinism contract as making the same calls in the same sequence given the same input. Non-deterministic work belongs in an activity.
The TypeScript SDK is the exception. It runs workflow code in a V8 isolate and replaces the clock, the random source and the timer with deterministic versions, so replay stays correct without hand discipline. Go and Java leave that discipline to you. Same contract, different enforcement.
Inside the runtime, the failure does not look like a bug. The command your new code emits is compared with the event occupying that position, so divergence arrives as a journal mismatch rather than a stack trace naming a function.
Why does a Temporal workflow that ran fine for a month throw a non-determinism error after a code change?
The changed branch only executes when an old history reaches it. New executions may never traverse it, so the deploy looks healthy and the defect waits. Then an execution resumes, retries or relocates, reissues a command the history does not hold, and the runtime raises a non-determinism error.
Three shapes cover the common cases: a new activity in the path, a reordered branch, a changed timer. The timer makes it concrete: a definition that waited then called an activity now calls it first, so the command after the pause is an activity schedule where history holds a timer. Keith Tenzer states the rule: you can always alter the future of a running workflow, but changing the past breaks determinism. What the runtime reports is a command it did not expect at that position.
Patching and Worker Versioning make late edits survivable, and both have to be declared before the deploy, because neither is discoverable afterwards. A patch writes a version marker into the history so old and new executions each take the branch they were written for. Versioning routes pinned executions to a compatible build instead. The agent case differs: what changes is usually the prompt or tool set, not the control flow, so the command sequence matches and no error fires. A version marker on the paused run records which definitions produced it, and stops a later resume running under the wrong prompt.
In October, Temporal hired Oso’s team, including founder Graham Neray, to build authorisation into the execution layer. Two days later Oso Cloud’s assets sold to Tiny for roughly $1.9 million. The inference: whoever holds the record can approve what it replays, and a relationship-based access-control model (ReBAC) fits “who may approve this run”. That ties back to who owns the record and how you get it back.
That is the strategic question. The operational one is the deploy itself.
What happens to in-flight agent runs when you deploy?
A deploy replaces your worker fleet, so any run you have in flight whose state lived only in a worker stops mid-step. The Service still holds the event history, so the execution can sit Running with nobody making progress, because workflow code is oblivious to worker downtime. Replay restores it only when a compatible worker picks the task up. An activity that was mid-flight may run again.
One team running roughly 28 production agents hit this: every deploy killed runs mid-flight, and Mastra re-drove the orphans on boot, reissuing model calls and tool calls. The recovery grace window is the mitigation, and its duration is a guess. Rivet defaults to about fifteen minutes. Vercel’s durable workflow defaults treat 120 seconds of silence as a crash. Two vendors, two guesses, and they differ by a factor of seven.
The missing primitive is a distributed lease. Without one, replicas race to recover the same run and the side effect fires twice. An idempotency key protects the effect, and leaves the duplicated orchestration in place.
A record that must replay identically to be trusted is an authority artefact. That is the governance side of the replayable record in its plainest form, and audit route history, the execution graph’s record of which branch ran and why, serves as a governance document, and as a debugging tool second. Provenance is what keeps it trustworthy: source, code version, authorising identity, superseded decisions. A decision that was overridden still has to be readable months later, which is what stops a resumed run acting on a rule the business has since replaced. LittleHorse reported agent controls in September, with no published detail.
The authority to bind future action to past consent gives a replayable record its value, and the right to approve what it replays is the asset being bought. The wider argument sits in the cluster hub.
Log, protocol, then authority
It moved from log, to protocol, to authority, and each move narrowed who can touch it. A deploy re-issues an interpretation of histories the new code did not write, which is why “can I deploy on a Friday” and “who is allowed to approve this run” are the same question at different distances.
Frequently Asked Questions
Does the determinism constraint apply to Activity code as well as workflow code?
No. Activities sit outside the replay path, so they can read the clock, call an API, generate a UUID or query a database without breaking anything. The runtime never re-executes them during replay; it feeds their recorded results back to the workflow. The constraint applies only to the orchestration code that issues commands, which is why the standard advice is to push everything non-deterministic into an Activity.
Can I edit or delete part of an event history to clear a non-determinism error?
No. Event History is append-only by design, and no SDK or Service API exposes a way to rewrite or remove events. Termination ends an execution; it does not repair its record. The sanctioned routes are patching, Worker Versioning, or starting a new execution with fresh history. Editing the journal would also destroy the property that makes it worth keeping: an unaltered account of what happened.
How large can an event history get before it becomes a problem?
Temporal bounds each execution by event count and total payload size, in the order of tens of thousands of events and tens of megabytes, after which the execution must continue as new. Long-running agent loops reach those limits quickly, because every Activity result, timer and signal adds an event. Continue-As-New closes the current history and starts a successor with a clean one, carrying state forward deliberately rather than accumulating it.
Is replay the same thing as retrying an Activity?
No. A retry re-invokes an Activity against the outside world, with all the side effects that implies. Replay re-executes the workflow’s orchestration from the beginning and feeds back recorded results instead of calling anything. Replay is a matching exercise against the journal; a retry is a new attempt at a real effect. Confusing the two is how teams end up expecting replay to duplicate emails or payments, or expecting retries to rebuild in-memory state.
Do Activity timeouts and retries make a workflow non-deterministic?
No. Timeouts and retry attempts are recorded as events, and the workflow sees the eventual outcome as a value in its history, not as a live decision. A retried Activity may run several times, but the orchestration observes one recorded result. What would break determinism is reading the clock or a random value inside the workflow to decide whether to retry, rather than letting the retry policy do it.
What happens to a durable timer or a sleep when I deploy mid-wait?
Nothing happens to the timer itself, because it lives in the Temporal Service rather than in your worker. It fires on schedule and the execution becomes ready to progress. What it cannot do is make progress without a compatible worker to replay it and continue. A workflow can therefore sit in Running with a fired timer and no movement, which reads as a stall in dashboards but is really a fleet compatibility gap.
Is Worker Versioning the same thing as patching?
No. Patching is a code-level declaration: you record a version marker in Event History and branch behaviour on it, so old and new histories each take the path they were written for. Worker Versioning works at the fleet level, assigning Build IDs and routing pinned executions to workers that can still satisfy their history. The two are complementary, and most teams end up using both, patching for logic and versioning for routing.
Do I need to version every deploy if my workflows never change?
No. The risk is tied to changes in the command-producing sequence, not to deploys as such. If you only change an Activity’s implementation and keep its name, input and output the same, the commands still match and old histories replay cleanly, though the new behaviour applies to future calls. Version markers exist for control flow changes, and adding them everywhere is noise that makes the real ones harder to find.
Would changing an agent’s prompt cause a non-determinism error?
Usually not, and that is the problem. A prompt change normally sits inside an Activity, so the command sequence stays identical and replay succeeds. The run then continues under a different set of instructions from the one it started with, which no error will flag. Recording the prompt, tool set and model on the paused execution is what lets you tell, months later, which definition produced the state you are about to resume.
Is the agent’s conversation memory stored in event history?
Only partially. Event History holds what passed through commands and Activity payloads: the inputs you sent and the results you got back. It does not hold model weights, vector store contents, retrieved documents or whatever short-term memory your framework keeps in process. That gap is the governance problem, because replay can reconstruct the workflow’s position but not the context that made the agent’s earlier decisions sensible.
Can I replay a completed workflow to debug what went wrong?
Yes, provided you can run the code version that produced the history. Replay from the recorded events rebuilds state step by step, and the SDK’s replay tests let you assert that a new build still matches a captured history. Replaying against a newer build is exactly where non-determinism errors surface, so a finished execution is useful evidence only while its compatible worker image still exists.
Does Temporal guarantee exactly-once execution of Activities?
No. The workflow’s orchestration is effectively exactly-once, because commands and results are recorded and replay matches them. Activities are at-least-once: a worker can die after the effect landed but before the result was recorded, and the retry will fire it again. Idempotency keys on the external call are what make that safe, and they protect the effect, not the orchestration around it.
