33  Deployment Topologies and Reliability

Chapter status: drafted. One small measurement, about a minute of compute: 72 short calls — three tasks × three sampling conditions × eight retries — asking a model to emit the same tool call repeatedly, to find out whether retrying an agent step reproduces it. Everything else here is ordinary distributed-systems practice applied to a non-deterministic caller, and is cited rather than re-derived. Software: ollama 0.24.0 serving llama3.2:3b, Python 3.14.5, Apple Silicon.

Learning objectives. After this chapter you can:

33.1 Where the agent actually runs

The first decision is a boring one, and teams routinely make it by habit. There are three shapes, and the honest way to choose is to name which constraint is binding.

Three shapes, and the constraint that should pick one
you get you give up choose it when
Managed agent runtime session state, retries, scaling and traces without writing them control over the loop, the ability to run in your own network, and portability speed to ship matters more than control, and no data-residency rule blocks it
Your own containers full control of the loop, the network and the data path building session storage, cancellation, retry semantics and observability yourself residency or an existing platform makes it necessary — not because it feels more serious
Serverless functions scale-to-zero and no idle cost long runs, because execution is time-boxed traffic is spiky and every task finishes in seconds

The third is the one that surprises teams. An agent loop is not a request handler: it can take minutes, and a platform that caps execution at a few seconds will kill it in the middle, after the tools have already run. If your runtime has a hard execution ceiling, the loop has to move behind a queue — which is Section 33.3, and is a change to the product, not to the deployment.

33.2 Where the conversation lives

An agent is a loop over messages, so the messages have to be somewhere. Three places, in increasing order of what they let you do:

In the process. Simple, fast, and it means one instance owns a session forever. Any restart, deploy or scale-in loses conversations, and load balancing has to pin a user to an instance — which is the constraint that eventually forces the change.

In a session store. The loop reads the history at the start of a turn and writes it back at the end. Any instance can serve any turn, restarts are survivable, and the store’s latency is now on every turn. This is the default that fits almost everything.

In the client. The caller sends the whole history each turn. The server is genuinely stateless and trivially scalable. The price is a request that grows on the wire every turn, and a client that can edit the history, which Chapter 36 has opinions about. The token bill is the same for all three options (see the margin note).

The token cost of carrying a history is easy to underestimate, and it is not specific to where the history lives. A twenty-turn conversation resends nineteen turns on the last request, so the last prompt grows linearly and the session’s total prompt tokens grow roughly quadratically — the same arithmetic whether the client sent the history or a server-side session store did, because what you pay for is what reaches the model. What client-held state changes is who can edit it, not what it costs — and Chapter 31’s measurements show prefill dominating latency well before that becomes a bill problem. Prompt caching helps exactly here, provided the history is append-only and nothing volatile sits at the top.

33.3 Agents that outlive a request

Once a task takes minutes, holding an HTTP connection open for it stops being reasonable: clients time out, load balancers cut idle connections, and a deploy kills whatever was in flight.

The shape that works is the one every long-running system converges on. Accept the request, write a job, return an id. A worker picks the job up. The client polls or subscribes for progress. Three things that are easy to skip and painful to add later:

  • Progress that means something. “Step 4 of an unknown number” is not progress. Emit the step, the tool just called, and tokens spent so far — Chapter 34 is where those become traces.
  • Cancellation that actually stops work. A cancel flag the loop never checks is decoration. The loop must read it between steps, and the tools must be cancellable or at least idempotent, because a cancel arriving mid-tool-call cannot undo what the tool has already started.
  • A dead-letter path. Jobs that fail repeatedly must land somewhere a human sees, rather than being retried forever — the same problem as Section 33.5, in a different form.

33.4 Durable execution, and what replay demands of your code

If a worker crashes halfway through a twelve-step agent run, what happens? The naive answer — start again — re-executes every tool call, including the ones that already sent something.

Durable execution engines answer this by recording each completed step and replaying the history after a crash, so completed work is not repeated. Temporal and Restate are the usual examples, and the idea long predates agents.

One caveat decides how much this buys you. A step counts as completed only once its result is recorded, so a step that was running when the worker died is run again. If the tool acted and the worker crashed before the engine recorded the result, the effect happens twice. Temporal’s documentation says this directly: an activity is recorded as completed once but may be executed more than once. Durable execution removes the re-running of finished steps; it does not remove the need for idempotent tools (Section 33.6).

The part that matters for adopting one is not the engine, it is the contract it imposes on your code:

Replay means your workflow function runs again from the top, and every completed step returns its recorded result instead of executing. For that to be sound, the function has to be deterministic: same history, same sequence of decisions. Reading the clock, generating a random id or calling an API directly from workflow code all break it, because the replay would take a different path than the original.

This is awkward for agents, and it is worth being blunt about it. An LLM call has to be treated as an external, potentially non-deterministic operation — even at temperature 0, where batching, hardware and a silent model update all move the output. The resolution is structural rather than clever: the model call is a step — an activity, in Temporal’s vocabulary — whose result is recorded and replayed, not a decision made inside workflow code. The workflow orchestrates; the non-determinism lives in the steps it records.

That is a real constraint on tool design (Chapter 24), and adopting a durable engine after the fact usually means rewriting the loop rather than wrapping it.

33.5 Retrying a step is not retrying a request

Chapter 32 dealt with retries against a provider: the failure there is amplification, and the fix is backoff, jitter, circuit breakers and a retry budget. This section is about retrying an agent step, where the failure is different and worse.

A provider retry that succeeds twice costs you money. An agent-step retry that succeeds twice sends the email twice.

The interesting question underneath is one people assume rather than measure: when you retry the step, do you get the same tool call? The measurement: a model was asked to emit one send_email call, eight times, for each of three tasks under three sampling conditions. The three tasks leave the model progressively more to decide: clear names the recipient, vague is loosely worded, and choice (quoted below) leaves the recipient to the model. In the seed column, 0 is a fixed seed and None means no seed was set. identical is byte equality of the whole tool-call payload (the tool name and all its arguments) across all eight retries; distinct to and distinct subj count unique values of the recipient and subject arguments only.

    task  temp   seed  identical  distinct to  distinct subj
   clear   0.0      0       True            1              1
   clear   0.0   None       True            1              1
   clear   0.7   None      False            1              1
   vague   0.0      0       True            1              1
   vague   0.0   None       True            1              1
   vague   0.7   None      False            1              6
  choice   0.0      0       True            1              1
  choice   0.0   None       True            1              1
  choice   0.7   None      False            5              2

Read the last row. The task was “the pump failed, notify whoever needs to know”, and across eight retries at temperature 0.7 the model chose five different recipients:

  3x  maintenance@factory.com
  2x  maintenance@company.com
  1x  pump_repair_team@example.com
  1x  maintenance-team@company.com
  1x  engineering@company.com

A retry did not repeat the action. It performed a different one. Every address is invented, and a retry loop with no idempotency protection would have mailed several of them.

Read five as a floor, not an estimate. Eight draws cannot bound how many addresses the model is willing to invent. Ecologists face the same problem when counting species, and their Chao1 estimator uses the values seen exactly once (\(f_1\)) and exactly twice (\(f_2\)) to estimate how many were never seen. With this distribution — one address three times, one twice, three once each, so \(f_1 = 3\) and \(f_2 = 1\) — the classic form \(5 + f_1^2/(2f_2)\) gives 9.5, and the small-sample bias-corrected form \(5 + \frac{n-1}{n}\cdot\frac{f_1(f_1-1)}{2(f_2+1)}\) gives about 6.3 with \(n = 8\) draws. Eight draws are far too few to choose between them, but both are above five, because three singletons in eight draws is the signature of a long tail you have not sampled. In the other direction, eight identical outputs at temperature 0 give a Wilson 95% interval of about [68%, 100%] on the repeat rate, so “byte-identical eight times” is compatible with a true repeat rate near two thirds. Neither correction changes the design conclusion. Both change what may be claimed: this establishes that the failure exists and reproduces easily, not how often it happens on your endpoint.

Three things this does not say, all of which matter.

Temperature 0 is not the fix, it is a different risk. At temperature 0 all eight retries were byte-identical for every task — including choice, which deterministically invented maintenance@company.com. Reproducible and still wrong. Determinism makes a retry repeatable; it does not make the underlying decision correct.

And a condition this table is missing settles a related question. Chapter 17 ran the same check at larger scale and found that a fixed seed at temperature 0.7 also produced identical output on all 30 documents across ten repeats. So the axis here is not really temperature; it is whether the sampler is pinned. That is worth knowing before you reach for temperature 0 as a reliability lever, because it costs you diversity you may want and buys you something a seed buys more cheaply — on a server that supports a seed and batches your request the same way each time. Many hosted APIs offer no seed, or only a best-effort one, for the reason in the next paragraph.

And this determinism is a property of a single-stream local server, not a guarantee. The load-bearing condition is batch invariance: the reduction kernels behind matmul, normalisation and attention take different code paths at different batch sizes, so the same prompt in a batch of one and a batch of seventeen can decode differently even at temperature 0 (He and Thinking Machines Lab 2025). Hosted APIs batch your request with strangers. Treat temperature 0 as a variance reduction, never as an idempotency mechanism.

The pattern in the table is worth more than the individual numbers, and it is finer than “temperature is the problem”. At 0.7 no task was byte-identical — including clear, which named the recipient and still varied in the body. What ambiguity changed is which field moved: clear kept the recipient and the subject, vague produced six subject lines, choice produced five recipients. Temperature perturbs everything; the task decides whether the perturbation lands on a field that has a side effect. This is also why a key derived from any part of the payload is unsafe, not only one built from the recipient: which field moves depends on the task.

33.6 Idempotency keys, and where the key comes from

The standard fix for a duplicated side effect is an idempotency key: the caller supplies a key, the receiver records it, and a second request carrying a key it has already seen returns the first result instead of acting again.

The measurement above settles the design question that usually gets hand-waved:

The key cannot be derived from the model’s output. Hashing the tool arguments seems natural and fails exactly when you need it: the choice retries produced five different argument sets, so five different hashes, so five emails. A key computed from a non-deterministic value is not a key.

So derive it from something upstream and stable:

Four candidate keys, and the two that work
key derived from dedupes a retry? notes
Hash of tool arguments no changes when the model changes its mind — the failure this section measured
Random id per attempt no a new key every retry, which is the same as having none. A random id generated once per operation and stored before the first attempt is fine — that is what Stripe suggests; the mistake is generating it per attempt
(session id, step index) yes stable across retries of the same step — add a sub-call index when one step makes several side-effecting calls, or they collide
Upstream request id yes right when one request must produce at most one side effect end to end

The step index is stable only if it comes from a history that survived the crash — a session store or a durable engine’s record. If a restart runs the agent from scratch, the model may make the same call at a different step, and the key changes with it.

Scope the key to the tenant as well, so two customers cannot collide on the same logical operation id.

Three implementation notes decide whether it works in practice.

Decide what happens when the key repeats with different arguments. The tempting behaviour — return the stored result — is wrong: it silently answers a different request with an old answer. Stripe’s API stores a fingerprint of the request alongside the key and rejects a reuse whose parameters do not match, and that is the behaviour to copy. Store the arguments with the record, compare on replay, and fail loudly on a mismatch.

The record and the effect must commit together. When the effect is a row in your own database that is one transaction and the problem is solved. When the effect is a remote call — which is the case for every interesting tool, including the send_email this chapter keeps using — it is not, because an HTTP request cannot join your transaction. This is the dual-write problem, and the standard answer is the transactional outbox: in one transaction, write the idempotency record and a row describing the intended call; a separate publisher reads the outbox and performs the call at least once — it may repeat the call after a crash, but it never skips it — carrying the key so the receiver dedupes it. You have not removed the duplicate. You have moved it somewhere a key can absorb it.

That last step assumes the receiver honours idempotency keys. Payment APIs such as Stripe’s do; plain SMTP does not, and many email and messaging APIs do not either. Against such a receiver, at-least-once means “occasionally twice”. What remains is to make the window small — record “sending” before the call, and after a crash check the provider’s own log before sending again — and to know that the window exists.

The retention window must outlast your longest retry. Easy to set to five minutes and then discover was needed for six hours, because the queue was backed up and the job was redelivered the next morning.

33.7 Compensation: there is no rollback for a sent email

Idempotency stops an action happening twice. It does nothing about an action that should not have happened once.

Databases give you transactions. The tools an agent calls mostly do not, so the pattern is a compensating action: for each side effect, the operation that undoes it — and the discipline is to write that column before shipping the tool, not after the incident.

Compensations, and the two rows with no real undo
tool compensation honest?
Create a record delete it mostly — the row goes, but anything that already fired on its creation (a webhook, a downstream event, a sent notification) does not un-fire
Charge a card refund mostly — the fee and the statement line survive
Book a slot cancel usually — but anyone turned away while the slot was held stays turned away
Post to a channel delete the message no — people read it
Send an email nothing no

The bottom two rows are the design constraint. When a tool has no compensation, the control has to move earlier — an approval gate before the action rather than an undo after it, which is Chapter 37’ subject. That is a product decision disguised as an engineering one, and the compensation table is what forces somebody to make it deliberately.

A useful habit when writing tools (Chapter 24): give every side-effecting tool a dry-run mode. It makes the compensation table testable and it gives the agent a way to check its own plan before committing to it.

33.8 Three ceilings every agent needs

Chapter 23 measured why they are needed: with one protocol, sixty percent of the easiest tasks never stopped on their own and ran until the step cap — although each run already had the answer at step 0 — so an accuracy column that looked like a reasoning result was really a termination result. Three limits, and they catch different failures:

Three ceilings, three different failures
ceiling catches what to do at the limit
Steps the loop that keeps calling tools stop, return partial work, mark it unfinished
Tokens the run that stays cheap per step and expensive overall stop and report spend — Chapter 32 enforces a related budget per virtual key
Wall clock the tool that hangs, which neither other limit sees cancel, and make sure cancellation reaches the tool

All three are needed because each is blind to the others’ failure. A step cap does not stop one tool hanging for an hour; a wall-clock timeout does not stop sixty fast steps burning a budget; a token cap does not stop a loop that reads the same small file forever.

And hitting a ceiling is a result, not an error to swallow. It must appear in the response, in the trace, and in your evaluation as a distinct outcome — Chapter 30’s termination rate is exactly this number, and an agent that looks 85% accurate while hitting its cap on a third of runs is not an 85% accurate agent.

33.9 Shipping a change: versions, canaries and rollback

Everything so far is about surviving a failure while a run is in progress. The other common way an agent breaks in production is a deploy: a new prompt, a new model version, a reworded tool description. Four habits make a deploy survivable.

Version the whole behaviour, not only the code. What an agent does is set by its code, its prompts, the pinned model version (Section 32.6), the tool definitions, and settings such as temperature and the three ceilings. Release them together, under one release id, and record that id on every trace (Chapter 34). A prompt edited in a dashboard without a new release id is a deploy that nobody can attribute or roll back — and the “step change that lines up with a release” that Chapter 35 diagnoses is visible only if the release was recorded.

Roll out to a slice first. A canary sends a small share of live traffic — say 5% — to the new release, and compares it with the old release on the same traffic: error rate, latency, cost per run, the share of runs that hit a ceiling, and whatever quality signal you have. The share grows in stages (5% → 25% → 100%) only while the canary is not worse, and traffic moves back automatically when a threshold is crossed. A shadow deployment — the new release runs on a copy of live inputs, but nobody is shown its output — is safe for a chatbot and dangerous for an agent, because a shadow agent with real tools sends real emails. Its side-effecting tools must run in dry-run mode (Section 33.7). What each of these rungs can and cannot detect is Chapter 35’s subject.

Agree the rollback trigger before the deploy. A service level objective (SLO) is a target over a time window, measured by a service level indicator (SLI) — for example, “99.5% of agent runs in a 30-day window finish without an error or a ceiling”. The 0.5% that may fail is the error budget: at 200,000 runs a month, 1,000 runs. A canary that burns that budget much faster than the old release is rolled back by rule, not after an argument during an incident. Choose the indicator with care: an HTTP 200 is not a successful run, because a run can return 200 and still have hit a ceiling or given a wrong answer (Chapter 34). Mind the size of the slice, too. A 5% canary on 200,000 runs a month sees about 330 runs a day, so at a 0.5% baseline it expects fewer than two failures a day. That is enough to catch a release that breaks badly, and far too few to notice one that is slightly worse. An A/B test run on the same 5% share has the same power problem: small effects need a larger share, a longer run and a sample size worked out in advance.

Rollback undoes the code, not the effects. Rolling back restores the old behaviour for new runs. It does not unsend the emails the bad release sent — that is the compensation table — and it does not settle what happens to runs that are still in flight. A durable workflow that started under the new code and is replayed by the old code (or the reverse) can take a different path, which is the determinism rule of Section 33.4 again. Durable engines provide workflow versioning for this (in Temporal’s Python SDK, workflow.patched()); the simple rule is to let every in-flight run finish on the version it started with.

33.10 Intuition checks

Answer each one before you open it. Two of them turn on the retry measurement in this chapter — what temperature 0 does and does not buy you, and why the idempotency key cannot come from the model — and the rest are the questions that decide whether a working prototype survives being deployed at all.

You deployed a loop onto a request handler.

Serverless platforms cap execution time, many load balancers cut a connection that stays idle too long, and an agent loop is not a request: it can take minutes, and the cap arrives mid-run — after some tools have already executed. The symptom is a truncated run with real side effects and no result.

If the work reliably finishes inside a bound you control, raising the timeout to match is a legitimate fix and much cheaper. Past that, the fix is structural: accept the request, enqueue a job, return an id, and let a worker run the loop. That change also gives you the place to put progress, cancellation and a dead-letter path.

No, and it confuses two different properties.

Measured here, temperature 0 made eight retries byte-identical for every task — including one where the model deterministically invented a recipient address. Reproducible and wrong. Determinism makes a retry repeatable; it does not make the decision correct, and it does nothing about the effect having already happened.

It is also not a guarantee. Hosted APIs do not promise bit-identical output at temperature 0, because batching and reduction order vary with what else is in flight. Idempotency has to come from a key, not from sampling settings.

Because the arguments are the model’s output, and the model’s output is exactly what varies between retries.

Measured: for the task “the pump failed, notify whoever needs to know”, eight retries at temperature 0.7 produced five different recipient addresses. Five argument sets, five hashes, five emails — the key would have deduped nothing at the moment it was needed.

Derive the key from something upstream and stable: (session id, step index), or the request id when one request must produce at most one effect.

Usually the one that talks to a person.

Creating a record can be deleted; a charge can be refunded, allowing for the fee and the statement line; a booking can be cancelled. But deleting a posted message cannot undo the fact that people have already read it, and a sent email cannot be undone at all.

That is why the compensation table has to be written before shipping rather than after an incident. Where the compensation column says “none”, the control moves earlier — an approval gate before the action (Chapter 37) — and somebody outside engineering has to agree to it.

Something non-deterministic is running inside workflow code.

Replay re-runs the workflow function from the top, returning recorded results for completed steps. That is only sound if the function takes the same path every time, so reading the clock, generating a random id, or calling an API directly from workflow code all break it.

For agents the specific trap is obvious once stated: the LLM call is the least deterministic thing you have, so it must be a recorded step, not a decision made in workflow code. The workflow orchestrates; the non-determinism lives in the steps it records.

Yes, because they are blind to each other.

A step cap does not stop one tool hanging for an hour — that is zero additional steps. A wall-clock timeout does not stop sixty fast steps burning a budget. A token cap does not stop a loop that reads the same small file forever.

Set all three, and treat hitting any of them as a distinct outcome that appears in the response, the trace and the evaluation. Chapter 30’s termination rate is that number, and an agent reported as 85% accurate while capping out on a third of runs has not been measured honestly.

33.11 Interview questions

On sourcing — the same caveat as Chapter 2. The questions below recur in the two credible open sources this book uses throughout, plus the ones this chapter’s own measurement makes answerable.

Testing whether you understand the failure

Q: Your agent’s worker crashes mid-run. What did it leave behind?

Whatever the completed tool calls already did, with no record that they completed. A naive restart re-executes them. Name the two answers: durable execution, which records each step and replays the history so completed work is not repeated; or idempotency keys on the side-effecting tools, so re-execution is harmless. Most systems need the second regardless, because durable engines run each step at least once: a crash after the tool acted but before the engine recorded the result re-runs that step.

Q: Is retrying an LLM step idempotent?

Not by default, and not for the reason people expect. Beyond the effect having already happened, the retry may not even repeat the same action — measured here, a vague task produced five different recipient addresses across eight retries. Temperature 0 made retries byte-identical on a local server, but that is a variance reduction, not a guarantee, and hosted APIs do not promise it.

Q: Where does an idempotency key come from?

Somewhere upstream and stable — (session id, step index), or the request id. Explicitly not a hash of the tool arguments, because those are the model’s output and vary between retries. Then two details that decide whether it works: the key must commit atomically with the effect — which for a remote call means an outbox, since an HTTP request cannot join your transaction — and the retention window must outlast your longest retry.

Testing whether you have shipped one

Q: Design deployment for an agent whose tasks take twenty minutes.

Queue-backed, because a twenty-minute HTTP request does not survive a deploy or a load balancer. Accept, enqueue, return an id; a worker runs the loop; the client polls or subscribes. Then the three things that are painful to add later: meaningful progress, cancellation the loop actually checks between steps, and a dead-letter path so failures reach a human instead of retrying forever. Add the three ceilings and make hitting one a reported outcome.

Q: Where does the conversation live, and what does that cost?

In the process is simplest and pins a user to an instance, so restarts lose sessions. A session store is the default: any instance serves any turn, at the cost of the store’s latency on every turn. Client-held history makes the server stateless but lets the client edit the history, which is a security question (Chapter 36). None of the three changes the token bill: the model is sent the whole conversation every turn wherever it is stored, so prompt tokens grow over the session — Chapter 31’s cost arithmetic.

Q: An agent posted a wrong message to a customer channel. What is your review about?

Not the model. Whether that tool had a compensation, whether the compensation is real — deleting a message people already read is not an undo — and if it is not, why the action was not behind an approval gate. Then whether it could have been a duplicate rather than a decision, which is the idempotency question, and whether the run hit a ceiling and was reported as a success.

Q: How would you roll out a new prompt for an agent that sends emails?

As a release: the prompt, the pinned model, the tool definitions and the settings under one id recorded on every trace. Then a canary on a small slice, compared with the old release on error rate, ceiling hits, cost and a quality signal, with a rollback trigger taken from the SLO’s error budget and agreed in advance. Not a plain shadow deployment, because a shadow agent with a real send_email sends real mail — shadow only with tools in dry-run mode. And say what rollback cannot do: it does not unsend what the canary already sent.

33.12 The 2026 toolbox

Almost nothing in this chapter needed a library — the measurement was 72 calls to a local model, and everything else is a property of your own loop. The second table is the one that matters, and it splits cleanly in two: durable execution engines that impose the determinism contract described above, and plain task queues that do not.

The stack these numbers came from. Python 3.14.5, Apple Silicon
library version used what it is for here
ollama 0.24.0 serving llama3.2:3b for the retry-determinism measurement

Worth knowing about, not used above:

The wider ecosystem, versions as of writing
project version when you would reach for it
temporalio 1.32.0 durable execution with replay; the mature option, and the one whose determinism contract this chapter describes
restate-sdk 1.0.4 durable execution with a lighter operational footprint
dbos 2.31.0 durable workflows backed by Postgres, when you already run one
prefect 3.8.4 workflow orchestration with retries and observability, closer to data pipelines than to agents
celery 5.6.3 the long-standing Python task queue when you need queue-backed workers and nothing more
rq 2.12.0 a smaller queue on Redis, when Celery is more than the job needs
ray 2.58.0 distributed execution when the agent fans out across machines

33.13 Lab: make a step safe to retry

Take an agent with at least one side-effecting tool. If you do not have one, a send_email that appends to a file is enough — the point is to see the duplicate.

Step 1 — reproduce the duplicate

Run the agent, kill the worker after the tool executes but before the result is recorded, and restart. Count the effects. You should get two. If you cannot make it happen on purpose, you cannot claim it will not happen by accident.

Step 2 — measure your own retry determinism

Ask your model for the same tool call ten times at your production temperature. Count distinct argument sets. Then repeat with one deliberately under-specified task. The gap between those two numbers is how much your prompt is protecting you.

Step 3 — add an idempotency key from the right place

Key on (session id, step index), not on the arguments. Repeat step 1. The second execution must return the first result without acting.

Step 4 — put the key and the effect in one transaction

A file append cannot join a database transaction, so for this step make the fake send_email insert a row into the same database (SQLite is enough) that holds the idempotency records, and write both in one transaction. For a real remote call, use the outbox described above instead. Then re-run step 1, killing the worker between the effect and the record. If the duplicate comes back, the write was not atomic — which is the bug this step exists to expose.

Step 5 — write the compensation table

Every side-effecting tool, its compensating action, and an honest note on whether the compensation is real. Circulate it. The rows with no compensation are the ones that need an approval gate, and that is a conversation with somebody outside engineering.

Step 6 — set the three ceilings and trip each one

Steps, tokens, wall clock. Write a task that trips each ceiling separately — a loop, an expensive run, a hanging tool — and confirm the agent stops, reports which ceiling it hit, and that the outcome is distinguishable from success.

What to hand in

The duplicate count before and after the key; your distinct-argument counts for a clear and a vague task; the compensation table with its honest column; and evidence that each of the three ceilings fires and is reported.

33.14 Exercises

  1. Force the duplicate. Kill a worker between a tool executing and its result being recorded. Count the effects. Then add an idempotency key and show the count returns to one.

  2. Your determinism number. Ask your production model for the same tool call ten times at your production temperature, and count distinct argument sets. Repeat with a deliberately vague instruction. Report both.

  3. The compensation table. One row per side-effecting tool in a system you work on: the tool, its compensation, and whether the compensation is honest. Identify the rows that need an approval gate instead.

  4. Atomicity. Find out whether your idempotency record commits together with the effect it guards. If the effect is a remote call it cannot, so implement the outbox instead and show that the duplicate is now absorbed by the receiver’s key rather than prevented by your transaction. Either way, construct the interleaving that produces a duplicate and say how likely it is at your traffic.

  5. Trip all three ceilings. Write three tasks that exhaust steps, tokens and wall clock respectively. Confirm each stops the agent, and that the three are distinguishable in your logs.

  6. The state decision, priced. For a twenty-turn conversation, compute the total prompt tokens and use Chapter 31’s arithmetic to turn them into latency and cost. That number is the same for client-held history and a session store, so price what does differ: the store’s latency on every turn, what a deploy loses, and what a client that can edit its history can do. Then say which you would choose and why.

  7. Cancellation, honestly. Send a cancel to a running agent and measure how long until work actually stops, and whether the in-flight tool call completed. If nothing checks the flag between steps, you do not have cancellation.

  8. Size the canary. Write an SLO for an agent you know, with its SLI stated precisely, and compute the monthly error budget. Then compute how many runs a 5% canary sees per day, and how many failures it should expect at your baseline failure rate. How much worse would a release have to be before a day of canary traffic could tell?

33.15 Where to go deeper

Before reading: pick the one tool in your system whose effect you would least like to see happen twice, and find out today whether anything stops it. That answer decides which of these you need first.

This chapter rests mostly on operational practice. The first table is the operational literature and the engines that implement it, and nothing in it is a research result; the second is the one research source the chapter cites.

Beyond this chapter
resource what it is
Making retries safe with idempotent APIs the clearest write-up of the key, the record and the transaction boundary this chapter argues for
Stripe’s idempotent requests the same design as a shipped public API, including the retention window most people set too short
Handling Overload the client-side retry budget, which stops the retries of failing jobs from turning into an outage
Service Level Objectives the Google SRE book’s chapter on SLIs and SLOs; its error budget is the rollback trigger of Section 33.9
Temporal’s workflow determinism rules the replay contract stated precisely; read this before adopting any durable engine, not after
temporalio/sdk-python the workflow/activity split in code, which is where the LLM call has to land
restate / dbos-transact-py two lighter takes on durability, useful for seeing which parts of Temporal’s contract are essential
The saga pattern compensation as a named pattern, with the failure modes the compensation table above only gestures at
celery / ray the queue-backed shape without durability, when steps are already idempotent
The primary source
topic source
why temperature 0 is not deterministic on batched inference He and Thinking Machines Lab (2025)

33.16 What to remember

  • An agent loop is not a request handler. It takes minutes, so a platform with an execution ceiling will kill it mid-run, after some tools have already fired. Once tasks outlive a request the shape is queue-backed: accept, enqueue, return an id, and add progress, cancellation and a dead-letter path while it is still cheap to add them.

  • Where the conversation lives is an operational and trust decision, not a token-cost one. In-process pins a user to an instance and loses sessions on deploy. A session store is the default, with its latency on every turn. A client-held history makes the server stateless and lets the client edit it. Wherever it lives, the model receives the whole history each turn, so prompt tokens grow every turn — Chapter 31’s arithmetic, and prefill dominates latency long before it dominates the bill.

  • Durable execution imposes determinism on your workflow code. Replay re-runs the function and returns recorded results for completed steps, so clocks, random ids and direct API calls inside workflow code break it. The LLM call is the least deterministic thing you have, so it belongs in a recorded step, not in workflow code. Adopting an engine later usually means rewriting the loop. And a step that was running at the crash runs again, so the tools still need to be idempotent.

  • Retrying a step is not retrying a request. Chapter 32’s retry problem is amplification; this one is duplication. And the retry may not even repeat the same action: asked to notify whoever needs to know, a model produced five different recipient addresses across eight retries at temperature 0.7.

  • Temperature 0 is variance reduction, not idempotency. All eight retries were byte-identical at temperature 0 in this measurement — including the run that deterministically invented an address. Reproducible is not correct, and hosted APIs do not promise bit-identical output at temperature 0 anyway, because batching and reduction order vary with what else is in flight.

  • Derive the idempotency key from upstream, never from the model’s output. Hashing tool arguments fails exactly when it is needed — five argument sets means five hashes means five emails. Use (session id, step index) or the request id, commit the record together with the effect — via an outbox when the effect is a remote call, because an HTTP request cannot join your transaction — and give it a retention window longer than your longest retry.

  • Write the compensation table before shipping the tool. A record can be deleted, a charge refunded, a booking cancelled. Deleting a posted message cannot undo its having been read, and a sent email cannot be undone at all. Where the compensation column says “none”, the control moves earlier — an approval gate rather than an undo — and that is a decision for somebody outside engineering.

  • Set all three ceilings, because each is blind to the others. A step cap does not stop a hanging tool; a timeout does not stop sixty cheap steps burning a budget; a token cap does not stop a loop reading the same file forever. Hitting one is a result: it belongs in the response, the trace and the evaluation, and Chapter 30’s termination rate is where an agent that looks accurate while capping out gets caught.

  • Ship a change as one versioned release, to a slice first. Code, prompts, pinned model, tool definitions and settings share one release id on every trace. A canary with a rollback trigger taken from the SLO’s error budget catches a release that breaks badly; a shadow agent needs its tools in dry-run mode; and a rollback restores the code, not the effects the bad release already had.