23  Agent Anatomy: Loop, State, and Control

Chapter status: drafted. The measurements run qwen3.5:9b through ollama 0.24.0 against a database of invented projects, so a correct answer can only have come from a tool call. Tool design — schemas, descriptions, error messages — belongs to Chapter 24 and is deliberately held back here; this chapter is about the loop itself. Software: NumPy 2.5.2 on Python 3.14.5.

Learning objectives. After this chapter you can:

23.1 The loop, and nothing else

An agent is not magic: it is an agent loop — a while-loop around a language model with access to functions.

Agent loop — the model chooses an action, the runtime executes it, the result is appended to the transcript, and the model chooses again. It ends when the model stops calling tools. ↩︎

Everything else — planning, memory, orchestration — is engineering around that loop. So here is the whole thing, with nothing removed:

def agent(question, tools, model, max_steps=10):
    messages = [{"role": "system", "content": SYSTEM},
                {"role": "user",   "content": question}]

    for step in range(max_steps):                 # 1. the budget
        reply = model.chat(messages, tools=tools) # 2. the model decides

        if not reply.tool_calls:                  # 3. the exit
            return reply.content

        messages.append(reply.model_dump())       # 4. remember the decision
        for call in reply.tool_calls:             # 5. the runtime acts
            try:
                args = json.loads(call.function.arguments) # a JSON string, not a dict
                result = tools[call.function.name](**args)
            except Exception as e:
                result = f"ERROR: {e}"            # 6. failure is an observation
            messages.append({"role": "tool",
                             "tool_call_id": call.id,
                             "content": str(result)})

    raise StepLimitExceeded(question, messages)   # 7. it did not finish

That is an agent. Fourteen statements, the def included — count them — and this loop is the pattern underneath the frameworks you will meet. (The numbers in the comments are labels for the discussion below, not statement counts.) model.chat is a schematic client, and tools does double duty here, standing for both the schema list sent to the model and the name → function lookup; in real code keep them apart (say, tool_schemas and tool_impls). What frameworks add is mostly opinions about the lines marked 1, 4 and 7, plus the state management, validation, scheduling and recovery that a fourteen-statement loop does not have.

Three details in there are not decoration. First, the arguments. On an OpenAI-compatible endpoint call.function.arguments arrives as a JSON string, so **call.function.arguments raises TypeError. Because that happens inside the try, the model receives an error message that blames its own call for the runtime’s bug. The reverse trap exists too: Ollama’s native /api/chat and Anthropic’s Messages API hand you an already-parsed object, and there it is json.loads that raises. Check which one your client returns. Second, the tool_call_id is what makes parallel tool calling expressible at all: five results come back in one turn and only the id says which answers which. Third, the exception carries messages, because whatever you decide to do when the cap fires, you can only do it if the transcript survives the raise.

The message shapes are the provider’s protocol, and they differ in detail though not in idea. Before any of this, the model is sent the list of tools — for each one a name, a description and a JSON Schema for its arguments, the same schemas Section 18.4 constrains. In the OpenAI-style format used above, the assistant message carries a tool_calls list and each result goes back as a separate role: "tool" message with the matching tool_call_id. In Anthropic’s format, the assistant message contains tool_use blocks (an id, a name and an input object), the response’s stop_reason is "tool_use", and the results go back as tool_result blocks, each with the matching tool_use_id, inside the next user message. The loop is the same in both; only the field names change.

The loop, and where each failure lives
question TRANSCRIPT system + user + every decision and observation this is the entire state MODEL chooses one action no tool call answer tool calls RUNTIME executes, or catches the exception as text append the result STEP CAP the admission that it may not stop WHERE THINGS GO WRONG the exit is a prediction nothing checks whether the task is done the transcript only grows cost rises per model call; mistakes stay the runtime is the boundary model emits arguments, not actions

Every framework you will meet is this diagram with opinions about three of its edges: what goes into the transcript, when the loop is allowed to stop, and what the runtime is permitted to execute.

Six things in it are worth naming, because each is a decision people make by accident.

The model never executes anything. It emits arguments; your runtime decides whether to run them. This is the entire security boundary, and Chapter 36 is about what happens when people forget that it is the only one.

messages is the state, and it only grows. There is no other memory. Every observation is appended, so the context gets longer with every step — which makes cost grow superlinearly in the number of steps, and eventually collides with the context window. The fourteen statements do not handle that collision: when the window overflows, model.chat raises outside the try, and the run dies with the transcript unsaved. A production version wraps that call too, persists messages before re-raising, and hands the overflow to a compaction step — which is where Chapter 25 starts.

The line marked 3 is the termination condition, and it is a model decision. The loop ends when the model chooses not to call a tool. Nothing checks whether the task is complete. That is the single most surprising property of this architecture for someone arriving from ordinary software: the stopping rule is a prediction. It also catches more than a finished answer. A reply that was cut off by the output-token limit, or blocked by a content filter, has no tool calls either, so this loop returns the fragment as if it were the answer. Check the reason the provider gives for stopping (finish_reason or stop_reason) before you trust reply.content.

The line marked 6 turns an exception into text. A failed tool call is not a crash; it is an observation the model gets to react to. Whether it reacts well is Chapter 24’s subject, and it is largely decided by what that string says. Note also what else that except swallows: a KeyError from tools[call.function.name] is your bug, and the model receives it as though the tool had failed. A runtime worth trusting distinguishes tool failures from failures of the harness — your own code around the model — before it writes the observation.

The line marked 7 is the honest part. A step cap is not a performance tuning knob,

Step cap — the maximum number of loop iterations. Not a tuning parameter but an admission that the loop’s exit is a model decision and may never be taken. ↩︎

it is an admission that the loop may not terminate on its own. What you do when it fires — retry, escalate, return partial work — is a product decision that deserves to be made deliberately rather than by whatever the framework defaults to.

There is no plan. The model chooses one action, sees one result, and chooses again. Anything that looks like planning is either the model writing a plan into its own context and then following it, or your code imposing structure from outside. Both are legitimate; neither is in the loop. The distinction is worth keeping: writing a plan into the transcript is in-context planning, and it works about as well as the model’s coherence over the run. Imposing a decomposition your code then executes is not planning at all — it is a pipeline, and Section 23.4 argues it is usually the better trade.

Three ways to make the exit a condition again

The exit does not have to stay a prediction. Three cheap moves put a checkable condition back into the loop, in increasing order of intrusiveness.

Make finishing a tool. A submit_answer(answer) in the schema turns “I am done” into a call the runtime can recognise, so the loop terminates on an observable event rather than on an absence. For the exit to become a real condition, a plain-text reply must stop counting as one: where the API offers it, force a tool call on every turn (tool_choice: "required" in OpenAI’s API, {"type": "any"} in Anthropic’s). A finish tool also fixes the problem of the text protocol measured in Section 23.3 directly: there the answer command CEVAP (“answer”) competes with the search command ARA (“search”), and it stops competing once both are the same kind of thing.

Detect repetition. If a call is byte-identical to one already in the transcript, the loop has probably stopped making progress — with a real exception to carve out: polling a job, re-reading a file the agent just wrote, or re-querying data that changes are all legitimate repeats. Exempt the tools where that is the point, and apply the check to the rest. Return an observation that says so rather than the same result again — the chapter’s own worst failure below is the same call repeated until the step cap, and one line of bookkeeping catches it.

Bound three budgets, not one. Steps, wall-clock seconds and tokens fail differently: a run can stay under ten steps and still spend a minute or fifty thousand tokens. Each needs a defined behaviour when it fires, and the fourteen statements above bound only the first.

It is worth writing this loop yourself once, in whatever language you use, against two real tools. It takes an afternoon and it permanently changes how you read framework documentation — every abstraction you meet afterwards is recognisably a wrapper around these fourteen statements, and you can ask what it adds instead of what it does.

What we will look at. First what the loop’s state actually is — a transcript that only grows — and the three consequences that fall straight out of that, including why “just retry it” is a far more expensive suggestion here than in ordinary software. Then a measurement of how far the loop gets as tasks need more steps, where the surprise is that tool use stays perfect while accuracy collapses. Then the same loop against a hand-rolled text protocol, which fails worst on the easiest task rather than the hardest. And finally the decision that should have been made before any of it: whether this task wants a loop at all.

23.2 The state is a transcript, and that is the whole problem

Ask what this agent’s state is and the honest answer is uncomfortable: it is the list of messages, and nothing else. There is no working memory, no variable bindings, no scratch space the model can address. Everything the agent knows at step 7 is whatever survived into the transcript by step 6.

That is a property of the minimal loop, not a law about agents. Three of the chapters that follow buy their way out of it: Chapter 25 moves state into files the agent reads back, Chapter 26 gives it a store that outlives the run, and Chapter 28 hands sub-tasks to workers with transcripts of their own. Read the three consequences below as what you get by default, and as the reason those chapters exist.

Three consequences fall directly out of that, and they explain most of what goes wrong in practice.

Cost grows with the number of model calls, not the number of tool calls. Each model call re-sends the whole transcript, so a genuinely sequential run of n steps costs roughly the sum of the first n context lengths — quadratic rather than linear. The saving grace, measured in Section 23.3, is that a model which emits several tool calls at once collapses many steps into one, and in this chapter’s measurement that is the largest lever on what the agent costs.

Nothing is ever forgotten, including the mistakes. A wrong tool call and its error message stay in context for the rest of the run, and the model continues to condition on them. This is why an agent that goes wrong often keeps going wrong in the same direction — the evidence of its earlier confusion is now part of what it is reasoning from.

There is no rollback of the transcript by default. In ordinary software you would retry a failed step from a clean state. Here the failure is in the state, and this loop keeps every message. You can build the rollback — snapshot the message list before a step and truncate back to it — and that is cheap and worth doing. What no snapshot undoes is the side effects: the email that went out, the row that was deleted, the payment that cleared. Keep the two apart in your head, because only one of them is under your control, and it is why “just retry it” is a more expensive suggestion for an agent than for an HTTP request.

One thing softens the quadratic term and it is worth knowing before you optimise anything else. Every request in an agent loop begins with the whole previous request, unchanged, which is the pattern prompt caching was built for: on current price sheets a cached read often costs around a tenth of a fresh one. The exact discount varies by provider and model, some providers charge extra to write the cache, and by default a cache entry expires after a few idle minutes. The quadratic does not go away; its constant shrinks, by up to an order of magnitude — and the corollary is uncomfortable, because compacting a transcript rewrites the prefix and invalidates the cache. Chapter 25’s compaction is therefore not free, and Chapter 31 has the mechanics.

Which is why Chapter 25 exists as a chapter at all. Once you see that the transcript is the state, deciding what goes into it — and what gets summarised, dropped, or never added — stops being tidying and becomes the main design activity.

Control: who decides what happens next?

The transcript settles what the agent knows. It does not settle who gets to choose the next action, and that is a separate axis with a spectrum on it — choosing a point on that spectrum deliberately is most of the engineering.

Five levels of autonomy
who chooses the next action when it is right
Fixed pipeline you, at design time the steps are known and stable. Cheapest, fastest, and testable
Router the model, once, from a small set a few known paths, and the input decides which
Constrained loop the model, each step, from a restricted toolset with a step cap the path varies but the space is bounded
Open loop the model, each step, with broad tools genuinely open-ended tasks. The most capable and the least predictable
Loop with an approval gate the model each step, but named tools stop for a human before executing the action is irreversible: money moves, data is deleted, a message is sent

The gate belongs in the runtime, not the prompt: a model can be talked out of asking for permission, a requires_approval flag on the tool cannot. Where that gate goes and who is on the other side of it is Chapter 37.

The last row is not a further step up in autonomy — an approval gate can be added to any loop — so read the first four rows as the scale. The mistake worth naming is starting with the open loop. An open loop is the most impressive thing to build and almost never the thing the problem needs, and because it works in a demo the cost only shows up later, in the form of a system nobody can debug.

Start at the top and move down only when a measurement forces you to.

23.3 How far does the loop get?

Everything above is architecture. The question a practitioner actually has is how reliable is this, and the answer depends on something measurable: how many steps the task needs.

The setup keeps everything else fixed. A database of invented projects — codes like ZRG-14, budgets no model could guess — reachable only through one tool. A question needs k lookups and one addition, for k from 1 to 5. Twenty-five tasks at each k, temperature 0, exact-match scoring. Two loops over the same model: the provider’s structured tool-calling API, and a hand-rolled text protocol where the model writes ARA(ZRG-14, butce) and the runtime parses it.

A run that hits the cap is scored wrong, not excluded — worth stating, because for the text protocol it is most of the story.

qwen3.5:9b, 25 tasks per step count, step cap 14
95% intervals are Wilson on n = 25

   loop  lookups  accuracy  95% CI          model calls  tool calls  ideal  hit cap   tokens
 native        1    100.0%  [86.7, 100.0]           2.0         1.0      1     0.0%      881
 native        2     96.0%  [80.5,  99.3]           2.0         2.0      2     0.0%      999
 native        3     80.0%  [60.9,  91.1]           2.0         3.0      3     0.0%    1,112
 native        4     68.0%  [48.4,  82.8]           2.0         4.0      4     0.0%    1,227
 native        5     40.0%  [23.4,  59.3]           2.0         5.0      5     0.0%    1,338

   text        1     40.0%  [23.4,  59.3]          11.1        10.7      1    60.0%    3,240
   text        2     76.0%  [56.6,  88.5]           6.5         5.7      2    24.0%    1,815
   text        3     52.0%  [33.5,  70.0]           7.1         6.4      3    28.0%    2,019
   text        4     60.0%  [40.7,  76.6]           5.3         4.3      4     0.0%    1,322
   text        5     28.0%  [14.3,  47.6]           7.0         6.1      5    12.0%    1,936

Twenty-five tasks per cell is a small sample and the intervals say so: 40% at five lookups is [23, 59], and 80% at three is [61, 91]. What this measurement supports is the direction — one lookup against five is far apart (Fisher exact, p = 3 × 10⁻⁶) — and not the exact shape of the curve between them. No pair of adjacent step counts separates on its own; the closest is 4 against 5 at p = 0.088. Read the intermediate points as a slope rather than as landmarks.

The decay curve, and what it costs

The tool calls are perfect and the answers are not

Read the native rows carefully, because the interesting column is not accuracy.

Tool use is flawless at every step count. Exactly k calls for a task needing k, zero malformed calls, zero runs that hit the step cap. The model always asks for the right thing. And accuracy still falls from 100% to 40% across the same range.

The loop is not what breaks. The reasoning over what the loop collected is. The agent retrieves five correct numbers and then adds them up wrong.

That is worth internalising because it redirects the debugging. When an agent fails, the instinct is to inspect the tool calls — and here the tool calls were never the problem. Aggregation over accumulated observations is.

Two rival explanations are worth taking seriously, and they end in different places. The first is context length. Truncation is unlikely: the native loop never averages more than 1,338 tokens per task, far inside the window, and Section 25.2 runs the same model on a similar task out to 11,048 tokens and stays between 88.3% and 100%. That makes running out of window an unlikely culprit. It does not rule out the effect of accumulated observations — how the model uses what it has collected, their format, what sits between them — which is a different thing from running out of room, and this design cannot separate it.

The second cannot be ruled out with this design, so it has to be stated. k controls two things at once — how many observations the loop must carry, and how many numbers have to be added in a single forward pass. Part of the fall may be arithmetic width rather than agency. The control that separates them is cheap and has not been run: the same 25 tasks with the five figures pasted straight into the prompt and no tool at all. Until that exists, read the curve as “reliability falls with the amount that has to be combined at the end”, and treat the agentic reading as the interpretation rather than the finding.

What the curve is not is steady per-step attrition. If every lookup succeeded independently with the same probability \(p\), a task needing \(k\) lookups would succeed with probability \(p^k\). Solving \(p^k\) = measured accuracy for two to five lookups gives \(p\) = 0.980, 0.928, 0.908 and 0.833 — falling, not constant — and a flat 0.90 per step would predict \(0.9^5 \approx\) 59% at five lookups against the 40% measured. The load per step is getting heavier, which is the same conclusion from a different direction. Externally, Kwa et al. (2025) measure how success falls as tasks get longer, at scale, by the time a task takes a human. Yao et al. (2024) measure a different dimension of reliability, consistency over repeated trials, with pass^k: the probability that an agent succeeds on all \(k\) independent attempts at the same task (here \(k\) counts trials, not lookups): the best function-calling agent they tested in 2024 (gpt-4o) succeeded on fewer than half the tasks, and in the retail domain its pass^8 was below 25%.

Parallel tool calling collapses the loop

The same table holds a second surprise, and this one is good news. The model calls column is 2.0 at every step count, because the model uses parallel tool calling.

Parallel tool calling — a model emitting several tool calls in one turn. It collapses many loop iterations into one, which makes it one of the largest levers on what an agent costs when the calls are independent. ↩︎

It does not take five turns to make five lookups; it emits all five requests in one turn, reads all five results, and answers. So the loop runs exactly twice however much work the task needs.

One distinction the name hides: the calls are parallel, the execution in the fourteen statements is not. The loop above runs them in a for, so batching cuts the token bill and the number of round trips, but not the wall-clock time spent inside the tools. Making that parallel too — an asyncio.gather over the calls — is a separate change, and it is the one that matters once your tools are network calls rather than dictionary lookups.

That has a large consequence for cost. Five times the work costs 1.52× the tokens — 881 to 1,338 — because the expensive thing is re-sending the transcript, and the transcript is only sent twice. A genuinely sequential loop pays that cost on every step, which is exactly what the text protocol does: 3.5× the model calls at k = 5, and up to 3.7× the tokens at k = 1.

This is also the honest limitation of the experiment. Because the five lookups are independent, the model can batch them — so what this measures is aggregation under load, not sequential planning. A task where the second step depends on the first result would force a real loop and, I would expect, worse numbers and a quadratic cost curve. That experiment is worth running before trusting any of these figures on a task shaped differently.

The structured API is not a convenience

Both loops in that table ran the same model over the same tasks; only the protocol differed. So the second block is a clean answer to a question that usually gets settled by taste: does the provider’s tool-calling API buy you anything a well-written prompt could not?

The text protocol is deliberately the shape Yao, Zhao, et al. (2023) introduced. In ReAct (“reason + act”) the model writes, in plain text, a Thought (a short piece of reasoning), then an Action (a tool name and its arguments); a parser extracts the action, the runtime runs it, and the result comes back as an Observation line, after which the cycle repeats. In 2022 there was no alternative — no provider exposed a typed tool schema — and the contribution was showing that interleaving reasoning with acting beat acting without reasoning, and that combining it with chain-of-thought gave the best question-answering results. So what follows is not “ReAct is bad”. It is what the typed transport added once it arrived, on one small model. Note too that this protocol has no Thought: step: it is ReAct’s transport without ReAct’s scratchpad.

The native loop is ahead at every step count — by 60, 20, 28, 8 and 12 accuracy points — while using about a third of the model calls on average (between a fifth and two-fifths, depending on the step count). But only the first of those gaps survives a test on 25 tasks: 25/25 against 10/25 is Fisher exact p = 3 × 10⁻⁶, while the 8-point gap at four lookups is 17 correct against 15 (p = 0.77) and means nothing. The text protocol’s own curve — 40, 76, 52, 60, 28 — is not even monotone. It does vary more than chance alone would explain (a χ² test across the five cells gives p ≈ 0.009), but not as a steady decay: its low points are one lookup, where most runs never stopped, and five. The two findings the totals cannot hide are the ones to keep: the native loop used about a third of the model calls, and it never once failed to terminate. The reason for both is visible in the transcripts rather than in the accuracies.

Here is a text-protocol run on the easiest task, one lookup:

SORU: BRS-26 projesinin bütçesi kaçtır?

  step 0: ARA(BRS-26, butce)      -> 3168      ← it already has the answer
  step 1: ARA(BRS-26, personel)   -> 7
  step 2: ARA(BRS-26, sure)       -> 8
  step 3: ARA(BRS-26, butce)      -> 3168
  step 4: ARA(BRS-26, butce)      -> 3168
  ...  repeats until the step cap

It got the answer at step 0 and never stopped. Sixty percent of the one-lookup tasks hit the step cap — the easiest condition had the worst termination behaviour, and it made 10.7 tool calls for a task needing one.

Read that against the accuracy column and an identity falls out. At one and two lookups the text protocol’s accuracy is exactly 100% minus its cap rate: 40 + 60, and 76 + 24. Every run that managed to stop was correct. The protocol was not reasoning worse there; it was failing to say it was done. So the 60-point gap at one lookup is a termination result rather than an accuracy result — which is the cleanest evidence in this chapter that the exit is the fragile part of the loop. From three lookups on the identity breaks (52 + 28, 60 + 0, 28 + 12) and real reasoning errors appear alongside the cap.

This is Section 23.1’s third point, measured. The exit is a model decision, and in the native API “stop calling tools and answer” is a structurally different action from calling a tool.

In a text protocol it is not. CEVAP(...) is just another string competing with ARA(...) — and once the transcript contains three ARA lines, the most likely next line is a fourth one. The loop’s own history argues against leaving it.

Why stopping is easy in one protocol and hard in the other
STRUCTURED TOOL API one model response, two fields tool_calls keep going content stop TWO FIELDS — STOPPING FILLS A DIFFERENT ONE runs that hit the step cap 0.0%, at every step count HAND-ROLLED TEXT PROTOCOL one model response, one field ARA(...) vs CEVAP(...) two strings in the same field ONE FIELD — STOPPING IS JUST ANOTHER STRING runs that hit the step cap 60.0%, on one-lookup tasks THE EASIEST TASK HAD THE WORST TERMINATION BEHAVIOUR once the transcript holds three ARA lines, the likeliest next line is a fourth

Nothing about the model changed between these two panels — only how the decision to stop is expressed. On the left it is a different field; on the right it is a string competing with the strings already in the transcript, and the transcript wins.

The text protocol scored 0.0% at every step count on the first run, with every step unparseable and every task hitting the cap. That is a suspiciously total failure, which is the shape a harness bug usually has.

It was mine. I had written the protocol’s signature as ARA(proje, alan), intending those as placeholders. The model read them as literal labels and emitted ARA(proje, ZRG-14), which my regex correctly refused. Adding one concrete example — ornek: ARA(ZRG-14, butce) — fixed it.

The lesson is not “write better prompts”. It is that a hand-rolled protocol makes you specify a calling convention unambiguously, in prose, and a typed schema does that for free. The 0% was a real cost of the text approach; it just was not the cost I was trying to measure, and publishing it as a model result would have been wrong.

One honest limit survives the fix. This is a single protocol prompt, and the first version showed exactly how sensitive the result is to wording. An instruction as small as “if you already have the answer, emit CEVAP immediately”, or a stop sequence on CEVAP, might recover much of the termination gap; no ablation was run. What a typed schema gives you is not a better prompt — it is the removal of that variable from the experiment.

23.4 Agent or pipeline?

The decision that matters most is made before any of the above: does this task want an agent at all?

An agent buys you one thing — the sequence of steps can depend on what the earlier steps found. That is genuinely valuable and it is the only thing it buys. Everything else on the list of agent benefits is available from a pipeline that calls a model at fixed points.

So the question is not “is this task complex” but “is the control flow data-dependent?” Complexity is not the criterion; branching is.

The criterion is branching, not difficulty
the task what to build
Extract fields from a document, then validate them Pipeline. The steps never change
Classify a ticket, then route it Router. One model decision, then fixed paths
Answer from a knowledge base Pipeline — retrieve, then generate. Chapter 20 is a pipeline, not an agent
Answer, but sometimes the first search is not enough Constrained loop. The branch is real but the space is small
Debug why a customer’s order failed, across five systems Agent. What you check second genuinely depends on what you found first

The measurement in Section 23.3 makes this concrete. A fixed pipeline for that task — parse the project codes, look each one up, add them — moves all the risk into one place, the parse. Once the codes are extracted the lookups and the sum are exact at every step count, cost nothing per call and take microseconds; the agent reaches 40% at five lookups. That is the trade in its clearest form, and note what it is not: it is not “no model”, and it is not “a regex beats a model”. It is that the model does one bounded thing instead of five sequential ones. Where the steps are known, writing them down beats asking a model to rediscover them every time, and by a wide margin.

Anthropic (2024) makes the same argument from production experience, and puts it more bluntly: add complexity only when it demonstrably improves outcomes. You will meet its vocabulary everywhere, so map it onto this chapter’s: what it calls workflows — model calls and tools orchestrated through predefined code paths — are the pipelines and routers above, and it reserves the word agent for systems where the model directs its own steps.

The failure modes are different in kind

A pipeline that breaks throws an exception at a line number. An agent that breaks returns a plausible answer having done something you did not expect, and you find out from the transcript.

That difference is why the levels of autonomy in Section 23.2 are worth taking seriously as an engineering choice rather than a maturity ladder. You are trading debuggability for the ability to handle inputs you did not enumerate — and if you can enumerate them, you gave up the debuggability for nothing.

There is a version of this argument that goes too far — “just write a pipeline” — and it is wrong for the tasks agents are actually good at. The honest position is that most things people build as agents are pipelines with extra steps, and that the residue is real and worth the machinery. The criterion above is how you tell which one you have.

23.5 Intuition checks

Answer each one before you open it. Two are definitions this chapter argues for; the other four it settled with a measurement rather than an argument.

A while-loop around a model that can call functions. The model emits arguments and your runtime executes them; the loop ends when the model chooses not to call anything. Everything else — planning, memory, orchestration — is engineering around those fourteen statements, and being able to say so is what keeps framework documentation readable.

Not at the tool calls, probably. Measured here, tool use stayed perfect — exactly the right number of calls, none malformed, none hitting the cap — while accuracy fell from 100% to 40%. The failure was in aggregating what came back. Note the shape of the task before generalising: the five lookups are independent, so the model batched them into two model calls and never ran five sequential steps. What fell apart is aggregation under load, not a chain of five dependent decisions. Check the reasoning over collected observations before you rewrite a schema, and decompose the task if you can: fewer things to hold at once is the fix that actually works.

In the minimal loop, the transcript and nothing else — no working memory, no variable bindings. Three things follow: cost grows with the number of model calls because each one re-sends everything; nothing is ever forgotten, including mistakes, which is why a confused agent tends to stay confused in the same direction; and there is no rollback of the transcript unless you build one, and rolling it back does not undo external effects, which need their own transaction or compensation mechanism where one exists. Then say what the rest of this part adds: external files, a memory store, and sub-agents with their own transcripts are all ways of not living inside one message list.

It helps, and by more than convenience. On the same model and task the native API beat a hand-rolled text protocol at every step count — by up to 60 accuracy points, though only that largest gap is statistically clear on 25 tasks — while using about a third of the model calls. The mechanism is termination: in the structured API “answer instead of calling a tool” is a different kind of action, whereas in a text protocol the answer command competes with the tool command, and the transcript’s own history argues for another tool call.

Not if the model batches them. Measured here the native loop used 2.0 model calls at every step count from 1 to 5, because it emitted all the lookups in one turn. That is why five times the work cost only 1.52× the tokens. Count the two separately: tool calls tell you what the agent did, model calls tell you what it cost.

When the control flow is not data-dependent. The criterion is branching, not difficulty: if you can write the steps down, write them down. A fixed pipeline for this chapter’s task has no decision to get wrong at any step count, costs nothing per call, and runs in microseconds, against an agent that reaches 40% at five lookups. (It can still have bugs — it is code — and it needs the same tests as any other code; what it does not have is a model choosing what to do next.) Reach for the loop when what you do second genuinely depends on what you found first.

23.6 Interview questions

On sourcing — the same caveat as Chapter 2. These recur in the two credible open sources this book uses, plus the ones this chapter’s own measurements make answerable.

Testing whether you understand the mechanism

Q: Walk me through what happens when an agent calls a tool.

The model emits a structured call — a name and arguments — as its output. It executes nothing. Your runtime validates and runs it, and appends the result, or the error, back into the message list as an observation. Then the whole transcript goes back to the model. Say the two consequences that show you have built one: the runtime is the only security boundary, and, in the minimal loop, the transcript is the only state.

Q: How does an agent decide it is finished?

It does not decide in any checked sense — the loop ends when the model emits no tool call, so the stopping rule is a prediction rather than a condition. That is why a step cap is not optional, and why the structured API’s separation between “call a tool” and “produce content” matters: measured here, a text protocol where the two compete failed to terminate on 60% of one-step tasks, having already retrieved the answer.

Q: Why does an agent get more expensive as tasks get longer?

Because every model call re-sends the entire transcript, so a sequential run of n steps costs roughly the sum of n growing contexts. The nuance worth volunteering is that this depends on whether the model batches its tool calls: in this chapter’s measurement the native loop used two model calls whatever the task size, and five times the work cost 1.52× the tokens — while the sequential text loop paid 3.5× the model calls for the same five-lookup task.

Testing whether you have shipped one

Q: How would you decide between an agent and a pipeline?

By asking whether the control flow is data-dependent, not whether the task is hard. If the steps are known and stable, write them down — a fixed, tested pipeline takes the model out of deciding the next step, is cheap, and is debuggable at a line number. It still needs its inputs, computations and external effects validated; code can be silently wrong too. Reach for a loop when what you check second genuinely depends on what you found first. Then say what it costs you: an agent that fails returns a plausible answer having done something unexpected, which is a much worse debugging position than an exception.

Q: Your agent works in testing and loops forever in production. What do you do?

First, look at the transcripts of the runs that hit the cap rather than theorising — the failure is usually visible in three lines. Expect one of two patterns: it is repeating a call whose result it already has, which is a termination problem and points at how the “done” action is expressed; or it is retrying something that keeps failing, which points at the error message the tool returns. Then bound it properly: a step cap, a wall-clock budget, and a token budget, with a defined behaviour when each fires.

23.7 The 2026 toolbox

Everything in this chapter ran locally with the versions below, and notably without any agent framework at all.

The stack these numbers came from
library version used what it is for here
ollama 0.24.0 serving qwen3.5:9b with tool calling on /api/chat
numpy 2.5.2 the invented database and the scoring

Worth knowing about, not used above:

The wider ecosystem, versions as of writing
library current version when you would reach for it
langgraph 1.2.11 when the control flow deserves to be a diagram rather than a while-loop
openai-agents 0.22.0 a small, readable implementation with handoffs and guardrails built in
smolagents 1.26.0 the one to read rather than only import
pydantic-ai 2.35.3 typed tools and validated outputs, if your stack is already Pydantic
llama-index 0.14.24 agents over your own data, with the retrieval already wired
crewai / autogen-agentchat 1.15.18 / 0.7.5 multi-agent orchestration — Chapter 28, and worth postponing until one agent works

23.8 Lab: build the loop, then break it

Build it before you read another framework’s documentation, because the point of the exercise is to have something concrete for that documentation to be about.

Step 1 — write the fourteen statements yourself

No framework. Two real tools, one model, one step cap. You are done when it answers a two-tool question end to end.

Step 2 — print the transcript at every step

Not a log line — the actual message list. Watch it grow. This is the moment the state stops being abstract.

Step 3 — measure the decay

Build tasks needing 1, 2, 3, 4 and 5 tool calls, with answers you can check exactly. Twenty-five each. Plot accuracy against step count and find where your model falls off.

Step 4 — count model calls separately from tool calls

They are different numbers and only one of them predicts your bill. If they are equal, your model is not batching, and your cost curve is the steep one.

Step 5 — make a tool fail

Return an error string from one tool. Does the loop recover, give up, or invent a value? Then change the error message and see whether the behaviour changes — that is Chapter 24’s whole subject, previewed in ten minutes.

Step 6 — remove the step cap

Run it on a task the model cannot complete and watch what happens. Put the cap back. You will never treat it as a tuning parameter again.

What to hand in

The accuracy-versus-steps curve, the model-call and tool-call counts side by side, one transcript of a run that failed, and one sentence naming the step count beyond which you would not trust this agent unsupervised.

23.9 Exercises

  1. Fourteen statements. Implement the loop from scratch against two tools you already have. Do not use a framework. Time yourself; it is an afternoon.
  2. Find your decay curve. Build tasks needing 1 to 6 tool calls with checkable answers, and plot accuracy against step count. Mark the point where it crosses the threshold your product needs.
  3. Separate the counters. Instrument model calls and tool calls separately on a real workload. Compute the ratio. If it is near 1, you are paying the sequential cost curve.
  4. Kill the batching. Force your agent to make one tool call per turn and re-measure cost. The difference is what parallel tool calling is worth to you in money.
  5. Termination probe. Give your agent a task it can finish in one call, and count how often it stops immediately. This chapter’s text protocol failed this test 60% of the time on the easiest possible task.
  6. The pipeline control. Write the fixed pipeline for a task you currently solve with an agent. Compare accuracy, cost and latency. Report honestly whether the agent earned its place.
  7. Break the state. Inject a wrong observation halfway through a run and see whether the agent recovers. Then explain, from Section 23.2, why it usually does not.
  8. Schemas first, decision second. Take a three-step manual workflow you know well. Write the tool schemas an agent would need, then argue whether the agent is worth it — the schemas usually make the answer obvious.

23.10 Where to go deeper

Before any of this: write the loop yourself, against two real tools, with no framework. Everything below reads differently afterwards.

Beyond the papers
resource what it is
Building effective agents — Anthropic the best short piece on this subject; its central argument is this chapter’s, argued from production experience
Agents — Chip Huyen a long, well-organised treatment of planning, tools and failure modes
LLM Powered Autonomous Agents — Lilian Weng the survey-depth post that popularised much of the vocabulary people now use
smolagents small enough to read in an afternoon, which makes it the best framework for learning what a framework does
langgraph the loop as an explicit state machine — useful precisely because it makes the control flow something you draw
pydantic-ai typed tools and typed results, with the validation boundary made obvious
openai-agents a minimal reference implementation of the loop, handoffs and guardrails
The primary sources
topic source
reasoning interleaved with acting Yao, Zhao, et al. (2023)
the model calling functions at all Schick et al. (2023)
self-correction, and what it does not fix Madaan et al. (2023); Huang et al. (2024)
search over reasoning, not over actions Yao, Yu, et al. (2023)
self-correction inside the loop Shinn et al. (2023)
how far reliability falls over repeated trials Yao et al. (2024)
the same decay, measured as a task-length horizon Kwa et al. (2025)
the agent’s own memory problem Packer et al. (2023)

23.11 What to remember

  • The minimal agent is fourteen statements. A while-loop around a model with tools. The model emits arguments and never executes them, so the runtime is the only security boundary. The loop ends when the model chooses not to call a tool. In this minimal loop the transcript is the only state; later chapters move state into files, a memory store and sub-agents.
  • The exit is a prediction, not a condition. Nothing checks whether the task is done, which is why a step cap is an architectural component rather than a tuning parameter.
  • Accuracy fell with the number of lookups in this experiment. Here: 100% → 96% → 80% → 68% → 40% for one to five lookups, at temperature 0 on a task with exact answers — n = 25 per cell, so the 40% is [23, 59] and it is the direction that is solid, not the shape. The lookups were batched into two model calls, so this is not five sequential decisions; Kwa et al. (2025) measure decay with task length at scale, while the pass^k of Yao et al. (2024) measures consistency over repeated trials.
  • The loop was not what broke. Tool use was perfect at every step count — exactly the right calls, none malformed, none capped — while accuracy fell 60 points. Aggregation over collected observations is where the failure lives, so debug there first.
  • Count model calls separately from tool calls. Parallel tool calling collapsed this loop to 2.0 model calls at every step count, so five times the work cost 1.52× the tokens. A sequential loop pays on every step: at five lookups the text protocol used 3.5× the model calls for the same work.
  • The structured tool API is not sugar. It was ahead at every step count, by up to 60 accuracy points — though only the one-lookup gap survives a significance test on 25 tasks — and it never failed to terminate, mainly because it makes “stop and answer” a structurally different action from “call a tool”.
  • The termination failure was worst on the easiest task. The text loop hit the step cap on 60% of one-lookup tasks, having retrieved the answer at step 0 and then kept calling tools until the budget ran out.
  • Choose an agent on branching, not difficulty. If the steps are known, write them down: a fixed, tested pipeline removes the model from choosing the next step, is cheap, and fails at a line number — though it still needs its inputs and outputs validated. An agent that fails returns something plausible instead.