24  Tool Design as API Design

Chapter status: drafted. The measurements run qwen3.5:9b through ollama 0.24.0 at temperature 0, one pass per condition, n = 60 per cell. ollama does not grammar-constrain enum values in a tool definition, so the enum measured below is a hint the model reads, not a decoding constraint — which changes what the result means, and is spelled out where it matters. Nothing is executed in the schema experiment — it scores the call the model emits, so the model’s arithmetic cannot contaminate a result about schemas. Chapter 23 owns the loop; this chapter owns what the loop reaches for. Software: NumPy 2.5.2 on Python 3.14.5.

Learning objectives. After this chapter you can:

24.1 The description is prompt text

A tool schema looks like documentation and behaves like a prompt.

Tool schema — the name, description, types and enums a model sees before choosing a call. Prompt text rather than documentation, because the model gets one shot from it and cannot check a guess. ↩︎

That single sentence is the chapter; everything below follows from it.

The distinction matters because the two are written by different instincts. Documentation is written for a reader who can ask a colleague, read the source, or try it and see.

A model gets one shot, from the schema alone, with no way to check. It cannot open your codebase. It cannot see an example call. Whatever the schema does not say, the model guesses — and a guess that type-checks is indistinguishable from an understanding.

So the schema carries four separate jobs, and it is worth being able to name which one you are doing:

Four jobs, one object
the field the job it does
name tells the model when this tool applies, before it reads anything else
description resolves the cases the name leaves ambiguous, and rules out the ones it should not be used for
parameter types and enums put your API’s own vocabulary in front of the model — and, if your runtime constrains decoding, make out-of-set values unrepresentable rather than merely discouraged. Which of the two you get depends on your stack, not on your schema — Chapter 18
parameter descriptions carry the mapping from how a user phrases a thing to what your API calls it

The last row is the one people skip, and it is where most misuse comes from. Your users say how much was set aside for it; your API calls it budget_allocated. Something has to bridge that, and if the schema does not, the model invents a bridge.

What we will look at. First a measurement of which part of a schema actually earns its keep, where the answer is uncomfortable for anyone who has spent an afternoon polishing a tool description. Then the same treatment applied to error strings, which produced the largest single effect measured in this chapter — and where the raw exception most tools return by default turns out to be a half-written error message. Then what a tool should hand back, which is a question about the agent’s memory rather than about output. Then what to do when there are too many tools to list at all. And finally how to tell whether a tool is any good, which is not a question you can answer without asking a model.

Measured: which part of the schema does the work?

Four versions of one tool, differing only in how they are described. The task is to fetch one numeric field about a project, and the questions never name the field literally — how much money was allocated has to become alan="butce". So the schema is the only thing that can carry the mapping. Nothing is executed; what is scored is the call the model emits.

Two of the four differ by four words. Here they are in full, variant B (named, free string) first:

{"name": "veri_getir",
 "description": "Fetch one field of one project.",
 "input_schema": {"type": "object",
   "properties": {"proje": {"type": "string"},
                  "alan":  {"type": "string"}},
   "required": ["proje", "alan"],
   "additionalProperties": false}}

Then variant C, where the only change is the enum:

{"name": "veri_getir",
 "description": "Fetch one field of one project.",
 "input_schema": {"type": "object",
   "properties": {"proje": {"type": "string"},
                  "alan":  {"type": "string",
                            "enum": ["butce", "personel", "sure"]}},
   "required": ["proje", "alan"],
   "additionalProperties": false}}

Variant A replaces every name with f1, a, b and empties the description; variant D is C plus a written-out description on each parameter.

qwen3.5:9b, 60 questions, one tool, four descriptions
95% intervals are Wilson on n = 60

 variant         called a tool  project right  field right   95% CI          fully correct
 A. opaque               58.3%           0.0%         0.0%   [ 0.0,   6.0]            0.0%
 B. named               100.0%         100.0%        25.0%   [15.8,  37.2]           25.0%
 C. enum                100.0%         100.0%       100.0%   [94.0, 100.0]          100.0%
 D. described           100.0%         100.0%       100.0%   [94.0, 100.0]          100.0%

Listing the accepted values did all of the measurable work. Going from free strings to an enumerated list took field accuracy from 25% to 100% — 15/60 against 60/60 on the same 60 requests, so the data are paired and McNemar is the test. Because variant C got every request right, the pairing is forced: 45 requests improved and none got worse, and exact McNemar gives \(p = 2 \times 2^{-45} \approx 5.7 \times 10^{-14}\). (An earlier draft quoted Fisher’s \(p < 10^{-19}\), which treats the variants as independent samples: the conclusion is the same, the number is not.) Careful prose descriptions on top of the enum added nothing observable — though with C already at 60/60 there was no room left to add anything. That is the opposite of where most effort goes: people write the description and leave the parameter as a string. The paired test is what the close comparisons later in the chapter need, and it is what you should run on your own tool.

What the enum is doing here, and what it is not. ollama does not grammar-constrain enum values in a tool definition, so variant C’s 100% is not the decoder refusing to emit mali buyukluk. It is the model reading butce, personel, sure in the schema and using them. That matters, because it makes this finding and the error-message finding later in the chapter the same finding: what fixed both was putting the API’s vocabulary where the model could see it. If your runtime does constrain decoding — hosted APIs call this a schema-enforced or strict mode (Chapter 18) — the enum becomes a guarantee and this experiment tells you nothing about the model at all. Say which case you are in before you generalise.

And one cell is missing — the one that decides how far this generalises. The four variants are a ladder, not a factorial: nothing → names → names + enum → names + enum + prose. The untested rung is names + prose, no enum. Without it, the honest statement is not “prose does nothing” but “prose adds nothing once the enum has already put your vocabulary in front of the model”. For a parameter you cannot enumerate — a free-text query, a date, an id, an open-ended filter — a description is the only bridge you have, and this experiment says nothing about it. Read rule 3 below as an argument rather than a measurement.

Two rows deserve their own explanation, because each teaches something the totals hide.

The opaque schema did not make wrong calls. It made no calls. With a tool named f1 taking a and b, the model called it 58.3% of the time and, when it did not, answered from parametric memory (what it absorbed in training) instead — inventing a paragraph about what ZRG-14 supposedly is. That is a far worse failure than a malformed call: a wrong call errors, and an invented answer looks like an answer.

Without the enum, the model echoes the user’s vocabulary into your API. Here is the whole of variant B, one row per phrasing (the rows summarise phrasings; the per-question rate is the table’s 15/60):

  MISS  wanted butce     got "bütçe"           ← "ne kadar para ayrildi?"
  MISS  wanted butce     got "mali buyukluk"   ← "mali buyuklugu nedir?"
  OK    wanted personel  got "personel"        ← "kac kisi calisiyor?"
  MISS  wanted personel  got "ekip mevcudu"    ← "ekip mevcudu nedir?"
  OK    wanted sure      got "sure"            ← "kac ay suruyor?"
  MISS  wanted sure      got "takvim"          ← "takvimi ne kadar?"

Neither of the two it got right contains the API’s word: kac kisi calisiyor has no personel in it, and kac ay suruyor has no sure. There the model mapped the meaning — how many people, how many months — onto a field name, and the name it chose was the API’s own. In three of the other four it passed the question’s vocabulary straight through — mali buyukluk, ekip mevcudu, takvim; in the fourth, the first row, it produced the Turkish spelling of the API’s own word, bütçe. The model is not guessing badly; it has no way to know that your API says butce, so it says what the user said.

And note the first row, which is Chapter 16 arriving in a new place: the model wrote bütçe where the API expects butce. Correct Turkish, wrong identifier. Any API whose parameter values are ASCII-folded versions of real words has this failure waiting in it, and an enum removes it here.

Not everywhere, though, and the difference is worth being precise about. An enum in a schema is a hint your model usually follows; normalising at the tool boundary is a guarantee you control. Do both: apply NFC, then fold diacritics through an explicit alias table (bütçe → butce) before you validate. And do not leave the folding to Python’s .lower() or .casefold(), which take no locale and know nothing of Turkish casing — "IŞIK".lower() is işik, not the Turkish ışık, and "İSTANBUL".lower() picks up a combining dot. Map the Turkish letters explicitly (str.translate), or use ICU (PyICU) when you need Turkish-aware case conversion. Accept what the model plausibly sends; store what your API needs. Chapter 16 has the rest of the traps; this is the one that reaches your function signature.

The schema is the only bridge between two vocabularies
WHAT THE USER ASKED FREE STRING → THE MODEL SENT ENUM → THE MODEL SENT "ne kadar para ayrildi?" bütçe butce "mali buyuklugu nedir?" mali buyukluk butce "ekip mevcudu nedir?" ekip mevcudu personel "takvimi ne kadar?" takvim sure ACROSS ALL 60 QUESTIONS not just the four above field right, variant B 25.0% field right, variant C 100.0% PROSE ON TOP OF THE ENUM ADDED NOTHING OBSERVABLE variant D also scored 100.0% — C had already reached the ceiling

The four phrasings variant B got wrong. In every one the model passed the user’s own word into the parameter, because nothing in the schema told it what the API calls that field. The first row is the sharpest case: bütçe is correct Turkish and a wrong identifier.

The honest scope: one model, one tool, three fields. And note what the ladder does and does not establish: it shows that putting the API’s vocabulary in the schema is worth far more than describing it in prose on top of that. It does not rank constraints against prose in general, because the prose-only rung was never run. What generalises is that the vocabulary has to reach the model somehow — not the exact numbers, and not an ordering the design cannot see. A larger model might infer butce from context; a schema with twelve fields would make the enum matter more, not less. Run the variant sweep on your own tools; it is an afternoon and it is the highest-yield afternoon in this chapter.

Which change buys what — schema variants and error styles

So write schemas in this order

The measurement gives an ordering, and it is roughly the reverse of the one most people follow.

  1. Enumerate everything enumerable. Every parameter with a fixed value set becomes an enum. This is the change with the measured effect.
  2. Name the tool for when it applies, not for what it wraps. find_orders over db_query_orders.
  3. Write the parameter description as a mapping, from the words users use to the value your API wants: butce = the money allocated.
  4. Then write the tool description, for the cases the name leaves open — including what the tool is not for, which is the part that stops a model reaching for it wrongly.
  5. Make optional parameters rare. Every optional parameter is a decision the model has to make with no information, and a default you did not document is a guess it will make differently each time.

24.2 Error messages are prompts

The line marked 6 in Chapter 23’s loop turned an exception into an observation. That makes the error string prompt text too — the model reads it and decides what to do next — and almost nobody writes it that way, because error messages are written for a human who will be reading a log.

The claim is testable. Hold everything fixed and change only what the tool returns when it fails. To make every condition face exactly the same failure, the first tool call is rejected unconditionally, whatever the model asked for; the score is on the second call. One artefact comes with holding the string fixed: a model that sent butce correctly is still told that 'bütçe' was invalid. That keeps the four conditions comparable, which is the point, but every condition is slightly less coherent than a real error would be, and the traceback condition — which quotes the value — suffers most from it.

qwen3.5:9b, 60 tasks, first call always rejected, four error styles
95% intervals are Wilson on n = 60

 error style     retried        right field              gave up
 bare          14/60  23.3%   1/60   1.7% [ 0.3,  8.9]  46/60  76.7% [64.6, 85.6]
 traceback      9/60  15.0%   7/60  11.7% [ 5.8, 22.2]  49/60  81.7% [70.1, 89.4]
 vague         22/60  36.7%   2/60   3.3% [ 0.9, 11.4]  38/60  63.3% [50.7, 74.4]
 actionable    60/60 100.0%  60/60 100.0% [94.0,100.0]   0/60   0.0% [ 0.0,  6.0]

 A `valid field` column is not shown because it was identical to `right field`
 in all four conditions: whenever the model recovered far enough to emit one
 of the three legal values, it emitted the correct one. Recovery here is
 all-or-nothing.

 The traceback row does not reconcile: 9 retried plus 49 abandoned is 58, not
 60. Two of that condition's tasks are unaccounted for, so one of its three
 figures is off by up to two tasks. It is flagged rather than quietly patched,
 and re-running that condition is the first thing on this chapter's list.

The four strings were:

 bare        "Error"

 traceback   Traceback (most recent call last):
               File "server.py", line 118, in veri_getir
                 return RECORDS[proje][alan]
             KeyError: 'bütçe'

 vague       "HATA: gecersiz alan adi."

 actionable  "HATA: 'bütçe' gecerli bir alan degil.
              Gecerli alanlar sunlardir: butce, personel, sure."

The scope of that 100%, stated honestly: the valid set here has three elements, so the actionable error could list all of it, and a ceiling that high is close to guaranteed once the model reads it. What this measures is the presence of a recovery instruction, not how well one scales. The interesting untested case is a set of two hundred values, where all you can offer is the nearest few — measure that on your own tool before assuming the same jump.

One sentence took recovery from 1.7% to 100%. The system prompt and the model were unchanged; the only difference was the text the tool returned on failure. Which is the section’s point rather than an exception to it: that returned string is prompt, written by the tool author instead of the prompt author.

Three things in that table are worth taking seriously.

The stack trace is the most interesting of the four, not the worst. It is also close to what your tools return today if nobody wrote an error message: a raw exception. (Chapter 23’s loop sends back ERROR: {e}, which for this failure is just ERROR: 'bütçe' — the traceback’s last line, with the same missing half.) It produced the fewest retries — 15.0% against a bare "Error"’s 23.3% — but the retries it produced were mostly the ones that worked: 7 of its 9 retries landed on the right field, against 1 of 14 for the bare error and 2 of 22 for the vague one. Read those as descriptive: which requests retried at all depended on the error message, so the three retrying subsets are self-selected and need not be equally hard, and any test on the conditional rates inherits that. The comparison that holds up is over all 60 requests, paired. A traceback states what failed — KeyError: 'bütçe' names the offending value — and says nothing about what would work, so it is a half-written error message, and the pattern is consistent with that: less often acted on, but informative when it is. Read that as a hypothesis, not a finding.

What it is not is worse than nothing. Retrying and giving up are two sides of one outcome — a task is either retried or abandoned — so “fewest retries” and “most abandonment” are the same comparison (up to the traceback row’s two missing tasks). Against a bare "Error" the gap is 9 retries against 14, and 49/60 abandoned against 46/60, with intervals of [70.1, 89.4] and [64.6, 85.6]. On the same 60 requests the right test is paired, and even the most favourable pairing — every changed request moving in one direction — gives exact McNemar \(p \approx 0.06\) for retries and \(p = 0.25\) for abandonment. This experiment cannot resolve that comparison, and an earlier draft of this chapter called the ordering a result six separate times. What the four rows do separate, unambiguously, is the actionable string from all three others.

This fits, and complements, a known result. Huang et al. (2024) found that models cannot reliably correct their own reasoning from intrinsic feedback — being told to try again does not help. What this experiment adds is the boundary: the same model recovers 1.7% of the time from feedback that carries no content and 100% of the time from feedback that names the answer set. The difference is not the model’s capacity for self-correction. It is whether the feedback contains information.

Giving up is the dominant failure, not looping. Three of the four styles produced abandonment in the majority of cases — the model stopped calling tools and answered without the data. That is worse than a retry loop, which at least announces itself in the transcript. Chapter 23 found the opposite failure, an agent that would not stop; here the same architecture will not continue, and the deciding factor is a sentence.

“Invalid” is barely distinguishable from nothing. The vague style names the problem in words and still recovered 3.3% of the time, against 1.7% for the bare error — a gap too small to resolve. Saying that something is wrong is not the useful part. Saying what would be right is.

The rule this gives you

An error message a model can use has three parts, and the last one is the one that matters:

  1. what failed        'bütçe' is not a valid field
  2. why                (optional; often unnecessary)
  3. what would work    valid fields are: butce, personel, sure

Write the third part and the rest is optional. Omit it and you have written a log line, not a tool.

Two corollaries worth building in:

  • Enumerate the valid options in the error, exactly as you would in an enum. If the set is too large to list, list the nearest few — did you mean butce? is a complete recovery instruction.
  • Never return a raw exception. Catch it, and translate. The translation is a one-line function and, on this evidence, it bought the largest single improvement measured in this chapter.

24.3 What a tool returns is part of its design

A tool’s return value goes straight into the transcript, which Chapter 23 established is the agent’s state. So a return value is not output — it is a write to shared memory that every subsequent step will read and pay for.

Three rules follow, and all three are routinely broken by tools that were written for humans first.

A tool that returns 50,000 tokens has spent a large share of the context budget in one call. Whether that breaks the agent depends on the budget and the task — 50,000 tokens is most of a small window and 5% of a million-token one, re-sent on every later call — but the costs are the same in either case: context later steps needed, earlier observations pushed toward the middle where they are used less well (Chapter 25 measures that), and a higher price on every remaining model call. Return what the task needs: the three fields, not the whole record; the matching rows, not the table.

Paginate, and say so in the result. "showing 10 of 247 matches; call again with offset=10 for more" is a sentence the model can act on. Silently truncating to ten is a tool that lies, and the model has no way to find out.

Summarise on the tool side when the raw form is unusable. A tool that wraps a 40-page PDF should not return 40 pages. What it returns is a design decision about what the agent is for, and making it inside the tool keeps it out of every prompt that uses the tool.

And some things a tool must never return. Everything a tool returns is read by the model and, in most products, can end up in front of the user. So a tool never returns secrets — API keys, passwords, access tokens, internal connection strings — and never returns records the current user is not entitled to see; filter by permission inside the tool, before the text reaches the transcript, because the model cannot be trusted to keep a secret it has read. The reverse direction matters too: text a tool fetches from outside (a web page, an email, a ticket) is data, not instructions, and a model cannot reliably tell the two apart. Chapter 36 treats this as indirect prompt injection; the tool author’s part is to mark where fetched text begins and ends, and never to pass it off as the system’s own words.

Actions need different semantics from lookups

Everything above is about reading. Tools that do things need three properties, and the cheapest time to add them is before the first incident. They matter most for tools with side effects; the two exceptions after the table are why “read-only” does not always let you skip them.

Three properties a writing tool needs
property why an agent specifically needs it
Idempotent the same call will arrive twice. A timeout that fires after the action actually succeeded, a run restarted after hitting its step cap, or a model that did not see the result will all produce a second call — and “create the invoice” must not create two
Dry-run mode lets the agent, or you, see the effect before committing to it. A first-class dry_run: true parameter is worth more than any amount of prompt instruction
Confirmation-gated some actions should not execute on a model’s say-so at all. That gate belongs in the runtime, not the prompt — Chapter 37 is where this becomes a chapter

The first row has a name worth knowing: idempotency. A call is idempotent when making it twice has the same effect as making it once. Reads are idempotent by nature; “set the status to paid” is too; “add 100 TL to the balance” is not. The standard way to make a writing call idempotent is an idempotency key: the runtime attaches a unique id to each intended action, and the tool remembers which ids it has already carried out, so a repeat returns the first result instead of acting again.

idempotency — a call that can be repeated without repeating its effect; what makes a retried agent step safe. ↩︎

A read-only tool can usually skip all three, with two exceptions worth naming: a read that costs real money per call still wants a dry run, and a read over data the user may not be entitled to still wants a gate. “Read-only” bounds the damage; it does not always make the call free or harmless.

The general principle is Chapter 23’s, restated for the tool author: the model emits arguments and the runtime decides. Anything you would not let an unreviewed pull request do automatically, do not let a tool call do automatically.

24.4 Scaling past fifty tools

The previous section was about what one tool puts into the context when it is called. This one is about what the tools cost before any of them is called.

Every tool definition sits in the context of every call, so a flat list of fifty tools is a large fixed tax on every step, paid whether or not any of them is relevant. The published figures are larger than most people expect: Anthropic (2025) reports a catalogue of 50-plus tools whose definitions alone cost about 72,000 tokens — some 77,000 with the rest of the starting context — before a single call, and an 85% reduction from loading definitions on demand. Chapter 27 measures what the same tax looks like once your tools arrive over a protocol rather than out of your own codebase.

It also gets less accurate — more near-duplicate options means more chances to pick a neighbouring one — and the same report puts numbers on that too: tool-selection accuracy on a large surface rose from 49% to 74% for one model, and from 79.5% to 88.1% for a stronger one, purely by searching the catalogue instead of pasting it in. That is the unusual case where the cheap fix and the accurate fix are the same fix.

Three ways out, in increasing order of how much they change your architecture:

  • Namespace and prune. Large tool surfaces often contain two or three tools that do the same thing for historical reasons. Merging them is free accuracy. The same instinct sets a tool’s granularity: build one tool per task a user has, not one per endpoint of your API — a single schedule_meeting rather than list_users, list_events and create_event, which the model would have to chain correctly on its own.
  • Search, then load — progressive disclosure. Expose one find_tools(query) and load only the matching definitions for that step. This is Chapter 19 applied to your own API surface, and it is what Anthropic (2025) ships as a first-class API feature. The threshold at which to reach for it is not a constant. Anthropic’s documentation says selection accuracy degrades once a model sees more than about 30–50 tools, and suggests search from as few as ten tools or 10,000 tokens of definitions; verbose definitions reach that sooner. Measure your own definition tax before picking a number.
  • Give it an interpreter instead. If your tools are mostly small transformations, a sandboxed code environment replaces fifty definitions with one — the model composes what it needs rather than picking from a menu.

Progressive disclosure — loading only the tool definitions relevant to the current step instead of all of them, so a large surface does not tax every call. ↩︎

Code mode, and what it costs

That third option — code mode — deserves naming clearly, because providers now ship it: Cloudflare’s Code Mode (Cloudflare 2025) and Anthropic’s programmatic tool calling (Anthropic 2025) both let the model write code that calls the tools, instead of calling them one round-trip at a time.

Two forms need separating. In the first, the tools stay and are exposed inside the sandbox as a typed code API — which is what both products do — and the bindings the sandbox can reach are still an allowlist. In the second, a general interpreter replaces the tools, and that is where the allowlist is really lost.

Code mode — letting the model write sandboxed code that calls the tools instead of calling them one at a time; at the far end, a general interpreter replaces the tool list. Composition, loops and intermediate variables come free; some or all of a readable whitelist does not. ↩︎

At the far end, instead of filter_rows, sum_column and join_tables, you expose a sandbox and let the model write the composition. It is genuinely more expressive: composition, loops and intermediate variables come free, and they are exactly what a fixed tool list makes awkward.

The cost of that end is that you have replaced a whitelist with an interpreter. A tool list is a security boundary you can read; a sandbox is a security boundary you have to build and keep believing in. That is a real trade and it belongs to Chapter 36 rather than to a paragraph here — but the design decision is made in this chapter, when you choose what surface to expose.

24.5 Evaluating a tool

A tool is not correct because it works. It has to be right on two counts: the model has to call it right, which is a property of the pair and can only be measured by asking, and the tool has to do the right thing when called right — return the correct result, and no side effect nobody asked for. This section is about the first, because it is the one tool authors skip; do not let that stand in for the ordinary tests you would write for any other function.

The evaluation is small enough to build in an afternoon and it is the same shape as this chapter’s two experiments:

  1. Collect twenty real requests that should reach this tool, phrased the way users phrase them rather than the way your API is named. This is where the misuse hides.
  2. Score the call, not the outcome. Was the right tool chosen; are the arguments right? Executing muddles a schema problem with everything downstream — the schema experiment above deliberately executed nothing.
  3. Count the three failure kinds separately, because they have different fixes:
Three failure kinds, three different fixes
failure what it means what fixes it
Not called the model answered without the tool, or picked a neighbour the name, and the description’s “use this when”
Called wrong right tool, wrong arguments types and enums, then parameter descriptions
Not recovered a failed call the model could not act on the error string
  1. Add the twenty requests to your regression set. A tool description is prompt text, so changing it is a change to system behaviour and deserves the same treatment as changing a prompt — Chapter 22 sized how many items you need before a difference means anything.

The “not called” row is the one worth instrumenting in production, because it is invisible otherwise. A tool that is never selected produces no errors and no logs; it just quietly does not participate, while the model answers from memory — which is exactly what the opaque schema did, 41.7% of the time, in prose that looked like an answer.

24.6 Intuition checks

Answer each one before you open it. Four are questions this chapter settled with a measurement rather than an argument; the other two are design decisions it argues for and cannot measure.

On enums, before prose. Measured here, moving one parameter from a free string to an enumerated list took correct calls from 25% to 100%, and adding careful descriptions on top added nothing observable. Be exact about why: the runtime here did not enforce the enum, so listing the values supplied vocabulary the model followed, and the experiment never tested prose without the enum. A decoder that does enforce it guarantees only that the value is in the set — not that it is the right one.

Usually not a wrong call — no call. With an opaque schema the model here invoked the tool only 58.3% of the time and otherwise answered from parametric memory, producing a confident paragraph about something it could not know. A wrong call raises an error you can see; an unused tool produces a plausible answer and no signal at all.

Because nothing told it otherwise, so it echoes the user’s vocabulary into your parameter. In this chapter’s measurement only two phrasings worked, and neither contains the API’s word: the model mapped their meaning onto a field name and landed on the API’s own. One case is worth noticing specially: the model wrote bütçe where the API expects butce — correct Turkish, wrong identifier, and a failure waiting inside any API whose values are ASCII-folded words.

What would have worked. Measured across four styles, recovery went from 1.7% to 100% purely by naming the invalid value and listing the valid ones. The stack trace names the value that failed and never what would have worked: a half-written error message. Here it could not be told apart from a bare "Error" — its retry and abandonment gaps do not survive a test — so it is not worse than nothing, only unfinished. Catch it and add the missing sentence.

Yes, and it is a problem for the whole run rather than that step. The return value goes into the transcript, which Chapter 23 showed is the agent’s state, so it is re-sent on every subsequent model call and it crowds out what later steps need. Return the fields the task needs, paginate with the pagination stated in the result, and summarise on the tool side when the raw form is unusable.

When the tools are mostly small transformations that users want to compose — filter, then sum, then join — because composition is exactly what a fixed list makes awkward and what code makes free. Be explicit about the price: you have replaced a whitelist you can read with a sandbox you have to build and keep trusting. That is a security decision made in this chapter and paid for in Chapter 36.

24.7 Interview questions

On sourcing — the same caveat as Chapter 2. These recur in the two credible open sources this book uses, plus the ones this chapter’s own measurements make answerable.

Testing whether you understand the mechanism

Q: What makes a good tool schema?

Start from the framing that decides everything else: the schema is prompt text, not documentation, because the model gets one shot from it with no way to check. Then order the work by effect. Enums first — measured here, enumerating one parameter took correct calls from 25% to 100% while prose descriptions on top added nothing. Then a name that says when the tool applies. Then parameter descriptions written as a mapping from users’ words to your API’s words, because otherwise the model passes the user’s vocabulary straight through.

Q: Your agent ignores a tool that would answer the question. Why?

Most likely the name and description do not say when it applies, so the model does not recognise the situation as this tool’s. The important half of the answer is what that failure looks like: not an error, but an answer from parametric memory. In this chapter’s measurement an opaque schema went uncalled 41.7% of the time and produced confident invented prose instead. So instrument selection, not just failures.

Q: What should a tool return on error?

A sentence saying what would have worked, with the valid options enumerated. Measured across four styles, that took recovery from 1.7% to 100%. A raw stack trace names the value that failed and never what would work, so it is a half-written error message — and in this measurement it could not be told apart from a bare “Error”. Catch the exception and add the missing sentence.

Testing whether you have shipped one

Q: How do you evaluate a tool?

By whether the model calls it correctly, which is a property of the pair. Twenty requests in users’ phrasing, scored on the emitted call rather than the outcome so a schema problem is not confounded with everything downstream, bucketed into not-called, called-wrong and not-recovered — because those have three different fixes. Then keep the twenty in the regression set, since a description change is a behaviour change. Say what this does not cover: it measures the model–tool interface, and the tool’s own implementation and side effects need testing separately, like any other function.

Q: You have eighty tools and accuracy is falling. What do you do?

First merge and prune, because large surfaces accumulate near-duplicates and every extra near-neighbour is another chance to pick the wrong one. Then move from a flat list to search-then-load, so only the relevant definitions occupy the context — Chapter 19 applied to your own API. If the tools are mostly small composable transformations, consider replacing them with a sandboxed interpreter, and be explicit that you are trading a readable whitelist for a security boundary you now have to maintain.

24.8 The 2026 toolbox

Everything in this chapter ran locally with the versions below, and the schema experiment executed nothing at all — it only scored the calls.

The stack these numbers came from
library version used what it is for here
ollama 0.24.0 serving qwen3.5:9b with tool calling
numpy 2.5.2 task generation and scoring

Worth knowing about, not used above:

The wider ecosystem, versions as of writing
library current version when you would reach for it
pydantic-ai 2.35.3 typed tools where the validation error is already a good error message
outlines 1.3.3 enforcing the schema at decode time rather than hoping
jsonschema 4.26.0 validating arguments before execution, which is where the runtime boundary lives
openai-agents 0.22.0 tool definitions with guardrails attached
smolagents 1.26.0 comparing a tool list against code mode on the same task
mcp 2.1.1 exposing tools over a standard protocol — Chapter 27

24.9 Lab: fix a tool by measuring it

Six steps, one afternoon, and the point of the sequence is that you change one thing at a time — so that at the end you can say which change bought what.

Step 1 — pick a tool the model gets wrong

If you do not know which one, you have found the first task: log tool calls with their arguments for a day.

Step 2 — write twenty requests in users’ words

Not in your API’s words. The gap between the two is the thing being measured.

Step 3 — score the call, executing nothing

Right tool, right arguments, both or neither. Record the baseline.

Step 4 — change one thing: add enums

Every parameter with a fixed value set. Re-measure. On this chapter’s tool that single change was worth 75 percentage points (25% to 100%).

Step 5 — now rewrite the descriptions

Re-measure again. Report the delta honestly, including if it is zero, as it was here.

Step 6 — break it on purpose and fix the error string

Force a failing call. Try the bare error, then the actionable one. Count how often the model recovers.

What to hand in

The four-row table of your own — baseline, enums, descriptions, error string — and one sentence saying which change you would have predicted was largest, and whether you were right.

24.10 Exercises

  1. The enum sweep. Take one tool with a free-string parameter that has a fixed value set. Measure correct calls, add the enum, measure again. Report both numbers.
  2. Users’ words, not yours. Write twenty requests for one of your tools in the vocabulary your users actually use. Count how many produce the right arguments. The gap is your parameter descriptions’ job.
  3. Rewrite the worst error. Find the ugliest error string a tool of yours can return. Rewrite it in the three-part form. Measure recovery before and after.
  4. Find the silent tool. Instrument which tools are never selected. For each one, decide whether it is redundant or badly named — both are fixable, and both are invisible without the instrumentation.
  5. Budget a return value. Take a tool that returns a whole record and compute what it costs across a ten-step run. Then decide what it should return instead.
  6. Dry-run. Add a dry_run parameter to a tool that writes something. Measure whether the agent uses it when asked to be careful, and whether the instruction is even necessary once the parameter exists.
  7. The ASCII trap. If your API uses ASCII-folded identifiers derived from words in a language with diacritics, write the test that catches bütçe/butce before a user does.

24.11 Where to go deeper

Do this before reading anything: take one tool, add enums to every parameter that has a fixed value set, and measure the change. It is twenty minutes.

Beyond the papers
resource what it is
Writing tools for agents — Anthropic the closest thing to a specification for this chapter’s subject, from people who watch tool misuse at scale
Building effective agents — Anthropic the surrounding argument: what to build before you need tools at all
pydantic-ai tools as typed functions, with validation errors that are already model-readable — the error-message finding, built in
outlines Chapter 18’s constrained generation, which is the mechanism that makes an enum a guarantee rather than a suggestion
smolagents includes code-mode agents, so you can compare the two surfaces on your own task
JSON Schema documentation the constraint vocabulary you are actually writing in; most people use a tenth of it
The primary sources
topic source
models learning to call functions Schick et al. (2023)
reasoning interleaved with tool use Yao et al. (2023)
making the schema a hard constraint Willard and Louf (2023)
what self-correction does and does not fix Huang et al. (2024)
tool use at scale, and its failure modes Qin et al. (2024)
what a large tool surface costs, and searching it instead Anthropic (2025)
replacing a tool menu with an interpreter Cloudflare (2025)

24.12 What to remember

  • A tool schema is prompt text, not documentation. The model gets one shot from it, cannot read your source, and cannot check a guess. Whatever the schema does not say, it invents.
  • Enumerate before you describe. Moving one parameter from a free string to an enumerated list took correct calls from 25% to 100%; prose on top added nothing observable once the enum had reached 60/60. Prose without an enum was not tested, and for parameters you cannot enumerate it is the only bridge you have.
  • A bad schema produces no call, not a wrong call. The opaque variant went uncalled 41.7% of the time and answered from parametric memory instead — a confident invented paragraph, which is worse than an error because it looks like an answer.
  • Without constraints the model echoes the user’s vocabulary into your API. mali buyukluk, ekip mevcudu, takvim — and bütçe where the API wanted butce, which is Chapter 16’s problem arriving inside a function signature.
  • The error string made the largest single difference measured here. Recovery went from 1.7% to 100% by naming the invalid value and listing the valid ones. Write what would have worked; the rest is optional.
  • A stack trace is a half-written error message. It names the value that failed and never what would have worked. It produced the fewest retries of any style, yet of the three incomplete styles its retries were the most often correct, 7 of 9 — both descriptive patterns that do not survive a test at n = 60. Do not read it as worse than nothing. Read it as unfinished, and add the third sentence.
  • The dominant failure after an error is giving up, not looping. Three of four error styles made the model stop calling tools and answer without the data — a quieter failure than a retry loop and a worse one.
  • What a tool returns is a write to the agent’s state. Return the fields the task needs, state the pagination in the result, and summarise on the tool side. A 50,000-token result can consume context later steps need and is carried into every later call; judge it against the task’s context and cost budgets. Never return secrets or records the user may not see.
  • A tool that acts must survive being called twice. Make it idempotent, give it a dry run, and put the confirmation gate in the runtime, not the prompt.