30  Evaluating Agents

Chapter status: drafted. One small measurement: the same thirty runs scored two ways, minutes to produce. Chapter 22 established how to size an eval set and how to audit a judge; this chapter is about the thing agents add, which is that an answer now has a path behind it and the path can be wrong while the answer is right. Software: ollama 0.24.0 serving qwen3.5:9b, NumPy 2.5.2 on Python 3.14.5.

Learning objectives. After this chapter you can:

• separate outcome evaluation from trajectory evaluation, and say what each one misses on its own;
• score tool use with a concrete metric set rather than a gut feeling;
• tell pass@k from pass^k, and say which one a deployed agent needs;
• build scenario tests that catch regressions before users do;
• convert a production failure into a permanent regression test.

30.1 An answer now has a path behind it

Everything in Chapter 22 still applies: size the set from the difference you need to detect, audit the judge before believing it, report intervals. Agents add one thing, and it changes what “correct” means.

A classifier produces an answer. An agent produces an answer and a trajectory — the sequence of decisions, tool calls, observations and retries that led there.

Trajectory — the sequence of decisions, tool calls, observations and retries that led an agent to its answer. Two runs can share the answer and differ in the trajectory. ↩︎

Two runs can agree on the answer and differ completely in what they did to get it, and only one of those differences shows up in an accuracy number.

Two evaluations, neither sufficient

	what it asks	what it misses
Outcome evaluation	did it produce the right answer?	how much it cost, how many wrong turns it took, whether it got there by luck, whether it did something destructive on the way
Trajectory evaluation	did it get there sensibly?	whether the answer was actually right — an efficient path to a wrong answer scores well

Outcome-only evaluation is the default, and it hides expensive nonsense. An agent that answers correctly after fifteen redundant tool calls scores identically to one that answers correctly in two — until the bill arrives, or until one of those redundant calls writes something.

What we will look at. First the six numbers that cover almost every agent evaluation, two of which everyone skips and both of which hide something specific, and the two ways of summarising repeated runs of the same task. Then the same thirty runs scored both ways, which settles what the second axis actually buys. The usual argument for trajectory evaluation is that it catches runs that got the right answer by a bad route; the measurement shows a different, sturdier reason. Then the part no exact metric can reach, where Chapter 22’s judge comes back with one extra caution attached. Then scenario tests, which are ordinary unit tests once you have fixtured the world properly. Then public benchmarks, and what a leaderboard position is evidence of. And finally the loop that turns every production failure into a permanent test.

30.2 The metric set

The path needs numbers too, and there are not many of them. Six cover almost every agent evaluation, and they are worth defining precisely, because “tool accuracy” means four different things to four people.

Six metrics

metric	definition	what it catches
Outcome accuracy	the final answer is correct	the thing you actually want
Tool-call precision	of the calls made, the share that were necessary	redundant work, repeated queries, exploration that went nowhere
Tool-call recall	of the calls needed, the share that were made	the agent that answered without checking — often, though not always, from parametric memory
Step count	tool calls, and model calls, counted separately	Chapter 23’s finding: they are different numbers, and the model-call count is the one that predicts your bill, because each model call re-sends the whole transcript
Termination rate	the share that stopped on their own rather than hitting the cap	the failure mode that looks like success in an outcome-only report
Cost per correct answer	mean tokens per request ÷ accuracy as a 0–1 rate (in money rather than tokens if the two systems use different models or prices)	the only cost number that compares two systems fairly

Tool-call precision and recall need one thing defined before they mean anything: which calls were necessary. A single golden path is the easy version and usually the wrong one — two tools may be interchangeable, a cached result may make a call unnecessary on the second run, and some exploration is legitimate. Write down an acceptable set of trajectories rather than one, or score against the facts a run had to establish rather than the calls it made to establish them.

The two that get skipped are recall and termination, and both hide specific disasters. Low tool-call recall usually means the agent is answering from memory rather than consulting the required source — not the only possible cause, but the common one. Chapter 24 measured an opaque schema going uncalled 41.7% of the time while producing confident invented prose. A low termination rate means runs are ending because the budget ran out, and an outcome metric records the ones that happened to be right by then as successes.

One run is one sample: pass@k and pass^k

An agent run is a sample, not a fact. At any temperature above zero, and often even at zero, the same task can pass on one run and fail on the next. So the honest unit is not “the agent solved task 17” but “it solved task 17 in \(c\) of \(n\) runs”. Two summaries of those repeated runs answer opposite questions:

• pass@k — the probability that at least one of \(k\) attempts succeeds. It is the right number when something checks every attempt and you keep the one that worked, as a test suite does for generated code.
• pass^k — the probability that all \(k\) attempts succeed. It is the right number for a deployed agent: each user gets a single attempt, and the same kind of request arrives again and again.

If a task succeeds independently with probability \(p\) on each run, then pass@k \(= 1-(1-p)^k\) and pass^k \(= p^k\). At \(p = 0.8\) and \(k = 4\) that is 99.8% against 41%. The same agent “almost always gets there” or “gets all four attempts right less than half the time”, depending only on which question you ask. From \(n \ge k\) runs of a task with \(c\) successes, the unbiased estimates are (the first from Chen et al., 2021, the second from τ-bench)

\[ \text{pass@}k = 1 - \binom{n-c}{k}\Big/\binom{n}{k}, \qquad \text{pass}^k = \binom{c}{k}\Big/\binom{n}{k}, \]

computed per task and then averaged over tasks. The order matters. Two tasks with \(p = 1\) and \(p = 0.6\) average to \(p = 0.8\), and \(0.8^4\) is 41%; the true pass^4 is \((1 + 0.6^4)/2\), which is 56%. For \(k = 1\) both summaries reduce to ordinary accuracy. Chapter 23 quoted the benchmark that introduced pass^k: on τ-bench the best agent tested in 2024 succeeded on fewer than half the tasks, and in the retail domain its pass^8 was below 25% (Yao et al. 2024).

The measurement below ran each task once, so it reports pass^1 and says nothing about consistency. That is its largest limitation, and running every task five or more times is the first thing to add to it.

Scoring the same runs both ways

So how much does the second axis actually add? The test set is thirty tasks, each needing exactly two lookups and one addition, over invented data so that nothing is guessable. Outcome passes if the sum is right. Trajectory passes if the agent made exactly the two necessary calls, no repeats, no wrong targets, and stopped on its own. The two systems are the two loops from Chapter 23: the native loop, which uses the provider’s tool-calling API, and a hand-rolled text protocol, where the model writes its tool calls as plain text that our code parses. This is a separate run with its own tasks, so its numbers do not match that chapter’s two-lookup row.

qwen3.5:9b, 30 tasks, 2 lookups needed

                        native loop    text protocol
 outcome pass                100.0%            36.7%
 trajectory pass             100.0%            33.3%

   both pass                 100.0%            33.3%
   outcome only                0.0%             3.3%
   trajectory only             0.0%             0.0%
   neither                     0.0%            63.3%

 tool calls (ideal 2)           2.00             7.10
   of which repeats             0.00             5.00
 hit the step cap               0.0%            63.3%
 tokens                          829            1,557

The first thing to say is what this measurement cannot settle. The disagreement everyone argues about — a run that passes on outcome and fails on trajectory — happened once. But the denominator for that event is not thirty: a run that failed on outcome cannot exhibit it. One of the eleven runs that passed on outcome failed on trajectory: 9.1%, Wilson interval [1.6, 37.7]. That is too wide to support a claim in either direction. If the true rate were near the upper bound, 38%, trajectory evaluation would be worth building for that reason alone; if it were near 2%, it would not.

And the native loop adds little to the estimate, which is the deeper problem. It passed 30/30 on both axes, so no disagreement was observed in that arm. That is still a measurement, with its own uncertainty: 0/30 is compatible with a true disagreement rate up to about 11% (the Wilson upper bound). The regime the usual argument is actually about — a system that is mostly right but sometimes gets there badly — is not measured here. What this table settles is different, does not depend on that count, and is worth more:

It tells you what the system is doing. The text protocol made 7.10 tool calls for a task needing 2, and 5.00 of them were repeats — 70% of all its tool calls asked for something it had already been told. The outcome number, 36.7%, contains none of that. You would ship this system knowing it works a third of the time and not knowing it spends most of its effort asking the same question again.

It tells you why runs failed. A 63.3% cap-hit rate says these are not wrong answers, they are unfinished runs — Chapter 23’s termination problem, and a completely different repair from a reasoning problem. Accuracy alone cannot distinguish “answered wrongly” from “never stopped”.

That reading has a consequence for the rest of the table. If two runs in three ran out of budget, then 36.7% is what this protocol delivers at this step cap — a real measurement of the system as configured, but partly a fact about the cap. That is why the cap value should always be reported next to such a number. Performance at a larger cap is unmeasured, and not guaranteed to be higher: extra steps can also produce wrong answers, repeats or new failures.

The experiment that settles it, a sensitivity check on the cap, is one line of configuration: double the cap and re-run. Either accuracy climbs, and the text protocol is slow rather than incapable. Or it does not, and the protocol is stuck in a loop that more budget only feeds — the more interesting result, and the more damning one. That run has not happened. Until it does, the 5.1× below inherits the same uncertainty, because the cap limits the tokens a run can spend (the numerator) and shapes the accuracy (the denominator).

It changes the cost comparison. On raw tokens the text protocol looks 1.9× more expensive. Per correct answer it is 5.1× — 4,246 tokens against 829. The cost metric that ignores outcome understates the real gap by nearly three-fold, which is why tokens-per-correct-answer is the only cost number worth putting in a comparison.

Small measurement: thirty tasks, one model, one run per task. Wilson 95% intervals, because a proportion near a boundary needs them and Section 22.5 says so: outcome 100% → [88.6, 100]; outcome 36.7% → [21.9, 54.5]; cap-hit 63.3% → [45.5, 78.1]; the single outcome-only disagreement → [0.6, 16.7] on thirty runs, or [1.6, 37.7] on the eleven where it could have happened. What the table establishes solidly is the size of the waste the outcome metric conceals — 7.10 calls against 2, and 70% of them repeats — and the direction of the cost correction. The outcome gap itself is not in doubt: the two systems ran the same thirty tasks, so the comparison is paired, and McNemar’s exact test on 19 discordant pairs all one way gives p = 4 × 10⁻⁶.

The outcome number hides the waste and understates the cost

The two rose bars are the same system. Against raw tokens it looks 1.9× more expensive; against tokens per correct answer — the same numerator, divided by the share that were right — it is 5.1×. Neither figure appears in an accuracy report, and the top panel shows where the difference went: five of every seven calls it made asked for something it had already been told.

30.3 Judging a trajectory

Some trajectory questions have exact answers — call counts, repeats, termination. Others do not: was that a reasonable thing to try? And for those the instrument is Chapter 22’s judge, with every caution from that chapter carried over, plus one that is specific to trajectories.

Chapter 22 measured a judge flipping its verdict on 38% of items when the two candidates were swapped. A trajectory is a sequence of many steps, so there are plausibly more places for position effects to act — early steps may weigh more than late ones, for instance — although nobody has measured that here. Before trusting any rubric score:

• Run the identical-trajectory control. Score a trajectory against itself. Both judges in Chapter 22 passed this and then diverged sharply on the probes that mattered, so passing it establishes only that the instrument is not broken.
• Run both orders when you compare two trajectories. If the judge sees trajectory A then B, also show it B then A, and keep only the verdicts that agree. This is Chapter 22’s position probe, and it is the one that caught the 38% flip rate.
• Score criteria separately, not holistically. Did it call the right tools, did it avoid redundant work, did it recover from errors — three scores you can act on, instead of a single 1-to-5 score that moves for reasons nobody can name.
• Let the exact metrics do the work they can do. Do not ask a judge to count tool calls. Count them.

The judge’s real job is the residue: whether an exploratory path that looks wasteful was actually a reasonable hypothesis, and whether a recovery was sensible or lucky.

30.4 Scenario tests and simulated users

Metrics tell you how a system did on runs you already have. Scenarios are how you get runs before your users generate them for you.

A unit test for an agent is a scenario: a fixed environment, a fixed request, and an assertion about what should happen. The environment is the hard part, and it is worth building properly because everything else depends on it.

• Fixture the world, not just the prompt. A scenario needs a database in a known state, tool responses that are deterministic, and a fixed clock, so that “today” is the same date on every run. Otherwise yesterday’s pass and today’s fail differ for reasons unrelated to your agent.
• Record and replay real traces. Capture production tool responses and replay them. This makes a regression test out of an incident, and it is the cheapest source of realistic scenarios you will ever have.
• Simulate the user for multi-turn work. A second model playing the user makes multi-turn evaluation possible — and inherits every judge caveat above, because a simulated user is a model with a rubric.
• Assert on the trajectory, not only the reply. It must call check_inventory before place_order is the kind of assertion that catches a real regression, and an outcome-only test cannot express it.
• Check the end state of the world, not only the text. For an agent that acts, the outcome is what changed: the order row, the refund, the booking. τ-bench scores a run by comparing the database at the end of the conversation with an annotated goal state (Yao et al. 2024). Assert also on what must not have changed — a correct reply that cancelled the wrong booking on the way is a failure.

30.5 Benchmarks, and what a leaderboard tells you

Public benchmarks are how most people first hear that a model is good at agentic work. Each of the six below measures something real, and each of them measures it somewhere that is not your product.

Six benchmarks

benchmark	what it measures	what it does not
τ-bench	tool-agent-user interaction in retail and airline domains, with rule-following and a simulated user; success is the final database state, and reliability is pass^k	your domain, your tools, your rules
SWE-bench	resolving real GitHub issues, verified by the repository’s own tests	work outside its repositories (the original set is twelve Python projects), and any behaviour those tests do not check
GAIA	multi-step reasoning with web and file tools, on questions easy for humans and hard for models	whether the capability transfers to your task
WebArena	agents operating realistic self-hosted websites end to end	API-based work, and any site not in the suite
BFCL	function-calling accuracy across many schemas, including parallel and multi-step calls	whether the model uses your tools well — Chapter 24 measured how much the schema decides that
AgentDojo	robustness to prompt-injection attacks in agentic settings, reported next to utility on the same tasks, with and without attack	capability outside its few simulated environments (email, banking, travel, messaging)

These are worth knowing and they are weak evidence about your system, for Chapter 22’s reasons: contamination you cannot check, saturation at the top of the table, and construct mismatch — a benchmark measures a task and you have a system. Even the outcome checks can leak: a 2024 audit of one agent’s successful SWE-bench patches found that about a third had the fix already written in the issue report or its comments, and about a third passed only because the tests were too weak to catch a wrong patch (SWE-Bench+). Use benchmarks to screen candidate models down to two or three. Decide between those on your own suite.

30.6 From incident to regression test

The highest-value evaluation work is not designing metrics. It is the loop that turns every production failure into a permanent test, and it is mostly logistics:

  1. a failure is reported            with its trace id
  2. pull the full trajectory         every call, argument, result, and decision
  3. reduce it to a scenario          fixture the tool responses, keep the request
  4. assert what should have happened  outcome AND trajectory
  5. confirm it fails                 a test that passes on day one has not
                                      yet shown it can catch the failure
  6. fix, and keep the test forever

Step 5 is the one people skip and it is the one that makes the rest real. Demonstrate that the test fails on the faulty behaviour before you rely on it — run it against the version with the bug, or reintroduce the fault deliberately. A regression test that has never been seen to fail has an unknown sensitivity, and the common case is that it passes because it does not exercise the path at all.

Step 2 is the one that has to be designed in advance. A trace id (or correlation id) is an identifier attached to every model call, tool call and log line that belongs to one request, so that all of them can be pulled together later. If you are not persisting full trajectories with correlation ids, an incident gives you a complaint rather than a test. Chapter 28 said the same thing about distributed traces, and it is the same instrumentation.

Two habits that compound:

• Every incident becomes a test the same day. The suite grows from real failures rather than from imagined ones, which is the difference between an eval set that predicts production and one that flatters it.
• Keep the failing traces, not just the tests. When a fix breaks something else, the original trace explains what the test was protecting, which a three-line assertion cannot.

30.7 Intuition checks

Answer each one before you open it. Three are settled by the thirty runs this chapter scores twice — what an accuracy number conceals, whether the second axis catches different runs, and which cost figure to report — and the other three are judgement calls, two of them resting on measurements from Chapter 24 and Chapter 22 rather than on anything run here.

Note1. Your agent is 85% accurate. What have you not measured?

Everything about how it got there: how many tool calls it made against how many it needed, how many were repeats, whether it stopped on its own or ran out of budget, and what a correct answer costs. Measured here, a system at 36.7% outcome accuracy was making 7.10 tool calls for a task needing 2, 70% of them repeats, and hitting its cap on 63.3% of runs. None of that is visible in an accuracy number.

Note2. Does trajectory evaluation catch runs that outcome evaluation passes?

This measurement cannot tell you. It happened once, out of the eleven runs where it could have happened — 9.1%, with an interval running from 2% to 38% — and the arm that scored 100% on both axes had no room to disagree at all. So the honest answer to the question as asked is “not settled at this sample size”. Trajectory evaluation earns its place for reasons that do not depend on the answer: it tells you what the system is doing, why runs failed (a 63.3% cap-hit rate is unfinished runs, not wrong ones), and what correctness actually costs.

Note3. Which cost number should you report?

Tokens per correct answer. On raw tokens one system here looked 1.9× more expensive than the other; per correct answer it was 5.1×. The outcome-blind figure understated the real gap by nearly three-fold, and it is the figure most comparisons quote.

Note4. Why measure tool-call recall?

Because low recall usually means the agent answered without consulting your data — from parametric memory, fluently, with no error anywhere. Not always: an interchangeable tool or a cached result can lower it too, so check the trace before concluding. Chapter 24 measured an opaque schema going uncalled 41.7% of the time while producing confident invented prose. An uncalled tool generates no logs and no failures, so nothing surfaces it except counting the calls you expected.

Note5. Can an LLM judge score a trajectory?

For the part that has no exact answer — was that a reasonable thing to try — with every caution from Chapter 22 and one more. A trajectory is a sequence, so it plausibly offers more places for position effects, and that chapter measured a judge flipping on 38% of items from a swap alone. Score criteria separately rather than holistically, run the identical-trajectory control first, run both orders when comparing two trajectories, and never ask a judge to count something you can count.

Note6. Your agent tops a public agent benchmark. Should you ship it?

You have learned it is not obviously unfit. Benchmarks carry Chapter 22’s three problems — contamination you cannot check, saturation at the top, and construct mismatch, since a benchmark measures a task and you have a system with a prompt, a parser and a fallback. Use leaderboards to screen to two or three candidates and decide between them on scenarios drawn from your own traffic.

30.8 Interview questions

On sourcing — the same caveat as Chapter 2. These recur in the two credible open sources this book uses, plus the ones this chapter’s own measurement makes answerable.

Testing whether you understand the measurement

Q: How do you evaluate an agent?

Two axes, and say what each misses alone. Outcome: did it answer correctly. Trajectory: tool-call precision and recall, repeats, step count separated into tool and model calls, and termination rate. Then the cost metric that ties them together — tokens per correct answer, because the per-request figure flatters a system that is often wrong. Run each task several times and report pass^k next to accuracy, because each user gets a single sample and the same request arrives again and again; if the agent is inconsistent, some of those users are certain to hit a failure. The detail that shows experience: a low termination rate means runs are ending on a cap, which is a different repair from a reasoning error and invisible in accuracy.

Q: Why is trajectory evaluation worth building?

Not primarily to catch runs outcome evaluation passes — on one task that happened once in the eleven runs where it could, an interval too wide to conclude anything from. It is worth building because it tells you what the system is doing: a system at 36.7% accuracy was making 7.10 tool calls for a task needing 2, with 70% of them repeats. You would ship that knowing it works a third of the time and not knowing where the effort went.

Testing whether you have shipped one

Q: A user reports the agent did something wrong. What happens next?

Pull the full trajectory by trace id, reduce it to a fixtured scenario, write assertions on both outcome and trajectory, and confirm the test fails before fixing anything — a regression test that passes on day one has not yet shown it can catch the failure it was written for. Say that step two is an architectural prerequisite: without persisted trajectories and correlation ids an incident produces a complaint rather than a test, and that instrumentation has to exist before the incident.

Q: How would you catch an agent regression before users do?

Scenarios from real traffic with a fixtured environment and deterministic tool responses, asserting on trajectory as well as answer — it must call check_inventory before place_order is a regression an outcome test cannot express. Then validate the suite by breaking the agent deliberately in a few known ways and checking which metric catches which. A suite that has never caught a planted break is a suite with unknown sensitivity.

30.9 The 2026 toolbox

Thirty runs against one local model, scored twice, is small enough that the stack it came from is two lines. Everything in the second table matters once the suite has to run on every commit rather than once in a notebook.

The stack these numbers came from

library	version used	what it is for here
ollama	0.24.0	serving qwen3.5:9b for both loops
numpy	2.5.2	task construction and scoring

Worth knowing about, not used above:

The wider ecosystem, versions as of writing

library	current version	when you would reach for it
langsmith	0.11.2	trace capture, dataset management, and trajectory diffing
arize-phoenix	20.4.0	open-source tracing and evaluation, self-hosted
braintrust	0.35.0	eval runs wired into CI with per-run scoring
openinference-instrumentation	0.1.59	OpenTelemetry conventions for LLM and agent spans
ragas	0.4.3	the retrieval half of an agentic RAG system, scored separately
deepeval	4.2.0	judge-based metrics that fail like unit tests

30.10 Lab: an eval suite that catches a break

Step 5 is the point of the whole thing. A suite you have never watched catch anything has unknown sensitivity, and breaking the agent yourself is the only cheap way to find out what it would miss.

Step 1 — twenty scenarios from real requests

Fixtured environment, deterministic tool responses. Real phrasing, including the messy ones.

Step 2 — score outcome

Accuracy. This is the number you already have.

Step 3 — add the trajectory metrics

Tool-call precision and recall against the calls each task needs, repeats, step count, termination rate. Record per run, not aggregated. Run each scenario at least five times, so that you can report pass^k as well as accuracy.

Step 4 — compute cost per correct answer

Not per request. Compare it to the raw token figure. Here the gap between the two systems grew from 1.9× on raw tokens to 5.1× per correct answer, a factor of 2.7.

Step 5 — break the agent on purpose

Three ways, one at a time: remove a tool description, halve the step cap, and make one tool return a bare "Error" instead of an actionable one — the change Chapter 24 measured taking recovery from 100% to 1.7%.

Step 6 — check which metric caught each break

Report the table. If outcome accuracy caught none of the breaks but the trajectory metrics did, you have just justified them. If outcome accuracy caught all three, do not jump to “the scenarios are too easy” — these three breaks all damage the answer directly, so catching them is what an outcome metric is for. The scenarios that separate the two kinds of metric are the ones where the agent gets the right answer badly: by a route that costs ten times as much, or by a side effect nobody wanted. Add one of each and re-run before drawing a conclusion about your suite.

What to hand in

The per-run scores, the cost-per-correct-answer figures, and the table from step 6 showing which metric detected which break.

30.11 Exercises

1. Two scorings, one set of runs. Score twenty runs on outcome and on trajectory. Report the 2×2. Predict the disagreement rate first.
2. Count the repeats. Instrument one agent for a day and compute what share of its tool calls asked for something it already had.
3. Cost per correct answer. Compute it for two of your systems, alongside raw tokens per request. Which comparison changes the decision?
4. Incident to test. Take one real failure trace and turn it into a regression test. Confirm it fails before the fix.
5. Break it three ways. Remove a tool description, halve the step cap, and degrade an error message. Record which metric caught which.
6. Termination audit. What share of your runs end because the model stopped versus because a cap fired? If you cannot answer, that is the first thing to instrument.
7. Simulate a user. Build a two-model multi-turn scenario, then run Chapter 22’s position probe on the simulated user itself. It is a model with a rubric and it has the same biases.
8. pass@k against pass^k. Run ten tasks five times each. Compute both with the estimators in this chapter for \(k = 1\) to \(5\), and plot both curves. Predict first: how far apart will they be at \(k = 5\)?

30.12 Where to go deeper

Before reading: take twenty runs you already have and count tool calls against tool calls needed. That ratio is usually the first surprise.

Beyond the papers

resource	what it is
τ-bench	the closest public benchmark to a real tool-and-user agent task; read how it scores a run — final database state against a goal state, repeated as pass^k — even if you never run it
AgentDojo	prompt-injection robustness as a measurable property, which most suites omit entirely
SWE-bench	verification by the repository’s own tests — an outcome metric that fluent prose alone cannot pass, though weak tests can still let a wrong patch through
Hamel Husain on evals	the applied case for building the suite from real failures rather than imagined ones
langsmith / arize-phoenix	trace capture and trajectory inspection; the instrumentation the regression loop depends on
ragas	for the retrieval half, scored separately, per Section 20.4

The primary sources

topic	source
a model as judge, and its biases	Zheng et al. (2023); Wang et al. (2024)
the paired test for comparing two agents	McNemar (1947)
benchmark contamination, measured	Sainz et al. (2023)
holistic benchmarking as a methodology	Liang et al. (2023)
tool-use benchmarks at scale	Qin et al. (2024)
pass^k, and scoring by final database state	Yao et al. (2024)
utility and attack success measured together	Debenedetti et al. (2024)
the loop being evaluated	Yao et al. (2023)

30.13 What to remember

• An agent produces an answer and a path, and accuracy scores only one of them. Outcome evaluation asks whether it was right; trajectory evaluation asks what it did to get there.
• Whether trajectory metrics flag different runs is not settled by a measurement this size. One of eleven outcome-passing runs failed on trajectory here — an interval from 2% to 38% — and the arm that passed everything could not have shown a disagreement at all. They earn their place for a reason that does not depend on that number: they show what the system is doing and why it failed.
• The waste outcome hides can be enormous. A system at 36.7% accuracy made 7.10 tool calls for a task needing 2, with 70% of them repeats, and hit its step cap on 63.3% of runs — which also means that 36.7% is partly a fact about the cap. Performance at a larger cap is unmeasured, and doubling the cap is the experiment that would show it.
• Report tokens per correct answer. Raw tokens made one system look 1.9× more expensive; per correct answer it was 5.1× — the outcome-blind figure understated the gap by nearly three-fold.
• Measure tool-call recall and termination rate, the two everyone skips. Low recall usually means answering from parametric memory with no error anywhere — check the trace, since an interchangeable tool or a cache can also lower it; a low termination rate means runs ending on a cap, which accuracy cannot distinguish from wrong answers.
• A judge can score the residue, not the countable part. Never ask it to count tool calls, score criteria separately rather than holistically, and run the identical-trajectory control first and both orders when comparing — a trajectory is a sequence, so it plausibly offers more places for the position bias Chapter 22 measured at 38%.
• One run per task measures pass^1 only. Run each task several times. pass@k (at least one of \(k\) succeeds) flatters; pass^k (all \(k\) succeed) is what a deployed agent faces — at a per-run success of 0.8, four attempts give 99.8% against 41%.
• Benchmarks screen; your scenarios decide. Contamination, saturation and construct mismatch all apply, and a benchmark measures a task while you have a system. For an agent that acts, check the end state of the world, not only the reply.
• Every incident becomes a test the same day — and confirm it fails before you fix anything, because a regression test that passes on day one has not yet shown it can catch the failure. That requires persisted trajectories with correlation ids, designed in before the first incident.

Debenedetti, Edoardo, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. 2024. “AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents.” Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track.

Liang, Percy, Rishi Bommasani, Tony Lee, et al. 2023. “Holistic Evaluation of Language Models.” Transactions on Machine Learning Research.

McNemar, Quinn. 1947. “Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages.” Psychometrika 12 (2): 153–57.

Qin, Yujia, Shihao Liang, Yining Ye, et al. 2024. “ToolLLM: Facilitating Large Language Models to Master 16000+ Real-World APIs.” Proceedings of ICLR.

Sainz, Oscar, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. “NLP Evaluation in Trouble: On the Need to Measure LLM Data Contamination for Each Benchmark.” Findings of EMNLP.

Wang, Peiyi, Lei Li, Liang Chen, et al. 2024. “Large Language Models Are Not Fair Evaluators.” Proceedings of ACL.

Yao, Shunyu, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2024. \(\tau\)-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. https://arxiv.org/abs/2406.12045.

Yao, Shunyu, Jeffrey Zhao, Dian Yu, et al. 2023. “ReAct: Synergizing Reasoning and Acting in Language Models.” Proceedings of ICLR.

Zheng, Lianmin, Wei-Lin Chiang, Ying Sheng, et al. 2023. “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.” Advances in Neural Information Processing Systems.