22  Evaluating LLM Systems

Chapter status: drafted. The judge audit runs qwen3.5:9b and gemma3:27b through ollama 0.24.0 on 60 XL-Sum Turkish articles. The sizing tables are computed exactly rather than simulated — the paired columns sum McNemar’s exact test over the joint distribution of the discordant cells — after a simulated version of the paired table survived a full draft carrying a column that cannot exist. The margin note in Section 22.2 says what happened. Software: NumPy 2.5.2, scipy 1.18.1, ollama 0.24.0 on Python 3.14.5.

Learning objectives. After this chapter you can:

This is the chapter your statistics training was made for: applied LLM evaluation is, at bottom, measurement design under distribution shift. Most of what follows is not new statistics. It is ordinary statistics applied to a setting where almost nobody applies it.

22.1 Benchmarks are not your evaluation

A public benchmark answers how does this model compare to other models on a task someone else chose. Your evaluation answers does this system do my job well enough to ship. Those are different questions and the first one is only occasionally evidence about the second.

Three specific reasons, in the order they will bite you.

Contamination. A benchmark published before a model was trained may be inside that model’s training data, and you cannot open the training set to find out (Sainz et al. 2023).

Benchmark contamination — test items that were in a model’s training data, so its score can reflect memorisation rather than capability. You cannot inspect a closed model’s training set, but that does not make contamination unmeasurable: Oren et al. (2024) turns it into a hypothesis test you can run through an API. ↩︎

This book measured the effect on a smaller scale in Chapter 13: 13.06% of WikiANN’s Turkish test set is byte-identical to a training sentence, and the score bonus that leakage produced fell monotonically with how much a system memorises — +26.50 span F1 for a gazetteer, +16.25 for a CRF, +7.65 for a fine-tuned encoder. Nobody had to be dishonest for that to happen.

Two things follow, and the second is why this is a section rather than a warning. Contamination is testable, even on a model you cannot open: Oren et al. (2024) permutes a benchmark’s item order and checks whether the model gives higher likelihood to the canonical ordering, which is a real hypothesis test you can run through an API — provided the API returns token log-probabilities, which the test needs and not every closed API exposes. Chapter 5 and Chapter 8 both point at it; this is the chapter where you would run it, before quoting a benchmark number you did not generate yourself. And the structural answer since 2024 has been benchmarks that rotate their items faster than models are trained — White et al. (2025) replaces about a sixth of its questions every month, trading comparability across time for a contamination guarantee. Neither replaces your own eval set. Both make a leaderboard number slightly less of a rumour.

Saturation. A benchmark is saturated when the best models score close to its ceiling, so there is little room left for them to differ. When the headline models are at 88, 89 and 90, one point of difference is competing with everything else that varies at that scale: the benchmark’s own label errors, its ambiguous items, and the prompt each lab used. Close scores do not by themselves distinguish a small real difference from measurement uncertainty — the models may genuinely differ by a point. Assess the benchmark’s uncertainty and its validity before reading anything into the ranking.

Construct mismatch. The benchmark measures a task; you have a system, which is a prompt, a retrieval step, a parser, a fallback, and a model. A benchmark score cannot tell you that your JSON parser drops 2% of responses — Chapter 18 measured a case where formatting failures were being counted as reasoning failures, and enforcement moved accuracy from 70.7% to 88.0% with no change to the model at all.

The useful role for public benchmarks is screening: they cheaply eliminate models that are obviously unfit, and they tell you what is broadly possible. Treat a leaderboard as a way to pick three candidates, never as a way to pick one — and remember that a rank is a statistic with a sampling process behind it. Singh et al. (2025) documents undisclosed private testing on the arena that lets a provider evaluate many variants and publish only the best, 27 of them before one release. Nobody has to cheat for a selection effect that size to move a ranking. Liang et al. (2023) is worth reading for the opposite reason: what it looks like when the sampling process is written down.

What we will look at. First the sizing question, which has an exact answer almost nobody computes and which turns out to disqualify the eval set most teams already have. Then how to build one — where to sample from, why the labelling order matters, and the harness bugs that report a plausible number instead of crashing. Then an audit of an LLM judge against probes whose right answers are known in advance, where the control everyone thinks to run is the one that separates two judges least. And finally how to report a result someone can act on, which is mostly a matter of never quoting a number without also saying what it could not have detected.

22.2 How big does your eval set need to be?

Here is the question almost every LLM project answers by feel, and it has an exact answer. Given a system that is right about 80% of the time, how large must an eval set be to detect an improvement of d points, at 80% statistical power and the conventional 5% significance level?

Statistical power — the probability that your evaluation detects a real difference of a given size. An underpowered eval mostly produces false negatives — a real improvement reported as “no difference” — and one more subtle failure: among the comparisons that do clear significance, the effects are systematically overstated, because only the luckily-large ones get there. So low power costs you both the improvements you miss and the size of the ones you find. ↩︎

smallest difference detectable at 80% power (two independent proportions,
baseline 80%, two-sided alpha = 0.05). The count is per system: 100 means
each of the two systems is scored on 100 items. Whether those are the same
100 items is the paired question, and the table below answers it.

     items      smallest detectable difference
        30            none — see below
        50                    17.5 points
       100                    13.4 points
       200                    10.0 points
       500                     6.6 points
     1,000                     4.8 points
     2,000                     3.4 points
Size your own eval set

At thirty items there is no answer, because no difference is large enough. Against a system that is right 80% of the time, even a perfect replacement — one that is right 100% of the time — reaches only 57% power on thirty items. That figure is from an exact test, and the exact test is the one to quote here: the normal approximation everywhere else in this section returns 75%, but at p = 1.00 its variance term is exactly zero and there is nothing left for it to approximate. Both exact routes agree — Fisher’s exact test with thirty items per arm and McNemar’s exact test on thirty paired items both give 0.5725. A hundred items per system detects 13.4 points. Most real improvements are two to five.

That is worth sitting with, because thirty items is a common size; it is the size this chapter’s own exercise asked for in an earlier draft.

This does not mean small eval sets are useless. It means they answer a different question, and the distinction is the most practically useful thing in this chapter:

Three different jobs, three different sizes
eval set size what it is genuinely for
20–50 items finding failure modes. Read every output by hand. You are looking for categories of wrongness, not for a number, and thirty items surfaces them well — this is a real job, and the size is right for it
100–300 items catching large regressions — a broken parser, a prompt that stopped working, a model swap that went badly
500+ items comparing two systems that are actually close, which is the case whenever you are tuning rather than fixing

The failure to distinguish these is why so much prompt engineering is folklore. Thirty examples were tried, one prompt looked better, and the difference was smaller than the noise floor by an order of magnitude.

What pairing buys, and when it does not

Chapter 17 used McNemar’s test (McNemar 1947) on a 14.67-point spread that overlapping confidence intervals would have called inconclusive — raw \(p = 0.0009\), and 0.062 after correcting for the 66 pairs the extreme one was picked from. The comparison there was paired — the same 150 reviews through every prompt — and pairing is the standard advice.

The advice is right and it is not unconditional, because McNemar’s test throws away every item the two systems agree on. It looks only at the items where exactly one of the two systems is right, and asks whether those split 50/50 between them, as they would if neither system were better. So what pairing buys depends on how often they disagree, which is an empirical property of your systems:

power to detect a 5-point difference: 80% baseline against 85%,
two-sided alpha = 0.05

            unpaired,     McNemar on n paired items, by how often
   items    n per arm     the two systems disagree
            (2n items)      10%      20%      30%      35%
       30       0.07        0.01     0.03     0.04     0.04
       50       0.10        0.06     0.07     0.06     0.06
      100       0.15        0.24     0.14     0.11     0.10
      200       0.26        0.54     0.30     0.21     0.19
      500       0.55        0.94     0.68     0.50     0.44
    1,000       0.84        1.00     0.94     0.81     0.75

The table stops at 35% because two systems at 80% and 85% cannot disagree
more than that: delta_max = 2(1 - p_B) + (p_B - p_A) = 0.35. Past it the
"both wrong" cell of the 2x2 table would have to hold a negative
probability. Note the item counts: the unpaired column is a study with n
items in each arm, so 2n items in total, while the paired columns run the
same n items through both systems. Note also the tests: the unpaired column
is the normal-approximation z-test, the paired columns are exact McNemar.
Fisher's exact test on the unpaired side would give 0.04, 0.06, 0.11, 0.22
and 0.52 for the first five rows.

Pairing pays more the more the two systems agree, and the threshold is one you can compute. Write \(\Delta = p_B - p_A\) for the true accuracy difference, \(\bar{p} = (p_A + p_B)/2\) for the average accuracy and \(\bar{q} = 1 - \bar{p}\). McNemar’s asymptotic statistic is roughly \(\sqrt{n}\,\Delta/\sqrt{\delta}\), where \(\delta\) is the disagreement rate; the two-arm unpaired statistic is roughly \(\sqrt{n}\,\Delta/\sqrt{2\bar{p}\bar{q}}\). So pairing wins, to this approximation, when \(\delta < 2\bar{p}\bar{q}\), which for an 80% system against an 85% one is 28.9% — and it wins while using half the items. That number is not arbitrary either: it is almost exactly the disagreement rate two independent systems would show, \(p_A(1-p_B) + (1-p_A)p_B = 0.80 \times 0.15 + 0.20 \times 0.85 = 0.29\) (the two differ by \(\Delta^2/2\), here 0.00125). Below it your systems are positively correlated and pairing exploits the correlation. Above it they are anti-correlated, which is rare between two versions of the same system.

Read the table row by row and the practical rule falls out. At 10% disagreement, 200 paired items reach 0.54 where 200 items in each of two arms — twice the labelling — reach 0.26. At 30% the paired design on n items has fallen behind the unpaired design on 2n items. So: comparing two prompts, two checkpoints, or two decoding settings — cases where the systems agree on most items — pair, and get the answer on half the labelling. Comparing two genuinely different architectures that disagree constantly, pairing buys little and the honest move is a bigger eval set rather than a cleverer test.

What does not happen is pairing costing you anything on data you already have. An LLM eval is paired whenever you run both systems on the same items. That is the normal design, but check it rather than assume it: a comparison against a number from someone else’s run, or against a different sample, is not paired and cannot be analysed as if it were. When the items are shared, analysing them as two independent samples looks more powerful past the independence point, but only because the test no longer has the error rate it claims. Computed exactly at n = 500 under a true null of 80% against 80%, where independence sits at 32% disagreement, the unpaired analysis rejects 0.05% of the time at 10% disagreement, 5.0% at 32%, and 6.4% at 36% — it buys the extra power with false positives. McNemar’s exact test stays between 3.7% and 4.2% across the same range, conservative in the way an exact test on a discrete statistic is. On a fixed eval set the paired test is never the wrong choice. It is only sometimes not much of a gain.

Where pairing wins, and where the table has to stop

I got this backwards twice. First I fixed the disagreement rate at Chapter 17’s 30.7%, found that pairing bought nothing, and briefly believed I had refuted a textbook result. What I had actually found is that 30.7% sits just above 28.9%, the rate two independent systems at these accuracies would show, so there was no correlation left to exploit — a fact about the input I chose, not about McNemar. And it was the wrong input. 30.7% is the share of reviews on which the twelve variants did not all agree, not how often two prompts disagree: two variants disagreed on a median of 12% of reviews and at most 28.7%, and the best and worst prompts on 42 of 150 (28.0%). Then, fixing it, I ran the simulation out to 40% disagreement and published a column that cannot exist at these marginals. The closed-form check in the script caught neither, because it only ever covered the unpaired side. The table above is now computed exactly, and the general lesson is cheap: when a paired design stops helping, compute \(p_A(1-p_B) + (1-p_A)p_B\) and see whether you have simply reached independence.

22.3 Building an eval set

An eval set is a sample, and everything your statistics training says about samples applies. Three decisions do almost all the work.

Sample from real traffic, not from imagination. Invented test cases are systematically cleaner, better spelled, and more typical than what users send. If you have no traffic yet, use whatever is closest — support tickets, search logs, a colleague’s real questions — and label the gap explicitly. A hundred real items beat a thousand invented ones, and this is the single highest-value decision in the chapter.

Stratify deliberately, then say what you did. Real traffic is dominated by easy cases, so a uniform sample spends most of its items confirming that the easy cases still work. Oversample the hard slices, the rare-but-costly categories, and the languages you actually serve — and then, when you report a headline number, either reweight to the traffic distribution or say plainly that the number is on a stratified sample. Both are fine; silence is not.

Include the cases with no good answer. Questions your system should refuse, inputs that are out of scope, documents that do not contain the answer. Chapter 20 found this to be the test nobody runs, and Chapter 21 found a model that refused 90 times out of 90 when asked for something absent — a result you only learn by asking.

Label the data before you look at model output

The order matters more than it sounds. Labels written after seeing a system’s answers drift toward those answers, and the drift is invisible afterwards. Write the rubric, label a sample against it, measure how often two labellers agree, and then run anything. Report that agreement as Cohen’s \(\kappa\) (Cohen 1960) rather than as a raw percentage, because a raw percentage includes the agreement two people would reach without reading the items: on an eval set where both labellers call 80% of items “correct”, two labellers assigning those rates independently already agree 68% of the time (\(0.8^2 + 0.2^2\)), so a raw 70% is \(\kappa \approx 0.06\) — essentially no agreement beyond that baseline. Report the confusion table beside it, because on a heavily skewed set kappa is low and unstable even for careful labellers (Chapter 5 shows both effects). For more than two labellers, or an ordinal rubric, use Krippendorff’s \(\alpha\) (Krippendorff 2018). Human agreement is not a hard ceiling on system scores — measured against a reconciled gold standard a system can beat any single labeller — but a system scoring far above the labellers is usually fitting their shared quirks rather than the task. Chapter 5 has the rest of the machinery.

Version the eval like code

The eval set is a dependency of every number you report, so it belongs in the repository, in a file, under review, with a changelog. Two rules that pay for themselves:

  • Never edit an item in place. Add a new one and retire the old, so a score from March remains comparable to a score from June, and a score that moved because the test changed is distinguishable from one that moved because the system did.
  • Keep a held-out slice you look at rarely. An eval set you tune against every day is a training set. Chapter 12’s split discipline is the same discipline, one level up.

The harness bugs that produce good numbers

A broken evaluation harness usually does not crash. It reports something plausible, and there is an asymmetry that makes this dangerous: a bad number gets investigated, a good one gets published. Two bugs from this book’s own work, both caught late:

  • A reasoning model was given a 24-token generation budget. ollama returns chain-of-thought in a separate thinking field, so the whole budget went there, the answer field stayed empty, every response failed to parse, and each failure fell through to a majority-class default. The run reported 93.75% accuracy — which was exactly the majority baseline.
  • A tuned threshold was applied to the wrong side of a probability, scoring a model at 51.50 macro-F1 instead of 80.23.

Neither raised an error. The guards that would have caught them are cheap: log the parse rate per run and abort below 50%, smoke-test every model×condition on two dozen items before an overnight queue, and quote median seconds per item rather than the mean — machine stalls in this book’s own runs made one mean nine times its median.

22.4 LLM-as-judge, audited

Those guards catch a harness that lies to you. They do not catch an instrument that lies to you — and the most common instrument in 2026 is another model.

Using a model to grade another model’s output is now the default for anything that cannot be scored with an exact match. It is a reasonable idea and it is also a measuring instrument you did not calibrate. So calibrate it — with probes whose right answer you already know, which means you can score the judge rather than trust it.

Four probes, on 60 Turkish news articles, comparing summaries. The “professional summary” below is the one-sentence summary XL-Sum ships with each article, written by BBC Türkçe’s journalists — the dataset’s reference summary (Chapter 14). Three of the probes were run for this chapter:

  1. Identical. A and B are the same text. The only correct verdict is a tie. This is Chapter 14’s copied-text control, moved from metrics to preferences.

  2. Position. A real comparison — the professional summary against the article’s first sentence — run in both orders. A judge without position bias (Wang et al. 2024) returns the same winner both times.

  3. Verbosity. The same summary, padded with contentless filler. No information is added, so preferring the padded version is preferring length.

  4. Self-preference. Have each candidate model summarise the same 60 articles, then ask each judge to compare its own family’s output against the other’s, in both orders. A judge with no self-preference splits 50/50 once the orders are averaged. This is the probe I did not run, and it is the one that matters most when the judge and the system come from the same lab: Panickssery et al. (2024) found the strength of the bias tracks how well a model recognises its own writing, which suggests it may grow, not shrink, as models get better at recognising themselves. The setup here — two models, both able to summarise — could run it in an afternoon.

Position bias — a judge preferring the candidate in a particular slot (first or second, depending on the model) regardless of content. Measured directly by running the same comparison in both orders and counting how often the verdict flips. ↩︎

                                      n     qwen3.5:9b     gemma3:27b   Fisher p

 1. identical texts → called a tie    60       100.0%         100.0%          —
                                           [94.0, 100]    [94.0, 100]

 2. same winner in both orders        60        61.7%          85.0%    0.00674
                                          [49.0, 72.9]   [73.9, 91.9]
    picked whichever was in slot A   60x2       57.5%          54.2%          —
    preferred the professional
    summary, both orders pooled      60x2       51.7%          74.2%    0.00048
                                          [42.8, 60.4]   [65.7, 81.2]

 3. preferred the padded copy         60        28.3%           0.0%    <0.0001
                                          [18.5, 40.8]     [0.0, 6.0]

Intervals are Wilson at 95%. The rows marked 60x2 pool two judgments per
article -- the same comparison in both orders -- and those two judgments are
not independent observations, so the intervals printed for them are narrower
than they should be. Computed on the 60 articles instead, the 51.7% becomes
[39.3, 63.8] -- itself approximate, since a per-article score of 0, 1/2 or 1 is
not a Bernoulli trial. The Fisher column treats the two judges as independent
samples, which they are not: both read the same 60 articles.

Both judges pass the test everyone runs

The identical-text control is the sanity check a careful team would think to run, and both models score 100%. Sixty pairs of byte-identical text, sixty ties, no exceptions. Even that deserves an interval, by this chapter’s own rule: 60/60 has a Wilson interval (a confidence interval for a proportion that stays sensible at small \(n\) and at 0% or 100%) of [94.0, 100], so the probe is also consistent with a judge that misses as many as one comparison in seventeen. It is a floor test, not a certificate. And if it were your whole audit you would conclude you have two sound judges and pick on price.

Then the probes that discriminate:

The small judge changes its answer on 38% of items when you swap the order. Same two texts, same question, different verdict. That is not a hard-comparison problem — it is a property of the instrument, and it means more than a third of its verdicts carry no information about the summaries at all.

It also preferred the professional summary only 51.7% of the time over the article’s opening sentence. Read that as a preference rate rather than an accuracy: calling 50% “chance” presumes the professional summary is the correct answer, and no human judgement in this run establishes that. The interval is [42.8, 60.4] if you count all 120 judgments and [39.3, 63.8] if you count the 60 articles, and neither is quite right: the two orders of one article are not two independent facts, and averaging them into a 0, ½ or 1 per article does not make a Bernoulli trial either. An article-level bootstrap is the honest interval, and this run did not keep what it needs. Every version of the interval comfortably contains 50, which is the only thing the conclusion needs.

In fairness to the model, the comparison is not trivial: an article’s opening sentence is fluent, on-topic text written by the same journalist. But it is not a strong summary on this corpus — Chapter 14 found lead bias weak here, with the first sentence scoring 12.75 ROUGE-1 against 24.89 for the best single sentence, and being the closest match to the reference only 15.8% of the time. So ROUGE, at least, does not rate the two as close. Until a human rubric says which of the two should win, a judge whose preference between them is indistinguishable from a coin has not shown that it can rank your system’s outputs either.

The larger judge looks better on every probe: 85.0% order-consistency, a 74.2% preference for the professional summary, and it never once fell for the padding. How firmly each gap stands is less settled than the p-value column suggests, because those p-values come from the wrong test — see the next paragraph — and the margins alone cannot supply the right one. The padded-copy gap is safe under any pairing: the larger judge never preferred the padding, so all 17 disagreements run one way and exact McNemar gives \(p = 2 \times 2^{-17} \approx 1.5 \times 10^{-5}\). The order-consistency gap is not: 37/60 against 51/60 is consistent with anything from 14 disagreements all one way (\(p = 0.00012\)) to 23 against 9 (\(p = 0.020\), or 0.060 after Bonferroni over three tests). This chapter therefore does not claim that all three gaps survive correction. And say how many tests you ran even when it seems to change nothing: five comparisons at \(\alpha = 0.05\) flag something about a quarter of the time, if they are independent, on a run where nothing differs.

What the p-values in that table assume, and should not. Fisher’s exact test compares two independent samples. These two judges read the same 60 articles, so the comparison is paired and the right test is McNemar’s — the test this chapter spends a whole section recommending. Running it needs the item-level agreement between the two judges, and this run stored only the margins, so it is the first thing to add. The direction of the correction is not obvious either: pairing normally sharpens a comparison of correlated raters, while the two pooled rows count each article twice and that works the other way. What survives without the item-level table is the ordering and the padding result; what does not survive is the precision of the two smaller p-values.

A rough internal check on Section 22.2: the sizing table says 50 items per system detects 17.5 points, and the gap here on order-consistency is 23.3 points — 85.0 against 61.7 — over 60 articles. Rough because the table’s rows are unpaired and per-arm while this comparison is 60 articles read by both judges, so the two are not the same design. The conclusion survives that sloppiness in either direction: the experiment was adequately powered for the difference it found, and would have been badly underpowered for a 5-point one, which is worth remembering before reading anything into the slot-A numbers (57.5% against 54.2%), where the design has no power at all.

Four measurements, and only three of them tell the judges apart
qwen3.5:9b gemma3:27b 0% 25% 50% 75% 100% THE PROBE EVERYONE RUNS identical texts → called a tie both judges at 100.0% THE THREE ALMOST NOBODY RUNS same winner in both orders 61.7% 85.0% preferred the professional summary 51.7% 74.2% preferred the padded copy lower is better here 0.0% 28.3% THE PROBE EVERYONE RUNS IS THE ONE THAT SEPARATES THE JUDGES LEAST on order-consistency the judge gap is 23.3 points, over n = 60 articles

The control a careful team thinks to run is the flat one: both judges land on the same dot. Everything that tells them apart lives on the three rows below it — which is the argument for running probes whose right answers you already know, rather than the one probe that makes any judge look sound.

Verbosity bias did not reproduce

Judges have been measured preferring longer answers (Zheng et al. 2023). Neither of these did: qwen3.5:9b preferred the padded copy 28.3% of the time [18.5, 40.8] and gemma3:27b never — 0/60, which is a Wilson interval of [0.0, 6.0], so the claim available from that cell is that its rate of falling for obvious padding is under about one in sixteen, not that it is zero. I will state the rest narrowly too, the way Chapter 20’s needle result had to be stated: my filler was obviously contentless boilerplate — three sentences of “the situation is as described above” — and detecting that is a much easier task than resisting a genuinely more detailed, better-organised, longer answer.

“Did not reproduce” means neither judge preferred the longer copy on balance (in more than half the cases). It does not mean qwen3.5:9b handled the probe well: choosing a strictly worse copy more than one time in four is still an error rate, just not a length bias. What I measured is that both judges mostly reject obvious padding. What I did not measure is what they do with elaboration that adds real structure but no correctness, which is the case the folklore is actually about.

What to do about it

Five changes, and the first has to happen before you collect any numbers rather than after.

  • Run the probes before you run the eval. The three that ran here take an afternoon between them, the fourth about the same, and they change which judge you use.
  • Randomise or average over order. Present each pair in both orders and count only the items where the verdict agrees, or randomise and accept the extra variance. If you drop the items that flip, report how many are left: a verdict that changes with order is an abstention, and the coverage rule below applies to it. Reporting a single-order pairwise result from a judge with a 38% flip rate is reporting the coin, not the systems.
  • Prefer pairwise over absolute scoring, but know what it costs. Asking “which is better” is more reliable than asking for a 1–5 rating, because a rating scale drifts between calls and has no anchor — Liu et al. (2023) is the careful version of the absolute-scoring approach if you need one. The first cost is that pairwise is exactly where position bias lives — an absolute score of one answer has no slot A — so pairwise only pays if you also follow the both-orders rule above. The second is that pairwise comparisons do not aggregate into a score without more structure, and the structure has a name: a Bradley–Terry model (Bradley and Terry 1952), which fits one latent strength per system from the win matrix and is what an Elo leaderboard is doing underneath. If you need an absolute number rather than an ordering, the alternative worth trying is a rubric — an item-specific checklist scored yes/no, which gives the anchor a 1–5 scale lacks while still adding up.
  • Calibrate against human labels on a subsample. Fifty items labelled by hand tells you the judge’s agreement rate with the thing you actually care about — as \(\kappa\), like the labellers’ agreement. Read it against the human–human \(\kappa\): a judge that agrees with you about as well as a second person does is as good as a second labeller, and one far below that is not. Without that number, a judge score is a measurement in unknown units.
  • Never let the judge be the same model as the system under test (Panickssery et al. 2024), or at least never without running the fourth probe above and measuring the self-preference that setup invites.

22.5 Reporting a result someone can act on

Report uncertainty next to every point estimate, so a reader can judge how much the result actually supports. Five rules, all cheap:

Report an interval, and say how you got it. For a proportion, Wilson (Wilson 1927) beats the textbook normal approximation at the small sample sizes this field actually uses — it does not produce impossible bounds below 0 or above 1. Every percentage in this chapter carries one. For anything that is not a proportion — a ROUGE mean, an average 1–5 rating, cost per item — resample the items instead:

import numpy as np
from scipy.stats import bootstrap
res = bootstrap((scores,), np.mean, confidence_level=0.95,
                n_resamples=10_000, method="BCa", rng=0)
print(res.confidence_interval)

And resample at the level the data is clustered at, not the level it is stored at. The judge probes above produce two judgments per article; resampling 120 judgments understates the interval, and resampling 60 articles does not. Miller (2024) works through the general case — cluster-robust standard errors, paired differences, and the planning arithmetic of Section 22.2 — and is the single most useful reference on this page.

Use the paired test when the systems mostly agree, per Section 22.2, and say which you used. Two overlapping marginal intervals are not evidence of no difference: Chapter 17’s 14.67-point spread had overlapping intervals and McNemar found p = 0.0009 — 0.062 once corrected for the 66 pairs it was picked from, so pairing sharpened the evidence without settling it.

Quote the median for latency, and the tail — not the mean alone. One stalled request rewrites a mean. This book’s own timing runs produced a mean nine times its median, and a concurrent job made one configuration look nineteen times slower than an identical one, which is why no chapter here publishes a latency number collected while something else was running. The median plus p95 and p99 is what describes the user experience. Keep the mean where it is the right statistic — expected cost per request and capacity planning are both sums, and a sum is a mean times a count — but never let it stand in for the typical case.

Report coverage alongside accuracy whenever anything can abstain, fail to parse, or time out. An accuracy computed only over successful responses is not an accuracy — Chapter 18’s abstention result moved 86.75% to 92.50% purely by declining the hardest fifth, which is an improvement only if you also say you answered 80% of the questions.

Say how many comparisons you made. One headline number needs no correction, provided it was not itself picked from many. Twelve slices at \(\alpha = 0.05\) on a run where nothing changed flag 0.6 of them on average and, if the twelve tests are independent, flag at least one 46% of the time (\(1 - 0.95^{12}\)); slices that overlap change the second figure. Two different statements, and the second is the one that matters when you are staring at a single report. A regression report is usually twelve slices. Holm’s step-down correction — statsmodels.stats.multitest.multipletests(p, method="holm") — is never less powerful than Bonferroni and, like it, does not need the tests to be independent. The judge table above reports three tests, and what survives correction is the shorter list given under that table rather than the p-value column; saying which is the whole of the discipline.

Regression testing, which is the point of all this

The eval set earns its keep when a change lands. Run it on every prompt change, model version, and dependency bump, and store the per-item results rather than the summary — because the useful question after a drop is which items changed, and a stored aggregate cannot answer it.

Two thresholds worth agreeing in advance, before anyone has a stake in the answer: the size of drop that blocks a release, and the size that merely opens an investigation. Set them from Section 22.2’s table rather than from a round number, because a rule that blocks on a 2-point drop in a 100-item eval will fire constantly on noise and then be ignored, which is worse than having no rule.

There is a second reason a threshold fires on noise, and it is not sample size. Running the same test on every merge is repeated significance testing: if each run draws fresh randomness — a new sample of items, or sampled generations — a pipeline that changed nothing still trips a 5% alarm about once in twenty runs, and with better than even odds within fourteen (\(1 - 0.95^{14} = 0.51\)). Re-scoring the same deterministic outputs on the same fixed set is not a new chance to fail; it gives the same answer every time. Two cheap fixes. Set the alarm from the eval’s own noise level: on a 100-item eval at 80%, a swing of about 11 points happens by chance one run in twenty, and a real drop has to be about 13 points before the eval catches it reliably, so a rule below that range fires on noise. And treat the alarm as a trigger for a second, larger run rather than as a verdict. If what you actually want is a threshold you can watch continuously as items accumulate, the tool is an always-valid sequential test (Johari et al. 2022) — a fixed-\(\alpha\) test looked at repeatedly no longer has the \(\alpha\) it claims.

22.6 Intuition checks

Answer each one before you open it. All six are questions this chapter settled with a measurement rather than an argument.

That it is not obviously unfit, and little else. You have not learned whether the benchmark was in its training data — Chapter 13 found 13.06% of one benchmark’s test set duplicated in its own training split — nor whether the benchmark is saturated, nor whether it measures your task. Benchmarks are a screen for picking three candidates. Your eval decides between them.

Yes, for the right question — but not for comparing systems. At thirty items there is no difference detectable at 80% power against an 80%-accurate baseline: even a replacement that is right every single time reaches only 57%, on an exact test. (The normal approximation says 75% and should not be believed at p = 1.00, where its variance term is zero.) What thirty items is excellent at is finding failure modes: read all thirty outputs by hand and you will find categories of wrongness that no aggregate score would surface. Use small sets to learn what breaks, large sets to decide what ships.

Yes, on data you already have — but know what it buys. McNemar discards every item the systems agree on, so its gain depends on the disagreement rate, and the threshold is computable: pairing on n items beats an unpaired study with n in each arm roughly when the disagreement rate is below \(2\bar{p}\bar{q}\), which for 80% against 85% is 28.9% — almost exactly the rate two independent systems would show. Comparing two prompts, which agree on most items, pairing wins on half the labelling. Comparing two very different systems it buys little, and a bigger eval set is the honest fix. What it never does is cost you power: the unpaired analysis of paired data only looks stronger above the independence point because its false-positive rate has drifted to about 6%.

You have learned that it passes the easiest possible check, and nothing more. Both judges here scored 100% on that probe and then diverged sharply: one flipped its verdict on 38% of items when the order was swapped and preferred the professional summary 51.7% of the time; the other flipped on 15% and preferred it 74.2% of the time. Those are preference rates, not accuracies — which summary should win is a question for a human rubric. The control everyone runs is the one that discriminates least.

You cannot tell from those numbers. A 100-item eval detects about 13 points at 80% power, and this is 5. Pairing does not rescue it. If the same 100 items went through both prompts, McNemar’s exact test on the disagreements is the right test, but a 5-item gap is at best 5 disagreements against 0, which gives a two-sided \(p = 2 \times 0.5^5 = 0.0625\); more disagreements in both directions only raise it. Paired or not, the honest report is “no detectable difference at this sample size”, which is a different statement from “no difference”.

The parse rate. That exact number came from a run where a reasoning model spent its whole token budget in a thinking field, every answer failed to parse, and each failure defaulted to the majority class — so the harness was reporting the majority baseline as a result. A bad number gets investigated and a good one gets published, so a suspiciously good number deserves the same scrutiny a bad one gets automatically.

22.7 Interview questions

On sourcing — the same caveat as Chapter 2. These recur in the two credible open sources this book uses, plus the ones this chapter’s own measurements make answerable.

Testing whether you understand the measurement

Q: How would you evaluate an LLM feature?

Start by separating the two questions — is the model capable, and does my system work — and say that public benchmarks answer only the first. Then give the structure: a versioned eval set sampled from real traffic, sized from the difference you need to detect, including cases the system should refuse, with human labels collected before any model output is seen. Finish on reporting: intervals, paired tests where the systems mostly agree, and coverage alongside accuracy.

Q: Your eval set has 100 items and the new prompt scores 5 points higher. Ship it?

Not on that evidence. A hundred items detects about 13 points at 80% power, so 5 points is inside the noise. Pairing helps less than people expect here: McNemar on the disagreements can find effects the marginal intervals cannot, but not this one — a 5-item gap is at best 5 disagreements against 0, and the exact two-sided p is then 0.0625. The answer that shows judgement is naming what you would do next: either enlarge the eval, or ship it as a low-risk change while explicitly not claiming an improvement.

Q: What is wrong with using GPT-4-class models as judges?

Nothing, provided you calibrate one first. The failure is treating it as an oracle rather than an instrument. Name the specific biases and how you would measure each: position, by running both orders — measured here at a 38% flip rate for a small judge and 15% for a large one; verbosity, by padding an answer without adding information; self-preference, by never letting the judge grade its own family without checking. And calibrate against human labels on a subsample, because otherwise the score has no units.

Testing whether you have shipped one

Q: How do you catch a broken evaluation?

With controls that have known answers, because a broken harness usually returns something plausible rather than crashing. Score a reference against itself and require a perfect result. Log the parse rate and abort below a threshold. Compare against a majority-class baseline every run — if your model matches it exactly, that is usually a bug: inspect the harness and the model’s raw outputs at once. And say the thing that makes all of it necessary: a bad number gets investigated, a good one gets published.

Q: Your production quality dropped and the eval did not move. What happened?

Most likely the eval no longer resembles traffic — inputs drifted, a new customer segment arrived, or the eval was built from imagined cases in the first place. Check the input distribution before touching the model. The second candidate is that the failure is in a part of the system the eval does not exercise: a parser, a retrieval step, a timeout. That is why per-item results are stored rather than aggregates, and why coverage is reported alongside accuracy.

22.8 The 2026 toolbox

Everything in this chapter ran locally with the versions below.

The stack these numbers came from
library version used what it is for here
scipy 1.18.1 the power calculations, Fisher exact tests, and Wilson intervals via binomtest(...).proportion_ci(method="wilson")
numpy 2.5.2 the sampling
ollama 0.24.0 serving both judges locally

Worth knowing about, not used above:

The wider ecosystem, versions as of writing
library current version when you would reach for it
lm-evaluation-harness 0.4.x reproducing a public benchmark number the way everyone else computed it
lighteval 0.13.0 the same job, smaller and easier to modify
promptfoo 0.122.2 (npm, checked 2026-06) sweeping prompts and models against assertions in CI; the PyPI package of this name is a thin wrapper, so take the npm version
ragas 0.4.3 the retrieval/generation split of Section 20.4, scored
deepeval 4.2.0 judge-based metrics wired into a test-runner, so evals fail like unit tests
statsmodels 0.15.0 proportion_confint and mcnemar if you would rather not write them

22.9 Lab: audit a judge, then size your eval

Do the audit before the eval, in that order, because a judge you have not scored turns every number that follows into a measurement in unknown units.

Step 1 — write 50 items from real traffic

Real inputs, and include five your system should refuse. Label them yourself before running anything.

Step 2 — measure human agreement

Have a second person label 25 of them, then compute Cohen’s \(\kappa\) (sklearn.metrics.cohen_kappa_score) as well as raw agreement, because raw agreement on a skewed eval set is mostly the agreement two labellers would reach without reading the items. Treat 0.4 as a rough warning line, not a verdict: at 25 items the interval around kappa is wide, and on a skewed set kappa is low even for careful labellers (Chapter 5). A low kappa with a confusion table full of genuine disagreements says the rubric is the problem, not the system — fix it before building an eval on it.

Step 3 — run the identical-text probe

Ask your judge to compare an output with itself, 30 times. If it does not call a tie, stop and fix the prompt before doing anything else.

Step 4 — run the position probe

Take 30 real comparisons and run each in both orders. Report the flip rate with a Wilson interval, because at 30 pairs the interval is wide — an observed 6/30 spans [9.5, 37.3] — and a bare 20% threshold would be deciding on noise. Treat this step as a screen rather than a measurement: if the interval’s lower bound clears 10%, you have a judge whose single-order verdicts are partly coin flips and you should be averaging over orders. If you need the flip rate itself to within a few points, Section 22.2’s table says what that costs.

Step 5 — calibrate against your own labels

On the 50 labelled items, how often does the judge agree with you — again as \(\kappa\), not as a raw percentage? That is the number that converts judge scores into something meaningful. Put it next to the human–human \(\kappa\) from Step 2: the judge is good enough when it agrees with you about as well as your second labeller did.

Step 6 — size the next eval

Decide the smallest difference that would change a decision. Use Section 22.2’s table to find the item count it requires. If the count is larger than you can label, say so explicitly rather than shipping an underpowered comparison and calling it a result.

What to hand in

The two probe results, the judge–human agreement rate, the human–human agreement rate, and one sentence naming the smallest difference your eval set can actually detect.

22.10 Exercises

  1. Size your current eval. Take an eval set you already use, find its item count in Section 22.2’s table, and write down the smallest difference it can detect. Compare that to the differences you have been reporting.
  2. Audit your judge. Run the identical-text and position probes on whatever judge you use now. Report the tie rate and the flip rate.
  3. Both orders. Take a pairwise evaluation you have already run and rerun it with the order swapped. How many verdicts change? What does that do to your conclusion?
  4. Find the agreement ceiling. Have two people label the same 30 items independently. Compute Cohen’s \(\kappa\) as well as raw agreement, and note how far apart they are. Compare both to the accuracy you have been reporting for your system.
  5. Build the absence slice. Add ten items your system should refuse. Measure how often it does. This is usually the fastest way to find a real problem.
  6. Break your own harness. Set a generation budget too small for a reasoning model and see what your pipeline reports. Then add the parse-rate guard that would have caught it.
  7. Power in reverse. Pick the smallest improvement that would change a decision at your organisation. Compute the eval size it needs. If you cannot afford it, write down what you will do instead — that sentence is a more useful deliverable than most eval reports.

22.11 Where to go deeper

Before reading anything: count the items in your current eval set and look up the smallest difference it can detect. If you have been reporting differences below that, everything else here is urgent.

Beyond the papers
resource what it is
Hamel Husain on evals and on LLM judges the best applied writing on this subject anywhere — start here, before any paper
Eugene Yan on evals a well-sourced tour of task-specific evaluation, with the failure cases named
HELM Stanford’s holistic benchmark; worth reading for its methodology even if you never use the leaderboard
lm-evaluation-harness the standard harness; read how it does answer extraction before you write your own
lighteval a lighter alternative with the same idea, easier to read end to end
promptfoo run the same eval across prompts and models, with assertions; makes Section 22.9 routine
ragas / deepeval judge-based metrics with the decomposition already built
Chip Huyen’s blog the product framing — what an eval is for, and when the answer is not a number
The primary sources
topic source
a model as the judge, measured against humans Zheng et al. (2023)
and the biases that come with it Wang et al. (2024)
holistic benchmarking, done deliberately Liang et al. (2023)
why benchmark contamination has to be measured Sainz et al. (2023)
the paired test this chapter keeps using McNemar (1947)
the interval that behaves at small n Wilson (1927)
judge-based metrics for generation Liu et al. (2023)
what automatic summarization metrics miss Fabbri et al. (2021)
whether models know what they know Kadavath et al. (2022)
black-box test for contamination Oren et al. (2024)
a leaderboard rank has a sampling process Singh et al. (2025)
error bars, clustering and planning for evals Miller (2024)
a judge that recognises its own writing Panickssery et al. (2024)

22.12 What to remember

  • A public benchmark screens; your eval decides. Contamination is real — Chapter 13 found 13.06% of one benchmark’s test set duplicated in its training split — and it is testable rather than merely suspectable, even on a closed model (Oren et al. 2024). On a saturated leaderboard, close scores cannot by themselves separate a real difference from measurement noise, and a leaderboard rank is a statistic with a sampling process behind it (Singh et al. 2025).
  • Size the eval from the difference you need to detect. On a system that is right 80% of the time, 30 items do not reach 80% power even for a perfect replacement — it reaches 57% on an exact test — 100 items per system detects 13.4 points, and 1,000 detects 4.8. Most real improvements are 2–5 points, so most eval sets in use cannot see the thing they were built to see.
  • Small eval sets are for finding failure modes, not for comparing systems. Twenty to fifty items read by hand will tell you what breaks. Telling you which is better takes hundreds of items for the 2–5-point differences most changes produce — how many depends on the effect size and on whether the design is paired.
  • Pairing helps more the more the systems agree. At 10% disagreement, 200 paired items reach 0.54 power against 0.26 for 200 items in each of two arms — twice the power on half the labelling. The gain shrinks to nothing at about 28.9% disagreement, which is essentially where two systems would be statistically independent, and it never goes negative on data you already have.
  • Audit the judge with controls that have known answers. Both judges here scored 100% on identical-text ties — 60/60, which is still only [94.0, 100] — and then one flipped its verdict on 38% of items when the order was swapped and preferred the professional summary 51.7% of the time, while the other flipped on 15% and preferred it 74.2% — preference rates, not accuracies. The control everyone runs is the one that discriminates least.
  • Run pairwise comparisons in both orders, count only the items where the two agree, and report how many items that leaves. A single-order result from a judge with a 38% flip rate is reporting the coin.
  • Verbosity bias did not reproduce here — qwen3.5:9b picked the obviously padded copy 28.3% of the time and gemma3:27b never, so neither favoured it — but that is a narrow claim about obvious padding, not about the elaborate-but-empty answers the folklore is really about.
  • A bad number gets investigated; a good one gets published. That asymmetry, not the bug, is what to design against. Log parse rates, keep a majority-class baseline in every run, score a reference against itself, and treat a suspiciously good result with the suspicion a bad one gets for free.