35 Evaluation in Production
Learning objectives. After this chapter you can:
- place a change on the ladder from offline eval to A/B test, and say what each rung catches and what it cannot;
- measure your own gate’s noise floor before trusting any score it reports;
- design a gate threshold that blocks regressions without blocking noise;
- recognise that a cosmetically identical prompt edit is not a cosmetically identical change;
- curate a golden set from real traffic, and know when it has aged out;
- score a sample of live traffic without scoring all of it;
- treat user feedback as the biased sample it is, and size an A/B test before running it;
- separate the three kinds of drift that look the same on a dashboard.
35.1 The ladder, and what each rung can catch
Everything up to now evaluated systems in a room by themselves. Production adds a constraint that changes the whole design: the thing you are evaluating is already serving users while you evaluate it. That turns evaluation from a single event into a ladder. The table lists the rungs from the cheapest to the most expensive, and each rung sees less than the one after it.
| rung | what it costs | what it catches | what it cannot see |
|---|---|---|---|
| Offline eval | seconds, no users | regressions on cases you thought of | anything your suite does not contain |
| CI gate | one build | a bad change before merge | whether the change helps real users |
| Shadow | double inference, no user impact | crashes, latency, malformed output on live inputs | the effect on users — nobody is served the output, so nothing about behaviour, satisfaction or downstream action is observable |
| Canary | a slice of real users | quality failures, while a failure still reaches only a few users (Chapter 33) | small effects — the slice is too small to resolve them |
| A/B test | days, and real exposure | whether it actually helps | fast-moving regressions — unless you wire in guardrail metrics and a stopping rule, which you should; an A/B test is not a commitment to run it out |
Four of the rung names are engineering jargon. CI (continuous integration) is the set of automated checks that runs on every proposed change to the code before that change is merged into the main version; a CI gate is a check that can refuse the merge. In a shadow deployment the new version receives a copy of live requests, but its answers are only stored, never shown to users. A canary serves the new version to a small share of real users first, so a failure is seen early and rolled back (Section 33.9 covers both). An A/B test splits users at random between the old version (A) and the new one (B) and compares outcomes; Section 35.7 sizes one.
The mistake is not skipping a rung. It is believing a rung about something it cannot see. A shadow deployment that runs clean tells you the new prompt does not crash; on its own it says nothing about whether the answers are worse, because nobody looked at one. That last part is a choice, not a limit of the method: shadow outputs are stored, and you can score them afterwards against a reference, with a validated judge, or by reading a sample. What shadow can never give you is how users would have responded to them. A green CI gate tells you the change did not regress the cases in your suite, which is a claim about your suite.
This chapter is mostly about the second rung, because it is the one teams get wrong in a specific and measurable way.
35.2 A prompt change is a code change
A prompt is the single highest-leverage, least-reviewed artefact in an LLM system. It is a text file; it changes behaviour more than most refactors; and it is routinely edited on a branch, skimmed in review, and merged on the strength of “reads fine to me”.
The discipline that fixes this is borrowed wholesale from testing:
- The suite lives in the repository, next to the prompt it exercises, and moves with it.
- The gate runs on every change to the prompt, not on a schedule and not when someone remembers.
- The gate blocks the merge, or it is not a gate — it is a dashboard, and dashboards do not stop anything.
- A production failure becomes a case in the suite before the fix is merged — and you confirm that the case fails before the fix and passes after it — which is Chapter 30’s regression discipline applied to prompts.
All of that is standard practice imported from software engineering, and none of it is the hard part. The hard part is that an eval score is a random variable and a unit test is not. assert add(2, 2) == 4 returns the same answer every time. A suite score does not, and a gate built as though it does will either block changes at random or wave regressions through. Which of those you get depends on a number almost nobody measures: how much the score moves when nothing changed.
35.3 What the gate can actually see
So measure it. The suite is 100 Turkish customer messages generated from five templates with randomised slots — order code, day, month, city, amount — which means the ground truth is known by construction rather than by hand labelling, and there is no annotation error to argue about. The task is extraction to JSON: five fields per message, 500 field decisions per run. Scoring is exact string match per field, plus a stricter all-five-correct exact match per item.
Three prompts, all extracting the same five fields:
- A — the one that shipped. Six rules: one on output format, then one per field, in the order the fields are listed.
- B — a rule deleted. Identical to A with the
cityrule removed. This is the regression a reviewer waves through, and it is here as a control that the measurement can detect something. - C — the same rules reordered. Byte-for-byte the same six rules as A, listed in reverse. No rule added, none removed, no wording changed. This is the edit that gets approved without comment.
Each prompt ran the full 100-item suite five times at temperature 0.7, and A ran twice more at temperature 0.
The noise floor is smaller than folklore claims
Prompt A, five identical runs of the same suite against the same model:
prompt A, temp 0.7, five runs
field accuracy 95.4 95.4 95.6 95.8 96.0 mean 95.64 sd 0.26
exact match 80 81 81 82 82 mean 81.20 sd 0.84The worst pair of identical runs differs by 0.60 points of field accuracy. That is an estimate of the noise floor, and it is worth being precise about what it is not: a bound. The range of five draws is not a ceiling on the sixth. Two fresh runs exceed 0.60 points about 10% of the time at this sd, and the expected widest gap across fifty runs — which is what a CI gate accumulates in a couple of months — is 1.17 points rather than 0.60. Five runs are also barely enough to estimate a standard deviation at all: the \(\chi^2\) interval on \(\sigma\) from four degrees of freedom runs from 0.16 to 0.75, a factor of 4.8.
So write down sd = 0.26, not the range, because sd is the quantity that scales. The floor is still small — small enough that the common objection to eval gates, that LLM scores are too noisy to gate on, is simply not true for a suite of this size and a task of this shape. It is worth measuring rather than assuming, in both directions: folklore overstates it here, and will understate it on a harder task with a judge in the loop (in Chapter 22 a small judge flipped its verdict on 38% of items when the candidates’ order was swapped).
The deleted rule is trivially visible
Prompt B, the missing city rule, against A:
A B delta
field accuracy 95.64 79.80 -15.84 pt = 26x the noise floor
exact match 81.20 21.40 -59.80 pt = 30x the noise floor
per field (of 100) A B
order_id 100 100
day 91 86
month 99 98
city 100 29 <-- the deleted rule
amount 88 86No subtlety here, and that is the point of including it: a change that removes a rule is 26 times the noise floor and is caught by any gate at all. If this were the only kind of regression, gate design would not be interesting.
The cosmetic edit is not cosmetic
Prompt C reorders the same six rules and changes nothing else.
A C delta
field accuracy 95.64 92.24 -3.40 pt = 5.7x the noise floor
exact match 81.20 72.00 -9.20 pt = 4.6x the noise floor
per field (of 100) A C
order_id 100 100
day 91 74 <-- the entire regression
month 99 99
city 100 99
amount 88 89Reordering six rules that were not otherwise touched cost 3.4 points of field accuracy and 9.2 points of exact match, concentrated almost entirely in one field. The reason this is easy to wave through is that the diff contains no new text: every line already existed and only their order moved, which is the one kind of change a reviewer has no content to react to. The rules are the same rules.
Which of those two numbers survives a test, and which does not. Both prompts ran the same 100 items, so the comparison is paired, and the per-item pairings were not stored — but the margins (the two totals, 91 and 74) bound the answer. A discordant item is one where the two prompts disagree: right under one, wrong under the other. On the day field, 91 correct under A against 74 under C leaves a net 17 discordant decisions in one direction, and across every split consistent with those margins McNemar’s exact test (the paired test of Chapter 22, which looks only at discordant items) gives p between 1.5 × 10⁻⁵ and 0.006. That regression is real on any reading. The exact-match difference does not fare as well: 81 against 72 leaves a net 9, and the same sweep gives p anywhere from 0.004 to 0.24, so it can be significant or nothing at all depending on pairings nobody wrote down. Store the per-item results, which costs nothing and is the difference between a claim and an anecdote.
This is why “prompt engineering” resists being called engineering: the artefact has no interface, so no edit can be proven local. Ordering is load-bearing and invisible.
The failure mode is specific enough to name. A separate probe took 40 items, ran them under each prompt, and inspected every wrong day value:
prompt A: day wrong on 4/40 — 4 of 4 are the leading digits of the amount
prompt C: day wrong on 13/40 — 13 of 13 are the leading digits of the amount
want day 18 got 6 amount 6.842,50
want day 12 got 1 amount 1.289,50
want day 26 got 5 amount 5.669,99
want day 8 got 29 amount 290,50In that probe every single wrong day, under both prompts, is the leading digits of the amount. The model bleeds one field into another, and moving the amount rule above the day rule roughly triples how often it does. The bug was always there — prompt A has it at 10% — and the reorder amplified it.
That is a much more useful finding than “reordering is risky”. It says the regression had a mechanism, the mechanism was visible in the per-field breakdown, and a suite that reported only an aggregate score would have shown a 3.4-point wobble with no way to tell whether it mattered. Report per-field or per-category numbers or you will not be able to act on the aggregate.
Temperature 0 buys reproducibility, not correctness
Prompt A, twice at temperature 0:
field accuracy 95.20 95.20 per-item results byte-identical
exact match 80 80Perfectly reproducible: not merely the same score, but the same result on every one of the 100 items. That is worth having in a gate, because it removes the run-to-run noise and makes a failing run reproducible on a laptop.
Two caveats, both worth stating. Temperature 0 scored 95.20, which sits below the entire temperature-0.7 range of 95.40–96.00 — greedy decoding is one particular sample, not the best one. And it does not fix anything: the leading-digit bug survives greedy decoding, which scored day 90/100 with 8 of its 10 misses being the same amount-bleed. Determinism makes a gate reproducible; it does not make it right. Your users are probably not served at temperature 0 either, so a temperature-0 gate measures a configuration you do not ship.
Two further limits keep “no noise” from meaning “every delta is real”. First, the byte-identical result is a property of this setup — one local server answering one request at a time. Hosted APIs and servers that batch many requests together often return slightly different text at temperature 0, so check it on yours, as Chapter 18 also recommends. Second, any prompt edit, even one that changes no meaning, flips the greedy output on a few items. A small delta between two temperature-0 runs of different prompts can therefore still be chance, and it still needs the paired test above.
35.4 Designing the gate so it fires on the right things
Given a noise floor, a gate is a threshold decision, and the obvious threshold is wrong in an instructive way. To see how a threshold behaves, the measured per-item results were resampled 2,000 times per cell of the tables below. Each draw compares one baseline run against one new run, as a real gate does, and the tables report how often the gate fired. One caveat on provenance: the per-item pairing across variants was not stored (see above), so this resampling cannot have been item-matched across A and C. Read the rates below as an illustration of how a threshold behaves, not as measured operating characteristics of a paired gate.
| suite size | naive gate: A vs A (false block) | A vs B (caught) | A vs C (caught) |
|---|---|---|---|
| 20 | 22.9% | 100% | 93.0% |
| 50 | 35.6% | 100% | 99.6% |
| 100 | 42.5% | 100% | 100% |
The naive gate — block if the new run scores lower than the baseline — catches almost every regression, and blocks 42.5% of changes that changed nothing. Worse, it gets worse as the suite grows: at 20 items the two runs often tie exactly, and ties pass; at 100 items exact ties are rare, so the comparison converges on a coin flip. A team that responds to a flaky gate by adding more test cases makes it flakier.
In this resampling experiment, a one-point threshold reduced the false-block rate:
| suite size | 1-point threshold: A vs A (false block) | A vs B (caught) | A vs C (caught) |
|---|---|---|---|
| 20 | 20.5% | 100% | 90.8% |
| 50 | 6.6% | 100% | 96.2% |
| 100 | 4.3% | 100% | 99.4% |
At 100 items the gate blocks 4.3% of no-op changes and still catches the reordered prompt 99.4% of the time. Now the suite size helps, as it should. The rule that generalises:
The deleted rule is not the interesting case. The reordered prompt is: a diff whose every line is a line that was already there, nearly six times the five-run range, and entirely inside one field.
Set the threshold from your measured noise floor, with headroom. Run your suite against an unchanged system five times and record the run-to-run sd — not the range, which is a five-draw estimate that future runs will exceed. A threshold placed just above the observed range barely helps: at this sd two fresh runs clear 0.60 points a tenth of the time. Roughly 4 × sd is the working rule, which here is the 1.0-point threshold used above.
The headroom is large because sd itself is uncertain. If the five-run sd were exact, the difference between two runs would have a standard deviation of \(\sqrt{2}\times\)sd, and a threshold of \(k \times\) sd would falsely block about \(\Phi(-k/\sqrt{2})\) of no-op changes (\(\Phi\) is the standard normal cumulative distribution): roughly 8% at 2 × sd, 2% at 3 × sd and 0.2% at 4 × sd. The resampling above found 4.3% at 1.0 point — twenty times that 0.2%. It behaves as if the true run-to-run sd were nearer 0.45 than 0.26. (Solving the formula for 4.3% gives about 0.45 once you account for scores moving in 0.2-point steps, so “more than one point lower” means at least 1.2 points; ignoring that discreteness gives about 0.41.) Either value is well inside the \(\chi^2\) interval of 0.16 to 0.75. Four times a five-run estimate is what keeps false blocks at a few percent even when the estimate came out low. That is a twenty-minute experiment that turns a gate people learn to ignore into one they trust. A gate that is overridden by habit is worse than no gate, because the record then shows a green check where a human actually skipped the test.
35.5 The golden set has a lifecycle
The suite above was generated, which is honest for measuring a noise floor and useful for exactly the capabilities its templates define — and not representative of production quality, because it contains exactly the phenomena those templates contain and nothing else. A real golden set comes from real traffic, and it is not a static artefact.
It is born from failures. The first entries should be things that actually broke. Chapter 34’s traces are the raw material: the run someone complained about is a scenario with its inputs already recorded.
It must be sampled deliberately, not conveniently. Traffic is heavy-tailed, so uniform sampling gives you a hundred variations of the most common request. Stratify — by intent, by length, by language, by customer segment — and deliberately oversample the rare and the hard, or the set will be easy in exactly the way production is not.
It carries personal data, and that is a decision. Real traffic contains real customers. Chapter 34’s redaction argument applies, and pseudonymising before the set enters the repository is the only place it can be done once; Chapter 38 covers the obligations. A golden set is a copy of production data with a long retention period, which is precisely the thing your policy is about.
It ages, and the failure is silent. Two mechanisms. The set gets easier as you fix the things in it, so a stable score means less each quarter — the same saturation Chapter 22 described for public benchmarks, arriving in your own repository. And the traffic moves out from under it: users find new phrasings, the product ships new features, and the set describes last year’s product. A golden set that has not changed in six months is not stable, it is stale. Budget refresh as routine work: add items from recent failures and record when each item entered. Retire items nothing fails any more, but only from the part of the set sampled to represent current traffic; items that entered as real failures are regression tests, and they stay.
Its size is set by Chapter 22’s arithmetic, not by taste. That chapter measured it: at 80% power on a system that is right 80% of the time, 30 items detect nothing at all, 100 items detect 13.4 points, and 1,000 detect 4.8. Those figures belong to an 80% baseline, so redo the arithmetic for your own metric rather than borrowing them. Exact match here sits at 81%, close to that baseline, and the reordered prompt cost it 9.2 points — less than the 13.4 points that 100 items per system can resolve without pairing. Field accuracy is a different case: near 96% the binomial variance \(p(1-p)\) is about a quarter of its value at 80%. This suite has 500 field decisions per run, and the same calculation says 500 unpaired decisions per system resolve a drop from 95.6% of only about 4.4 points (1,000 would bring that down to about 2.9). The reorder’s 3.4-point drop sits below that. Yet the gate on these same 100 items fires on the reordered prompt 99.4% of the time.
Those two numbers answer different questions, and it is worth keeping them apart. The 13.4 and 4.8 are minimum detectable effects: they carry the uncertainty of estimating an unknown effect from a fresh sample. The 99.4% is an operating characteristic of a decision rule: given these fixed 100 items and this measured difference in means, how often does the threshold trip? It conditions on the observed 3.40 being the true effect, so it is not power and does not generalise past this suite. Both are worth reporting — the gate’s false-block and catch rates for operations, the paired test above for the claim that the regression is real. Pairing is what makes the second possible at this sample size, which is why a gate compares runs on identical items rather than on fresh samples.
35.6 Scoring live traffic without scoring all of it
Offline evaluation asks whether the system handles cases you collected. Online evaluation asks whether it is handling the cases arriving right now. You cannot score all of them — with a judge about as large as the model it judges, judging every request roughly doubles inference cost — so the question is what to sample.
The cheap signals first, because they need no judge and cover all traffic: schema validity and parse rate (Chapter 18), refusal rate, latency and token counts (Chapter 34), tool error rate, retry rate, and truncation at the step ceiling (Chapter 33). None of these tell you whether an answer was good, and all of them move sharply when something has broken.
Then a judge, on a sample, with three things sampled together:
- a small uniform slice — a fixed percentage, so the baseline is comparable over time;
- everything that already looks wrong — errors, retries, ceiling hits, thumbs-down, escalations;
- a stratified slice of the rare cases you would otherwise never see.
Two disciplines make the number usable. Audit the online judge exactly as Chapter 22 audits an offline one — it is the same instrument with less supervision, and a judge that flips on 38% of items when you swap the order will do so in production too. And record the judge’s version and prompt alongside the score, because a judge upgrade shifts the whole series; Chapter 32’s pinning argument applies to the evaluator as much as to the model.
35.7 User feedback and A/B tests
Two more kinds of evidence exist only in production: what users tell you, and what they do when you change the system under them.
User feedback is a biased sample, not a metric. Explicit signals — a thumbs-up or thumbs-down, a star rating, a written complaint — come from a small minority of users, and not a random one: people who are annoyed, or delighted, rate more often than people who simply got their answer and left. That is selection bias, and it means a thumbs-down rate describes the users who chose to click, not your traffic. Implicit signals — the user regenerates the answer, copies it, edits it, asks the same question again, abandons the conversation or asks for a human — cover everyone, but each is a proxy with more than one meaning: a regenerate can mean “wrong” or “show me another version”. A thumbs-down also mixes “wrong” with “not what I wanted”: a correct refusal, or an accurate answer the user did not like, collects one too. Optimise a system directly for thumbs-up and you meet Chapter 9’s reward hacking again: answers that please rise, answers that are right but unwelcome fall. And the rate moves when the interface moves — a button that is easier to reach changes the rate with no change in quality. So use feedback the way the previous section does: as a sampler that sends suspicious cases to the judge and to people, and as a source of golden-set candidates — not as the quality score.
An A/B test answers the question the other rungs leave open: does the change help real users? It is the only rung that can, because random assignment makes the two groups alike in everything except the version they received. Four decisions are written down before it starts:
- The unit of randomisation. Usually the user, not the request, so one person does not meet two behaviours in one conversation. Analyse at the same level: one user’s requests are correlated, so treating 50,000 requests from 5,000 users as 50,000 independent observations overstates your precision.
- One primary metric — the outcome the change is meant to move, such as the share of conversations resolved without a human. Choosing it after seeing the data is the many-tests problem Chapter 22 warns about: check enough metrics and one of them crosses p < 0.05 by chance.
- Guardrail metrics — what must not get worse while you pursue the primary one: latency, cost per conversation, escalation rate, complaint rate. The name is A/B-testing vocabulary, not the runtime guardrail of Chapter 37, although a clearly breached guardrail metric should stop the test just as fast.
- The sample size and the stopping rule.
The sample size is Chapter 22’s arithmetic again, but real effects on user outcomes are small, so the numbers become large. For two independent groups, detecting a change from \(p_1\) to \(p_2\) at 80% power and a two-sided 5% level needs about
\[ n \approx \frac{(1.96 + 0.84)^2 \,\bigl[p_1(1-p_1) + p_2(1-p_2)\bigr]}{(p_1 - p_2)^2} \]
users per arm. If 20% of conversations now end in an escalation to a human and you hope to cut that to 18%, that is about 6,000 users per arm; to detect a one-point change, from 20% to 19%, about 24,600 per arm. Halving the effect you want to see multiplies the sample by about four, because the difference sits squared in the denominator. At 5,000 eligible users a day split evenly, the one-point test needs about ten days — which is why the A/B test is the slow rung.
The stopping rule matters as much as the size. Checking the p-value every day and stopping at the first p < 0.05 — peeking — inflates the false-positive rate far beyond 5%: in a simulation of an A/A test (both arms identical) with ten equally spaced looks, about 19% of runs crossed the line at some look. Fix the duration in advance and test once at the end, or use a sequential test designed for repeated looks. Guardrail metrics are watched continuously for the opposite reason: stopping early for harm is cheap, while a false win ships. And before the first real test, run an A/A test through the whole pipeline — it is this chapter’s noise-floor measurement, applied to the experiment platform.
35.8 Three kinds of drift that look identical on a dashboard
A quality metric declines. The dashboard shows one line going down, and there are three unrelated causes with three unrelated fixes.
| what moved | what it looks like | how to tell | what to do |
|---|---|---|---|
| The model | a step change on a specific day, across all traffic | gen_ai.response.model changed while gen_ai.request.model did not (Chapter 34) |
pin versions (Chapter 32); re-run the golden set against both |
| The inputs | a gradual slide, concentrated in some segments | input distribution moved: new phrasings, lengths, languages, a new customer | refresh the golden set; the system did not change, the job did |
| The system | a step change you can attribute to a deploy | it lines up with a release (Chapter 33) | the gate should have caught it; add the case that escaped |
Recording the responded model and the input distribution is what separates these; without them, all three are “quality is down” and the response is a guess. The distinguishing evidence is cheap and must be collected in advance — you cannot reconstruct last month’s input distribution after the fact.
How do you see the inputs move? Compare recent traffic with a reference window on a few cheap descriptors: input length, language, the share of each intent (from a classifier, or from clusters of input embeddings as in Chapter 11), and the share of inputs that sit far from every golden-set item in embedding space. A chi-square test on the binned counts, or simply a chart of the shares over time, shows which segment has grown; the last descriptor also points to the new inputs your golden set does not cover.
A fourth case deserves naming because it is often the most common: nothing drifted, and your metric is noisy. That is what the noise floor is for. A weekly re-run of a fixed suite that moves 0.5 points is not a signal on a suite whose measured floor is 0.6. An online score has a larger floor still, because each week it is computed on different requests: on top of run-to-run noise it carries ordinary sampling error. For a judged sample of 500 requests with an 80% pass rate that is a standard error of about 1.8 points — compute it before you read a weekly wobble as drift.
35.9 Routing the uncertain cases to people
Automated scoring runs out where it matters most: the cases where the system is unsure, the judge is unreliable, or the cost of being wrong is high. Those go to people, and the pipeline is Chapter 5’s, pointed at production.
What makes it work is that routing is a policy, written down and measured, not a queue somebody watches. Three inputs decide it: model uncertainty where you have it, disagreement between judges or between a judge and a rule, and stakes — a refund, a medical claim and a shipping query do not deserve the same scrutiny. Chapter 37 is where the enforcement side of that policy lives.
The output of human review is not one artefact but three, and teams routinely capture only the first: the immediate decision on that case; a labelled item for the golden set; and, when the same failure appears a third time, a change to the system rather than a third correction. If human review produces only decisions, you have bought a manual process, not a feedback loop.
The cost model that follows is worth stating plainly, because it is what makes the case to a budget-holder: review capacity is fixed, so the question is never “should humans check this” but “which slice of traffic do the humans we have look at”. That makes routing a resource-allocation problem with an obvious objective — send them the cases where their judgement changes an outcome.
35.10 The loop this part builds toward
Every chapter in this part has been one arc of the same loop, and it is worth seeing assembled:
- Serve it (Chapter 31) and route it (Chapter 32).
- Deploy it so failure is survivable (Chapter 33).
- Instrument it (Chapter 34), so a complaint reaches a span.
- Turn that span into a test — this chapter, and Chapter 30’s discipline.
- Gate the next change on the test, with a threshold set from a measured noise floor.
- Sample live traffic to find the failures nobody reported.
- Route the uncertain ones to people, and feed their labels back to step 4.
The loop is not exotic. It is the operational rhythm of any mature software system, with two adjustments for this setting: the tests are statistical rather than deterministic, and the artefact under test is a text file that nobody thinks of as code.
35.11 Intuition checks
Answer each one before you open it. The first three carry the chapter’s measurements — the noise floor, the cosmetic edit that was not cosmetic, and the gate that gets worse as it grows. The last three are about what a green gate does and does not license you to believe.
Measure it before believing either answer. Here they were not.
The same prompt, run five times over the same 100-item suite, gave sd = 0.26 and spanned 0.60 points of field accuracy. That is the noise floor, and it is small enough to gate on — but set the threshold from the sd with headroom rather than from the range, because the range of five draws is an estimate the sixth run will sometimes beat. About 4 × sd is the working rule.
The number is task-specific and will be larger with a judge in the loop (in Chapter 22 a small judge flipped on 38% of items when the order was swapped) or a smaller suite. The transferable part is not 0.26; it is that a twenty-minute experiment — run the unchanged system five times, record the spread — replaces an argument with a threshold.
No, and this is the chapter’s central measurement.
Reordering six rules — none added, none removed, no wording changed — cost 3.40 points of field accuracy and 9.20 points of exact match. Name the denominator when you quote a multiple: that is roughly six times the five-run range of 0.60 points, and about thirteen times the run-to-run sd of 0.26. Every “× noise” figure in this chapter is against the range, because that is what a reader compares a single new run to; sd is the number to build the threshold from. The whole regression sat in one field: day fell from 91 to 74 out of 100.
The mechanism is nameable. Across a 40-item probe, every wrong day under both prompts was the leading digits of the amount — the model bleeds one field into another, and moving the amount rule above the day rule tripled how often it did. The bug existed in the shipped prompt at 10%; the reorder amplified it.
A prompt has no interface, so no edit to it is provably local. That is the argument for gating prompt changes at all.
Not on its own — with a naive gate, more cases make it flakier.
Measured: a gate that blocks whenever the new score is lower falsely blocks 22.9% of no-op changes at 20 items, 35.6% at 50, and 42.5% at 100. At small sizes the two runs often tie exactly and ties pass; as the suite grows, exact ties disappear and the comparison converges on a coin flip.
Add a threshold first. At one point and 100 items, false blocks fall to 4.3% while the reordered prompt is still caught 99.4% of the time — and now the extra cases help, as you expected them to. Set the threshold from your measured noise floor rather than from a round number.
It is a real benefit with two real limits.
At temperature 0 the two runs here were byte-identical per item — not just the same score, the same result on all 100. That removes the run-to-run noise on this setup (batched or hosted serving may not be as exact) and makes a failing gate reproducible on a laptop, which is worth a lot.
But greedy decoding scored 95.20, below the entire temperature-0.7 range of 95.40–96.00: it is one particular sample, not the best one. And it fixed nothing — the leading-digit bug was still there. You also probably do not serve users at temperature 0, so the gate measures a configuration you do not ship.
Reasonable resolution: gate at temperature 0 for reproducibility, and separately track the sampled configuration you actually serve.
Almost certainly not — it is the most common way an eval program dies quietly.
Two things happen at once. The set gets easier as you fix what is in it, so a flat score reflects a solved set rather than a healthy system: Chapter 22’s benchmark saturation, arriving in your own repository. And traffic moves out from under it — new phrasings, new features, new customers — so the set describes last year’s product.
A stale set is worse than no set, because it produces a green check. Keep two sets rather than one: a regression set that only ever grows, where an item earned its place by once being a real failure and stays forever precisely so an old bug cannot return unnoticed, and a representative set sampled from current traffic and refreshed on a schedule. Retiring items nothing fails is right for the second and wrong for the first. Record when each item entered, and treat refresh as routine work rather than a project.
Four candidates, and you cannot distinguish them after the fact without evidence collected in advance.
The model moved — a step change on one day, visible as gen_ai.response.model changing while gen_ai.request.model did not (Chapter 34). The inputs moved — a gradual slide in some segments; the system is unchanged and the job is not. The system moved — a step change that lines up with a deploy, and a case your gate did not contain. Or nothing moved and the metric is noisy.
Rule out the fourth first, using the noise floor. Then the responded model and the input distribution separate the other three — both cheap, and both impossible to reconstruct retrospectively.
35.12 Interview questions
On sourcing — the same caveat as Chapter 2. The questions below recur in the two credible open sources this book uses throughout, plus the ones this chapter’s own measurement makes answerable.
Testing whether you understand the problem
Q: How is evaluating an LLM system in CI different from running unit tests?
The assertion is statistical rather than deterministic. A unit test returns the same answer every run; a suite score is a random variable, so a gate needs a threshold and the threshold needs a measured noise floor. Say how you would get one: run the unchanged system five times over the same suite and record the run-to-run sd, not only the widest gap. Measured here on 100 items that was sd = 0.26 points, with an observed range of 0.60 — small enough to gate on, which is the opposite of the usual assumption. The threshold goes at roughly 4 × sd, because later runs will exceed a five-run range.
Q: A change to a prompt only reorders existing instructions. Does it need an eval run?
Yes, and this is the case that convinces people. Reordering six rules with no other edit cost 3.4 points of field accuracy and 9.2 of exact match here — about six times the noise floor — with the entire regression inside one field. A prompt has no interface, so no edit is provably local, and a reviewer has no way to tell a cosmetic prompt change from a behavioural one by reading it.
Q: How would you size a golden set?
From the difference you need to detect, using Chapter 22’s arithmetic: at 80% power on an 80%-accurate system, 100 items resolves 13.4 points and 1,000 resolves 4.8 — and redo that for your own baseline, because near 96% accuracy the variance is about four times smaller. Then note the distinction that matters for gating — those are unpaired minimum detectable effects, and a gate compares two runs on the same items. Do not cite the 99.4% catch rate as evidence for pairing: the resampling behind it could not match items across prompts, and it takes the observed 3.4-point drop as the true effect, so it illustrates a threshold rather than measuring a paired gate. What pairing does buy is a test on the same items: McNemar’s test finds the day regression real on every pairing consistent with the margins, and only stored per-item results would settle the exact-match difference.
Testing whether you have run one
Q: Your eval gate blocks good changes and people started overriding it. Fix it.
First diagnose rather than loosen it: the usual cause is a gate with no threshold, which measured here falsely blocks 42.5% of no-op changes at 100 items — and gets worse as the suite grows, because exact ties become rare. Set a threshold from the measured noise floor; at one point that fell to 4.3% while still catching a real regression 99.4% of the time. Then say why it is urgent: a habitually overridden gate is worse than none, because the override is recorded as a green check.
Q: Offline evals are green and users are complaining. What do you do?
Accept that the suite is the hypothesis being falsified. Go to traces (Chapter 34), find the complained-about runs, and check whether the failure class exists in the suite at all — usually it does not, which is a curation problem rather than a scoring one. Add those cases, then ask why they were absent: convenience sampling from a heavy-tailed distribution, or a set that has aged out. Also check the fourth possibility — that the offline metric is measuring something users do not care about.
Q: How much production traffic would you score with an LLM judge, and which?
Not all of it: with a judge of similar size, judging every request roughly doubles inference cost. Take the free signals on 100% — parse rate, refusals, latency, tool errors, ceiling hits — then judge three overlapping samples: a fixed uniform slice for a comparable baseline, everything that already looks wrong, and a stratified slice of rare cases. Then the two disciplines: audit the online judge the way you would an offline one, and pin and record its version, because a judge upgrade shifts the entire series.
Q: Product wants the thumbs-down rate as the headline quality metric. What do you say?
That it is a biased sample, not a measurement. Only a small, self-selected minority clicks; a thumbs-down mixes “wrong” with “unwelcome”; and the rate moves with the interface. Optimising it rewards pleasing answers over correct ones. Keep it as a sampler that routes cases to the judge and to reviewers, and measure quality with a judged uniform sample — or, for a specific change, with an A/B test sized in advance, with guardrail metrics and a stopping rule fixed before the first look.
35.13 The 2026 toolbox
The measurement needed a model, a scoring loop and the standard library, so the first table is short. Read the second as two layers: harnesses that run and score a suite, and platforms that additionally store the runs so you can compare them over time — the second group is what a golden set’s lifecycle actually needs.
| library | version used | what it is for here |
|---|---|---|
ollama |
0.24.0 | serving qwen2.5:7b for all 1,700 suite items |
pytest |
9.1.1 | the gate this chapter’s discipline borrows from; the suite runs as ordinary tests |
Worth knowing about, not used above:
| project | version | when you would reach for it |
|---|---|---|
promptfoo |
0.122.2 | declarative prompt test suites with assertions and a CI runner; the shortest path from this chapter to a working gate |
inspect-ai |
0.3.261 | the UK AI Security Institute’s (formerly AI Safety Institute) eval framework; solvers and scorers as composable pieces, strong on reproducibility |
deepeval |
4.2.0 | eval metrics packaged as pytest assertions, which fits the gate-as-test framing directly |
ragas |
0.4.3 | retrieval-specific metrics when the thing under test is a RAG pipeline (Chapter 20) |
langfuse |
4.15.1 | traces, datasets and eval runs in one place, so a golden set and the traces it came from stay linked |
arize-phoenix |
20.5.0 | open-source tracing plus evaluation you can run locally; useful for the trace-to-test step |
braintrust |
0.36.0 | hosted eval platform built around comparing runs across versions over time |
35.14 Lab: measure your gate before you trust it
Step 1 — build a suite with ground truth you did not label
Fifty to a hundred items, generated from templates with randomised slots, on a task you care about. The point of generating rather than labelling is that annotation error cannot contaminate the noise-floor measurement. Keep it in the repository next to the prompt.
Step 2 — measure your noise floor
Run the unchanged system over the suite five times and record both the widest gap and the standard deviation. Do this before writing any gate logic. Write the sd down where the gate’s threshold is defined, with the date and the model version, and set the threshold at roughly four times it. Five runs pin the floor only loosely — the \(\chi^2\) interval on \(\sigma\) from four degrees of freedom puts the true sd anywhere from about 0.6 to about 3 times the estimate — so if the threshold decision is close, run twenty.
Step 3 — make a regression you would approve in review
Not a deleted rule — that is too easy. Reorder the instructions, or reword one sentence without changing its meaning. Run it five times. Report the delta as a multiple of your noise floor, and report per-field or per-category numbers, not only the aggregate.
Step 4 — find the mechanism
If the change moved the score, find out how. Print the wrong predictions and look for a pattern; in this chapter every wrong day turned out to be the leading digits of the amount. A regression with a named mechanism becomes a test case; a regression that is only a number becomes an argument.
Step 5 — build the gate and measure its false-block rate
Wire the suite into CI so it blocks a merge. Then submit no-op changes and record how often it blocks them — and pick the count from the precision you need, because five will not do it: zero blocks out of five still leaves a 95% Wilson upper bound of 43%, which cannot distinguish “a few percent” from “disastrous”. Resolving a few-percent rate takes on the order of a hundred trials, which is cheap when the no-op is a whitespace commit and a scripted re-run. If the measured rate is above a few percent, your threshold is too tight — go back to step 2 rather than to the override button.
Step 6 — run it at temperature 0 as well
Compare reproducibility and score. Confirm whether your per-item results are identical across runs, and whether the greedy score sits inside or below your sampled range. Decide deliberately which configuration the gate uses, and write down why.
What to hand in
Your measured noise floor with the five run scores behind it; the delta from a review-invisible edit, as a multiple of that floor, with per-field breakdown; the mechanism you found for it; your gate’s threshold with the reasoning; and its measured false-block rate, with the number of no-op trials and the 95% Wilson upper bound. Use enough trials to resolve the rate you need (step 5); five cannot.
35.15 Exercises
Your own noise floor. Run an unchanged system five times over your real eval set. Report the run-to-run
sdand the widest gap, and compare them to the size of the last improvement your team claimed. If the claim is smaller than the floor, say what that means.The invisible edit. Take a prompt in production and reorder its instructions without changing any of them. Run both versions five times. Report the delta with per-field numbers, and whether a reviewer could have predicted the direction.
Threshold sweep. Using resampled runs, plot false-block rate and catch-rate against threshold for suite sizes 20, 50 and 100. Pick your operating point and justify it in one paragraph.
Pairing versus power. Compute the unpaired sample size needed to resolve your measured regression at 80% power (Chapter 22), then compare it to the suite size at which your paired gate catches it 95% of the time. Explain the gap.
Golden set age. For each item in your eval set, record when it was added, whether it entered as a regression item or a representative one, and whether anything has failed it in the last quarter. Retire the representative items nothing fails, keep every regression item, and report how many of each are left.
Trace to test. Take three real production complaints, find their traces (Chapter 34), and convert each into a suite case. Then check whether the original suite contained anything in that failure class.
The three drifts. Instrument enough to distinguish a model change, an input-distribution change and a system change. Then simulate one of each and confirm your dashboard tells them apart. Note which evidence you had to collect in advance.
Size an A/B test, then peek at it. Pick a yes/no outcome your system logs, and compute the users per arm needed to detect a one-point and a two-point change at 80% power (Section 35.7), and how many days that takes at your traffic. Then simulate an A/A test checked after every tenth of the data and report how often it “finds” a difference at p < 0.05.
35.16 Where to go deeper
Before reading: run your existing eval suite twice, unchanged, and look at the two numbers. Whether you can say which differences are real is the whole subject of this chapter.
This chapter cites no papers, so there is one table rather than the usual two. The statistical foundations it stands on are in Chapter 22; what follows is the tooling that turns them into a gate.
| resource | what it is |
|---|---|
promptfoo |
declarative prompt evals with a CI runner; the shortest path from this chapter to a gate that blocks a merge |
inspect_ai |
the UK AI Security Institute’s framework; read it for how it separates solvers from scorers, which is the abstraction most homegrown harnesses are missing |
deepeval |
eval metrics as pytest assertions — the gate-as-test framing in library form |
ragas |
retrieval-specific metrics for when the system under test is a RAG pipeline (Chapter 20) |
langfuse |
traces, datasets and eval runs in one system, which is what keeps a golden set linked to the traces it came from |
phoenix |
open-source tracing and evaluation you can run locally; the cheapest way to practise the trace-to-test step |
braintrust-sdk |
a hosted platform organised around comparing eval runs across versions over time |
openai/evals |
an early, widely-copied eval registry; useful mostly for reading how other people specified their suites |
35.17 What to remember
A prompt change is a code change, and an eval score is not a unit test. The suite lives in the repository, runs on every prompt change, and blocks the merge — but the assertion is statistical, so the gate needs a threshold, and the threshold needs a number almost nobody measures.
Measure your noise floor; it is a twenty-minute experiment. Five identical runs of the same 100-item suite gave a run-to-run
sdof 0.26 points of field accuracy, with an observed range of 0.60. That is small enough to gate on, which contradicts the folklore. Set the threshold from thesd(about 4 ×sd), not from the range, which later runs will exceed — and without a measured floor no later delta is interpretable.A cosmetic prompt edit is not a cosmetic change. Reordering six rules with nothing added, removed or reworded cost 3.40 points of field accuracy and 9.20 points of exact match — about six times the noise floor — and the entire regression sat in one field,
dayfalling from 91 to 74. A prompt has no interface, so no edit to it is provably local.Report per-field numbers, not only an aggregate. Across a 40-item probe, every wrong
dayunder both prompts was the leading digits of the amount: the model bleeds one field into another, and moving theamountrule above thedayrule tripled how often it did. The aggregate showed a wobble; the breakdown showed a mechanism, and only a mechanism becomes a test.A gate with no threshold gets worse as your suite grows. Blocking whenever the new score is lower falsely blocked 22.9% of no-op changes at 20 items and 42.5% at 100, because exact ties vanish and the comparison converges on a coin flip. A 1-point threshold at 100 items gives 4.3% false blocks and still catches the reordered prompt 99.4% of the time.
Temperature 0 buys reproducibility, not correctness. Two runs were byte-identical per item — but greedy decoding scored 95.20, below the whole sampled range of 95.40–96.00, and the leading-digit bug was still there. It is also a configuration you probably do not serve.
The golden set has a lifecycle, and staleness is silent. It is born from real failures, stratified rather than conveniently sampled, and it carries personal data. It gets easier as you fix what is in it and the traffic moves out from under it, so a set that has not changed in six months is not stable, it is stale — and it still produces a green check.
Score live traffic in layers. Free signals on everything — parse rate, refusals, latency, tool errors, ceiling hits — then a judge on three overlapping samples: a fixed uniform slice, everything that already looks wrong, and a stratified slice of the rare. Audit that judge, and pin and record its version, because a judge upgrade shifts the whole series.
User feedback samples; it does not measure. A/B tests measure, slowly. Thumbs come from a self-selected minority and reward pleasing answers, so use them to route cases, not as the score. An A/B test randomises users, fixes one primary metric, guardrail metrics and a stopping rule in advance, and is large: a one-point change from 20% needs about 24,600 users per arm, and peeking daily at ten looks turns a 5% false-positive rate into about 19%.
Three drifts look identical on a dashboard. The model moved, the inputs moved, or the system moved — distinguishable only with the responded model and the input distribution, both of which must be recorded in advance. Rule out the fourth possibility first: that nothing moved and the metric is noisy.