17  Prompting and In-Context Learning

Chapter status: drafted. The measurements here were run on the same Turkish product-review corpus as Chapter 4 and Chapter 12 — 233,508 de-duplicated reviews — but on a balanced sample rather than the natural 93.7%-positive distribution, so that accuracy is readable against a 50% floor and the effect being measured is the prompt rather than the class prior. Every prompt variant measured here is zero-shot: the in-context learning half of the title is carried by the literature and by Chapter 12‘s few-shot numbers rather than by a measurement of its own. The \(k\)-shot curve this chapter ought to have — accuracy against number of demonstrations, run twice with gold and with random labels — is Section 17.10’ third exercise, and it is the measurement we would add first. Software: ollama 0.24.0 serving qwen3.5:9b, Python 3.14.5.

Learning objectives. After this chapter you can:

17.1 Why prompting works at all

A pretrained model predicts the next token. Nothing in that objective says it should follow an instruction, and the first large models mostly did not — you had to trick them, by writing a prefix whose natural continuation happened to be the answer you wanted.

Reynolds and McDonell (2021) is the historical document for that era, and reading it now is a good reminder of how much of “prompt engineering” was working around a missing capability.

Two things changed. Scale made models able to infer a task from examples in the context (Brown et al. 2020), and instruction tuning, followed by preference training (Chapter 9), trained them to treat an instruction as an instruction rather than as text to continue. The second is why modern prompting feels like asking rather than tricking.

in-context learning (bağlam içi öğrenme) — the model doing better at a task because examples of it are sitting in the prompt. No weight is updated and nothing survives the call, which is why learning is a slightly misleading word for it. ↩︎

In-context learning is the first of those: you put input-output pairs in the prompt, and the model does better on the next input, with no parameter updated and nothing retained after the call.

Two words you will meet constantly name how many examples you give. A zero-shot prompt has none — only the instruction and the input. A few-shot prompt adds a handful of demonstrations (input–output pairs), and \(k\)-shot names the exact number: an 8-shot prompt carries eight.

So it is not learning in the sense Chapter 9 uses the word — nothing is fitted, nothing persists. That is a claim about the mechanism, not about what the model can pick up: a mapping it genuinely acquires from the context is still acquired from the context, and a paragraph below names exactly that case. A useful working description of the common case is task inference: the examples narrow down which task you are asking for, out of the many the model already knows how to do.

That framing predicts something odd, and the prediction holds. Min et al. (2022) replaced the labels in the demonstrations with random ones and found performance mostly survived. What the examples were mainly conveying was the shape of the task — the label space, the format, the distribution of inputs — not the input-to-label mapping.

That is worth knowing before you spend a week curating perfect examples.

Two lines of work say how that narrowing happens, and they are worth having because “task inference” is otherwise just a phrase. Xie et al. (2022) formalises it as implicit Bayesian inference: the demonstrations sharpen a posterior over latent concepts the model already acquired in pretraining, which is why the distribution of the examples matters more than their correctness. Olsson et al. (2022) gives the circuit-level version — attention heads that complete [A][B] … [A] → [B], which appear at exactly the point in training where in-context learning ability jumps. After Chapter 7, that is the hardware answer to “how can examples in a prompt work without changing a weight”.

And a limit on Min et al. (2022) worth carrying, because this chapter repeats it several times. Pan et al. (2023) separates task recognition — the demonstrations tell the model which task — from task learning, where they teach it the mapping. Random labels leave task recognition intact, which is what Min et al. measured; task learning does exist, grows with model scale, and is exactly what random labels destroy. So “the labels do not matter” is true of the recognition part and false of the learning part, and it is not a licence to use wrong labels.

It also predicts the failure mode. If demonstrations mostly convey format, then a model that already knows the format gains little from them — which is exactly what Chapter 12 measured, where eight examples were worth +7.17 macro-F1 on one model and +0.51 (\(p = 0.62\)) on a stronger one. The technique did not stop working; the models stopped needing it.

The claim this chapter is built on

The prompt is part of the model. Not a message to it — a component of the system, sitting in the same position a feature-extraction function sits in Chapter 4’s pipeline. It has a version, it has a contract, it has regressions, and changing it changes your output distribution as surely as swapping the checkpoint would.

The rest of the chapter is what follows from taking that seriously, and Section 17.4 is the measurement that makes it hard to avoid.

What we will look at. First the pattern library — five things that work, and what job each one is actually doing, so you can tell when a pattern will not help. Then the shift from writing requests to writing specifications, which is what makes a prompt checkable at all. Then the measurement this chapter turns on: twelve prompts that mean the same thing, spanning almost fifteen points of accuracy on one model and one task, with the role prefix finishing eleventh of twelve. And finally the machinery that follows once you accept the number — reproducibility, a frozen set, paired tests, and a version string.

17.2 A pattern library

If the prompt is a component, these are its parts. Five patterns cover most of what works, and none of them is a magic word — each is doing an identifiable job, and naming the job is what tells you when the pattern will not help.

Role. You are an experienced customer-feedback analyst. The job is to select a region of the model’s distribution — vocabulary, register, what counts as a relevant detail. It is weakest when it asks for expertise the model does not have (a role prefix does not make a model a doctor) and strongest when it disambiguates style, which is a real and frequent need.

In a chat model, the role and any standing rules usually go in the system prompt — a separate message, before the user’s turn, that the chat template (Chapter 9) marks with its own tokens. Models are trained to give it priority over what the user writes, but it is still text in the same context window, not a setting the model cannot ignore: it makes instructions stronger, not tamper-proof.

Structure. Sections, delimiters, an explicit place for the input. The job is to make the boundary between your instruction and the user’s data unambiguous. <review>...</review> is not decoration; it is what tells the model which span is data. Be precise about what it is not: a delimiter marks that boundary, it does not enforce it. Text inside the tags is still text the model reads, and a review that closes the tag and issues its own instruction is the standard attack, prompt injection — Chapter 36 is where that is treated as the security problem it is, and structure alone is not a defence.

Demonstrations. Input-output pairs. The job, per Min et al. (2022), is mainly to fix the format and label space. Use them when the output shape is unusual or hard to describe, and expect less from them when the task is one the model has obviously seen.

Decomposition. Ask for one thing at a time, or ask for intermediate results before the answer. The job is to move work out of a single forward pass and into tokens — which is Chapter 10’s subject, and where the measurement of when it helps lives. This chapter’s contribution is the practical corollary: a prompt that asks for four things at once will do at least one of them badly, and splitting it is usually cheaper than fixing it.

Output contract. State the exact shape of the answer, and make it the easiest thing for the model to produce. This is the pattern with the most leverage and it gets its own section.

This is the prompted form of chain-of-thought: the model writes intermediate steps before its final answer. It belongs to Chapter 10, which measures it properly, and the short answer is that the phrase is largely a relic. Kojima et al. (2022) found it worked on models that had not been trained to reason; models that have been trained to reason produce the trace whether or not you ask, and adding the phrase to a reasoning model mostly wastes tokens.

The durable part is not the phrase but the mechanism: more tokens before the answer means more computation available to produce it. Ask for intermediate results when the task needs them, and skip the incantation.

17.3 Prompts as specifications

The most useful shift in this chapter is grammatical: stop writing prompts as requests and start writing them as specifications.

A request says what you want. A specification says what a valid output is — which means it can be checked, and a prompt whose output can be checked is a prompt you can test.

request         "Summarise the customer's complaint."

specification   "Return one JSON object with keys `issue` (string, max 15
                 words), `severity` (one of: low, medium, high) and
                 `product` (string, copied exactly from the review, or null
                 if no product is named). Return nothing else."

The second version gives you three things the first does not: a parser that either succeeds or fails loudly, a set of values you can validate, and a null that gives the model a legal way to say no product is named instead of inventing one — though, as Chapter 18 stresses, you still have to check that the model actually uses it.

Three rules that follow:

  1. Name the failure output. If the model cannot comply, what should it return? A prompt without an escape hatch gets a fabricated answer instead of an admission, because you left the model no legal way to say no.
  2. Put the constraint where it binds. An instruction 800 tokens above the input competes with the input for attention. Constraints that must hold go close to the end. The rule is about long inputs: in long contexts models use material at the start and end better than in the middle (Liu et al. 2024), and Section 25.3 shows why restating the goal at the end is nearly free. On this chapter’s short reviews, the review first variant — which moves the instruction towards the end — came last; an untested difference on 150 items does not refute the rule, but it is a reminder that the rule does not automatically transfer to short inputs.
  3. Make the contract machine-checkable, then check it. A schema you do not validate is a comment. Chapter 18 is where the enforcement machinery lives — grammars, JSON Schema, and the measurement of what constraining costs.

And a warning from Section 13.8, which is worth repeating in the chapter where the schema gets written rather than the one where it gets enforced: a schema guarantees shape, never truth. A grammar-constrained call happily returned {"answer": 1} for the capital of France. Validating the contract tells you the output is well-formed and nothing else.

17.4 How much of your score is the prompt?

Everything above is craft. This section is the measurement that makes the craft matter, and it is the one to run on your own task before you trust any number you have.

Twelve prompts were written for the same task — decide whether a Turkish product review is positive or negative. A competent reader would call them the same instruction.

They differ in formatting, in the surface form of the labels, in where the review sits relative to the instruction, in delimiters, and in politeness. None changes the task or the criteria.

Three of them do change what the labels are, and that turns out to be informative rather than a flaw in the design: A/B and 1/0 strip the labels of meaning entirely, and olumlu/olumsuz changes their language. Hold that in mind for the table — the semantically empty A/B finishes second, ahead of the meaningful Turkish pair by more than ten points. That suggests the meaning of the label word is not decisive here — yet olumlu/olumsuz, which differs from positive/negative only in the labels’ language, trails it by 12 points, so the words still matter. The two findings together want an explanation, and the label-bias discussion below is one candidate.

What each variant changes
variant what it changes against baseline
baseline — instruction, then the review, then Answer:; labels positive / negative
label positive/negative label surface form
label A/B label surface form, meaning removed
label 1/0 label surface form, meaning removed
label olumlu/olumsuz label surface form, language changed to Turkish
bulleted the instruction laid out as bullets
numbered the instruction laid out as numbered lines
triple quotes the review wrapped in """…"""
xml delimiters the review wrapped in <review>…</review>
review first the review moved above the instruction
polite a lütfen prefix and a teşekkürler suffix
role prefix a one-sentence persona prepended

One thing this chapter owes you and does not supply: the full text of all twelve. The table above names what changes; it does not let you reproduce the run, and a prompt experiment whose prompts are not printed is not reproducible. Print yours.

All twelve ran on the same 150 balanced reviews, same model, temperature 0, fixed seed. Every response parsed; nothing was dropped. The intervals below are Wilson intervals — a 95% confidence interval for a proportion that, unlike the textbook \(\hat p \pm 1.96\,\text{SE}\), stays sensible near 0% and 100%.

variant                        acc      95% CI (Wilson)
label positive/negative     90.00%   [ 84.2,  93.8]
label A/B                   88.67%   [ 82.6,  92.8]
bulleted                    87.33%   [ 81.1,  91.7]
numbered                    87.33%   [ 81.1,  91.7]
label 1/0                   86.67%   [ 80.3,  91.2]
triple quotes               82.00%   [ 75.1,  87.3]
baseline                    79.33%   [ 72.2,  85.0]
label olumlu/olumsuz        78.00%   [ 70.7,  83.9]
xml delimiters              77.33%   [ 70.0,  83.3]
polite                      76.67%   [ 69.3,  82.7]
role prefix                 76.00%   [ 68.6,  82.1]
review first                75.33%   [ 67.9,  81.5]

spread 75.33% – 90.00% = 14.67 points     mean 82.06   sd 5.57

A range needs something to be a range against. If all twelve prompts were equally good and each score were an independent draw at \(n = 150\), the standard error on one estimate would be 3.13 points and the expected spread of twelve such draws about 10 points — from sampling noise alone. So 14.67 is roughly one and a half times what chance would hand you, not an order of magnitude above it. The paired design makes the true null tighter than the independent calculation, so the effect is probably real; the number to report is a permutation one — within each review, shuffle which variant produced which of its twelve answers, recompute the spread, repeat ten thousand times, and read off where 14.67 falls. We did not run that, and it is the first thing to add if you repeat this experiment.

Twelve prompts, one task, one model — ranked

Almost fifteen points of accuracy, between two prompts that say the same thing.

Is that real, or is it 150 documents of noise?

The confidence intervals overlap, so the obvious next question is whether any of this survives contact with statistics. The observed spread of 14.67 points is a fact about these 150 documents; whether it generalises is the separate claim, and the paired analysis below is how you interrogate it — it uses the item-level disagreements directly instead of comparing two aggregate proportions.

best  'label positive/negative'  90.00%
worst 'review first'             75.33%

  documents where they disagree            42
    best right, worst wrong                32
    worst right, best wrong                10
  McNemar exact test                    p = 0.0009
  Bonferroni over 66 pairs              p = 0.062

The intuition behind McNemar’s test is a coin: if the two prompts were equally good, each of the 42 disagreements would be equally likely to go either way, and the exact test asks how surprising a 32–10 split is for a fair coin tossed 42 times. Bonferroni then multiplies that raw \(p\) by the number of comparisons you could have made — here 66.

The difference survives an uncorrected paired test — and best against worst is the extreme of the \(\binom{12}{2} = 66\) possible pairs, chosen after looking at the data, so the uncorrected number is not the one to quote. Corrected for those 66 comparisons it lands at p = 0.062: suggestive, and not significant at the 5% level after correction, on 150 items. That is Bonferroni. Holm would need the full ranking of all 66 p-values, and the largest accuracy gap is not guaranteed to be the smallest paired p-value, so we do not quote a Holm figure we have not computed.

The honest primary test is the omnibus one — Cochran’s Q across all twelve variants on the same 150 items — because that is the question actually being asked: does the prompt matter at all? Run Q first, then Holm-corrected pairwise McNemar for the pairs you want to name. We report the pair because it is the one people quote, but treat the effect as a spread. The next statistic measures that spread directly, avoids the multiple-comparison problem, and does not involve accuracy at all:

all 12 variants give the same answer on   104/150 = 69.3% of reviews
the prompt decides the answer on           46/150 = 30.7%

two variants disagree on: median 12.0% of reviews, max 28.7%

On three reviews in ten, the answer changed across the twelve tested prompt variants. The review, the model, the temperature and the seed were all held fixed, so the prompt is what moved — though “the prompt” here covers label wording, role, ordering and politeness, not formatting alone. Note what that first number is and is not: twelve variants agreeing does not mean the agreed answer is right, and this run does not report accuracy on the unanimous 104 separately. Add that row when you repeat the experiment — it is the difference between “the model has an opinion” as a measurement and as a figure of speech, and the next subsection shows how much rests on it.

Is the prompt deciding, or is it moving a threshold?

Follow the arithmetic before you accept the framing, because it points somewhere more useful. If the 104 unanimous reviews are all correct, then the mean score of 82.06% — 123.1 of 150 — leaves 19.1 of the 46 contested reviews correct. That is 41.5%, below the 50% floor of a balanced task. The best variant gets 31 of the 46 and the worst gets 9.

Twelve prompts answering at random on hard items would sit at 50%. Sitting below 50% on average, with the extremes at 67% and 20%, points to a different picture: twelve prompts at twelve different thresholds, each systematically leaning toward one label.

Hold that reading loosely, because it rests on the “all correct” assumption. If just four of the 104 unanimous reviews were wrong, the contested 46 would be at 50.2%, not 41.5%. And a below-chance score on a subset chosen because the prompts disagree on it can also come from reviews whose gold label is itself debatable. The threshold story is the hypothesis to test, not the conclusion.

Which points at a fix the prompt-engineering framing hides. Zhao et al. (2021) showed that much of this sensitivity is label bias — the model’s prior over the answer tokens, shifted by surface features of the prompt — and that estimating it on a content-free input (such as N/A) and dividing it out substantially reduces the variance across prompt formats. The fix needs the model’s probabilities for the label tokens, not just the text it returns; Chapter 18 shows how to read them. So the honest version of this section’s result is: before calibration the observed prompt spread is 14.67 points — in a design where sampling noise alone can produce roughly 10. How much of that survives after is the experiment to run next, and reporting each variant’s \(P(\text{predict positive})\) on a balanced set — which should be 50% and almost certainly is not — is the one-column diagnostic that would have shown this without any of the arithmetic above.

Both of those readouts describe the same 150 reviews, cut two different ways:

The same 150 reviews, cut two ways — and the 42 that decide the question
ALL TWELVE VARIANTS ON THE SAME 150 REVIEWS — WHERE THEY AGREE, AND WHERE THE PROMPT DECIDES 104 reviews — all twelve variants give the same answer 69.3% — all twelve variants agree 46 reviews 30.7% — the prompt decides THE SAME 150 REVIEWS, BEST VARIANT AGAINST WORST — AND THE 42 A PAIRED TEST ACTUALLY USES 108 reviews — both prompts give the same answer these carry no information about the difference 42 disagree 32 best right 10 of the 42: 32 to the best prompt, 10 to the worst Why the pairing matters Throwing away the 108 grey reviews is what turns two overlapping intervals into McNemar p = 0.0009 (uncorrected).

Every block is drawn at the same scale, so a review is the same width in both bands. The top band is the chapter’s headline number in a different form: all twelve variants give the same answer on 104 of the 150, and on the remaining 46 the choice of variant decides the answer. That is agreement, not accuracy. The bottom band is why the statistics work — a paired test looks only at the 42 reviews where the two prompts differ, which is how it separates scores whose confidence intervals overlap.

What that means for everything else you measure

Put this next to Chapter 12’s result on the same model. There, writing the instruction in English rather than Turkish — the single most repeated piece of prompting folklore — was worth +0.54 macro-F1, with a 95% interval running only to +2.25: not a demonstration that the effect is gone, but a ceiling of just over two points on whatever remains. Here, changing the bullet formatting is worth eight. The two numbers come from different tasks and metrics — complaint detection scored by macro-F1 on 2,000 documents there, balanced sentiment scored by accuracy on 150 here — so read the comparison as one of orders of magnitude, not as a precise ratio.

Read the table again before concluding that language does not matter, though, because it complicates the folklore rather than killing it. Changing the label from positive/negative to olumlu/olumsuz cost 12 points against the label positive/negative variant — the largest single-variable change in this experiment. (Against baseline the same two Turkish words cost only 1.33 points, so which reference you pick matters; the variants’ full texts, which this chapter does not print, are what would settle the cleaner comparison.) Instruction language did almost nothing; label language did more than any formatting change. “Does language matter?” is not one question, and the two halves of it gave opposite answers on the same model.

The variable everyone argues about was too small to measure where they argue about it (at most about two points), and the largest difference in the experiment came from two words nobody thinks of as a variable.

Three consequences, and the third is the one that changes how you work:

  1. A single-prompt model comparison is unreliable. If the prompt spread on one model is 14.67 points, then “model A beat model B by 5 points” measured with one prompt each is a statement about the prompts. Compare models over a set of prompts and report the distribution, or accept that you have not compared the models.
  2. Your reported accuracy needs an error bar you have not been computing. The sampling interval on 150 documents spans about 12 points end to end. The prompt spread, on the same data, is 14.67 — the same order of magnitude, which is the point: one of those error bars is computed and printed and the other is neither. Note also that they respond differently to more data. Add documents and the sampling interval keeps shrinking toward zero; the observed prompt spread shrinks only until the sampling noise inside it is gone, and then stays at the true prompt effect.
  3. Prompt changes need regression tests, not review. No reviewer reading “moved the review above the instruction” would predict −4 points, because the change is semantically empty. Only a measurement catches it, which is what makes this a testing problem rather than a writing problem.

Sclar et al. (2024) is the systematic version of this experiment across tasks and models, and it finds the same thing at scale: spurious formatting features move scores by amounts comparable to real model differences. Lu et al. (2022) found the same for the order of demonstrations, and their result is the sharper one: permuting a fixed set of examples moved scores from near state-of-the-art to near chance, the examples identical and only their sequence changed. The review first variant above is the zero-shot shadow of that effect; in a few-shot prompt the ordering degree of freedom can be larger than the formatting one measured here, and you are almost certainly not measuring it. The literature has been clear about this for years; the practice has not caught up, because the prompt still feels like a message rather than a component.

The patterns that lost

Two results in that table are worth naming because they contradict widely taught advice.

The role prefix came 11th of 12 (76.00%, against 79.33% for the same prompt without it). You are an experienced customer-feedback analyst is the most reproduced prompting pattern in circulation, and on this task it cost three points.

Politeness cost 2.67 points (a net four reviews of 150). Lütfen… Teşekkürler is harmless-looking, and here at least it did not help.

Neither result generalises, and neither is significant even on this sample: net differences of 5 and 4 reviews in 150 cannot reach the 5% level on an exact McNemar test even if every disagreement went the same way (the smallest possible two-sided \(p\) is 0.0625 and 0.125). They are observations on this sample, not established findings. The lesson is not “never use a role prefix”; it is that you cannot tell which prompt is better by reading them, and the belief that you can is what this whole section is arguing against.

17.5 Making prompt work reviewable

If the prompt is a component and its observed spread is 14.67 points, it needs the machinery any other component gets. Four pieces, in the order they pay off.

1. Pin down reproducibility first

Before measuring anything, find out whether your setup returns the same answer twice. Thirty reviews, ten repeats each:

setting                          identical output   documents that varied
temperature 0, fixed seed              100.0%              0 / 30
temperature 0, no seed                 100.0%              0 / 30
temperature 0.7, fixed seed            100.0%              0 / 30
temperature 0.7, no seed                 3.3%             29 / 30

Good news and a trap. At temperature 0 this setup returned identical output on all 30 documents across ten repeats, with or without a seed, and a fixed seed did the same at temperature 0.7. Read that as no variation in 300 calls rather than as a proof of determinism: zero events in 30 items still leaves room for a true variation rate of up to about 10%. Without either, 29 of 30 documents gave a different answer across ten runs — so an evaluation at default settings is measuring the sampler as much as the prompt.

That table is a single local server answering one request at a time, and that is the load-bearing condition — not that the server is local. What breaks determinism is a lack of batch invariance: the reduction kernels behind matmul, normalisation and attention take different code paths at different batch sizes, so the same prompt in a batch of one and in a batch of seventeen can decode differently even at temperature 0 (He and Thinking Machines Lab 2025). Hosted APIs batch your request with strangers, which is why they are frequently non-deterministic at temperature 0 — but a local server is not automatically safe either. Change the batch size, the context length, the KV-cache reuse or the backend and the same thing happens. Do not assume; run the ten-repeat check against the endpoint and the configuration you will actually ship on, and if it is not deterministic, your eval needs repeats and an interval rather than a single number.

2. Build a small evaluation set, and freeze it

Once you know the setup answers the same way twice, you need something to answer about. Fifty to two hundred items, labelled by you, covering the failure modes you have already seen. It does not need to be large — it needs to be fixed, so that two prompt versions are comparable.

The discipline from Chapter 12 applies unchanged: hold it out, look at it as rarely as possible, and never tune a prompt against the same items you report on.

A prompt tuned on its evaluation set is exactly as overfitted as a model tuned on its test set, and far easier to do by accident — because iterating on a prompt feels like writing rather than fitting.

3. Compare paired, not independently

This is the statistical point Section 17.4 turned on. Two prompts run on the same items are a paired comparison, so use the pairing: count the documents where they disagree, and test that count.

prompt A right, B wrong     32
prompt B right, A wrong     10
                            --
McNemar exact            p = 0.0009

Comparing two accuracy figures with overlapping confidence intervals would have called that inconclusive. The paired test can see, on the same 150 documents, a difference the unpaired comparison cannot, because it throws away the 108 items both prompts agree on — which carry no information about the difference. (That holds if you chose the pair before looking; if you picked best and worst afterwards, you need the multiple-comparison correction shown above.)

4. Version the template, not the string

Store the template with slots, in the repository, with a version. Log which version produced each output. When something regresses, the question which prompt was this has to have an answer, and “the one in the notebook” is not one.

PROMPTS = {
    "sentiment/v3": "...",     # bulleted, labels positive/negative
    "sentiment/v4": "...",     # + explicit null case
}

Two habits that fall out of it: the assembled prompt is a build artefact and should not be edited by hand, and a prompt change ships with its evaluation delta the way a code change ships with tests.

17.6 Intuition checks

Answer each one before you open it. All six are questions this chapter settled with a measurement rather than an argument.

You do not know, because you have not separated the model from the prompt. Measured on one model, one task and twelve semantically identical prompts, the spread was 14.67 points — three times the gap you are trying to interpret. Part of that observed spread is sampling noise (roughly 10 points of it could come from noise alone at 150 items), but the same noise sits in your 5 points.

Run both models over a set of prompts and compare the distributions. If that is too expensive, at least run each model on two or three prompts and report the range. A single-prompt comparison is a statement about the prompts.

Semantically yes; empirically you cannot rely on it. Here the score fell 4.00 points (79.33% → 75.33%; a net six reviews of 150). No paired test is reported for this pair, so the drop is not established — but you cannot predict its sign by reading, and you have to measure it.

That is the whole argument for treating prompts as testable components. No reviewer can predict that by reading, because there is nothing to read — the change is meaning-preserving. The only thing that catches it is running the evaluation, which makes prompt changes a testing problem rather than a writing problem.

On this task it came 11th of 12, at 76.00% against 79.33% for the same prompt without it.

Do not over-read that either — it is one task and one model, and a net difference of five reviews in 150 cannot be significant at the 5% level. The point is not that role prefixes are bad. The point is that the most-reproduced pattern in prompting failed to beat its own baseline here, and the sign could not be guessed in advance. Measure it on your task: one extra run and a paired test on a large enough set.

It depends on how they are wrong. Randomly noisy labels are less bad than you would think, which tells you something about what demonstrations are for. Min et al. (2022) randomised the labels in the demonstrations and most of the benefit survived, because what the examples mainly convey is the format and the label space rather than the mapping.

But random labels and systematically flipped labels are not the same thing. Task learning is real and grows with scale (Pan et al. 2023), and large enough models can follow a flipped mapping in the demonstrations and fall below chance (Wei et al., 2023). Fix wrong labels as soon as you know about them.

Practical readings: do not spend a week hand-curating examples before checking whether examples help at all; and if your task’s difficulty is in the mapping rather than the format, expect demonstrations to underdeliver.

What does move the number is which examples you show — Liu et al. (2022) found that retrieving demonstrations semantically close to the input beats choosing them at random, which turns example selection into a retrieval problem and hands it to Chapter 19’s machinery.

Locally, here, yes — 100% identical across ten repeats, with or without a seed. On a hosted API, often not, because batching changes floating-point reduction order and your request shares a batch with strangers.

Run the check rather than assuming: same prompt, same input, ten times, against the endpoint you will ship on. If it varies, every number in your evaluation needs repeats and an interval.

When you are maintaining a specification rather than writing a request. The signals: the prompt has a version history, its output is parsed by code, a change to it can break a downstream consumer, and you cannot safely edit it without running something.

At that point it is a component of your system and belongs under the same discipline as the rest of your code — repository, version, contract, regression test. Chapter 18 is where the output contract gets enforced rather than requested.

17.7 Interview questions

On sourcing — the same caveat as Chapter 2. These recur in the two credible open sources this book uses, plus the ones this chapter’s own measurements make answerable.

Testing whether you understand the mechanism

Q: What actually happens when you put examples in a prompt?

Nothing is learned in the training sense — no weight changes and nothing survives the call. The examples narrow which task the model performs, which is why it is better described as task inference than as learning. Then deliver the result that shows you have read past the tutorial: Min et al. (2022) randomised the demonstration labels and most of the benefit survived, so what examples largely convey is the format and label space rather than the mapping.

Q: Why did prompting stop feeling like a trick?

Because instruction tuning happened. A base model continues text, so early prompting meant constructing a prefix whose natural continuation was the answer. Instruction tuning, followed by preference training (Chapter 9), trained models to treat an instruction as an instruction. Candidates who cannot separate “the model got bigger” from “the model got trained to follow instructions” have not thought about where the capability came from.

Testing whether you have shipped one

Q: Two prompts, which is better?

Run both on the same frozen evaluation set and use a paired test — count the items where they disagree and test that count, not the two accuracies. Measured here, two prompts with overlapping confidence intervals were separated at \(p = 0.0009\) by McNemar, because the pairing discards the items on which both gave the same answer, right or wrong.

If the answer is “I read them and the second is clearer”, the candidate has not measured a prompt.

Q: How much of your reported accuracy is the prompt?

The honest answer is that most teams do not know, and the measurement is a morning’s work: write several semantically identical prompts and run all of them. Here twelve of them spanned 14.67 points, and all twelve agreed on only 69.3% of items — so on three items in ten the choice of prompt variant decided the output. That number is an error bar that almost nobody reports, and here it was larger than the sampling interval.

Q: How do you stop prompt changes from breaking production?

Treat the prompt as a versioned component: template in the repository, version logged with every output, a frozen evaluation set, and a delta reported with every change. Then say why review is not enough — a semantically empty edit lowered the score by 4 points here (on 150 items, untested for significance) and its sign could not have been predicted by reading.

Q: Your evaluation results changed and nothing was deployed. What happened?

Check reproducibility before anything else. Same prompt, same input, ten repeats. Locally, temperature 0 showed no variation in 300 calls here; without a seed at temperature 0.7, 29 of 30 items varied. And hosted endpoints are often non-deterministic even at temperature 0, because batching changes floating-point reduction order — so the answer may be that nothing happened and your eval never had an interval on it.

17.8 The 2026 toolbox

Everything in this chapter ran on one laptop with the versions below.

The stack these numbers came from
library version used what it is for here
ollama 0.24.0 serving qwen3.5:9b for every measurement in this chapter
Python stdlib 3.14.5 the experiment is a for loop over prompts and a paired test; it needed no framework

Worth knowing about, not used above:

The wider ecosystem, versions as of writing
library current version when you would reach for it
promptfoo 0.122.0 (npm) prompt A/B testing and regression suites as a config file — the tooling version of Section 17.5, and the fastest way to stop doing this in notebooks
dspy 3.3.1 stops you writing prompt strings at all: declare the signature, let it compile the prompt against a metric. The right answer once you accept that hand-tuning a brittle artefact is a poor use of a week
instructor / pydantic 1.15.4 / 2.13.4 the output contract of Section 17.3 as a Python type, validated on arrival
guidance 0.3.1 constrains generation to a template or grammar — Chapter 18’s machinery, applied at the prompt layer
jinja2 3.1.6 prompt templates with slots, versioned as files. Unglamorous, and it solves the “which prompt produced this” problem outright
langsmith 0.11.1 tracing and dataset management when the pipeline outgrows a spreadsheet
pytest 9.1.1 the point being that a prompt regression suite is just tests, and needs no LLM-specific framework to start

dspy deserves the second look its position does not give it. Its premise is exactly this chapter’s finding: if prompt wording moves the score by amounts you cannot predict, then hand-writing prompts is optimisation by guesswork, and a compiler with a metric will beat you. The cost is that your prompts stop being readable artefacts, which is a real trade rather than a free win.

17.9 Lab: turn a prompt into a component

Take a task you actually do — one where you have been iterating on a prompt by feel. The lab converts it into something with a version and an error bar.

Step 1 — freeze two sets, not one

Label a development set you will iterate on and a final set you will open once, at the end. Fifty each is a workable minimum. Include the failures you already know about; that is what makes them useful rather than decorative. Split before you look at anything, write both to files, and stop editing them.

Everything from step 2 to step 5 runs on the development set. Every number you report on the development set is a number you selected for, and reporting it as your system’s accuracy is the mistake this whole chapter is about — you would be quoting a score tuned on the set that produced it.

Step 2 — measure reproducibility before anything else

Same prompt, same input, ten repeats, against the endpoint you will ship on. If the answers vary, every number below needs repeats and an interval, and you need to know that now rather than after a confusing week.

Step 3 — write six prompts that mean the same thing

Not six better prompts — six that a colleague would call identical. Vary formatting, label surface form, delimiters, the position of the input, a role prefix. Run all six.

The spread you get is your prompt’s error bar. Write it down; it is the number that tells you how seriously to take every comparison you have made so far.

Step 4 — pick the winner with a paired test

Best against worst on the development set, McNemar on the disagreements. Report the number of items where they differ, not just the two accuracies — and if you picked the pair after seeing all six, correct for the \(\binom{6}{2} = 15\) comparisons you could have made.

Step 5 — rewrite the winner as a specification

Output contract, explicit failure value, constraints near the end. Re-run on the development set. Report whether the score moved and whether the parse failures went to zero, which is usually the larger win.

Step 6 — open the final set, once

Run the prompt you chose on the set you have not touched, and report that number as your system’s accuracy with an interval on it. Expect it to be lower than the development figure; the gap is what your iteration cost you in optimism, and it is worth knowing. If you go back and change the prompt after seeing it, you have spent the set and need a new one.

Step 7 — put it under version control and break it on purpose

Template with slots in a file, version string logged with each output. Then make a semantically empty edit — reorder two sentences — and run the evaluation. If the number moves, you have just demonstrated to yourself why this is a testing problem.

What to hand in

The six-prompt spread, the paired test for your winner, the before-and-after for the specification rewrite, and one paragraph on what your prompt’s error bar means for the model comparison you most recently believed.

17.10 Exercises

  1. Your own brittleness number. Six semantically identical prompts, one task, one model. Report the spread and the fraction of items where all six agree. Compare the spread against the difference between the two models you are choosing between.

  2. Paired versus unpaired. Take two prompts whose confidence intervals overlap. Compute an unpaired comparison and a McNemar test on the same data. Report both, and say which you would put in a report and why.

  3. The \(k\)-shot curve this chapter does not have. Run your task at \(k \in \{0, 1, 2, 4, 8, 16, 32\}\) demonstrations, twice: once with the gold labels and once with the labels shuffled. Plot both curves. If they rise together, the demonstrations are conveying format; if the gold curve pulls away, they are conveying the mapping — which is Min et al. (2022) against Pan et al. (2023) on your own data. Then permute a fixed set of four demonstrations eight ways at fixed \(k\) and report that spread too; per Lu et al. (2022) it may be the larger of the two effects.

  4. The role prefix, on your task. Run the same prompt with and without You are an expert…. Predict the sign before you run it. Report both.

  5. Reproducibility on your endpoint. Ten repeats at temperature 0 against a hosted API and against a local model. If they differ, explain the mechanism to a colleague in two sentences.

  6. Request to specification. Take a prompt written as a request, rewrite it as a contract with an explicit failure value, and count parse failures before and after. Then find an input where the failure value is the correct answer and check that the model actually returns it.

  7. The regression you cannot see. Make a meaning-preserving edit to a production prompt — reorder sentences, change bullets to prose. Run your evaluation. If nothing moved, try three more edits before concluding your task is robust.

17.11 Where to go deeper

One exercise beats any reading here: write your prompt twice, changing only formatting, and score both on the same hundred items. Whatever you read next will land differently once you have seen your own spread.

Beyond the papers
resource what it is
Anthropic’s prompt engineering guide vendor documentation, but unusually concrete and honest about what does and does not transfer
OpenAI Cookbook runnable recipes; useful precisely because you can measure whether each one helps on your task
promptfoo run the same eval across prompt variants and models; the tool that makes this chapter’s experiment routine
Hamel Husain on evals the best applied writing on why prompt work without measurement is guesswork
Stanford CME 295 transformers and LLMs, with prompting placed in the context of how the model actually works
Intro to Large Language Models — Andrej Karpathy the hour that explains why a prompt is a component of the system rather than a message to it
The primary sources
topic source
in-context learning, as first reported Brown et al. (2020)
formatting alone moves the score Sclar et al. (2024)
so does example order Lu et al. (2022)
what demonstrations actually convey Min et al. (2022)
which examples to show Liu et al. (2022)
zero-shot reasoning from one sentence Kojima et al. (2022)
prompting as an interface, thought through early Reynolds and McDonell (2021)
label bias, and calibrating it away Zhao et al. (2021)
a survey of the whole technique zoo Schulhoff et al. (2025)

17.12 What to remember

  • In-context learning updates no parameters. Nothing is fitted and nothing survives the call, which is why task inference describes the common case better than learning — with the caveat below that task learning is real and grows with scale. Min et al. (2022) randomised the labels in the demonstrations and most of the benefit remained, which says examples mainly convey format and label space rather than the mapping.

  • The prompt is part of the model. Twelve prompts that a competent reader would call the same instruction spanned 14.67 accuracy points on one model and one task — 75.33% to 90.00%. That is not a writing difference; it is a component to be measured — though sampling noise alone could produce roughly 10 points of that observed spread, and the accuracy-free evidence is the disagreement rate in the next point.

  • On 30.7% of items, the answer changed across the twelve variants. All twelve agreed on only 69.3% of the reviews; two variants disagreed on a median of 12% and as much as 28.7%. Your model has a stable answer on most items and the prompt moves the rest.

  • The spread is 14.67 points on these items, and the paired analysis is how you interrogate it. Best against worst: overlapping confidence intervals, 42 disagreements, McNemar \(p = 0.0009\) — but that pair was chosen after seeing the data, so the number to quote is the one corrected for all 66 pairs, \(p = 0.062\). Suggestive; not significant at the 5% level after correction. The pairing uses information the unpaired comparison throws away, and the correction says how much of the result to believe.

  • Which means a single-prompt model comparison is unreliable. If the prompt spread is 14.67 points, “model A beat model B by 5” measured with one prompt each is a statement about prompts. Compare over a set of prompts and report the distribution.

  • The folklore variable was too small to detect, and the ignored ones were larger on this sample. Chapter 12 measured English-versus-Turkish instructions at +0.54 (95% upper bound +2.25; \(p = 0.54\)) on this same model. Bullet formatting, here, moved the score eight points on 150 items. The role prefix — the most reproduced pattern in circulation — came 11th of 12, 3.33 points behind its own baseline: too small a gap to be significant on its own, but its sign could not be predicted.

  • You cannot tell which prompt is better by reading them. That is the whole case for treating prompt changes as a testing problem: a semantically empty edit lowered the score by 4.00 points (untested for significance), and its sign could not be predicted by reading.

  • Check reproducibility before you measure anything. Locally, temperature 0 gave identical output on every repeat with or without a seed, and so did a fixed seed at temperature 0.7 — but with neither, 29 of 30 items varied across ten runs. Hosted APIs are frequently non-deterministic even at temperature 0, because batching changes floating-point reduction order. Run the ten-repeat check on the endpoint you will ship on.

  • Write prompts as specifications. Name the output contract, name the failure value, and put constraints where they bind. Then validate the contract — a schema you do not check is a comment, and Section 13.8 already showed that a schema guarantees shape and never truth.