18 Decoding and Structured Output
Learning objectives. After this chapter you can:
- explain what each sampling parameter does to the probability distribution;
- make a model emit schema-valid output whenever it is allowed to finish;
- use logprobs for confidence, routing, and abstention;
- diagnose repetition and degeneration instead of tuning parameters at random.
18.1 Greedy, beam, temperature, top-k / top-p / min-p
A language model’s forward pass ends in a vector of logits — one raw score per vocabulary token. Everything in Chapter 7 was about producing that vector. What happens next is not learning; it is a sampling policy you control at inference time, and it changes the character of the output more than most people expect.
- Temperature \(\tau\) rescales logits before the softmax: \(p_i \propto \exp(z_i/\tau)\). Low \(\tau\) sharpens, high \(\tau\) flattens.
- Top-k keeps only the \(k\) highest-probability tokens, then renormalises.
- Top-p (nucleus) keeps the smallest set of tokens whose cumulative probability reaches \(p\), then renormalises (Holtzman et al. 2020).
The order these run in matters, and it is the part that surprises people. In transformers (5.x), one decoding step applies the repetition penalty (and processors such as no-repeat n-gram) to the raw logits, then temperature, then top-k, then top-p, then min-p, and then samples. vLLM also applies the penalties (repetition, frequency and presence) and then temperature first, but runs min-p before top-k and top-p; llama.cpp’s default chain applies temperature last, after all the cutoffs. So the order is fixed inside one library, not across libraries — check the one you use. Two consequences follow for the temperature-first order. Because temperature runs before top-p, raising the temperature changes which tokens fall inside the nucleus — so the common claim that temperature “only changes the contrast, never the ranking” is true of temperature alone and misleading in a pipeline that also truncates; the explorer below lets you watch the nucleus change membership. And because the penalties run before temperature (in all three libraries), the same penalty value bites far harder at \(\tau = 0.2\) than at \(\tau = 1.5\): an additive penalty \(\alpha\) on the raw logit becomes \(\alpha/\tau\) after scaling. If you are tuning both, tune the temperature first.
Explore: temperature, top-k, and top-p
The model has just read “After the lecture, the students were completely ____” and produced the logits below. Gray bars show the softmax distribution at the current temperature, before any cutoff; teal bars show what survives your sampling policy, after renormalisation. Try \(\tau = 0.2\), then \(\tau = 1.8\). Then cut the tail with top-p \(= 0.6\).
Notice what a low temperature does not do: it never changes the ranking, only the contrast. Top-k and top-p, by comparison, are hard cutoffs — they exist mainly to stop the model from ever picking recursive in a sentence about tired students. This trio is the cheapest lever you have in every applied system from here on.
Greedy, beam, and min-p
Three more policies complete the picture.
Greedy takes the argmax at every step. It is temperature 0 in the limit, and it is the right default whenever there is one correct answer. (APIs treat “temperature 0” as a request for greedy decoding, since dividing by zero is undefined. Even then a hosted API may not return the same string every time: batching and floating-point rounding on the GPU can change a near-tie between two tokens, and after one different token the rest of the text differs too.)
Beam search keeps the k best partial sequences (k is the beam width) and extends all of them, so it searches for a high-probability whole sequence rather than choosing the best token at each step. It is still a search with a limited width, so it can miss the single most probable sequence.
Chapter 15 measures it properly on translation, and two of its findings are worth carrying here. The gains die quickly — beam 1 to 2 bought +0.61 BLEU, and beam 2 to 64 bought +0.57 more for seven times the compute. And the widely repeated warning that wider beams make the output worse (the “curse of beam search”) did not appear at all, most likely because the decoder normalised by length during the search. Length normalisation divides a beam’s log-probability by its length; without it, every extra token adds another negative log-probability, so the search prefers outputs that are too short.
Min-p (Nguyen et al. 2024) is the newer cutoff. Where top-p keeps a fixed mass and top-k a fixed count, min-p keeps every token whose probability is at least a fraction of the top token’s:
\[ \text{keep } i \quad\text{if}\quad p_i \;\ge\; m \cdot \max_j p_j \]
That makes the cutoff adapt to the shape of the distribution rather than to a constant. Typical values of \(m\) are 0.05–0.1: with \(m = 0.1\) and a top token at 0.6, every token below 0.06 is cut.
When the model is confident — one token at 0.9 — min-p keeps almost nothing, which is what you want. When it is genuinely uncertain across twenty plausible continuations, min-p keeps them all, where top-p 0.9 would cut the last few by cumulative mass alone, however plausible they are.
The general rule underneath all of them: the policy should depend on how many good answers the task has. Extraction, classification and translation have one, so greedy or a narrow beam. Drafting, brainstorming and dialogue have many, so sample. Chapter 15 measured the cost of getting this backwards — a chat-default temperature of 1.0 took a translation model from 30.17 BLEU to 14.34.
What we will look at. First why greedy decoding walks into repetition loops, and why the usual penalties are the wrong tool once you are generating structure. Then constrained decoding, where a grammar makes invalid output unreachable rather than unlikely — measured on two models, because it buys almost nothing on one and seventeen accuracy points on the other. Then tool calls, the case where the strong-model result does not let you off. Then logprobs, which discriminate well, calibrate badly, and still buy you accuracy for the price of one threshold. And finally streaming, which wants the opposite of everything structured output wants.
18.2 Repetition and degeneration
Greedy and low-temperature decoding have a characteristic failure: the model falls into a loop and repeats a phrase until it hits the token limit.
The mechanism is not mysterious. Having produced a phrase once, the model has made that phrase part of its own context, and a language model trained on real text has learned that repeated phrases are more likely after the first repetition.
Holtzman et al. (2020) named this and showed the surprising part: the loop is not a low-probability accident. The repeated continuation is often the single highest-probability sequence available, which is why greedy decoding walks into it and sampling rarely does.
The usual controls, and what each costs:
| control | what it does | what it breaks |
|---|---|---|
| Sampling (top-p, min-p) | takes a non-argmax token sometimes, breaking the loop | variety you may not want in an extraction task |
| Repetition penalty | lowers the logit of any already-used token — dividing it when positive, multiplying when negative (see below) | punishes legitimately repeated words — names, technical terms, null |
| No-repeat n-gram | forbids any n-gram already emitted | forbids real repeated phrases; ruinous in structured output where {"type": recurs |
| Presence penalty | a flat subtraction the first time a token appears | same failure mode, milder; does not escalate |
| Frequency penalty | a subtraction proportional to how often it has appeared | escalates with every repeat, so long outputs drift |
The repetition penalty \(\theta\) acts on the raw logit, and its sign handling is the part implementations get wrong:
\[ z_i' = \begin{cases} z_i / \theta & z_i \ge 0 \\ z_i \cdot \theta & z_i < 0 \end{cases} \]
Dividing a negative logit by \(\theta > 1\) raises it towards zero — that is a reward, not a penalty — which is why the two branches are not cosmetic. The penalty comes from CTRL (Keskar et al. 2019), whose formula simply divides; the sign case was added by later implementations such as transformers. The OpenAI-style penalties avoid the problem by being additive and acting on counts instead: \(z_i' = z_i - \alpha_{\text{presence}}\,\mathbb{1}[c_i > 0] - \alpha_{\text{freq}}\,c_i\), where \(c_i\) is how many times token \(i\) has already appeared. Presence fires once; frequency keeps scaling.
The penalties are blunt instruments and they are the wrong tool for structured output. A JSON object legitimately repeats keys, braces and quote characters; a no-repeat-3-gram constraint on JSON whose keys recur (an array of objects, say) forbids writing a key the second time and makes schema-invalid output all but certain. If you are generating structure, the fix for repetition is the grammar in Section 18.3, not a penalty.
18.3 Constrained decoding
Everything above reshapes a distribution over the whole vocabulary. Constrained decoding does something stricter: at each step it computes which tokens could legally continue the output under a grammar, and sets the logits of all the others to \(-\infty\). Invalid output stops being unlikely and becomes unreachable — with three conditions on that word. The schema has to be one the library actually supports, since unsupported keywords are commonly ignored rather than enforced. The generation has to finish: a run that hits your token limit, a timeout, or a dropped connection leaves a prefix that is legal so far and invalid as a document. And a grammar constrains structure, not semantics, so a required field can be filled with a plausible fabrication. Parse and validate the result anyway; the guarantee is worth a lot and it is not “never check”.
constrained decoding — masking the logits at each step to the tokens a grammar allows. Guarantees the shape of a completed generation and says nothing about whether it is true. ↩︎
One property of that mechanism is easy to miss and worth stating before the measurements, because it bounds what they mean. Masking at each step is not the same as sampling from the model’s distribution conditioned on the grammar. A token that is locally legal may lead only to dead ends, and local masking has no way to renormalise for that — so what you get is a distribution over grammatical strings whose likelihoods are no longer proportional to the model’s own (Park et al. 2024). In practice it shows up as grammatical output the unconstrained model would have ranked lower. Grammar-aligned decoding fixes it at extra cost; every library in Section 18.9 ships the cheap, distorting version, which is the right default and worth knowing you are using.
One decoding step, with and without a grammar
The machinery is a finite-state machine over the tokenizer’s vocabulary (Willard and Louf 2023): compile the regex — or a JSON Schema whose nesting is bounded — into an automaton, track which state you are in, and precompute for each state the set of allowed next tokens. A general context-free grammar does not fit in a finite-state machine, because unbounded nesting needs a stack; libraries that accept a CFG run an incremental parser (a pushdown automaton) instead, which costs more per step and buys back arbitrary recursion. Geng et al. (2023) is the same idea for general structured NLP tasks.
What it actually buys
The interesting question is not whether a grammar can enforce a schema — it can — but where an unenforced model stops complying on its own, and what enforcement costs when it takes over. Four schemas of increasing awkwardness, 150 balanced reviews, two models, asked-versus-enforced:
qwen3.5:9b parses schema-ok accuracy s/item
L1 flat, natural enum asked 100.0% 98.0% 89.3% 1.23
L1 flat, natural enum forced 100.0% 100.0% 88.7% 1.30
L2 unnatural enum asked 99.3% 99.3% 86.7% 1.33
L2 unnatural enum forced 100.0% 100.0% 87.3% 1.36
L3 nested + array asked 100.0% 98.7% 88.0% 1.44
L3 nested + array forced 100.0% 100.0% 88.0% 1.62
L4 nested + nullable asked 100.0% 98.0% 88.7% 1.73
L4 nested + nullable forced 100.0% 100.0% 88.7% 1.74
llama3.2:3b parses schema-ok accuracy s/item
L1 flat, natural enum asked 98.0% 98.0% 83.3% 0.37
L1 flat, natural enum forced 100.0% 100.0% 85.3% 0.56
L2 unnatural enum asked 99.3% 97.3% 78.0% 0.43
L2 unnatural enum forced 100.0% 100.0% 80.0% 5.13
L3 nested + array asked 79.3% 79.3% 70.7% 0.37
L3 nested + array forced 100.0% 100.0% 88.0% 0.70
L4 nested + nullable asked 99.3% 98.7% 86.0% 0.53
L4 nested + nullable forced 100.0% 100.0% 86.7% 9.26Three findings, and the first is the one that reframes the technique.
On a strong model, the grammar buys almost nothing. qwen3.5:9b complied with the schema 98–99.3% of the time when merely asked, at every difficulty level including nested objects with a nullable field. Enforcement took that to 100% and left accuracy unchanged. Modern instruction-tuned models mostly produce the JSON you describe; the grammar is insurance on the last one or two percent.
Two qualifications on that null result (no difference found), and they matter more than the result itself. First, it is a null on 150 items: the 95% interval around an 88% accuracy is roughly ±5 points, so this experiment could not have detected a two- to five-point degradation even if one were there. Second, the task here is classification. Tam et al. (2024) found that format restriction helps classification and hurts multi-step reasoning — GSM8K-style tasks lost several points under JSON mode with near-zero parsing failures, which locates the loss in the reasoning rather than in the parsing. So this chapter’s result is a narrow subset of theirs rather than a contradiction of it, and it does not license the general claim. Do not carry “enforcement is free” into a chain-of-thought task; the safe pattern there is to let the model reason unconstrained and constrain only the final answer field.
On a weak model, it is the difference between working and not. llama3.2:3b fell to 79.3% schema-valid on the nested-plus-array schema — one output in five unusable. Enforcement took it to 100%, and accuracy rose from 70.7% to 88.0%, a 17.3-point gain, because the malformed fifth had been scored as wrong. The grammar did not make the model smarter; it stopped a formatting failure from being counted as a reasoning failure.
The cost is latency, and it is not where you would guess. Overhead ranged from 1.01× to 17.5×; the top of that range is the weak model at L4, where 0.53 s per item became 9.26 s. The overhead was not monotone in schema complexity — on llama3.2:3b L1 and L3 were cheap, L2 and L4 expensive; on qwen3.5:9b L3 was the most expensive. The two ends sit on different models, which is itself the finding: on qwen3.5:9b the grammar costs between 1.006× and 1.13× — essentially free — and on llama3.2:3b between 1.5× and 17.5×. A grammar is cheap where the model already agrees with the schema and expensive where it does not.
The two expensive ones share a property: the schema demands something the model does not want to produce, an unnatural enum in one case and a nullable field in the other. That is consistent with the grammar having to fight the model’s distribution rather than ride it, though this measurement establishes the pattern rather than the mechanism.
Which is a reason to measure rather than reason about it. If the grammar’s cost depended only on schema complexity you could budget for it from the schema; it appears to depend on the disagreement between the schema and the model, which you cannot see without running both.
What it cannot fix
A grammar constrains shape. It has no access to truth, and the failure this produces is worth seeing directly. From Section 13.8, where a schema demanded an object with an integer answer:
POST /api/generate format={"answer": integer}
prompt: "What is the capital of France?"
-> {"answer": 1}Schema-valid, parseable, type-correct, and meaningless. Constrained decoding converts a parse failure into a semantic failure, which is usually the trade you want — a wrong value you can validate beats a crash — but it is a trade, not a fix. Every guarantee it gives you is syntactic.
Two corollaries worth stating plainly:
- Validate values, not just structure. An enum is checkable; a free-text field is not. Push as much of your contract into enumerable types as the task allows.
- A schema is not a substitute for the escape hatch. If the model has no legal way to say I do not know, the grammar guarantees it will say something else instead. Add
nullto the type, and check that the model actually uses it; this book did not measure that, so treat it as a check, not a finding. As Section 11.11 puts it, an escape hatch that never fires is decoration.
18.4 Making tool arguments reliable
The highest-leverage use of everything above is not extraction; it is tool calls. A tool call is a structured output — a function name and a typed argument object — and it has a property that makes correctness matter more than usual: something executes afterwards.
A malformed summary is a bad summary. A malformed argument object is either a crash or, worse, a call that succeeds with the wrong value.
How this looks in practice: with function calling, you send the model a list of tool definitions, each a name, a description and a JSON Schema for its arguments, and the model replies with a tool name and an argument object instead of prose. Whether those arguments are enforced depends on a flag. Hosted APIs usually offer two strengths. A “JSON mode” guarantees only that the output is syntactically valid JSON, not that it matches your schema; a schema-enforced mode (OpenAI’s API, for example, calls it strict) compiles your schema into the kind of grammar described above. Know which one you have switched on.
Three rules follow from the measurements above.
Constrain the arguments, always. This is the case where the strong-model result does not let you off. 98% schema-validity is excellent for extraction and unacceptable for execution: at a thousand calls a day it is twenty failures a day, each one a stack trace or a wrong action. The grammar takes that to zero for every call that finishes. Its latency cost is real and varies by schema, but you can measure it before you ship.
Type the arguments as narrowly as the tool allows. An enum of four operations cannot be called with a fifth. A date typed as a string can be "yesterday"; typed with a pattern, it cannot. Every constraint you can push into the schema is a validation you do not have to write and an error class that becomes unreachable.
A valid call is still not a correct call. The grammar guarantees {"operation": "delete", "id": 4} is well-formed. It does not know whether 4 was the right id, and Chapter 24 is where that problem — and the confirmation and authorisation machinery around it — belongs.
18.5 Logprobs, confidence, and abstention
Alongside each token, the model reports the log probability it assigned — its logprob. This is the cheapest signal available about how sure the model is, it costs nothing extra to obtain where the API exposes it, and most pipelines throw it away. Not every hosted API or model returns logprobs, and those that do often return only the top few candidates (top_logprobs): check both before relying on it, because renormalising over A and B below needs both tokens in the list.
The question is whether it is usable. Measured with qwen3.5:9b on 400 balanced reviews, with the answer constrained to a single token — A for positive, B for negative — so that the first generated token is the answer and its probability can be renormalised over just those two alternatives into a proper binary probability.
logprob — the log probability the model assigned to a token. Interpretable as confidence only when the answer is a single token; for a long answer, the mean logprob mostly measures how ordinary the phrasing was. ↩︎
It discriminates
The first question about any confidence signal is whether it orders items correctly — whether the ones it is sure about really are the ones it gets right. Two AUCs answer that:
accuracy 86.75%
AUC of P(positive) against the true label 0.9492
AUC of confidence for predicting its own correctness 0.8268
mean confidence when the model was right 0.9481
mean confidence when the model was wrong 0.7793The score is good and the model knows when it is guessing. An AUC of 0.9492 says the probability ranks positives above negatives almost as well as a trained classifier would; 0.8268 says confidence separates the model’s own correct answers from its incorrect ones. Kadavath et al. (2022) is the general version of that second result.
It is not calibrated
Discrimination and calibration are different properties, and the second one fails. The table groups the 400 answers by stated confidence; ECE (expected calibration error, defined in Chapter 12) is the gap between confidence and accuracy in each bin, averaged with the bin sizes as weights:
confidence bin n accuracy mean confidence
[0.50, 0.55) 15 33.3% 52.4%
[0.55, 0.60) 10 50.0% 57.5%
[0.60, 0.65) 10 70.0% 62.5%
[0.65, 0.70) 8 62.5% 66.4%
[0.70, 0.75) 7 85.7% 72.7%
[0.75, 0.80) 8 87.5% 77.6%
[0.80, 0.85) 13 61.5% 82.7%
[0.85, 0.90) 20 85.0% 87.6%
[0.90, 0.95) 25 68.0% 92.6%
[0.95, 1.00) 284 95.1% 99.4%
ECE = 7.04%The table uses ten equal-width bins over \([0.50, 1.00)\). Always say which bins you used, because ECE is sensitive to that choice, and this binning is not flattering anyone: 284 of the 400 items land in the top bin, which alone contributes about 44% of the 7.04%. An equal-mass binning, or a Brier score — which needs no bins at all — is the more honest summary of the same data.
Two things to read here, and they are not equally solid. The model is overconfident in the middle: the bin it called 92.6% likely was right 17 of 25 times, a 68.0% rate whose 95% Wilson interval runs \([48, 83]\) — entirely below the confidence it stated. That is the firm finding.
The one people quote is the bottom bin, and it deserves more caution than it usually gets: 5 correct out of 15, a 33.3% point estimate with an interval of \([15, 58]\). That interval contains 50%, so this does not establish “below chance” — and a wide interval is not evidence of uselessness either. What this bin does establish is narrower: a label probability near 0.5 is not a calibrated estimate of being right, on fifteen items. (The model reports a log-probability; 0.5 is the probability, which is about \(-0.69\) in logs. Convert before you threshold.)
Do not read a label probability near 0.5 as “right half the time”. What it may still be good for is ordering: an uncalibrated score can rank documents well enough to route the doubtful ones to a human, which is exactly what the abstention experiment below does. Judge that use on an accuracy–coverage curve, not on this bin.
Two fixes, in order of effort. Temperature scaling (Guo et al. 2017) fits a single scalar \(T\) on a held-out slice and divides the logits by it — the same \(\exp(z/\tau)\) from Section 18.1, used after the fact instead of during generation. Because it is monotone it leaves the AUC and the abstention curve of the next section exactly unchanged while typically cutting ECE by a factor of two or more, so there is no reason not to try it first. What one scalar can fix is a systematic overconfidence, like the top bins here; it cannot remove the bin-to-bin zigzag in the middle of the table. If one scalar is not enough, Chapter 12’s isotonic or Platt map is the next step.
It is immediately useful for abstention
Even uncalibrated, the ranking is enough to route:
abstain on the least-confident kept accuracy on what is kept
0% 400 86.75%
5% 380 89.47%
10% 360 90.83%
20% 320 92.50%
30% 280 95.00%
50% 200 97.50%Sending the least-confident fifth to a human, or to a larger model, takes accuracy on the retained items from 86.75% to 92.50% without changing the model at all. Read the denominator: 92.50% is the model’s accuracy on the 320 it kept, not the accuracy of the whole pipeline. What the other 80 cost — a human’s error rate, or a bigger model’s — is not measured here, and the end-to-end number is the coverage-weighted combination of the two. Report accuracy and coverage together, always. Even so it is a larger gain than most prompt engineering produces, and it costs one threshold.
This is the cheapest routing signal in the book, and it composes with everything else: abstain below a threshold, escalate the abstained items to a bigger model, and report coverage alongside accuracy. Note that the threshold is a fitted quantity and belongs on a held-out slice — the same discipline Chapter 12 applied to its decision threshold. Choosing it on the data you report is the same mistake in new clothes.
Those last two tables describe the same 400 reviews. Side by side, they show why a broken calibration does not stop you from using the signal:
Badly calibrated, correctly ordered — and useful because of the second part
Left: each dot is one confidence bin, placed at the confidence the model stated and the accuracy it actually reached. A calibrated model would put every dot on the dashed diagonal; this one scatters, and the two rose dots are the failures worth remembering — the bin the model called a coin flip came in at 33.3% on only 15 items, and the bin it called 92.6% came in at 68.0% on 25. Right: the same 400 reviews sorted by that same miscalibrated confidence, then truncated from the least-confident end. Accuracy climbs at every step, which is the whole point — the ordering is right even where the numbers are wrong.
The caveat that limits all of it
This worked because the answer was one token from a two-token alphabet. That is the easiest possible case, and the technique degrades quickly away from it:
- Multi-token answers. The mean logprob over a sentence mostly measures how ordinary the phrasing was, not how likely the content is to be true.
- Open-ended generation. There is no “the answer token” to read.
- Long structured output. The probability of a well-formed JSON object is dominated by its punctuation, which the model is certain about.
Which is a design argument as much as a caveat: if you want a confidence signal, structure the task so the answer is one constrained token. Such labels need not cost accuracy: in Chapter 17’s comparison of twelve prompt variants, the one with the meaningless labels A/B finished second, more than ten points ahead of olumlu/olumsuz.
18.6 Streaming and incremental parsing
Streaming exists because users react to the latency they perceive, not the one you measure. A response that takes four seconds to arrive complete feels slower than the same response whose first words appear in two hundred milliseconds — even when both finish at the same moment, so no throughput metric can tell them apart.
The tension with everything above is that structured output and streaming want opposite things. A JSON object is only parseable once its last brace arrives, so the natural implementation buffers the whole response and streams nothing. Three ways out, in increasing order of effort:
| approach | what the user sees | cost |
|---|---|---|
| Buffer and validate | nothing, then everything | simplest; the full latency is perceived |
| Incremental parse | fields appear as they complete | a parser that accepts prefixes; partial states are visible |
| Split the call | prose streams, structure arrives after | two calls, and they can disagree with each other |
The middle row is the one worth building, and the thing to get right is that a partial parse is a different type from a complete one. {"duygu": "olu is not a bad object; it is an incomplete one, and a consumer that cannot represent not yet known will either crash or render garbage. Model it explicitly — a field is missing, pending, or present — rather than letting None mean two things.
One practical note follows from Section 18.3: a grammar and an incremental parser are the same automaton looked at from two ends.
If you are already constraining generation, the state machine that decides which tokens are legal is also the one that can tell a consumer which fields are final. Most stacks do not connect the two, and it is a reasonable thing to build when the interface warrants it.
18.7 Intuition checks
Answer each one before you open it. The first follows from the algebra; the other five are questions this chapter settled with a measurement rather than an argument.
Note1. Does lowering temperature make the model more accurate?
It makes it more deterministic, which is not the same thing. Temperature rescales the logits before the softmax, so it changes the contrast between options and never the ranking — the argmax token at \(\tau = 0.1\) is the argmax token at \(\tau = 1.9\).
So if greedy decoding gets an item wrong, lowering the temperature will not fix it; it will only make the model wrong more reliably. What temperature controls is variance. Lowering it can raise the accuracy of sampled output — each draw lands on the argmax more often, which is what Chapter 15 saw at temperature 1.0. But as the temperature falls, the accuracy of sampled output converges to greedy’s; in the limit the two are the same, and an item whose argmax is wrong stays wrong.
Note2. Your JSON parse failure rate is 2%. Is constrained decoding worth it?
It depends entirely on what happens next. For extraction into a table, 2% is a retry and a shrug. For tool calls, 2% is a wrong action or a stack trace on every fiftieth request, and the answer is yes without further discussion.
Then measure the cost rather than assuming it is small: overhead here ranged from 1.01× to 17.5×, worst where the schema demanded something the model did not want to produce.
Note3. A grammar guarantees valid JSON. What does it not guarantee?
Anything about the content. Measured in Section 13.8: a schema demanding an integer answer returned {"answer": 1} for the capital of France — valid, typed, parseable, meaningless.
Constrained decoding converts parse failures into semantic failures. That is usually the trade you want, because a wrong value can be validated and a crash cannot, but it is a trade. Push what you can into enums, validate values separately from structure, and give the model a legal way to say I do not know.
Note4. Your weak model is 79% schema-valid. Is it a bad model?
Partly, but less than the number suggests. Measured here: llama3.2:3b at 79.3% schema-validity on a nested schema scored 70.7% accuracy — and with the grammar turned on, 100% valid and 88.0% accurate.
Seventeen of those points were never a reasoning failure; they were a formatting failure being counted as one. Before concluding a small model cannot do your task, make sure you are measuring the task rather than its punctuation.
Note5. The model says it is 92% confident. Should you believe it?
Not as a probability. Measured on 400 items: the bin of answers the model rated 0.90–0.95 was right 68.0% of the time. Expected calibration error was 7.04%, and the bin nearest probability 0.5 was right 33.3% on 15 items — an interval wide enough to contain chance, and too wide to prove anything except that the number is not a calibrated probability.
But the ranking was good — AUC 0.8268 for predicting its own correctness — which is enough to route on. Use logprobs to order items, not to quote percentages, and calibrate on a held-out slice if you need the number itself.
Note6. You want a confidence score for a free-text answer. How?
Not by averaging the logprobs, which mostly measures how ordinary the phrasing was rather than how likely the content is to be right.
Restructure the task instead. Ask for the answer and a single constrained token that classifies it — a verdict, a category, a yes/no — and read the logprob of that token. This chapter’s abstention curve works because the answer was one token from a two-token alphabet; that property is something you design in, not something you find.
18.8 Interview questions
On sourcing — the same caveat as Chapter 2. These recur in the two credible open sources this book uses, plus the ones this chapter’s own measurements make answerable.
Testing whether you understand the mechanism
Q: What does temperature actually do?
Rescales the logits before the softmax, \(p_i \propto \exp(z_i/\tau)\). Say the consequence that shows you have thought about it: it never changes the ranking, only the contrast, so temperature cannot fix a wrong argmax. Then distinguish it from top-k and top-p, which are hard cutoffs that remove candidates entirely, and from min-p, whose cutoff scales with the top token’s probability so it adapts to how peaked the distribution is.
Q: How does constrained decoding work?
Compile the schema into an automaton over the tokenizer’s vocabulary, and at each step mask the logits of every token that could not legally continue. Invalid output becomes unreachable rather than unlikely. The good follow-up: it guarantees syntax and nothing else, and it costs latency — measured here from 1.01× to 17.5×, worst when the schema demands something the model does not want to produce.
Q: Why is beam search standard in translation and absent from chat?
Because beam search looks for the highest-probability sequence, which is what you want when the task has approximately one right answer and actively not what you want when it has many: in open-ended text the most probable sequence tends to be bland and repetitive (Holtzman et al. 2020). Chapter 15 measured the cost of confusing the two in the other direction, sampling where one answer was wanted: a chat-default temperature of 1.0 took a translation model from 30.17 BLEU to 14.34.
Testing whether you have shipped one
Q: Tool calls fail in production about 2% of the time. What do you do?
Constrain the arguments — this is the case where 98% is not good enough, because something executes afterwards. Then narrow the types: enums for operations, patterns for dates, so whole error classes become unreachable. Then verify the constraint is actually applied, because a server that ignores format fails silently. And say clearly that a valid call is still not a correct call: the id may be well-typed and wrong, which is Chapter 24’s problem rather than the decoder’s.
Q: How would you make your system say “I don’t know”?
Give it a legal way to — a null in the schema or an explicit abstain value — and then a threshold to decide when to use it. Constrain the answer to a single token, read its logprob, and pick a threshold on a held-out slice. Measured here: abstaining on the least-confident 20% took accuracy on the kept items from 86.75% to 92.50% (80% coverage), no model change involved — what the other 20% cost has to be measured separately. Report coverage alongside accuracy, or the number is meaningless.
Q: Your evaluation says the model is 92% confident and it is right 68% of the time. What is wrong?
Nothing is wrong; you are reading a discrimination signal as a calibration one. The logprob ranked answers well — AUC 0.8268 for predicting its own correctness — while being badly calibrated, at 7.04% ECE. Use the ordering to route and calibrate on held-out data if you need the number itself. Chapter 12 has the machinery.
18.9 The 2026 toolbox
Everything in this chapter ran on one laptop with the versions below.
| library | version used | what it is for here |
|---|---|---|
ollama |
0.24.0 | serving qwen3.5:9b and llama3.2:3b, and the format parameter that enforces a schema |
| Python | 3.14.5 | json.loads from the stdlib parses; the twenty lines around it are our own checks for required fields and enum values, since parsing is not schema validation. The abstention curve uses numpy, which is not stdlib |
Worth knowing about, not used above:
| library | current version | when you would reach for it |
|---|---|---|
outlines |
1.3.3 | the reference implementation of the automaton in Section 18.3; use it when you need the grammar in your own process rather than behind a server flag |
xgrammar |
0.2.5 | the same idea optimised for throughput; what to reach for when the 17.5× overhead measured above is unacceptable |
lm-format-enforcer |
0.11.3 | schema enforcement that plugs into transformers and vLLM generation loops |
guidance |
0.3.1 | interleaves generation with fixed template text, so the model only fills the holes |
pydantic / jsonschema |
2.13.4 / 4.26.0 | validating the values after the grammar has guaranteed the shape — the step Section 18.3 argues you still owe |
18.10 Lab: an extractor that cannot emit invalid output
Build the pipeline, then measure what the guarantee did and did not cover.
Step 1 — write the schema before the prompt
Fields, types, enums wherever the values are enumerable, and an explicit nullable for not present in the source. The schema is the specification; the prompt is how you ask for it.
Step 2 — measure the unenforced baseline
Ask for the schema in the prompt, no enforcement, on 150 items. Record parse rate, schema-validity, and accuracy. On a strong model expect 98%-ish, which is the point: you need this number to know what enforcement is buying.
Step 3 — turn the grammar on, and verify it is actually on
Enforce the schema. Then confirm the constraint is live by sending one request whose schema the model would otherwise violate — Section 13.8 found a server silently ignoring format on one endpoint while honouring it on another, and the failure is invisible without this check.
Step 4 — measure the latency cost, per schema
Time both modes. The overhead here ranged from 1.01× to 17.5× and was not monotone in schema complexity, so your schema’s cost is an empirical question. If it is large, find out which part of the schema is expensive by removing fields one at a time.
Step 5 — count how often valid output is wrong
This is the step the guarantee tempts you to skip. Take the schema-valid outputs and check the values against your labels. Report validity and accuracy as two separate numbers, because a pipeline can be 100% on the first and poor on the second, and only one of those is visible in your logs.
Step 6 — add abstention
Constrain the answer field to a single token, read its logprob, and plot accuracy against coverage. Pick a threshold on a held-out slice — not on the data you report — and state the coverage you are shipping at.
What to hand in
A table with validity, accuracy and latency for enforced and unenforced; the verification from step 3; the accuracy-versus-coverage curve; and one paragraph on which failures your guarantee does not cover.
18.11 Exercises
- Show algebraically that temperature scaling never changes the argmax token. What does this imply about greedy decoding?
- In the explorer, set \(\tau = 1\) and top-k \(= 12\). Find the smallest top-p that still keeps at least four tokens, and explain the cumulative-mass logic behind your answer.
- Set temperature to 0 and run the same prompt 20 times against a hosted API. Are the outputs identical? Explain what you find.
- Take an extraction task with a 40% JSON failure rate. Add constrained decoding and report the new failure rate — and the new error modes.
18.12 Where to go deeper
Before reading: take one prompt, generate at temperature 0 ten times, and confirm you get the same string. Whether you do decides how every other number in your evaluation should be reported.
| resource | what it is |
|---|---|
outlines |
constrained generation as a library; the source is the clearest explanation of the automaton in Section 18.3 |
| How to generate text — Hugging Face | the decoding strategies with runnable code and honest examples of each one failing |
| Controllable Text Generation — Lilian Weng | a survey-depth post covering the space this chapter samples from |
vllm |
production serving, with structured-output support; read its sampling parameters to see what is actually exposed |
| Stanford CS336 | includes inference and decoding from the implementation side |
| Let’s build GPT: from scratch — Andrej Karpathy | the generate function written early on, for the bigram baseline, is the simplest possible sampling loop (softmax, then draw a token), in code you can pause and edit; nanoGPT’s version adds temperature and top-k |
| topic | source |
|---|---|
| why the highest-probability text is not the best text | Holtzman et al. (2020) |
| a cutoff that scales with the distribution | Nguyen et al. (2024) |
| compiling a schema into a token mask | Willard and Louf (2023) |
| the same idea for general structured tasks | Geng et al. (2023) |
| whether models know what they know | Kadavath et al. (2022) |
18.13 What to remember
Temperature never changes the ranking. It rescales logits before the softmax, so it controls variance, and lowering it moves the accuracy of sampled output towards greedy’s, reaching it in the limit. If greedy gets an item wrong, a lower temperature makes it wrong more reliably.
Pick the policy from how many good answers the task has. One answer: greedy or a narrow beam. Many: sample. Chapter 15 measured the cost of getting it backwards at more than half the BLEU (30.17 → 14.34).
Repetition loops are high-probability, not accidental, which is why greedy walks into them. The penalties that fix them are blunt, and they are the wrong tool for structured output, where repeated keys and braces are correct.
On a strong model a grammar buys the last one or two percent.
qwen3.5:9bwas 98–99.3% schema-valid when merely asked, at every schema difficulty tested. Enforcement took it to 100% and left accuracy alone.On a weak model it is the difference between working and not.
llama3.2:3bfell to 79.3% valid on a nested schema; with the grammar it was 100% valid and accuracy rose 70.7% → 88.0%, because a formatting failure had been counted as a reasoning failure.The grammar’s cost is latency and it is unpredictable — 1.01× to 17.5× here, and not monotone in schema complexity. It was worst where the schema demanded what the model did not want to produce. Measure it on your schema.
A schema guarantees shape, never truth.
{"answer": 1}for the capital of France is valid, typed and meaningless. Constrained decoding converts parse failures into semantic failures, which is a good trade and not a fix.Constrain tool arguments without arguing about it. 98% is fine for extraction and unacceptable when something executes: at a thousand calls a day that is twenty wrong actions or stack traces.
Logprobs discriminate well and calibrate badly. AUC 0.9492 against the label and 0.8268 for predicting its own correctness, at 7.04% ECE — with the 0.90–0.95 bin right only 68.0% of the time and the bin nearest 0.5 right on 5 of 15 items, with substantial uncertainty. Rank with them; do not quote them as probabilities.
Abstention is the cheapest accuracy on kept items you will buy. Dropping the least-confident 20% took accuracy on the kept items from 86.75% to 92.50% (80% coverage) with no model change; the cost of the other 20% has to be measured separately. Choose the threshold on held-out data and report coverage with it.
A usable confidence signal is designed in, not discovered. It worked here because the answer was one token from a two-token alphabet. Averaging logprobs over free text measures how ordinary the phrasing was.