31  Serving and Inference Infrastructure

Chapter status: drafted. The measurements here are small on purpose — about 150 short requests against a 3B model on a laptop, a few minutes in total. They are enough because what matters is a ratio whose sign and cause are the same on every machine, at least for single or small-batch inference as here — prefill is compute-bound, decode is bandwidth-bound (limited by how fast weights can be read from memory, not by arithmetic). With large enough batches decode can become compute-bound too; Section 31.4 works out where. The magnitude does not transfer, and the chapter says where it stops transferring rather than pretending otherwise. Everything that needs a GPU fleet to observe is cited rather than claimed. Every condition is measured five times and reported as a median; the conditions are run in shuffled order, so drift cannot pile up in the long-prompt rows; and a fixed short prompt is re-timed at intervals through the session (11 times in all) as a control. On the run printed here that control held to 2.0% across the whole session, which is what makes the rest of the table worth reading. Software: ollama 0.24.0 serving llama3.2:3b (Q4_K_M, 28 layers, 24 attention heads, 8 KV heads, head dimension 128), num_ctx 8192, temperature 0, OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0, on Python 3.14.5 and Apple Silicon. The exact chip matters more than that name suggests, and the timings say which class it was: 190 decode tokens per second against a 2.0 GB weight file is about 380 GB/s of weight streaming, which only a high-bandwidth part sustains. The same model on a 120 GB/s laptop tops out near 60 tokens per second. Reproduce the ratio, not the rate.

Learning objectives. After this chapter you can:

• split a request into prefill and decode, and say which half your latency is actually made of;
• read the KV cache as the thing that makes generation fast and long context expensive, and size it in gigabytes before you rent the hardware;
• explain what batching buys the operator and what it costs the individual user;
• use prompt caching as the first lever, and recognise the prompt layouts that make it do nothing;
• build a cost-per-request worksheet from token counts you can measure today;
• order the remaining levers — smaller model, quantization, speculative decoding — by what each one actually trades away.

31.1 One request, two very different halves

Everything downstream of this chapter — the gateway of Chapter 32, the traces of Chapter 34, the bill in Chapter 38 — is easier to reason about once you can see that a single request is not one operation but two, with different costs and different bottlenecks.

prefill / decode — the two halves of a request: the prompt is read in one parallel pass, then output tokens are produced one at a time, each depending on the last. ↩︎

Prefill / decode is the split. Prefill reads the prompt. Every token is available at once, so the whole prompt goes through the network as one batched matrix multiplication — the hardware is busy and the work is compute-bound.

Decode writes the answer. Token n+1 cannot be computed until token n exists, so each one is a separate pass over the whole model, and — as long as the batch is small — the machine spends most of its time moving weights from memory rather than multiplying them.

That asymmetry is the chapter’s foundation, and it is easy to measure. Six prompt lengths, each with its own text so nothing is cached, generating 32 tokens:

prompt tokens   prefill   prefill tok/s      decode tok/s
           96    0.064s            1,433            191.3
          298    0.153s            2,007            192.2
          572    0.277s            2,047            190.9
        1,095    0.520s            2,104            182.9
        2,156    1.055s            2,043            177.9
        4,247    2.292s            1,848            161.2

Prefill runs at roughly 2,000 tokens per second; decode at roughly 190. On this hardware, prefill time per token was about eleven times lower than decode time per token — on the 4,247-token row, 0.54 milliseconds against 6.2. That is a statement about time, not about a provider’s price list, where output tokens usually cost several times input — and it is the ratio rather than the absolute numbers that survives a change of hardware. Take both figures from the same row: decode slows as the context grows (5.2 ms at 96 tokens, 6.2 ms at 4,247), so pairing the best prefill with the worst decode inflates the gap. Prefill also dips a little on the longest row, most likely for a related reason: each prompt token attends to every token before it, so the attention part of prefill grows faster than the prompt does.

The 1,433 tok/s on the first row is not a slower prefill; it is a fixed per-request overhead that a 96-token prompt cannot amortise. Five repetitions confirm it, and they also make it the noisiest row in the table — 34.5% spread against 2–7% everywhere else, because the fixed cost dominates a prompt that small. An earlier single-shot version of this table reported 105 decode tok/s on the last row; repeated five times it is 161, with 0.7% spread. That is what one measurement buys you. Each column is its own median over the five runs, so dividing one printed column by another (96 ÷ 0.064 = 1,500) can differ a little from the printed rate.

So a first, crude latency model, good enough to argue with:

\[ \text{latency} \;\approx\; \underbrace{\frac{\text{prompt tokens}}{\approx 2{,}000/\mathrm{s}}}_{\text{prefill}} \;+\; \underbrace{\frac{\text{output tokens}}{\approx 190/\mathrm{s}}}_{\text{decode}} \]

One request: one parallel pass, then a serial loop

The top half is not to scale: the decode boxes are enlarged so the per-token loop is visible at all. In wall clock those 32 passes take 0.20 s against prefill’s 2.29 s, so drawn honestly they would occupy about 25 pixels against prefill’s 286. The bar underneath is to scale, and it is the shape worth remembering. Prefill is one pass over 4,247 tokens; decode is 32 separate passes over the model, one per output token. That bar is a different request — 7,427 tokens of context, 64 tokens out — measured with streaming on, and it is the shape most production requests have: almost all of the wall clock is spent before the first character appears.

What we will look at. First where the wall clock actually goes, which at production prompt lengths is almost entirely prefill — and why streaming cannot touch that. Then the KV cache, which is what makes decode fast and what makes a long context expensive, in gigabytes you can compute before renting anything, and how it adds up with the weights to the memory a card needs. Then batching, which buys the operator throughput and charges the individual user for it. Then prompt caching, the one lever with no quality cost and the largest measured win in this chapter. Then the cost worksheet, which is four numbers and a multiplication. And finally the remaining levers, in the order to reach for them rather than the order they get proposed in.

31.2 Where the wall clock actually goes

time to first token (TTFT) — how long the reader waits before anything appears: queueing, prefill and the first decode step. Prefill dominates on long prompts; streaming does not shorten it, it makes it the wait the user feels. ↩︎

Users do not experience total latency. They experience time to first token (TTFT) and then a stream. Measuring it with streaming on, at four context lengths, each generating 64 tokens:

context tokens    TTFT     total    TTFT share
           90   0.132s   0.487s        27.1%
          960   0.506s   0.863s        58.7%
        3,731   2.061s   2.476s        83.2%
        7,427   4.715s   5.182s        91.0%

At 7,427 tokens of context, 91% of the request is over before the first character appears. The model is not “slow”; the prompt is long, and the user is waiting on prefill.

Two consequences that get missed.

Streaming does not fix TTFT. It fixes everything after it. Streaming converts a 5.2-second wait into a 4.7-second wait followed by text arriving at reading speed, which is a large perceived improvement — but the 4.7 seconds is untouched, and no amount of front-end work will touch it. The lever for TTFT is the prompt, and Section 31.5 is the cheapest version of that lever.

A long context makes decode slower too, not just prefill. Every generated token attends over everything before it, so the per-token cost grows with what is already in the window. Measured separately — five repetitions, as everywhere else, with 64 output tokens each so the rate is stable:

context tokens   decode tok/s   ms per token
            83          189.1           5.29
           965          186.7           5.36
         3,731          160.6           6.23
         7,414          140.9           7.10

A 25% slowdown between an empty window and a 7,400-token one. Worth knowing because it compounds: a long prompt costs you once in prefill and then again, quietly, on every token you generate.

When you compare this table with the first one, compare at matching context lengths. The first table’s 4,247-token row (161 tok/s) agrees, within noise, with this table’s 3,731-token row (161), not with the 7,414-token one (141). A number taken from the wrong row is still a wrong number; it just looks sourced.

Two names for what these tables measure, because dashboards use them. The right-hand column — milliseconds per generated token — is the time per output token (TPOT), also called inter-token latency; together with TTFT it describes what one user feels. Throughput is the other axis: tokens per second summed over every request the server is handling at once. It is what the operator pays for, and Section 31.4 shows why the two pull against each other.

31.3 The KV cache, and why context has a price in gigabytes

Decode would be far worse than 5.2 ms per token if each step recomputed the whole prompt. It does not, because of the KV cache: the keys and values computed for every previous token are kept, so step n+1 computes one new column and reads the rest.

KV cache — the stored keys and values from earlier tokens, so generation does not recompute the whole prompt at every step. Fast, and the reason long context costs memory. ↩︎

That is a straight time-for-memory trade, and the memory is computable before you rent anything. Per token the cache holds a key and a value, for every layer, for every KV head:

\[ \text{bytes per token} \;=\; 2 \times L \times H_{kv} \times d_{\text{head}} \times \text{bytes per element} \]

For this model at fp16 — 28 layers, 8 KV heads, head dimension 128 — that is \(2 \times 28 \times 8 \times 128 \times 2 = 114{,}688\) bytes, or 112 KiB per token. Multiply by the window:

What the window costs in memory, llama3.2:3b

context	cache, fp16	cache, q8_0	fp16 cache vs 6.4 GB of fp16 weights	fp16 cache vs the 2.0 GB Q4 file
8,192	0.88 GiB	0.46 GiB	0.15×	0.47×
32,768	3.50 GiB	1.86 GiB	0.59×	1.9×
131,072	14.00 GiB	7.44 GiB	2.3×	7.5×

The fp16 columns, which the two ratio columns also use, are the default and the right first calculation. The server these timings came from ran the q8_0 column instead: OLLAMA_KV_CACHE_TYPE=q8_0, an 8-bit cache at 1.0625 bytes per element once the block scale is counted, which puts it at 59.5 KiB per token and 7.4 GiB at the full window — a little over half the fp16 figure. Quantizing the cache is a separate lever from quantizing the weights, it is the one that moves the number that actually binds, and it is off by default. Check which one your server is running before you size a card, because nothing in the output tells you.

A 2.0 GB quantized model file with its 131k window full at fp16 needs 14 GiB of cache. Note what that last ratio compares before quoting it: four-bit weights against a sixteen-bit cache, which is the widest of the three honest comparisons. At matched precision the cache is 2.3× the weights; against the eight-bit cache this chapter actually ran it is 4.0× the file. All three say the same thing — the cache is the allocation that scales with usage, and it is the one nobody budgets for — but only one of them is the number to size a card with, and which one depends on what you are running.

This is why “can I run a 3B model?” is the wrong question and “at what context length?” is the right one, and why the default num_ctx in a local server matters so much: ask for the full 131k window and you have committed 14 GiB (about 15 GB) before the first request.

The 8 KV heads against 24 attention heads in that arithmetic are grouped-query attention, and they are the reason the number is 14 GiB rather than 42 — three heads share each key-value pair. It shrinks the per-request KV cache (and the cache bytes decode reads each step); Section 7.6 does the same arithmetic for a 70B-class model, 10.7 GB per request at 32k instead of 85.9 GB under hypothetical full multi-head attention. It is a recurring shape: architecture choices from Chapter 7 reappear here as line items.

Sizing a card, all together. The memory a server needs is roughly weights + KV cache per sequence × concurrent sequences + a working margin for activations and the runtime itself. Weights are parameters × bits per weight ÷ 8. This model’s 3.2 billion weights take 6.4 GB at 16 bits and 2.0 GB as Q4_K_M — about five bits per weight on average, not four, because some tensors are kept at higher precision and every small block of weights carries its own scale. Served at fp16 with one 32k-token conversation, that is 6.4 GB of weights plus 3.5 GiB (3.8 GB) of cache, about 10 GB before the margin. Four such conversations at once need 14 GiB of cache, and the cache is now more than twice the weights.

For a mixture-of-experts model (Section 7.7), put the total parameter count into that formula: every expert must be in memory, because any token may be routed to any of them. The active count is what sets per-token speed. That is how qwen3.6:35b-a3b in Chapter 12 needs a 23.9 GB file yet served documents slightly faster than the dense 9.7B model. (Under heavy batching, different tokens pick different experts, so more of the weights are read at every step and that speed advantage narrows.)

At fleet scale the same pressure produced PagedAttention, which manages the cache in fixed-size blocks like virtual memory instead of one contiguous reservation per request (Kwon et al. 2023). The waste it removes is fragmentation — room reserved for 2,000 tokens by a request that generated 200, and the unusable gaps left between such reservations — and removing it is what lets a server hold many more concurrent sequences on the same card. Because blocks can be shared, it also lets several requests point at one copy of a common prompt prefix, which is what Section 31.5 builds on.

31.4 Batching: throughput the individual user cannot feel

Why decode is bandwidth-bound, and where that stops

“Decode is memory-bound” has been asserted four times by now. Here is the derivation, because it also tells you the one thing the assertion does not: how far batching goes before it stops being free.

A decode step for a batch of \(B\) sequences does about \(2NB\) floating-point operations and moves about \(N \cdot b\) bytes of weights, where \(N\) is the parameter count and \(b\) the bytes per parameter. Their ratio — the arithmetic intensity — is

\[I \;=\; \frac{2NB}{N b} \;=\; \frac{2B}{b}\]

and note what cancelled: the model size. Intensity depends on batch and precision only. A chip is compute-bound above its ridge point, peak FLOP/s divided by peak bandwidth — for an H100-class part, roughly \(989 \text{ TFLOP/s} \div 3.35 \text{ TB/s} \approx 295\) FLOP/byte. So at fp16 decode stays bandwidth-bound until about \(B = 295\); at four-bit weights, until about \(B = 74\).

Prefill is the same formula with the prompt length in place of \(B\). For the 4,247-token prompt above, at fp16, that is an intensity of 4,247 — more than an order of magnitude past the ridge. (That ridge is for an H100; on the Apple Silicon chip measured here peak compute is far lower relative to bandwidth, and four-bit weights raise the intensity further, so the gap is larger still.) That is exactly why prefill is compute-bound and decode is not, and why the two halves of the table behave so differently.

That number is also the honest boundary on the claim that follows.

Decode is memory-bound: the hardware spends its time moving weights, not multiplying them. That means a second request costs almost nothing extra, because the weights being moved for one sequence can serve several at once. Batching converts that slack into throughput — and the second request is nearly free, and so is the fiftieth, and somewhere around the three-hundredth it is not, because the batch has crossed the ridge and the arithmetic starts to cost.

In practice the limit arrives sooner, for two reasons the formula leaves out. Each sequence also reads its own KV cache at every step, and that traffic is not shared across the batch, so at long context it grows with \(B\) and pulls intensity down. And every sequence’s cache must fit in memory: on real cards it is usually memory, not the ridge, that caps the batch.

continuous batching — new requests join a running batch as slots free, instead of waiting for the whole batch to finish. The throughput lever in every modern server. ↩︎

Naive batching waits for a full batch and then runs it to completion, so one long generation holds the whole batch hostage. Continuous batching — iteration-level scheduling, introduced by Orca (Yu et al. 2022) and standard since — lets a finished sequence leave and a queued one take its slot at the next step.

I did not measure this, and should say so plainly: a single-stream local server is the wrong instrument, and inventing a number here would be worse than citing one. What the literature reports is large: Orca’s authors measure order- of-magnitude throughput gains over naive batching at the same latency budget (Yu et al. 2022), and vLLM reports 2–4× over the serving systems that preceded it by combining continuous batching with paged cache management (Kwon et al. 2023).

The part worth internalising is the trade, not the multiple:

What batching does to each number

	throughput	single-request latency
No batching	worst	best
Static batching	better	worst — waits for the slowest sequence
Continuous batching	best	slightly worse than none, far better than static

Batching is a throughput optimisation that individual users pay for in latency. Every request shares the machine with its batch-mates, so a busy server is a slower server per request even when it is a cheaper server per request. If your product has a latency SLO, it is a constraint on batch size, and somebody has to own that number.

31.5 Prompt caching: the lever to reach for first

If prefill dominates latency and the prompt begins with the same thousands of tokens every time — a system prompt, a tool schema, a few-shot block, a retrieved document being asked about repeatedly — then that prefix’s KV cache can be computed once and reused.

prompt caching — reusing the KV cache of a prompt prefix across requests, so a repeated system prompt is not recomputed. ↩︎

Prompt caching is not a subtle optimisation. The same six prompts as the first table, each sent twice:

prompt tokens   cold prefill   warm prefill   speedup   cold total   warm total
           95         0.066s         0.016s        4x       0.300s       0.252s
          303         0.151s         0.016s       10x       0.386s       0.252s
          564         0.277s         0.014s       19x       0.514s       0.251s
        1,090         0.538s         0.016s       34x       0.781s       0.259s
        2,149         1.060s         0.012s       88x       1.311s       0.261s
        4,247         2.290s         0.015s      153x       2.562s       0.286s

Warm prefill is flat at 12–16 milliseconds regardless of prompt length, because nothing is being recomputed. At 4,247 tokens that is 153× on the prefill and 9.0× on the whole request.

The longer the prefix, the larger the win — which is the opposite of how most optimisations scale, and it is why this is the first thing to try.

Three conditions, and all three are design decisions rather than configuration:

The prefix must be byte-identical. A timestamp, a user name, or a session id near the top of the prompt invalidates everything after it. Cached prefixes match by prefix, so one changing token at position 12 discards the other 4,000.

Order the prompt by stability. System instructions, then tool schemas, then few-shot examples, then retrieved context, then the user’s turn. Most teams write prompts in the order they thought of them, which puts the volatile part first and turns the cache off without anybody noticing.

Measure the hit rate or you do not have one. A cache you have not instrumented is a belief. Chapter 34 is where that instrumentation belongs, and the metric is share of prompt tokens served from cache, per route.

What was measured, and what was not. The 153× above is a local server’s slot-level prefix reuse: the previous request’s KV cache is still resident and the next request happens to share its prefix. That is the same mechanism a hosted API uses, but not the same system, and three differences change the number you will see in production.

Eviction is the big one. ollama keeps one slot per model by default (OLLAMA_NUM_PARALLEL=1), so a single interleaved request from another user discards the prefix — and the measurement above sent the same prompt twice in a row, which is the friendliest case that exists. Two users alternating would have shown nothing. Servers built on paged blocks, such as vLLM and SGLang, keep many cached prefixes at once and share them between requests, which softens this but does not remove it. Control is the second: some hosted APIs cache automatically above a minimum prefix length, others require an explicit marker, and a local server gives you neither knob. Price is the third: hosted cached input tokens are billed at a discount rather than skipped, and some providers charge extra for the request that writes the cache. That discount is the line the worksheet in Section 31.6 depends on.

Take 153× as an upper bound on the mechanism, and instrument the real hit rate on the real system.

31.6 The cost worksheet

Latency was the first half of the argument. The bill is the other half, and both are made of the same tokens — which is why the first thing to establish is which tokens.

From Chapter 14, summarizing a 300-word news article measured 890 prompt tokens against 53 generated — a ratio of nearly 17 to 1. That shape is typical of anything with a system prompt, a schema and retrieved context, which is to say most production LLM features.

On this shape of workload, most of what you pay for is reading, and that inverts the usual intuition about where to optimise. One conversion step before you act on it: providers bill output tokens at several times input, so a 17:1 token ratio is nearer 3:1 or 4:1 in money, and a cache hit discounts the input side further. Do the multiplication with your own price sheet. On this ratio it survives — asking for shorter answers saves little, sending less saves a lot — and on a task that generates more than it reads it would not.

The worksheet has four lines and one multiplication:

Everything a cost estimate needs

line	where it comes from	example
prompt tokens	count them; do not estimate	890
of which cached	measure the hit rate	60%
output tokens	count them	53
requests per month	product, not engineering	2,000,000

Multiply out with your provider’s three prices — uncached input, cached input, output — and you have a monthly bill with a cache-hit-rate term you can move. With illustrative prices, not any provider’s, of $1.00 per million uncached input tokens, $0.10 per million cached and $4.00 per million output tokens, the example works out as:

The worksheet multiplied out, at illustrative prices

line	tokens per month	price per million	cost per month
uncached input (40% of 890)	712 M	$1.00	$712
cached input (60% of 890)	1,068 M	$0.10	$107
output (53)	106 M	$4.00	$424
total			$1,243

Without the cache the same traffic costs $2,204, of which input is $1,780. Two things to read off it: the hit rate is worth more than any change on the output side, and once the cache works, input is about two thirds of the bill even though it is 94% of the tokens.

The widget below computes the time side of that worksheet — seconds per request and machine-hours per month, not dollars — wired to the latency model from Section 31.1 so both move together; multiply by your own prices for the bill. It holds the measured rates fixed, so it ignores the decode slowdown at long context and extrapolates beyond the 7,400 tokens actually measured; read its long-prompt end as a lower bound.

Latency and machine time, on the numbers measured above — move the prompt, watch what happens

viewof srvPrompt = Inputs.range([100, 32000], {
  value: 4000, step: 100, label: "prompt tokens"})

viewof srvOutput = Inputs.range([16, 2000], {
  value: 200, step: 16, label: "output tokens"})

viewof srvHit = Inputs.range([0, 1], {
  value: 0.6, step: 0.05, label: "prompt-cache hit rate"})

srvRates = ({prefill: 2025, decode: 190, warm: 0.015})

srvCalc = {
  const uncached = srvPrompt * (1 - srvHit);
  const prefill  = uncached / srvRates.prefill + srvRates.warm;
  const decode   = srvOutput / srvRates.decode;
  return ({
    uncached,
    prefill,
    decode,
    total: prefill + decode,
    share: prefill / (prefill + decode),
    coldPrefill: srvPrompt / srvRates.prefill
  });
}

Plot.plot({
  width: 680, height: 132, marginLeft: 96, marginBottom: 34,
  x: {label: "seconds", grid: true, domain: [0, Math.max(srvCalc.coldPrefill + srvCalc.decode, 0.5)]},
  y: {domain: ["no cache", "with cache"], label: null},
  color: {domain: ["prefill", "decode"], range: ["#1F4E79", "#B3453E"], legend: true},
  marks: [
    Plot.barX([
      {row: "no cache",   part: "prefill", t: srvCalc.coldPrefill},
      {row: "no cache",   part: "decode",  t: srvCalc.decode},
      {row: "with cache", part: "prefill", t: srvCalc.prefill},
      {row: "with cache", part: "decode",  t: srvCalc.decode}
    ], {x: "t", y: "row", fill: "part", inset: 2.5}),
    Plot.ruleX([0])
  ]
})

html`<p style="font-size:0.88rem; color:var(--fig-muted); margin-top:0.5rem;">
  <b style="color:var(--fig-ink);">${srvCalc.total.toFixed(2)} s</b> per request
  — ${(srvCalc.share * 100).toFixed(0)}% of it prefill, and that share is what
  the reader waits through before the first character.
  ${srvHit > 0 ? html`Caching saves
  <b style="color:var(--fig-ink);">${(srvCalc.coldPrefill - srvCalc.prefill).toFixed(2)} s</b>
  by skipping ${(srvPrompt * srvHit).toLocaleString("en-US")} prompt tokens.`
  : html`With no cache the whole prompt is recomputed every time.`}
  At 2 million requests a month that is
  <b style="color:var(--fig-ink);">${((srvCalc.total * 2e6) / 3600).toLocaleString("en-US",
  {maximumFractionDigits: 0})} machine-hours</b> of serving, if requests are
  served one at a time; batching divides it.
</p>`

31.7 The levers, in the order to reach for them

The order matters more than the list, because the cheap levers are also the ones with no quality cost, and teams routinely start at the bottom.

Six levers and what each one trades away

lever	what it buys	what it costs	reach for it
Send fewer tokens	linear on prefill and on the bill	engineering time on prompts and retrieval	first, always
Prompt caching	up to 153× prefill, measured above	a prompt ordered by stability	first, always
Smaller model	3–10× on everything	quality, measurably	when the eval says it holds
Quantization	memory and bandwidth	some quality, task-dependent	when memory is the binding constraint
Speculative decoding	decode latency, primarily; cost depends on serving efficiency and pricing	a draft model to host	when decode is the slow half — long answers; little help when prefill dominates (details below)
Continuous batching	throughput	per-user latency	when the server is busy, not before

Two deserve a sentence each, because both are widely misunderstood.

Quantization stores weights at reduced precision — the model measured in this chapter is Q4_K_M, about five bits per weight on average. Because decode reads every weight once per step, fewer bytes per weight also raise the decode ceiling, roughly bandwidth ÷ weight bytes at short context (at long context the KV-cache bytes read each step add to the denominator): that is how the opening note got 380 GB/s from 190 tokens per second on a 2.0 GB file, and the same chip would top out near 60 tokens per second on the 6.4 GB fp16 weights. Modern post-training methods choose which weights to keep precise rather than rounding uniformly: GPTQ solves a layer-wise reconstruction problem (Frantar et al. 2023), AWQ protects the small fraction of weights that the activation distribution says matter most (Lin et al. 2024). Both report far less degradation than naive rounding, and both are still degradation. It is a memory optimisation with a quality cost, and the cost is task-dependent — which means it belongs in your eval suite, not in a blog post’s benchmark table.

Speculative decoding attacks the serial part directly. A small draft model proposes several tokens; the large model scores them all in a single parallel pass and a modified rejection-sampling step decides how many to keep, resampling the first rejected position from a corrected distribution (Leviathan et al. 2023). “Accepts the longest correct prefix” is the useful picture, and at temperature 0 it is exactly the mechanism: keep draft tokens while they match the large model’s own greedy choice. With sampling it is only approximately so: acceptance is stochastic, and a plausible draft token can still be rejected. The output distribution is provably unchanged — this is not an approximation — and the win is in wall-clock, typically 2–3× when the draft model agrees often. Note what it does not do: the large model still processes every token, and also scores the rejected drafts, so FLOPs rise slightly rather than fall; on per-token API pricing the bill does not fall either. Self-hosted, cost follows GPU time and throughput, so the speedup can lower cost per completed answer — measure it on your own serving load. What it buys first is latency.

For the arithmetic underneath all of this at scale — how model size, batch size and chip memory bandwidth combine to set what is achievable — Pope et al. work it through properly (Pope et al. 2023), and Chapter 8’s \(6ND\) gives the training-side counterpart.

31.8 Intuition checks

Answer each one before you open it. Four are settled by this chapter’s measurements and arithmetic — the two rates, the TTFT share, the KV cache arithmetic, and what a cold prefill costs against a warm one — and the last two are the ones this chapter deliberately did not measure, because a single-stream laptop is the wrong instrument for batching and a general benchmark is the wrong instrument for quantization.

Note1. Your feature is “too slow”. Where do you look first?

At the ratio of prompt tokens to output tokens, before you look at the model.

Prefill and decode run at very different rates — measured here, about 2,000 against 190 tokens per second — so a request with a long prompt and a short answer spends nearly all its time reading. At 7,427 tokens of context, 91% of the wall clock was over before the first character appeared.

If that is your shape, the levers are prompt length and prompt caching, and speculative decoding in particular has limited end-to-end benefit — it speeds up decode. A genuinely faster model does still help, because prefill is compute-bound and a smaller model has less compute to do; what will not help is anything aimed at the token-by-token half. If instead you are generating thousands of tokens, the answer is the opposite one, which is why the ratio is the first thing to measure rather than the last.

Note2. Does streaming reduce latency?

No. It reduces perceived latency after the first token, and changes nothing before it.

Streaming cannot start until prefill finishes, so TTFT is identical with it and without it. What streaming buys is that the remaining seconds are spent reading rather than waiting — a real and large product improvement, and not a performance one.

The give-away that a team has confused the two is a latency dashboard that improved when streaming shipped. Neither TTFT nor total time should have changed. If the number fell, the client is most likely now stopping the clock at the first chunk instead of the last one.

Note3. Why can a 2 GB model need more than 16 GB of memory?

Because the weights are not the whole footprint. The KV cache holds a key and a value for every token in the window, for every layer and every KV head.

For this model at fp16 — 28 layers, 8 KV heads, head dimension 128 — that is 112 KiB per token, so a 131,072-token window is 14 GiB (about 15 GB) of cache on top of a 2.0 GB model file. Say what that ratio compares, though: four-bit weights against a sixteen-bit cache. At matched precision the cache is 2.3× the weights. An 8-bit cache roughly halves the cache side, which is a lever worth knowing about and one that is off by default.

This is why the honest answer to “will it fit?” is a question about context length, and why a local server’s default window is a memory decision disguised as a convenience setting.

Note4. Your prompt cache hit rate is near zero and you cannot see why.

Something volatile is near the top of the prompt.

Prefix caches match by prefix: one differing token at position 12 invalidates everything after it. A timestamp, a user id, a session identifier or a freshly-shuffled few-shot block will all do it, and none of them look like a performance bug.

The fix is to order the prompt by stability — instructions, then schemas, then examples, then retrieved context, then the user’s turn — and the check is to send the same request twice and compare prefill times. Cold and warm should differ by an order of magnitude or more; measured here, 2.290s against 0.015s.

Note5. Continuous batching raised throughput 3×. Did latency improve?

Almost certainly not, and it probably got slightly worse per request.

Batching exploits the fact that decode is memory-bound: several sequences can share the weight movement that one sequence needs. That raises tokens per second across the server while each individual sequence advances no faster — and it now shares the machine.

So the two numbers move in opposite directions, and reporting only the one that improved is a common way to mislead a review. If you have a latency SLO, it is a ceiling on batch size, and somebody must own it.

Note6. When is quantization the wrong answer?

When memory is not your binding constraint, or when nobody has evaluated it on your task.

Quantization trades quality for memory and bandwidth. Modern methods lose much less than naive rounding — GPTQ and AWQ both choose which weights to protect rather than scaling everything uniformly — but “much less” is not “none”, and the loss is task-dependent in ways a general benchmark will not tell you.

If your constraint is prompt length, weight quantization does not shrink the problem: the KV cache is a separate allocation with its own lever (the 8-bit cache above), although memory freed from the weights can hold more cache. Reach for weight quantization when the model does not fit, and put it through the same eval you would use for a model swap.

31.9 Interview questions

On sourcing — the same caveat as Chapter 2. The questions below recur in the two credible open sources this book uses throughout, plus the ones this chapter’s own measurements make answerable.

Testing whether you understand the machine

Q: Walk me through what happens when a request hits an LLM server.

Two phases with different characteristics. Prefill reads the whole prompt in one parallel, compute-bound pass. Decode produces output tokens one at a time, each a separate memory-bound pass over the model, reusing the KV cache so earlier tokens are not recomputed. Name the rate difference — about eleven to one per token in this chapter’s measurements — and say which half dominates for the request shape you are being asked about. Candidates who describe one undifferentiated “inference” have not looked.

Q: Why is the KV cache the thing that limits concurrency?

Because it grows with tokens times sequences, while the weights are a fixed cost. Weights are loaded once and shared by every request on the card; the cache is per-sequence and grows with every prompt and generated token. Give the arithmetic — \(2 \times L \times H_{kv} \times d_{\text{head}}\) values per token, times the bytes per value — and note that grouped-query attention exists largely to shrink it. PagedAttention is the systems answer: manage it in blocks so a sequence that reserved 2,000 tokens and used 200 does not waste the rest.

Q: Where does batching help and where does it hurt?

It helps throughput because decode is memory-bound and the weight movement is shared. It hurts per-request latency because each sequence now shares the machine. Continuous batching removes the worst version of the trade — the head-of-line blocking in static batching — but does not remove the trade itself. A candidate who says batching is free has not run a latency SLO.

Testing whether you have shipped one

Q: The bill doubled last month and traffic did not. Debug it.

Input tokens first, because they dominate: 890 against 53 in this book’s summarization measurement. Look for a prompt that grew — a retrieval step returning more chunks, a system prompt that gained a section, a conversation history that stopped being truncated. Then the cache hit rate, because a volatile token moved to the top of the prompt turns caching off silently and looks exactly like a price change. Only then model or provider changes.

Q: How would you cut latency in half without changing the model?

Establish the split first — if it is 90% prefill, output-side changes cannot get you there. Then: prompt caching for the stable prefix, which measured 153× on prefill here; shortening the prompt itself, which is linear; and streaming to convert the remaining wait into reading time. Say what you would measure after each one, and say that speculative decoding is on the list only if you can host a draft model, and that what it buys is decode latency — whether it also lowers cost depends on how you pay for compute.

Q: We want a 99th-percentile latency SLO. What do you need?

A batch-size ceiling, an admission policy for what happens when the queue is long, a separate route for long-context requests so they cannot sit in front of short ones, and per-route TTFT instrumentation rather than a single global number. Then say the uncomfortable part: a p99 SLO and maximum throughput are in direct conflict, and somebody has to be told which one they are buying.

31.10 The 2026 toolbox

About 150 requests against a 3B model on a laptop — 130 timed calls across 20 conditions, each repeated five times (the cache condition sends a cold and a warm call per repeat), plus 11 canary (control) calls and 2 discarded warm-ups — needs exactly two things, listed in the first table below. The second table is what you would actually serve on. None of it was used here: the numbers in this chapter are ratios, and ratios are the part that survives the move to a real card.

The stack these numbers came from. Python 3.14.5, Apple Silicon

library	version used	what it is for here
ollama	0.24.0	the local server; its /api/generate reports the prefill and decode split per request, which is what made these measurements cheap
llama3.2:3b	Q4_K_M	the model under test; small enough to measure on a laptop without thermal throttling

Worth knowing about, not used above:

The wider ecosystem, versions as of writing

project	current version	when you would reach for it
vllm	0.28.0	the reference open-source server: continuous batching, paged KV cache, prefix caching (Kwon et al. 2023)
sglang	0.5.18	a serving runtime built around structured programs and aggressive prefix reuse
text-generation-inference	3.3.7	Hugging Face’s server, closest to their ecosystem; now in maintenance mode, and Hugging Face itself recommends vLLM or SGLang for new deployments
llama.cpp	b10690 (rolling)	CPU and Apple Silicon inference; the engine ollama was built on (newer ollama versions also run many models on their own engine over the same ggml library). Tagged per build, so pin one
tensorrt-llm	1.2.1	NVIDIA’s compiler-based runtime when the last 20% of throughput matters
llmcompressor	0.13.0	applying GPTQ and AWQ quantization (Frantar et al. 2023; Lin et al. 2024) to a model you intend to serve

31.11 Lab: measure your own two halves

Any local server that reports timings will do; the point is to produce your numbers rather than to reuse mine.

Step 1 — get the prefill and decode rates for your model

Six prompt lengths, each with distinct text so nothing caches, a fixed short output. Record prompt tokens, prefill seconds, output tokens, decode seconds. Two rates fall out, and their ratio is the number to remember.

Step 2 — find your TTFT curve

Turn streaming on and record time to the first token at three context lengths. Plot TTFT as a share of total. If it does not rise steeply, your prompts are shorter than most production prompts and you should retest with a realistic one.

Step 3 — measure your cache, then break it

Send the same long prompt twice and record both. Then send it twice more with a fresh timestamp at the top of the prompt on each call — the same timestamp twice would still cache. The second pair should lose the speedup entirely. Keep both numbers; the difference is what prompt ordering is worth.

Step 4 — size the KV cache in gigabytes

Get layers, KV heads and head dimension from your model’s metadata and compute bytes per token. Multiply by the context length you were about to configure. If the answer surprises you, that is the point of the step.

Step 5 — build the worksheet

Prompt tokens, cached share, output tokens, requests per month, three prices. Produce a monthly figure, then compute what a 50% prompt reduction and a 60% cache hit rate each save. Rank them.

Step 6 — argue for one change

One paragraph naming the single lever you would pull first, with the number from step 5 that justifies it and the quality risk you are accepting.

What to hand in

Your two rates and their ratio; the TTFT share at your realistic context length; the cached and uncached prefill for the same prompt; the KV cache size for your configured window; and the paragraph from step 6.

31.12 Exercises

1. Your two rates. Measure prefill and decode tokens per second for a model you can run. Report the ratio and predict, before measuring, the total latency for a 4,000-token prompt with a 100-token answer. Then measure it.

2. Break the cache on purpose. Take a prompt with a long stable prefix and measure cold and warm prefill. Move a timestamp from the bottom of the prompt to the top and repeat. Report both numbers and the design rule you would write for your team.

3. Size a window. For three models you might deploy, compute KV cache bytes per token from their metadata and produce a table of memory against context length. Say which model you would run at 32k context on a 24 GB card.

4. The worksheet, honestly. Take a feature you have actually shipped or specified. Count real prompt and output tokens — count, do not estimate — and compute the monthly bill at your traffic. Then compute it again with the prompt 40% shorter, and say which parts of the prompt you would cut.

5. TTFT against context. Plot time to first token against prompt length for your setup, over at least five lengths. Fit a line. Use the slope to predict TTFT at a context length you have not tested, then test it. Because prefill’s attention part grows faster than prompt length, a linear fit underestimates TTFT at long context; report the departure from linearity.

6. The output-length myth. Find a place where someone proposed saving money by asking the model for shorter answers. Using the input/output ratio you measured, compute what that change would actually have saved.

31.13 Where to go deeper

Before reading: take one real request from your own product and write down its prompt-token and output-token counts. Almost everything below is easier to judge once you know your own ratio.

Beyond the papers

resource	what it is
vllm	the reference open-source server; the code is readable and the scheduler is where continuous batching becomes concrete
vLLM’s paged-attention design note	the KV cache as pages, explained by the people who implemented it (Kwon et al. 2023)
Transformer Inference Arithmetic	where the prefill/decode asymmetry comes from, derived rather than asserted
How To Scale Your Model	memory bandwidth and arithmetic intensity worked through carefully; the book-length version of Pope et al. (2023)
Databricks on inference performance	batching, TTFT and throughput measured on real hardware, which a laptop cannot show you
llama.cpp	the engine family underneath the numbers in this chapter; read its KV cache handling if you serve on CPU or Apple Silicon
tensorrt-llm	what the last 20% of throughput costs in build complexity, so you can decide whether you want it
llm-compressor	applying quantization to a model you intend to serve, with the evaluation loop attached

The primary sources

topic	source
the KV cache as paged memory	Kwon et al. (2023)
continuous batching, and why static batching wastes a card	Yu et al. (2022)
post-training quantization to 4 bits	Frantar et al. (2023)
activation-aware weight quantization	Lin et al. (2024)
speculative decoding	Leviathan et al. (2023)
inference arithmetic for large models	Pope et al. (2023)

31.14 What to remember

• A request is two operations, not one. Prefill reads the prompt in one parallel pass; decode writes the answer one serial token at a time. Measured here at roughly 2,000 against 190 tokens per second — writing a token takes about eleven times as long as reading one. Every serving decision follows from which half your requests are made of.

• Latency is usually the prompt. At 7,427 tokens of context, 91.0% of the wall clock was over before the first character appeared. Streaming does not change that number; it only makes the rest of the request visible. The lever for time-to-first-token is prompt length, and after that, caching.

• Long context is charged twice. It costs prefill once, and then slows every generated token: decode fell from 189 to 141 tokens per second between an empty window and a 7,400-token one.

• The KV cache can dwarf the model. At 112 KiB per token, a 131k window is 14 GiB of cache — 7.5× a 2.0 GB four-bit file, 2.3× the same weights at fp16, 4.0× the file with the eight-bit cache actually used here. Quote whichever matches your setup, but “will it fit?” is a question about context length either way. Grouped-query attention is why the number is 14 and not 42. Size a card as weights + cache × concurrent sequences + a margin, and for a mixture-of-experts model count the total parameters, not the active ones.

• Prompt caching is the first lever and it is not subtle. Warm prefill was flat at 12–16 ms regardless of prompt length: 153× on prefill and 9.0× on the whole request at 4,247 tokens. It requires a byte-identical prefix, so order the prompt by stability and put nothing volatile near the top.

• Batching buys throughput and spends latency. Decode is memory-bound, so sequences can share weight movement; continuous batching keeps the slots full instead of waiting for the slowest sequence (Yu et al. 2022; Kwon et al. 2023). Reporting the throughput gain without the latency cost is how a review gets misled.

• Most of what you pay for is reading. 890 prompt tokens against 53 generated in this book’s own summarization measurement — a token ratio, which becomes nearer 3:1 or 4:1 once output tokens are priced higher. Asking for shorter answers saves little; sending less saves a lot. Boilerplate removal is the one form of “send less” with no quality risk; switching models may save more, but it is a quality decision — as is dropping retrieved context or few-shot examples — and belongs in the eval.

• Order the levers by what they cost you. Removing genuinely redundant tokens and reusing a cached prefix carry no quality risk and go first; dropping context the model uses is a quality decision. A smaller model, quantization (Frantar et al. 2023; Lin et al. 2024) and speculative decoding (Leviathan et al. 2023) each trade something specific — quality, quality, and a hosted draft model for latency respectively — and belong in an eval, not in a benchmark table someone else ran.

Frantar, Elias, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. “GPTQ: Accurate Post-Training Quantization for Generative Pre-Trained Transformers.” International Conference on Learning Representations (ICLR).

Kwon, Woosuk, Zhuohan Li, Siyuan Zhuang, et al. 2023. “Efficient Memory Management for Large Language Model Serving with PagedAttention.” Proceedings of the 29th Symposium on Operating Systems Principles (SOSP).

Leviathan, Yaniv, Matan Kalman, and Yossi Matias. 2023. “Fast Inference from Transformers via Speculative Decoding.” Proceedings of the 40th International Conference on Machine Learning (ICML).

Lin, Ji, Jiaming Tang, Haotian Tang, et al. 2024. “AWQ: Activation-Aware Weight Quantization for LLM Compression and Acceleration.” Proceedings of Machine Learning and Systems (MLSys).

Pope, Reiner, Sholto Douglas, Aakanksha Chowdhery, et al. 2023. “Efficiently Scaling Transformer Inference.” Proceedings of Machine Learning and Systems (MLSys).

Yu, Gyeong-In, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. “Orca: A Distributed Serving System for Transformer-Based Generative Models.” 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI).