25  Context Engineering

Chapter status: drafted. The measurements run qwen3.5:9b through ollama 0.24.0 on invented facts embedded in real Turkish news articles, so a correct answer can only have come from the context. This chapter is the consequence of Chapter 23’s finding that the transcript is the agent’s state; Chapter 26 covers what persists between runs. One reproduction note if you rerun the long-context probe: ollama silently truncates a prompt longer than the served num_ctx (its context-length setting) and warns about nothing. Set it explicitly above your longest condition, and check the served length rather than assuming it. An 11k-token prompt quietly cut to 4k is no longer a long-context condition, and the cut may remove a needle as well, so neither a flat curve nor a drop would mean what it seems to. Software: NumPy 2.5.2 on Python 3.14.5.

Learning objectives. After this chapter you can:

25.1 A budget, not a container

The window is usually described as a capacity: 200k tokens, so you can fit 200k tokens. That framing is wrong in a way that matters, because it suggests the only question is whether things fit.

Three results from earlier chapters say otherwise, and together they make the case for treating the window as a budget where every token competes:

  • Chapter 20 measured the price of not choosing: eighty documents instead of the one that mattered cost 148× the input tokens for identical accuracy. The model was not confused by the extra material. You simply paid for it, on every request, forever.
  • Chapter 23 showed that in an agent the context is not something you compose once — the transcript is the state, it only grows, and every model call re-sends all of it.
  • Chapter 24 showed that a tool’s return value is therefore a write to that state, which is why a tool returning 50,000 tokens can take space its later steps need, and is carried into every later call.

So there are two questions, not one: “does it fit” — capacity is a real constraint — and, for everything that does, “what is each token earning”. The second is the one nobody asks.

That reframing has an immediate practical consequence: you cannot answer the second question without instrumentation. The first move in context engineering is not a technique, it is a budget by category — how many tokens went to the system prompt, to tool definitions, to retrieved documents, to observations, to history. Almost everyone who runs that count for the first time finds one category they did not know was there and would not have guessed.

The categories that surprise people, in rough order of frequency: tool definitions, which are re-sent on every call and scale with the number of tools; a system prompt that accumulated instructions nobody has read in months; and retrieved documents whose k was set once and never revisited. None of these are visible without counting.

The one thing that changes the arithmetic

One qualifier before any of that arithmetic gets applied. Providers cache the prefix of a prompt: an unchanged system prompt and tool block, re-sent on the next call, is billed at a fraction of its first-call price and skips most of the prefill (the pass that reads the prompt before the first output token) — Chapter 31 measured warm prefill flat at 12–16 ms whatever the prompt length. So “every token is re-sent on every call” is exactly true of the tokens and only partly true of the bill.

Why only a prefix? Because attention is causal: the keys and values stored for token i (the KV cache of Section 7.6) depend on every token before it. Change one token and everything after it has to be recomputed, while everything before it can be reused. The practical rules follow from that one fact (Section 31.5 has the mechanics). The prefix must be byte-identical, so a timestamp or a user name near the top of the system prompt quietly switches caching off for everything below it. By default a cache entry also expires after a few idle minutes, and some providers charge a premium to write it.

A worked example makes the size of the effect concrete. Take a 10-call agent run whose system prompt and tool definitions total 5,000 tokens, and use multipliers that are typical of 2026 price sheets (they vary by provider and model): a cache write costs 1.25× the normal input price, a cache read 0.1×. Uncached, the fixed part costs what 10 × 5,000 = 50,000 input tokens cost. Cached, it costs what 1.25 × 5,000 + 9 × 0.1 × 5,000 = 10,750 cost — about 4.7 times less, for exactly the same tokens.

Two consequences follow, and both make cutting tokens worth less than the raw count suggests. Fixed costs are cheaper than they look, so pruning the system prompt returns less than multiplying it by the step count suggests. And compaction invalidates the cache: rewriting the middle of a context invalidates the cached prefix from that point on, so the first call after you compact pays full price — plus any write premium — for everything after the rewritten point. Design the context to be append-only where you can — stable prefix, growing suffix, the task restated at the end — and compact rarely in large steps rather than often in small ones.

What we will look at. First whether long contexts really do degrade, where this chapter’s own measurement finds nothing and the published benchmarks find a collapse — and the reason for the disagreement is the most useful thing in the section. Then the four techniques: compaction, which is a lossy compression problem where you choose the loss in advance; isolation, which is usually the largest single lever; just-in-time retrieval instead of loading everything up front; and the filesystem as somewhere to put what the transcript cannot hold. And finally the case where all of it still fails, because the data was never text.

25.2 Does a long context actually degrade?

The budget argument above is about money. There is a second, louder claim — that long contexts degrade accuracy — and it deserves separating from the first, because the two lead to different engineering.

Chapter 20 tested it and found nothing: 640 queries, sixteen conditions, up to 16,775 tokens, 100% at every context size and every position. I ran a harder local version for this chapter — two invented figures to find and add rather than one to locate — and the curve was still essentially flat:

qwen3.5:9b, 60 tasks, two invented figures to find and sum

 distractor docs  accuracy   95% CI          prompt tokens
               0    100.0%   [94.0, 100.0]             142
               4     93.3%   [84.1,  97.4]             863
              12     88.3%   [77.8,  94.2]           2,319
              28     88.3%   [77.8,  94.2]           5,228
              60     96.7%   [88.6,  99.1]          11,048

Sixty distractors scored higher than twelve. That non-monotone shape is consistent with noise, but it does not establish it — a real effect of content or position need not be monotone either. The intervals overlap heavily — but overlapping intervals are not a test, so here is one. The largest gap, 0 against 12 distractors (60/60 versus 53/60), sits on the same 60 tasks, so the right instrument is the paired one: all seven disagreements run the same way, and exact McNemar gives a two-sided \(p = 2 \times 2^{-7} = 0.0156\). (Fisher’s exact test, which treats the two conditions as independent samples, gives 0.013 — close here, and the wrong model of the data.) Across four comparisons the Bonferroni line is 0.0125, so it fails either way and I will not claim it. The comparison the headline rests on, 0 against 60 distractors, gives p = 0.50. So: adding 78× the tokens made no difference this sample can detect — which is not evidence that the conditions are equivalent — and the dip in the middle sits right at the edge of what 60 tasks can resolve either way.

Why the easy test finds nothing

The name for the louder claim is context rot: a model’s accuracy decays as its input grows, even when everything still fits in the window. It would be comfortable to conclude that context rot is folklore. It is not. Hong et al. (2025), the report that popularised the name, ran 18 frontier models on deliberately simple tasks at lengths inside their advertised windows, and found performance degrading as the input grew in every one of its experiments. My two experiments missed it for two reasons rather than one.

The simpler reason first: 11,048 tokens is about a third of the length at which the published benchmarks report their headline drops, so this probe never entered the regime they measure. The second reason is the interesting one, because it is the same reason most in-house evaluations miss the effect even when they do reach the length.

Both of my tests allowed literal matching. Both were needle-in-a-haystack (NIAH) tests: one fact — the needle — is hidden in a long filler text — the haystack — and the model is asked for it. The question contained ZRG-14; so did the needle. Attention is extremely good at finding a rare string that is already in the query. Modarressi et al. (2025) was built exactly to remove that crutch — its needles share minimal lexical overlap with the question, so the model has to infer the association rather than match it — and the result is stark: across 13 models that all claim 128K contexts, 11 fall below half their short-context baseline at 32K tokens. Even GPT-4o, one of the two exceptions that stayed above half, goes from 99.3% to 69.7%.

Hsieh et al. (2024) reaches the same place from a different direction, and states the trap plainly: models show near-perfect scores on vanilla needle-in-a-haystack, yet on a benchmark that extends the test beyond search — to multi-hop tracing and aggregation — almost all of them show large drops as the context grows. Across 17 models claiming 32K or more, only half maintain satisfactory performance at 32K.

So the honest synthesis, which is more useful than either result alone:

Three tasks, three different answers about the same window
what you measure what you will find
Locating a rare string that appears in the query little or no degradation at the lengths tested here; this is what my tests and vanilla NIAH measure
Locating information by meaning, not by string match progressive degradation, largest where it was measured: 11 of 13 models below half their short-context baseline by 32K, not starting there (Modarressi et al. 2025)
Tracing across several facts, or aggregating them large degradation, earlier (Hsieh et al. 2024)

A model’s advertised context length is a capacity, not a competence — its effective context length is an empirical quantity and usually much shorter.

Effective context length — the length at which a model still performs, as opposed to the length it accepts. Routinely far shorter than the advertised window. ↩︎

Why this might happen, mechanically. Two mechanisms are usually offered, and they are plausible rather than established — nothing below is measured in this chapter. Attention weights are a softmax over every key in the context, so adding keys gives the mass somewhere else to go — dilution. Note this is a tendency and not an identity: the scores are not fixed while the denominator grows, and a model that learns to score the relevant token higher in a longer context can hold or increase its share. A literal string match plausibly holds up better because it produces one large score where a semantic association produces several medium ones — but even a large score is diluted: with a target logit \(a\) and \(N\) equal distractor logits \(b\), the target’s weight is \(e^{a}/(e^{a} + N e^{b})\), which still falls as \(N\) grows for any finite gap. Treat this as a possible mechanism, unmeasured here. Separately, many advertised windows are only partly trained windows. A model pretrained at 8K is typically extended to 128K by rescaling its RoPE positions (Position Interpolation, YaRN — Section 7.4) plus a comparatively short continued training on long inputs. It works, but the model has seen far fewer examples of long distances than of short ones, and quality tends to fall with distance. That is the likely machinery behind capacity, not competence.

One published result points the other way and is worth holding onto. Shi et al. (2023) found that irrelevant sentences added to arithmetic word problems dropped accuracy substantially, with no length increase to blame. My distractors cost nothing and theirs cost a great deal — the difference is the task. Distractors are cheap when the job is to find a literal string and expensive when the job is to reason, because reasoning has to decide what is relevant and literal matching does not.

And the widely-run test that would reassure you — needle-in-a-haystack — is precisely the test that its built-in lexical overlap makes easy. If you take one operational thing from this section, make it this: evaluate long context on your own task with your own paraphrases, never on literal matches, because the literal-match version will pass and tell you nothing.

Which is Chapter 20’s lesson restated in a place where it costs real money: a test that everything passes is not a test. My local experiment is small — one 9B model, 60 tasks, up to 11k tokens — and it agrees with the published result about the easy case, which is the most it can establish. The published benchmarks are the evidence for the hard case, and they were run at a scale no laptop reaches.

Two shapes, and which one you get depends on the task
LITERAL MATCH — qwen3.5:9b, THIS CHAPTER 100% 80% 60% 0 4 12 28 60 distractor documents (not to scale) FLAT: 100.0% AT ZERO, 96.7% AT SIXTY 60 tasks, up to 11,048 prompt tokens MEANING MATCH — GPT-4o, NoLiMa 100% 80% 60% 99.3% 69.7% short 32K context length EVEN A TOP MODEL: 99.3% → 69.7% AT 32K 11 of 13 fell below half their baseline; GPT-4o did not THE WINDOW IS NOT WHAT CHANGED BETWEEN THESE PANELS — THE TASK IS both are well inside the advertised context; only the right one requires inference

Two different models, so the curves are not a head-to-head comparison and the left one is far smaller. What transfers is the shape. When the answer shares a rare string with the question you get the left panel, at least at these lengths; when it does not, you get the right one. The easy test is the one almost everyone runs.

25.3 Compaction: what to keep, summarise, and drop

Chapter 23 established that the transcript only grows. Compaction is what you do about that, and it is a lossy compression problem with an unusual property: you choose what to lose, and you choose it before you know what will be needed.

Compaction — summarising or dropping parts of an agent’s transcript so the context stops growing. Lossy, and the loss is chosen before you know what will be needed. ↩︎

The useful move is to stop treating the transcript as one thing. It has parts with very different value per token:

Not all tokens are worth the same
part of the transcript keep it how why
The task — what the user actually asked verbatim, always, and re-state it near the end it is short, it is the thing being optimised against, and it is what gets lost when history dominates
Decisions the agent made and their reasons summarised what was decided is needed; the deliberation that produced it usually is not
Tool results that are still live verbatim these are facts the answer depends on. Summarising a number is how you lose it
Tool results already superseded or consumed drop, or reduce to one line the ten search results you rejected are pure cost from the moment you rejected them
Errors already recovered from drop the traceback, keep one line of what it taught Chapter 23’s point: a stale error keeps conditioning the model toward the confusion it caused — but “the date field wants ISO-8601” is the sentence that stops it happening again, and it costs eight tokens

Three failure modes are worth knowing before you build this.

Summarising numbers. A summariser will happily turn the budget is 4,318 lira into the budget was discussed. Any value the final answer must contain should be exempt from compaction by rule, not by hope — the safest implementation keeps a structured facts list beside the prose and never summarises it.

Compacting the task away. As history grows, the original request drifts toward the middle of a long context. Liu et al. (2024) measured what that costs: accuracy on a retrieval task is highest when the relevant material sits at the beginning or the end of the input and degrades markedly in the middle. My own needle test did not reproduce this — 100% at every position — for the same reason it found nothing else: a literal string match is immune to where it sits. Re-stating the objective at the end of the context is nearly free — the task is short, and appending it leaves the cached prefix intact — and it largely stops the task getting lost in the middle. It does not fix evidence that sits there; the remedy for that is the selecting and cutting the rest of this chapter is about.

Compaction that is not idempotent. Summarising a summary loses more each time, and long-running agents do this repeatedly. Compact from the original where you can, or accept and measure the drift.

Rolling history, concretely

The pattern that works in practice is a three-tier window:

  [ system prompt + tools ]     fixed, always present
  [ compacted history      ]    everything older than the last N steps,
                                summarised once, with a facts list kept verbatim
  [ last N steps           ]    verbatim, because recent detail is what the
                                next decision is actually about
  [ the task, restated     ]    last, so it is never in the middle

Choosing N is an empirical question with a cheap answer: run your eval at N = 3, 5 and 10 and take the smallest that does not lose accuracy.

25.4 Isolation: give the specialist a clean window

Compaction reduces a context you have already polluted. Context isolation avoids polluting it.

Context isolation — running a read-heavy sub-task in its own window and returning only its conclusion, so the parent’s context never carries what it read. The reading is still paid for, once, by the sub-agent. ↩︎

The idea is simple and it is the single largest lever in this chapter: a sub-task that needs to read a great deal — search fifty documents, scan a log, try six queries — runs in its own context, and returns only its conclusion. The parent never sees the fifty documents. It sees one paragraph.

  without isolation           with isolation
  ─────────────────           ──────────────
  parent reads 50 docs        sub-agent reads 50 docs in a fresh window
  parent context: 40k         sub-agent returns 200 tokens
  and stays 40k forever       parent context: +200

The saving is not the 40k. On the parent’s side it is roughly \((40{,}000 - 200) \times M\) over the \(M\) model calls left in its loop. That is a gross figure. The sub-agent’s own 40k of reading, its fixed prompt and output, and any cache pricing still have to be subtracted, so treat the sketch as illustrative rather than as a bill. How large \(M\) is depends on something Chapter 23 measured rather than assumed. When the steps are independent the model batches its tool calls: five lookups collapsed into 2.0 model calls there, and the growth was nowhere near quadratic. When the steps are genuinely sequential — step n needs step n−1’s result — the multiplier is the step count and the isolation saving is at its largest. Isolation pays most exactly where batching cannot help.

What it costs is equally real and usually understated:

  • The parent cannot see what the sub-agent saw. If the returned paragraph is wrong or incomplete, the parent has no way to notice — it has lost the evidence. Return a short result and a pointer to where the full material lives.
  • Coordination is not free. Every isolated call is at least one extra round trip, and the specialist has to be given enough context to do its job, which is a prompt you now maintain.
  • Errors compound quietly. A summary that is 90% right, consumed by a parent that treats it as fact, is a failure mode with no error message. Chapter 28 is where this gets its own chapter.

The heuristic worth applying: isolate when the sub-task’s inputs are large and its output is small. Searching, scanning, filtering, verifying. Do not isolate when the parent will need the detail — you will only pay the round trip and then ask for it again.

25.5 Just-in-time retrieval beats eager loading

The instinct when building an agent is to load what it might need: the schema, the docs, the recent tickets, the user’s history. It is one fewer failure mode, and it feels like preparation.

It is also paying for everything on every call. The alternative — give the agent the ability to fetch and let it fetch what it needs — is almost always cheaper, and Chapter 20’s measurement is the reason to believe it: eighty documents instead of one cost 148× the input for identical accuracy. Eager loading is that trade, made silently, on every request.

Two things make just-in-time work, and both are Chapter 24’s subject:

  • The agent has to know what is available. A tool named find_orders with a description saying when it applies is what makes fetching-on-demand possible at all. This is why the two chapters are adjacent.
  • Identifiers beat payloads. Pass around order IDs, file paths and URLs; fetch the contents when needed. A pointer costs ten tokens and a document costs two thousand, and most pointers are never dereferenced.

What it costs is latency: a round trip you would not have paid, on the steps that need the data. That is a real trade and it has an obvious middle ground — eagerly load the one or two things needed on nearly every request, fetch everything else on demand. What makes it a decision rather than a habit is measuring the hit rate: if you eagerly load something that is used 15% of the time, you are paying for it 85% of the time for nothing.

25.6 The filesystem as external memory

There is a move that sounds like a workaround and is closer to the right architecture: let the agent write things down and read them back.

A scratchpad file, a notes directory, a structured JSON of findings so far. The context holds a reference — findings.md, 340 lines — and the agent reads the part it needs when it needs it.

This is better than a longer window for reasons that have nothing to do with capacity:

  • It survives compaction. Anything written to a file is still there after the transcript is summarised, which makes it the natural home for exactly the facts that compaction destroys.
  • It is addressable. The agent can re-read one section rather than re-reading everything, which a linear transcript cannot offer.
  • It is inspectable by you. A file is a debugging artefact and a transcript is a haystack. When a long run goes wrong, the notes file is usually where you find out why.
  • It survives the run. Which is where this chapter hands over to Chapter 26, whose subject is what should persist between runs rather than within one.

The discipline that makes it work: write structured, read selectively. A notes file that becomes free-form prose is a second transcript with the same problem. Give it headings, or make it JSON, so a later step can read one field rather than the whole file.

25.7 When text retrieval is not enough

One caveat on everything above, because it is where a well-built context pipeline still fails.

Chapter 20’s machinery — chunk, embed, retrieve — assumes the information is text with local meaning: a passage that means roughly the same thing on its own as it did in the document. Three common cases break that assumption outright, and no amount of chunking repairs them:

  • Tables and schemas. A row is meaningless without its header, and a column name is meaningless without the table’s purpose. Chapter 20 measured what chunking does to structure; the fix here is not better chunking but serialising each row into a self-contained sentence, or not retrieving at all and letting the agent query the table.
  • Code. A function’s meaning depends on its callers, its types and the module’s conventions — all of which live elsewhere in the repository. Retrieving the function body alone retrieves the least useful part.
  • Anything relational. Which customers churned after the March incident is a join, not a search. Embedding rows and hoping is the expensive way to get a worse answer than a query would have given.

The rule: when the data has structure, give the agent a way to query the structure. For the query shapes above — aggregates, joins, follow the callers of this function — a run_sql tool over a real schema beats embedding rows, and a grep-and-read pair beats embedding a codebase. It is a claim about those query shapes, not a general ranking: a question like where do we handle retries has no exact string to grep for, and that is what embedding search is good at. Most real systems want both. That is a context-engineering decision, because it determines what ends up in the window — and it is usually the difference between an agent that works on your data and one that works on a demo.

25.8 Intuition checks

Answer each one before you open it. Two rest on measurements — one published, one run here — and the other four on judgements this chapter argues for and tells you how to check on your own system.

No, for two separate reasons that are easy to conflate. The cost reason is close to unconditional: every token is re-sent on every model call, so a context you did not curate is a bill you pay forever — Chapter 20 measured 148× the input for identical accuracy. Prompt caching discounts the part of the prefix that does not change, which is why the part that does change is where the money is. The capability reason is conditional but real: a window is a capacity, not a competence, and Hsieh et al. (2024) found that of 17 models claiming 32K or more, only half held up at 32K.

Almost nothing about your task. NIAH lets the model match a rare string that is already in the question, which is the easiest thing attention does — Chapter 20’s run returned a flat 100% at every size and position, and this chapter’s harder local version stayed between 88.3% and 100% with no trend in context length. Modarressi et al. (2025) removed exactly that crutch and 11 of 13 models fell below half their short-context baseline at 32K; even GPT-4o, one of the exceptions, went from 99.3% to 69.7%. Test with paraphrases, not literal matches.

Count, by category — system prompt, tool definitions, retrieved documents, tool results, history. Not because counting is virtuous but because the largest category is routinely not the one people assume. The fixed costs (system prompt, tool definitions) are multiplied by every step in the loop, so their token count is larger than it looks — but a cached prefix bills those repeats at a discount, so rank categories by measured cost, not by tokens alone.

When its inputs are large and its output is small: searching, scanning, filtering, verifying. The saving is not the tokens the sub-task read, it is those tokens multiplied by every remaining model call in the parent’s loop — minus what the sub-agent itself costs. Do not isolate when the parent will need the detail — you pay a round trip and then ask for it again. And return a pointer alongside the summary, because the parent has otherwise lost the evidence and cannot notice a wrong conclusion.

Numbers, identifiers, and anything the final answer has to contain. A summariser will turn the budget is 4,318 lira into the budget was discussed without any signal that it did. Keep a structured facts list beside the prose and exempt it from compaction by rule. Also worth protecting: the original task, restated at the end of the context rather than left to drift into the middle.

Often because the real data has structure and the pipeline treats it as text. A table row means nothing without its header; a function means nothing without its callers; which customers churned after the March incident is a join rather than a search. Chunking does not repair any of those. Give the agent a way to query the structure — run_sql over a real schema, grep plus read over a repository — instead of embedding rows and hoping.

25.9 Interview questions

On sourcing — the same caveat as Chapter 2. These recur in the two credible open sources this book uses, plus the ones this chapter’s own measurements make answerable.

Testing whether you understand the mechanism

Q: Why does context engineering matter if windows keep growing?

Separate the two arguments, because they have different strengths. Cost is close to unconditional: the transcript is re-sent on every model call, so an uncurated context is multiplied by the step count — prompt caching discounts an unchanged prefix but not a rewritten middle, which is why compaction is not free — and Chapter 20 measured 148× the input tokens for identical accuracy when you skip the choosing. Capability is conditional but documented: a claimed window is not an effective one, and Hsieh et al. (2024) found half of 17 models failing to hold up at 32K, a length all of them claim to support.

Q: Someone shows you a perfect needle-in-a-haystack result. What do you say?

That it is the easy version, and say why: the needle shares a rare string with the question, so the model matches rather than infers. Cite the contrast if you can — Modarressi et al. (2025) removed the lexical overlap and 11 of 13 models dropped below half their short-context baseline at 32K. Then give the practical form: evaluate with paraphrases and with multi-fact aggregation, because those are the shapes real tasks have.

Q: What actually grows in an agent’s context, and why does it cost more than it looks?

The transcript, monotonically — every decision and every observation is appended and nothing is removed. It costs more than it looks because each model call re-sends all of it, so the total is the sum of a growing series rather than the final length — roughly quadratic in the number of sequential steps, as Chapter 23 showed. That is why fixed costs like tool definitions matter disproportionately: they are paid once per call, not once per run.

Testing whether you have shipped one

Q: An agent costs three times its budget. Walk me through fixing it.

Measure by category first, and say why: the largest category is usually not the assumed one. Then cut in order of measured cost, counting cache hits and token prices rather than raw tokens. The usual candidates: prune tool definitions and the system prompt, because they are multiplied by every step; cut what tools return to the fields actually used; isolate the read-heavy sub-task into its own context; then compact history with the facts kept verbatim. Re-measure accuracy at every stage with an interval, because each of those can cost points quietly.

Q: How do you decide between eager loading and fetching on demand?

On tokens the answer is usually “on demand”, and the first-order arithmetic is one line: eager loading costs T tokens on every request; just-in-time costs the fetch tool’s definition c on every request, plus T on the fraction h of requests that need it. So fetching wins whenever h < 1 − c/T. With a tool definition of about a hundred tokens against a two-thousand-token document, that threshold is 0.95 — only something needed on nearly every request is worth loading eagerly. The line leaves out one real cost: each fetch is an extra model call, which re-sends the context so far (cheaply, if it is cached). Say that, say what just-in-time costs in time — a round trip of latency on the steps that need the data — and say how to get most of both: pass identifiers rather than payloads, and dereference them only when a step actually needs the contents.

25.10 The 2026 toolbox

The local experiment in this chapter ran on one laptop with the versions below. The long-context results it leans on came from published benchmarks run at a scale no laptop reaches.

The stack these numbers came from
library version used what it is for here
ollama 0.24.0 serving qwen3.5:9b over contexts up to 11k tokens
numpy 2.5.2 task generation, scoring and the confidence intervals

Worth knowing about, not used above:

The wider ecosystem, versions as of writing
library current version when you would reach for it
langgraph 1.2.11 state as an explicit object, which makes compaction a design decision rather than an accident
openai-agents 0.22.0 sub-agent isolation as a first-class construct

tiktoken, RULER and NoLiMa are the other three things you will want. They are in Section 25.13 with links rather than versions.

25.11 Lab: cut an agent’s token usage without losing accuracy

The order of these steps matters — measure, then cut, then re-measure. Doing it in any other order produces savings you cannot attribute. Cut first the category that contributes most to measured cost and can be reduced safely. That is not always the one with the most tokens: the cache-hit rate and the token prices decide what each token costs. Steps 2–5 list the usual candidates; if your measurement ranks them differently, follow the measurement.

Step 1 — build the budget by category

Instrument one real run. Count tokens for: system prompt, tool definitions, retrieved documents, tool results, conversation history. Next to each count, record its cost, with cached and uncached input tokens priced separately. Print the table. Do not skip to the techniques; the point of this step is that you will be wrong about which category is largest.

Step 2 — fix the fixed costs, if they dominate the bill

System prompt and tool definitions are re-sent on every call, so their tokens are multiplied by the step count. The bill is a different matter: a cached prefix is billed at a fraction of its first-call price (Section 25.1.1), so check the cache-hit rate and the token prices before putting them first. When they do dominate measured cost, this is usually the cheapest large saving: prune tools nobody uses, and delete the instructions that accumulated.

Step 3 — cut what tools return

Per Chapter 24, return the fields the task needs rather than whole records. Re-measure. Also check the shape: a 40-row result the agent only reads two rows of should be paginated.

Step 4 — isolate one read-heavy sub-task

Pick the step with large input and small output. Run it in its own context and return only the conclusion. This is normally the single biggest number in the lab.

Step 5 — compact history, keeping facts verbatim

Summarise everything older than the last N steps, and keep a structured facts list untouched. Sweep N over 3, 5 and 10.

Step 6 — re-measure accuracy, not just tokens

Every step above can silently cost accuracy. Run your eval — sized per Chapter 22, because a 30-item set cannot detect the regression you care about — and report tokens and score together. A 60% token reduction that costs four points may or may not be worth it, but you cannot have that conversation without both numbers.

What to hand in

The category table before and after, the token reduction per step, the accuracy at each stage with an interval, and one sentence on which step you would keep if you could only keep one.

25.12 Exercises

  1. The budget table. Instrument one real agent run and produce tokens by category. Write down, before you look, which category you expect to be largest. Most people are wrong.
  2. Multiply by steps. Take your system prompt and tool definitions, and multiply their token count by the average number of model calls per run. That number is what pruning them is worth in tokens; if the prefix is cached, the saving on the bill is smaller.
  3. Paraphrase your NIAH. If you have a long-context evaluation, rewrite the questions so they share no distinctive words with the answers. Re-run. Report both curves.
  4. Isolate one step. Find your most read-heavy sub-task, run it in a separate context, and measure the parent’s token count before and after.
  5. Break compaction on purpose. Compact a transcript containing a number the answer needs, and check whether the number survived. Then add the structured facts list and check again.
  6. Hit-rate audit. For each thing you load eagerly, measure how often it is actually used. Anything below about half is a candidate for just-in-time. (The token arithmetic in the interview questions would allow a much higher threshold; the extra round trip and its latency are why half is a sensible place to start.)
  7. Notes file. Give an agent a scratchpad it must write findings to, and a tool to read sections back. Compare token usage and accuracy against the same agent without one, on a task long enough to need compaction.

25.13 Where to go deeper

Before reading: instrument one run and produce the token table by category. It takes an hour and it decides which of the sections above you actually need.

Beyond the papers
resource what it is
Effective context engineering for AI agents — Anthropic the most complete practitioner treatment of this chapter’s subject, including compaction and sub-agent isolation
Building effective agents — Anthropic the surrounding architecture, and when not to add machinery
RULER the benchmark that separates claimed from effective context length; runnable on your own model
NoLiMa the needle set built without lexical overlap — the evaluation to copy rather than the vanilla one
tiktoken counting tokens honestly, which is step one of everything here. It implements OpenAI’s tokenisers only; for other models use the model’s own tokeniser or the token counts the API reports
langgraph state as an explicit object rather than an accumulating list, which makes compaction a design decision
The primary sources
topic source
claimed context length versus effective Hsieh et al. (2024)
what the needle test hides Modarressi et al. (2025)
position effects in long contexts Liu et al. (2024)
irrelevant context as active harm Shi et al. (2023)
managing a fixed window like an operating system’s memory, paging context in and out Packer et al. (2023)
retrieval as the alternative to a longer prompt Lewis et al. (2020)
the report that popularised the name context rot, and measured it Hong et al. (2025)

25.14 What to remember

  • The window is a budget, not a container. Check both: whether the input fits, and whether each part earns its cost. Every token is re-sent on every model call, so an uncurated context is multiplied by the number of steps.
  • Count before you cut. Tokens by category — system prompt, tool definitions, retrieved documents, tool results, history. The largest category is routinely not the assumed one. Fixed costs are paid per call rather than per run, but a cached prefix is billed at a discount, so rank categories by measured cost.
  • A claimed context length is a capacity, not a competence. Of 17 models claiming 32K or more, only half held up at 32K (Hsieh et al. 2024).
  • Needle-in-a-haystack is the easy test, and it will pass. Chapter 20’s retrieval run scored 100% at every position and size, and this chapter’s 60-task version stayed between 88.3% and 100% with no monotone decline — both with a needle that shared a rare string with the question. Remove that overlap and 11 of 13 models fall below half their short-context baseline at 32K; even GPT-4o, one of the exceptions, goes from 99.3% to 69.7% (Modarressi et al. 2025). Evaluate with paraphrases.
  • Isolation is the largest lever when the shape is right. A read-heavy sub-task in its own context returns a paragraph instead of forty thousand tokens, multiplied by every remaining call in the parent’s loop. Chapter 28 modelled where it inverts: six steps producing 200-token results cost 1.20× as three agents, because the per-agent fixed cost is paid three extra times; twenty-four steps producing 3,000-token results saved two thirds. Large inputs, small outputs, many remaining steps — all three, or it is not a saving.
  • Compaction is lossy, and you choose the loss in advance. Keep the task and live tool results verbatim, summarise decisions, drop consumed results and resolved tracebacks — keeping a one-line lesson where it stops the same error recurring. Preserve exact values the answer requires.
  • Prefer identifiers to payloads. A pointer costs ten tokens and a document costs two thousand, and most pointers are never dereferenced. Load eagerly only what is used on nearly every request.
  • When the data has structure, query the structure. Aggregates, joins and exact lookups over tables are what run_sql over a real schema is for, and chunk-and-embed handles them badly; semantic search over code or prose is a different task, and most systems end up using the two side by side.