20  Retrieval-Augmented Generation

Chapter status: drafted. The measurements use 3,397 real Turkish news articles — the XL-Sum test split Chapter 14 worked on — because chunking is only interesting on documents that have internal structure. Section 20.1’s needle test uses invented facts, so a correct answer can only have come from the context and there is nothing to grade subjectively. Retrieval mechanics belong to Chapter 19 and are cited rather than repeated. Software: ollama 0.24.0 serving qwen3.5:9b, snowballstemmer 3.1.1, NumPy 2.5.2 on Python 3.14.5.

Learning objectives. After this chapter you can:

• say why retrieval beats a longer prompt, and which of the usual reasons survives measurement (Lewis et al. 2020; Liu et al. 2024);
• design the canonical pipeline and name what each stage is responsible for;
• choose a chunk size deliberately, knowing what it trades;
• attribute a wrong answer to the stage that caused it, before tuning anything.

20.1 Why not just paste everything into the prompt?

First, what the name means. Retrieval-augmented generation (RAG) is a simple recipe: when a question arrives, search your own documents for the few passages most likely to contain the answer, paste those passages into the prompt, and ask the model to answer from them. The search is Chapter 19’s subject; this chapter is about everything around it. The original formulation (Lewis et al. 2020) trained the retriever and the generator together; the version almost everyone builds today keeps them separate, and that is the version here.

RAG exists because models are frozen and your knowledge is not. That is the permanent reason. But context windows kept growing, and every time they did, somebody argued the problem had gone away: if the whole corpus fits, why retrieve?

There are three answers everyone gives — cost, because you pay for every input token on every request; access control, because a prompt cannot enforce who may see which document; and recency, because a cached prompt goes stale. All three are true, with one qualification worth making now: the second and third are things retrieval lets you build, not properties the label confers. A retrieval layer gives you a place to filter by permission and a place to re-index. But an index that does not filter out documents a user may not read still leaks them, just through one more step, and an index nobody refreshes is as stale as the prompt it replaced. The fourth answer, the one repeated most confidently, is that a model does not use a long context evenly.

That fourth claim is the only one of the four that is a factual assertion about model behaviour rather than about your infrastructure, so it is the one worth measuring. I measured it and did not reproduce it.

The needle test

The design has to leave no room for a generous grader, so it is deliberately crude. Take one invented fact — the approved 2025 budget for project ZRG-14 is 743,912 lira — that no model can know or guess. Put it in a context alongside genuine Turkish news articles as distractors. Ask for the number, and score by exact match, so there is no grading ambiguity. Then vary two things: how many documents are in the context, and where the fact sits. There is no retrieval step in this test — the documents go straight into the context — so the retrieval failures counter in the last line does not measure retrieval here.

qwen3.5:9b, 40 invented facts per cell, temperature 0, exact-match scoring

 documents  position  accuracy  prompt tokens
         1     start    100.0%            113
         5     start    100.0%            951
         5    middle    100.0%            953
         5       end    100.0%            949
        10     start    100.0%          2,005
        10    middle    100.0%          2,007
        10       end    100.0%          1,987
        20     start    100.0%          4,106
        20    middle    100.0%          4,115
        20       end    100.0%          4,112
        40     start    100.0%          8,318
        40    middle    100.0%          8,313
        40       end    100.0%          8,316
        80     start    100.0%         16,744
        80    middle    100.0%         16,720
        80       end    100.0%         16,775

16 conditions, 640 queries, 0 retrieval failures, 0 errors

Nothing. Not a gradient, not a dip in the middle, not a wobble at 80 documents and 16,775 tokens. Six hundred and forty queries and the model found every needle in every position.

What a null result is worth

The temptation is to conclude that lost-in-the-middle is a myth. That conclusion is not available from this table, and understanding why is more useful than the table itself.

A test that everything passes is not a test. What I built is the easiest possible version of the task: one verbatim string, a distinctive key (ZRG-14), a numeric answer, and a question that repeats the key exactly. Attention is extremely good at that.

This is not just my intuition about my own test; the controlled version has been run. Modarressi et al. (2025) removed the literal lexical overlap between the question and the needle and measured what happens: of thirteen models claiming 128K support, eleven fell below half their short-context baseline at 32K, and GPT-4o dropped from 99.3% to 69.7%. A needle you can match on a token is a different task from a needle you have to infer, and ZRG-14 is about as matchable as a needle gets.

Be careful with Liu et al. (2024) too, in the other direction. That paper contains two experiments, and only one is multi-document question answering. The other is key–value retrieval over UUIDs — a pure string lookup with no semantic cue at all, which is structurally the test I built — and there the U-shape appeared for some models and not for others. So a flat result on one 9B model is not a contradiction of that paper; it sits inside the variance the paper itself reports. What the multi-document half adds is the harder condition, where the relevant passage must be identified on meaning rather than matched on a rare token, and that is where the effect is most robust.

Note that context length is not the difference between us: their multi-document setting tops out at 30 passages of at most 100 tokens each, about 4,400–5,200 tokens on average depending on the tokenizer, which is between a quarter and a third of my longest condition. The difference is the needle, not the haystack. Their measurements were also taken on a 2023 generation of models, and long-context handling is precisely the thing that has had the most engineering attention since.

One caveat the table cannot show. With 40 queries per cell, a perfect score has a 95% Wilson lower bound of 91.2% — so this design can rule out a drop the size of the one Liu et al. (2024) reported for GPT-3.5-Turbo in the multi-document setting (about twenty points, from 75.8% with the answer first to 53.8% with it in the middle of twenty documents), and cannot rule out an eight-point one. The claim is “no large positional effect”, not “no positional effect”.

So the honest statement of what I measured is narrow, and I would rather write the narrow true thing than the wide familiar one: on this model, at up to 16,775 tokens, with a lexically easy needle, no positional effect large enough for forty queries per cell to see.

If your retrieval task looks like that, on a similar-sized model at similar context lengths, position is probably not your first worry. If it does not, this table tells you nothing about your case — and the way to find out is to run the same experiment with your needles, which is an afternoon.

The argument that survives

Look at the token column instead of the accuracy column, because that is where the measurement actually lands.

  1 document      113 prompt tokens    100.0%
 80 documents  16,775 prompt tokens    100.0%

                148x the input, for identical accuracy

The cost of not retrieving is not that the model gets it wrong. It is that you pay 148 times over to be equally right. You pay it on every request that cannot be served from a cache, and as the context grows, attention’s quadratic share of the compute grows with it, so latency rises faster than the token count. (At this model size and around 16K tokens, most prefill compute is still in layers that scale linearly; the quadratic term dominates at longer contexts.)

Prompt caching is the honest rebuttal to that number and it deserves stating plainly. A stable corpus prefix can be cached: cached input is billed at a fraction of the fresh price — a tenth is common in 2026, though the discount varies by provider and writing the cache can cost extra — and the prefill is skipped. At a tenth, that turns 148× into something nearer 15× — smaller, still not free, and available only while the prefix does not change, which is exactly the case retrieval exists for. Note also that the 113-token baseline assumes perfect retrieval of exactly one document; a real system passes \(k\) chunks and pays to build and query an index. The structural arguments below are the ones caching does not touch.

That reframes the whole chapter. Retrieval is not a workaround for a model that cannot read long inputs — this one reads them fine. Retrieval is how you stop paying for 79 documents you did not need. It is also the natural place to handle three things a single, shared, stuffed prompt handles badly: a corpus far larger than any context window (most real document collections are gigabytes, not thousands of tokens), knowledge that changes after training, and knowledge that a given user is not allowed to see.

What we will look at. First the pipeline, which is worth naming stage by stage only because each stage fails on its own and produces the same symptom, together with the prompt that turns retrieved chunks into a grounded answer with citations. Then chunking, which is the chapter’s real experiment: eleven ways of cutting the same corpus, queried two ways, where the best configuration under one kind of question is the worst under the other and the received default is a guess about questions its author never saw. Then the three failure modes and the afternoon of code that tells them apart — which is the thing you will reach for every week, and the thing almost nobody builds before they need it — and the standard metrics for scoring each half of the system.

20.2 The canonical pipeline

If retrieval is what stops you paying for 79 documents you did not need, the next question is what the machinery actually consists of. A typical RAG system has seven stages (some, such as reranking, are not present in every system). Naming them separately is what makes Section 20.4 possible, because each one can fail on its own.

The seven stages, and which of them can fail on its own

The upper row is built once; the lower row runs on every request. The three failure modes along the bottom — one at retrieval, two at generation — all produce the same symptom, a wrong answer, which is why Section 20.4 is about telling them apart before tuning anything.

Stages 3 to 6 — embedding, the index (exhaustive, or an approximate nearest-neighbour index, ANN, that trades a little recall for speed), top-k retrieval and cross-encoder reranking — were Chapter 19’s subject, and this chapter treats them as given. Two things about the diagram are worth saying out loud, because both are routinely skipped.

The pipeline is also the thing every RAG survey enumerates — Gao et al. (2023) is the most complete map of the variants, and worth skimming once you have built one yourself, so the taxonomy attaches to something. Note that the whole diagram assumes retrieval happens exactly once, before generation. Systems that decide whether and when to retrieve, and critique what came back — Self-RAG (Asai et al. 2024) is the reference design — are Chapter 29’s subject, and they only make sense once this fixed pipeline is working and measured.

Stage 1 is where most real projects actually lose. A PDF table becomes a column of disconnected numbers; a two-column layout interleaves; a scanned page produces nothing at all. No amount of embedding quality recovers text that was destroyed on the way in, and the failure is invisible downstream — the pipeline runs, the numbers look plausible, and the answer is wrong for a reason nobody is looking at. Read fifty parsed documents by hand before tuning anything else.

Stage 5 sets a ceiling on stage 7. Chapter 19 showed this and it is the single most useful fact in the chapter: if recall@k is 80% for the k you pass to the model, then 20% of questions cannot be answered from the retrieved text before generation starts, and no prompt engineering reaches them. (The model may still answer some of them from its own memory, but those answers are not grounded in anything you retrieved; and when the same fact appears in several documents, the bound is looser — Chapter 19 states the exact conditions.)

Which is why the two halves of a RAG system want different metrics and different people looking at them. The retrieval half is judged on not missing; the generation half on using what arrived and refusing when it did not. Section 20.4 is about telling those two apart, and it is harder than it sounds because both produce the same symptom: a wrong answer.

Stage 7: a grounded prompt, with citations

The generate stage is usually a single prompt, and its shape matters more than its wording. The goal is a grounded answer: one in which every claim can be traced to the retrieved text rather than to the model’s own memory. Three habits get you most of the way.

• Number the chunks and ask for citations. Give each chunk an ID and its source, and ask the model to mark each claim with the ID of the chunk that supports it. A citation turns “trust me” into something a reader, or a script, can check: does chunk [2] actually say that?
• Give the model a legal way out. State exactly what to write when the chunks do not contain the answer. Section 20.4 explains why a system without this option never abstains.
• Documents first, question last. Model providers commonly recommend putting long material at the top and the instruction and question at the end. Placing the highest-ranked chunks at the start of the block rather than in the middle is a free precaution against position effects, even though this chapter’s needle test did not detect one.

Answer the question using ONLY the numbered documents below.
After each sentence, cite the document that supports it, e.g. [2].
If the documents do not contain the answer, reply exactly:
"The documents do not answer this question."

[1] (source: budget-2025.pdf, page 14)
The approved 2025 budget for project ZRG-14 is 743,912 lira. ...

[2] (source: board-minutes-2025-03.docx)
...

Question: What is the approved 2025 budget for project ZRG-14?

A citation is itself a claim, and models do cite chunk [2] for a sentence that chunk 2 does not support. So check citations rather than display them: for each cited sentence, ask whether the cited chunk supports it — an exact match for numbers and names, a validated judge for paraphrase — and count the failures. That count is the faithfulness score defined at the end of Section 20.4.

20.3 Chunking is a modelling decision

Stage 2 is the one this chapter measures, because it is the stage where a number gets copied from a blog post and never revisited. A chunk is the unit you retrieve. That makes chunk size a decision about what is retrievable at all, not a preprocessing detail — and it moves two things in opposite directions at once:

• Smaller chunks are more precisely targeted. A 50-word chunk that matches the query is almost entirely about the query.
• Smaller chunks mean more of them, so the index is more crowded and each chunk carries less context to disambiguate itself.

Received wisdom picks a number — 512 tokens, 200 words, “about a paragraph” — and moves on. The number is worth measuring on your own corpus, because the trade above depends on how long your documents are and how localised the answers are within them.

The experiment

The retriever is held fixed so that only the cutting changes. 3,397 Turkish news articles, indexed eleven different ways with the same BM25 retriever, and queried under two kinds of question — because a single kind would have measured the wrong thing and produced a confident, wrong rule.

• A. article-level. The query is the article’s professional one-sentence summary. What is this document about?
• B. fact-level. The query is six content words drawn from one sentence of the article. Each word is then replaced by another inflected form of the same stem (as in seçim → seçimde), so the query does not repeat the text’s exact word forms. Where is this specific detail?

A hit means any chunk of the source article. Read that definition against each condition before reading the numbers. In condition A it is the right target — the question is the article. In condition B it is a proxy and a generous one: retrieving the correct article’s other paragraph counts as a success even though the sentence the query was drawn from is not in the returned chunk, so condition B’s numbers bound article-level retrieval rather than measuring fact-level evidence. Getting the stricter number needs supporting-span annotation — record which sentence each query came from and require the chunk to contain it — which this run does not have. Build times are not reported: an earlier run’s timings were contaminated by a concurrent job on the same machine, and a number I cannot stand behind does not belong in a table.

3,397 articles | A: 400 summary queries | B: 400 fact queries

chunking                    chunks  w/chk   A R@1   A R@5  A MRR    B R@1   B R@5  B MRR
whole article                3,397    366   71.8%   91.8%   80.2    96.0%   99.8%   97.7
fixed 50w, no overlap       24,266     51   57.0%   81.8%   67.1    96.5%   99.8%   97.8
fixed 100w, no overlap      12,390     99   66.5%   86.5%   75.2    95.5%   98.2%   96.7
fixed 200w, no overlap       6,459    187   70.0%   88.5%   78.3    94.0%   97.2%   95.4
fixed 400w, no overlap       4,074    297   72.5%   90.2%   80.1    93.8%   97.8%   95.6
fixed 100w, overlap 25      15,983     98   65.8%   87.0%   74.4    97.5%   99.8%   98.6
fixed 100w, overlap 50      21,890    101   66.0%   84.8%   73.7    97.2%   99.8%   98.5
fixed 200w, overlap 50       8,070    184   71.5%   88.0%   78.8    97.0%   99.8%   98.3
fixed 200w, overlap 100     10,166    192   70.5%   87.5%   77.8    96.2%   99.8%   98.0
sentence-aligned ~100w      14,500     86   66.0%   85.8%   74.3    97.8%  100.0%   98.8
sentence-aligned ~200w       7,674    162   69.5%   88.5%   77.9    97.2%  100.0%   98.5

What it says

Four things bound this table before any of it can be read, and all four are properties of the design rather than of chunking.

The sweep is narrower than eleven rows suggest. The mean article is 366 words, so a 400-word window leaves most articles intact — 4,074 chunks from 3,397 articles — and fixed 400w is close to a noisy copy of whole article. The range actually swept is 50 to 200 words.

Rank is not a fair axis when the units differ in size. At rank 1 the whole-article index hands the generator 366 words and the 50-word index hands it 51, so part of condition A’s ordering is simply more text wins. The comparison the table is missing is recall at an equal word budget — whole article at \(k{=}1\) against 200w at \(k{=}2\), 100w at \(k{=}4\), 50w at \(k{=}7\). Note the direction of that effect: condition B’s 50-word index already beats the 400-word one with a sixth of the text, so equalising the budget would strengthen that finding, not weaken it.

Recall@5 is not deduplicated by article. Five chunks from a 50-word index can be five slices of one document; five chunks from the whole-article index are necessarily five different documents. Fine-grained indexes are therefore penalised at rank 5 because several of their five slots can go to slices of the same article, and an unknown part of the ten-point R@5 gap in condition A is that artefact rather than chunking.

And \(n = 400\) per condition, so a 95% interval is about ±4.6 points in A and ±1.9 in B. Do not turn that into “smaller differences are indistinguishable”, which is the error this book keeps warning about: those are intervals on single proportions, and the systems here are compared on the same 400 queries. A paired test conditions on the queries the two systems disagree about and can resolve gaps far inside an unpaired interval. The queries are shared across all eleven systems, so a paired McNemar test could be computed from the existing data and is the right instrument; we report the unpaired numbers, which is the weaker choice, and the intervals above should be read as a caution against over-reading small gaps rather than as a test that settles them.

With those in hand: the best chunk size in one condition is the worst in the other. Chunks at or above article length lead condition A — 400-word chunks at 72.5% and whole articles at 71.8%, indistinguishable from each other — and fall to the bottom of condition B: 400-word chunks are last at 93.8%, whole articles eighth of eleven at 96.0%. Fifty-word chunks are last in A at 57.0% and mid-table in B at 96.5%, above both. The 15.5-point gap in A is far outside any interval above. The reversal in B is smaller — 2.7 points between 50-word and 400-word chunks, 4.0 between the best and the worst — and sits at the edge of what unpaired intervals can resolve, which is exactly where the paired test would earn its keep. The two columns are not noisy versions of one ranking: their rank correlation is negative (Spearman \(\rho \approx -0.62\)), so they come close to opposite orderings of the same eleven systems.

Ranked side by side, the two columns are hard to mistake for noise:

The same eleven chunkings, ranked twice — once per kind of question

Every line is one chunking, drawn between the place it finishes on article-level questions and the place it finishes on fact-level ones. The rose line is 400-word chunks going from first to last; the green one is sentence-aligned ~100-word chunks going from ninth to first. Read the numbers as well as the positions, though: condition B’s whole spread is four points against condition A’s fifteen, so the reordering on the right is real but much cheaper to get wrong — which is the same warning the margin note below makes about reading recall@1 when your generator consumes five chunks.

That is the whole lesson, and it is worth stating as a rule you can act on: chunk size is a bet on the granularity of the question.

Ask what a document is about and a chunk that holds only part of it can only match part of your query, so bigger wins. Ask where a specific fact is and a short chunk that contains it is almost entirely about it, so smaller wins — this is BM25’s length normalisation doing exactly what it was designed to do, seen from two sides.

Two more results in that table were not obvious to me before running it.

Overlap earns its cost, but only for facts. At 100 words, adding a 25-word overlap moved condition B from 95.5% to 97.5% — 18 errors in 400 down to 10, a net movement of eight queries. (How many queries changed in each direction is what the paired test above needs, and this run did not record it.) The same overlap moved condition A from 66.5% to 65.8%, three queries and well inside that condition’s noise band. A small difference is not the same as a demonstrated absence of one, so decide in advance how large a change would matter to you and read the result against that.

The pattern is a gain where a fact can straddle a chunk boundary, and nothing where the query describes the whole document. Overlap is therefore not a general-purpose safety margin. It is a specific repair for a specific failure, and if your questions are not fact-shaped you are paying for 29% more chunks for no return.

Structure beat arithmetic — with one confound named. Sentence-aligned chunks of about 100 words scored 97.8% in condition B against fixed 100-word windows’ 95.5%, and did it with fewer chunks than the overlapping variant that came second (14,500 against 15,983). One caveat before crediting all of that to structure: the sentence-aligned index averages 86 words per chunk against the fixed index’s 99, and condition B already rewards smaller chunks. Interpolating the fixed-size series, an 86-word fixed window would be expected somewhere near 95.8%, so roughly two of those 2.3 points look attributable to the boundary and the rest to size. A size-matched control — fixed 86w, no overlap — is the missing row in that table. What survives either way is that packing whole sentences costs nothing, beats the naive split, and matches the padded one (97.8% against 97.5% is a single query out of 400) with fewer chunks.

Note what the R@5 columns do. In condition B, every configuration is between 97.2% and 100% at rank 5, so if your generator reads five chunks, the B half of this table is close to a rounding error — for finding the right article, which is all these columns measure. Condition A is different: R@5 still spans ten points (81.8% to 91.8%), so for questions about a whole document the choice matters even at five chunks. Do not promote the flat B column into a claim about the system either: whether the chunk that arrives actually carries the answering sentence is a different question this run does not score. If your generator reads one chunk, the R@1 columns are your system. Chapter 19 made the same point about fusion, and it generalises: the metric that matters is the one your consumer actually reads.

Structure beats arithmetic

Whatever size you land on, chunk on structure where the document has any. Sections, articles, clauses, speaker turns, table rows. A boundary drawn at a fixed word count lands mid-sentence, mid-clause, mid-argument, and produces two chunks that are each locally plausible and jointly wrong — the failure Chapter 14 found in map-reduce, arriving here through a different door.

Overlap is the crude repair: repeat the last n words of each chunk at the start of the next, so a fact that straddles a boundary appears whole in at least one chunk — provided the fact is shorter than the overlap and does not straddle two boundaries. A 25-word overlap does not guarantee that a claim whose subject and qualifier are forty words apart lands whole in some chunk, though at some offsets a 100-word window will hold both. That is why it is a repair and not a guarantee.

It costs index size: a window of \(w\) words advancing by \(w - o\) words (overlap \(o\)) produces about \(w/(w-o)\) times as many chunks — 100/75 ≈ 1.33 for the 25-word overlap, 1.29 measured, since an article shorter than one window yields one chunk either way. The table above suggests that keeping the document’s own boundaries does the same job more cheaply: sentence-aligned chunks reached 97.8% while fixed windows padded with overlap reached 97.5% — the same, within one query — and the sentence-aligned index was 9% smaller. Paying for overlap to repair a boundary you did not have to draw is the expensive way round.

Two document types where fixed-size chunking simply fails:

• Tables. A row means nothing without its header. Chunk a table by rows and keep the header in every chunk, or serialise each row into a sentence before indexing.
• Anything with cross-references. As described in section 4 is meaningless once section 4 is a different chunk in a different part of the index. Either resolve the reference at ingestion or accept the loss and measure it.

So what should you do?

One boundary on all of this first: every number above is BM25. The mechanism behind condition B — short chunks winning because they are almost entirely about the query — is BM25’s length normalisation, which is a property of the scorer as much as of the cutting. A dense retriever has no length normalisation. It has a maximum sequence length that truncates, and a pooling step (often a mean over tokens) that squeezes a long chunk into one vector and dilutes its signal, and those produce a different optimum. The direction is probably similar; the numbers are not transferable, and the pipeline diagram at the top of this chapter shows an embedding stage, so a reader is entitled to assume otherwise unless we say this.

Measure it, on your corpus, with your retriever, with queries shaped like the ones you will get. But if you need a starting point before you have that:

• Chunk on structure — sentences, paragraphs, sections, clauses, rows — and pack to a target size rather than cutting at one.
• Target roughly 100–200 words if your questions are about specific facts, and larger, up to whole documents, if they are about what a document covers.
• Add overlap only for fact-shaped questions, and treat it as a boundary repair rather than a default.
• If your questions are genuinely both kinds, consider indexing at two granularities rather than compromising at one. Measure first: on this corpus the best compromise (200 words, 50 overlap) was only about a point behind the best configuration in each condition, which is within noise, but the gap is not guaranteed to be that small on yours.

20.4 Failure modes, and attributing them

A RAG system gives a wrong answer. There are three different causes and they have three different fixes, so the first job is not to tune anything — it is to find out which stage failed.

Three failures, one symptom

failure	what happened	how you detect it
Retrieval miss	the right chunk was never returned	check whether it is in the top-k at all; if not, nothing downstream matters
Grounding miss	the right chunk was returned and the model ignored it, or contradicted it	the chunk is in the context and the answer is not in the chunk
Synthesis miss	everything needed was there, spread across chunks, and the model combined it wrongly	all facts present, conclusion wrong

The diagnostic is mechanical and takes an afternoon to build:

for q, gold_doc, expected in evaluation_set:
    hits = retrieve(q, k=10)                   # the same k the generator reads
    answer = generate(q, hits)
    if answer == expected:
        continue                               # correct: nothing to attribute
    if gold_doc not in {h.doc_id for h in hits}:
        bucket["retrieval miss"] += 1          # fix retrieval
    elif not answer_supported_by(answer, hits):
        bucket["grounding miss"] += 1          # fix the prompt, or the model
    else:
        bucket["synthesis miss"] += 1          # fix decomposition, or the task
                                               # (or a bad source/label: inspect)

Read the three tests in order. Only wrong answers are attributed. If the gold document never arrived, nothing downstream could have saved it. If it arrived but the answer is not supported by the retrieved text, the model ignored or contradicted its evidence. If every claim is supported and the conclusion is still wrong, the facts were usually combined wrongly. But the last bucket also collects two other causes: the retrieved document itself may be wrong, stale or contradicted by another document (a corpus error), or the reference answer may be wrong (a label error). Their fixes differ from a synthesis fix, so inspect this bucket by hand; a supported but wrong answer drawn from a single chunk points at the source rather than at synthesis. In practice == becomes a normalised match or a judge, and answer_supported_by is the faithfulness check defined at the end of this section.

Do this before touching the embedding model. The default reflex when a RAG system disappoints is to swap the embeddings, and the measurement in Chapter 19 says that is often not the biggest lever — the choice of embedding model was worth 2.4 points on easy queries and 25.3 on hard ones, while the choice between lexical and dense was worth sixty to seventy, in whichever direction the query type dictated. And an embedding swap can only fix the first of the three buckets.

Two failures worth naming separately

Two of the ways a RAG system goes wrong sit awkwardly in that three-way table, because both are consequences of choices you made rather than stages that broke.

Distraction. Irrelevant retrieved context does not merely fail to help; it actively degrades answers (Shi et al. 2023). That reframes k as a real trade rather than a “more is safer” dial: raising it improves recall and imports noise, and past some point the second effect wins.

With the same caveat Section 20.1 earned, and it is worth being explicit rather than quietly citing past it: my own needle test put 79 irrelevant documents around the answer and cost nothing at all. Distraction did not appear there either. Shi et al. (2023)’s effect is measured on arithmetic and reasoning problems, where an irrelevant clause can be used — pulled into the chain of reasoning and acted on. Locating a string is not that kind of task, and neither is most extraction.

So treat k as a parameter with a measured optimum on tasks that involve reasoning over the retrieved text, and do not assume the penalty exists on tasks that do not: measure it before you budget for it, which is what Section 20.9’s fifth exercise asks you to do.

Refusal that never happens. A system with no legal way to say the documents do not answer this will answer anyway, from parametric memory, fluently. Chapter 17’s rule applies exactly here: name the failure output, make it easy to produce, and then check that the model actually uses it — on questions you know the corpus cannot answer. That last test is the one nobody runs, and it is the difference between a system that abstains and a system that has never been given the option.

Scoring the two halves

The three-way attribution works on single failures. To track a system over time you want numbers, and the evaluation frameworks — RAGAS (Es et al. 2024) and its relatives — offer a standard set that follows the same split. The names recur in every tool and job interview, so here they are in plain terms:

The four standard RAG metrics

metric	half	the question it asks	score
context precision	retrieval	are the retrieved chunks relevant, and are the relevant ones ranked near the top?	share of relevant chunks, weighted by rank
context recall	retrieval	does the retrieved text contain everything the reference answer needs?	share of the reference answer’s claims found in the context
faithfulness	generation	is every claim in the answer supported by the retrieved text?	supported claims ÷ all claims in the answer
answer relevance	generation	does the answer address the question that was asked?	a similarity score between the question and the answer

A worked example makes faithfulness concrete: an answer that makes four claims, three of which the retrieved chunks support, scores 3/4 = 0.75, even if the fourth claim happens to be true — faithfulness asks where a claim came from, not whether it is correct. The mapping to the attribution buckets is direct: low context recall is a retrieval miss, low faithfulness is a grounding miss, and a wrong answer with both high is a synthesis miss (or, as noted above, a source or label error).

Two cautions. First, a note on provenance: the RAGAS paper itself defines faithfulness, answer relevance and context relevance; context precision and context recall were added later in the library, and context recall needs a reference answer written by a person. Second, every one of these numbers is produced by an LLM judge that splits text into claims and checks each one, so its errors are judge errors. Chapter 14 found two automatic faithfulness measures that agreed on the ranking while flagging five summaries and eleven summaries respectively, and overlapping on exactly one. Use the metrics for the structure they impose, validate the judge on a sample you have labelled by hand (Section 22.4 shows how), and do not compare absolute scores across tools.

20.5 Intuition checks

Answer each one before you open it. Most of them are questions this chapter settled with a measurement rather than an argument; where it did not, the answer says so.

Note1. Context windows keep growing. Is RAG a temporary workaround?

No — and this chapter measured how large the cost is, which is the part usually asserted rather than shown: answering from 80 documents instead of the 1 that mattered cost 148× the input tokens for identical accuracy. The permanent reasons are different again: the model is frozen and your documents are not — a retrieval step is how new, private or changing information reaches a model that finished training before it existed, and most real corpora are far larger than any context window anyway. Access control is the other permanent reason: a prompt cannot enforce who may read which document, and an index with per-document permission filters can. Long context and retrieval are complements, not competitors — a bigger window changes how many chunks you can afford to pass, not whether you needed to choose them.

Note2. What chunk size should you use?

The honest answer is that it depends on the granularity of your questions, and this chapter measured how much: on the same corpus, 400-word chunks led on article-level questions at 72.5% recall@1 and came last on fact-level questions at 93.8%, while 50-word chunks did the reverse. If you need a default before you have measured, pack whole sentences to roughly 100–200 words. If your questions are genuinely of both kinds, consider indexing twice rather than compromising once — and measure whether the compromise actually costs you anything, because on this corpus it cost about a point.

Note3. Should you add overlap between chunks?

Only if your questions are fact-shaped. Measured here, a 25-word overlap on 100-word chunks moved fact-level recall@1 from 95.5% to 97.5% and moved article-level recall@1 down from 66.5% to 65.8% — while adding 29% more chunks to the index. It is a repair for facts that straddle a boundary, not a general safety margin. And chunking on sentence boundaries matched overlapping fixed windows (97.8% against 97.5%, one query apart) with a 9% smaller index.

Note4. Your RAG system gives a wrong answer. What do you do first?

Attribute it, before tuning anything. Check whether the supporting chunk was retrieved at all: if not, it is a retrieval miss and the prompt is irrelevant. If it was retrieved and the answer contradicts it, that is a grounding miss. If everything needed was present and the conclusion is still wrong, that is a synthesis miss. Three causes, three different fixes, one symptom — and the default reflex, swapping the embedding model, addresses only the first.

Note5. Is more retrieved context always safer?

No — but this is one of the answers the chapter did not measure for itself. Irrelevant context can actively degrade answers (Shi et al. 2023), so k is a real trade rather than a “more is safer” dial: raising it improves recall and imports noise, and past some point the second effect can win. That evidence comes from reasoning problems; this chapter’s own needle test put 79 irrelevant documents around the answer at no cost, so measure the penalty on your task before you assume it. There is a second reason to keep k honest — every chunk you pass is paid for on every request, so a k chosen by superstition is a permanent line item.

Note6. Your system never says “I don’t know”. Is that a prompting problem?

Usually it is a design problem: the system has no legal way to abstain, so it answers from parametric memory, fluently and without warning. Give it an explicit refusal output, make it cheap to produce, and then — the step nobody takes — test it on questions you know the corpus cannot answer. Until you have run that test you do not know whether you built a system that abstains or one that has simply never been given the option. Chapter 18 has the machinery for making the abstention a real, thresholded decision.

20.6 Interview questions

On sourcing — the same caveat as Chapter 2. These recur in the two credible open sources this book uses, plus the ones this chapter’s own measurements make answerable.

Testing whether you understand the mechanism

Q: Walk me through a RAG pipeline.

Name the stages and say what each one can break, because a list of seven nouns demonstrates nothing. Parse — where tables and scanned pages are destroyed, silently. Chunk — which decides what is retrievable at all. Embed and index — where an approximate index quietly caps recall. Retrieve, rerank, generate. Then give the relationship that matters: recall@k bounds everything downstream, so the pipeline has a ceiling set before the model is invoked.

Q: How do you choose a chunk size?

By measuring it against the questions you expect, and the good answer names why there is no universal number: chunk size trades targeting against context, and which side wins depends on whether a question is about a whole document or a specific fact. Cite the shape rather than a number — on one corpus 400-word chunks were best for article-level questions and worst for fact-level ones, on the same index and the same retriever. Then say what you would do before you had measured: pack whole sentences, target 100–200 words, add overlap only for fact-shaped questions.

Q: Why does adding more retrieved context sometimes make answers worse?

Because irrelevant context is not neutral — it can distract (Shi et al. 2023) — and because models do not always read a long context evenly (Liu et al. 2024). Both are reasons to treat k as a tuned parameter with a measured optimum rather than a safety margin. If you have run the needle test yourself, say what you found and under what conditions, because the effect’s size depends heavily on the model and the task.

Testing whether you have shipped one

Q: Your RAG system is wrong 30% of the time. How do you find out why?

Refuse to guess, and build the three-way attribution: retrieval miss, grounding miss, synthesis miss. Say that you would run it before changing the embedding model, because that is the reflex and it only addresses one of the three buckets. The follow-up worth volunteering is that this requires logging the retrieved context alongside every answer — without it, failures are unattributable after the fact.

Q: How would you evaluate a RAG system?

In two halves, because they fail independently: the retrieval half on recall@k and nDCG against labelled queries, the generation half on faithfulness to the retrieved context and on answer correctness. Frameworks like RAGAS (Es et al. 2024) impose exactly this decomposition — context precision and recall, faithfulness and answer relevance — and are worth using for the structure. Add the test almost nobody runs — questions the corpus cannot answer, scored on whether the system abstains — and say plainly that automatic faithfulness scores need validating before you trust them, which is Chapter 22’s subject.

Q: When would you not use RAG?

When the knowledge is small and stable enough to sit in the prompt; when the task needs the whole corpus rather than a few passages, such as “summarise every complaint this quarter”, where retrieval structurally cannot see enough; and when the requirement is really a database query wearing a natural-language costume, in which case text-to-SQL over a schema beats embedding rows and hoping. Knowing where the technique stops is the part that separates a practitioner from someone repeating an architecture diagram.

Q: RAG or fine-tuning?

Usually the wrong either-or. Use retrieval for knowledge — facts that change, facts a user must be able to check against a source, facts some users may not see — because updating an index is cheap, every answer can cite its evidence, and permissions can be enforced per document. Use fine-tuning for behaviour — format, tone, a narrow task done reliably — because that is what a small training run changes well, while it adds new facts only unreliably and cannot easily remove them again. The two compose: a model fine-tuned on your format, answering over retrieved documents that supply your facts. Section 9.9 walks through the decision question by question.

20.7 The 2026 toolbox

Everything in this chapter ran on one laptop with the versions below.

The stack these numbers came from

library	version used	what it is for here
snowballstemmer	3.1.1	the Turkish stemming behind every retrieval number in Section 20.3
numpy	2.5.2	BM25 over 24,266 chunks, and the whole sweep
ollama	0.24.0	serving qwen3.5:9b locally for the needle test

Worth knowing about, not used above:

The wider ecosystem, versions as of writing

library	current version	when you would reach for it
docling / unstructured	2.123.0 / 0.27.1	stage 1 — turning PDFs and HTML into text that survives; the stage most worth spending on
pymupdf	1.28.2	fast, low-level PDF text and layout extraction when you want to control the parsing yourself
langchain-text-splitters	1.1.2	the chunkers everyone starts with, including recursive and structure-aware splitters
chonkie	1.7.0	a chunking-only library, when splitting is the thing you want to tune rather than a side effect of a framework
llama-index / langchain	0.14.24 / 1.3.17	end-to-end orchestration; useful to start, worth understanding underneath before you depend on it
haystack-ai	3.1.0	pipeline-shaped alternative, with the stages made explicit
ragas	0.4.3	the two-halves evaluation of Section 20.4, implemented (Es et al. 2024)

20.8 Lab: a RAG system that tells you which stage failed

The deliverable is not a chatbot. It is a harness that, given a wrong answer, names the stage responsible — because that is the thing you will actually need every week and the thing almost nobody builds.

Step 1 — fifty questions, written before any code

Write fifty questions against your own corpus, and for each one record the document that answers it. Include five that the corpus cannot answer; those are the only way to test abstention. Do this first, because questions written after you have seen the system’s output are questions the system can already answer.

Step 2 — two chunkings, not one

Index the corpus twice: whole documents, and sentence-packed chunks of about 150 words. You now have the two ends of Section 20.3’s trade on your own data, and the comparison costs one afternoon.

Step 3 — measure recall@k before generating anything

For each question, is the gold document in the top-k? That number is your ceiling. If it is 70%, stop and fix retrieval; nothing you do to the prompt will reach the missing 30%.

Step 4 — generate, and log the context

Answer each question with the retrieved chunks in the prompt, and store the chunks alongside the answer. An answer without its context is unattributable after the fact, which makes the log useless exactly when you need it.

Step 5 — attribute every failure

For each wrong answer, apply Section 20.4’s three-way test in order: was the gold chunk retrieved, was the answer supported by what was retrieved, was the conclusion right given the facts present. Produce a table of counts.

Step 6 — test the refusal

Run the five unanswerable questions. Count how many produced a confident answer anyway. This is the single most informative number in the lab, and it is the one nobody measures.

What to hand in

The three-bucket attribution table, the recall@k that bounds it, the two chunkings compared on the same questions, and one paragraph naming the stage you would fix first and what it would cost.

20.9 Exercises

1. Reproduce the inversion. Take 500 documents of your own and build two query sets — one asking what a document is about, one asking where a specific fact is. Sweep chunk size. Does the ranking invert on your corpus as it did on this one? If it does not, what is different about your documents?
2. Find the ceiling. Measure recall@k for the k your system actually passes to the model. Then compute the best end-to-end accuracy your system could possibly reach. Compare it to what it does reach; the gap is your generation problem, and everything above it is not.
3. Attribute a week of failures. Take the wrong answers your system produced last week and bucket them three ways. Report the proportions. Predict, before you look, which bucket will be largest.
4. Break the parser on purpose. Take ten PDFs from your domain, extract the text, and read it. Count how many tables survived. This exercise has a 100% record of surprising the person who runs it.
5. Measure distraction. Hold retrieval fixed and vary only k from 1 to 20. Plot accuracy. Find the point where adding context starts to cost you (Shi et al. 2023).
6. Test the refusal. Write twenty questions your corpus cannot answer. Count confident answers. Then add an explicit abstention path and measure again.
7. Two granularities. Index your corpus at both whole-document and sentence-packed granularity, route each query to one of them, and measure whether routing beats either index alone — the same question Chapter 19 asked about retrievers, one level up.

20.10 Where to go deeper

Before reading anything: take fifty wrong answers from your own system and bucket them three ways. The distribution will tell you which half of this literature is relevant to you.

Beyond the papers

resource	what it is
ragas	the two-halves evaluation implemented; use it for the decomposition it forces, not the absolute numbers
Eugene Yan on LLM patterns	RAG placed among the other patterns, with the trade-offs stated
Chip Huyen’s blog	long-form applied writing on RAG and agent architectures from someone who has shipped them
llama-index documentation	worth reading the ingestion and chunking sections even if you build it yourself
Stanford CS224N	lectures covering retrieval-augmented models and their evaluation

The primary sources

topic	source
the original formulation	Lewis et al. (2020)
the map of everything since	Gao et al. (2023)
how models use a long context	Liu et al. (2024)
why a literal needle is the wrong test	Modarressi et al. (2025)
irrelevant context as an active harm	Shi et al. (2023)
evaluating the two halves separately	Es et al. (2024)
deciding whether to retrieve at all	Asai et al. (2024)

20.11 What to remember

• The needle test found nothing, and that is the chapter’s first lesson. Sixteen conditions, 640 queries, 100% at every context size and every position up to 16,775 tokens — with 40 queries per cell, which is enough power to exclude a large positional collapse and not a small one. Lost-in-the-middle did not reproduce here because a verbatim string with a rare key is the easiest thing attention can do, not because the phenomenon is imaginary: Modarressi et al. (2025) removes the lexical overlap and watches eleven of thirteen long-context models fall below half their short-context baseline. A benchmark everything passes is not measuring your risk.
• What survives is the cost, and it is large. Eighty documents instead of the one that mattered: 148× the input tokens for identical accuracy, paid on every request that cannot be served from a cache — prompt caching brings that nearer 15×, and only while the prefix is stable — with attention’s quadratic share growing as the context does. Retrieval is not a crutch for a model that cannot read; it is how you stop paying for 79 documents you did not need.
• Two reasons to retrieve are permanent and have nothing to do with window size. The model is frozen and your documents are not, and a prompt cannot enforce who may see which document. A third is practical: most real corpora are far larger than any window.
• Chunk size is a bet on the granularity of the question. On one corpus, 400-word chunks led article-level retrieval at 72.5% recall@1 and came last on fact-level retrieval at 93.8%; 50-word chunks did the reverse. The best configuration in one condition is the worst in the other, so a chunk size copied from a blog post is a guess about questions the author never saw.
• Overlap is a repair, not a safety margin. A 25-word overlap on 100-word chunks bought +2.0 points on fact-level queries and −0.7 on article-level ones, for 29% more index. It fixes facts that straddle a boundary and nothing else.
• Structure beat arithmetic, mostly. Sentence-aligned chunks of ~100 words reached 97.8% where fixed 100-word windows reached 95.5% — and matched the overlapping variant (97.5%, one query behind) with a 9% smaller index. The sentence-aligned chunks are also smaller (86 words against 99); interpolating the fixed-size series credits roughly two of the 2.3 points to the boundary, but the size-matched control has not been run. The cheapest way to stop cutting through a fact is to cut where the text already breaks.
• Watch the metric your consumer actually reads. On fact-level queries, recall@1 moved four points across these configurations and recall@5 stayed between 97.2% and 100%; on article-level queries, recall@1 swung fifteen points and recall@5 still spanned ten. If your generator reads five chunks and your questions are fact-shaped, the chunking choice barely moves article retrieval; if it reads one, the R@1 columns are your system. Whether the retrieved chunk carries the answering sentence is a separate measurement, and this run does not make it.
• Three failures, one symptom. A wrong answer is a retrieval miss, a grounding miss, or a synthesis miss, and they have three different fixes. Attribute before you tune — the reflex fix, swapping the embedding model, addresses one of the three.
• Number the chunks, ask for citations, and check them. A citation is a claim too. Faithfulness (supported claims ÷ all claims) and context recall score the two halves over time, but they come from an LLM judge, so validate the judge before trusting the number.
• A system with no way to abstain will not abstain. Give it an explicit refusal output and then test it on questions the corpus cannot answer. That test is the one nobody runs, and it is the difference between a system that declines and one that has never been given the option.

Asai, Akari, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. “Self-RAG: Learning to Retrieve, Generate, and Critique Through Self-Reflection.” Proceedings of ICLR.

Es, Shahul, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. 2024. “RAGAs: Automated Evaluation of Retrieval Augmented Generation.” Proceedings of EACL: System Demonstrations.

Gao, Yunfan, Yun Xiong, Xinyu Gao, et al. 2023. “Retrieval-Augmented Generation for Large Language Models: A Survey.” arXiv Preprint arXiv:2312.10997.

Lewis, Patrick, Ethan Perez, Aleksandra Piktus, et al. 2020. “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” Advances in Neural Information Processing Systems 33.

Liu, Nelson F., Kevin Lin, John Hewitt, et al. 2024. “Lost in the Middle: How Language Models Use Long Contexts.” Transactions of the Association for Computational Linguistics 12: 157–73.

Modarressi, Ali, Hanieh Deilamsalehy, Franck Dernoncourt, et al. 2025. “NoLiMa: Long-Context Evaluation Beyond Literal Matching.” Proceedings of ICML.

Shi, Freda, Xinyun Chen, Kanishka Misra, et al. 2023. “Large Language Models Can Be Easily Distracted by Irrelevant Context.” Proceedings of ICML.