19  Retrieval: Search Before Generation

Chapter status: drafted. The measurements extend Chapter 16’s retrieval experiment — the same 7,165 Turkish documents, the same query construction, the same seed — so the sparse numbers here and there are directly comparable. Two query conditions and two dense models are used deliberately; Section 19.2 says why running only one of each would have measured the wrong thing. Software: sentence-transformers 6.0.0, snowballstemmer 3.1.1, NumPy 2.5.2 on Python 3.14.5.

Learning objectives. After this chapter you can:

• build a BM25 baseline and say precisely which queries it wins;
• explain what a dense retriever adds, and construct a query type where it beats lexical search by seventy points and one where it loses by sixty;
• decide whether to fuse two retrievers, using a rule rather than a habit;
• tune the recall/latency dial on an approximate index deliberately;
• report retrieval quality with recall@k and nDCG, and know which of them bounds what your downstream system can possibly do.

19.1 Two families, and the question that decides between them

Retrieval has two families and they fail in opposite directions.

Lexical retrieval matches words. BM25 — the standard scoring function (Robertson and Zaragoza 2009) — is Chapter 2’s TF-IDF with two corrections that matter: term frequency saturates, so the tenth occurrence of a word adds far less than the second, and document length is normalised, so a long document does not win by containing more of everything. (The formula, and what its two knobs \(k_1\) and \(b\) do, is written out in Section 19.10.)

It is fast, exact, interpretable, needs no training, and it cannot match a word it has never seen.

Dense retrieval matches meaning. A bi-encoder (Karpukhin et al. 2020) embeds queries and documents separately into the same space, so documents can be encoded once and searched by vector similarity (usually the cosine similarity of Chapter 3). It handles paraphrase, synonymy and — as this chapter will show — different languages.

It needs a trained model, it cannot guarantee an exact match on a part number, and its failures are hard to explain.

bi-encoder — query and document encoded independently, so every document can be embedded ahead of time and search is a vector lookup. The independence is what makes it scale, and what limits its accuracy relative to a cross-encoder. ↩︎

The received framing is that dense is the modern one and lexical is the baseline you report out of politeness. The measurements below say something less convenient.

What we will look at. First the experiment, which needs two query conditions rather than one, because a single condition would have measured the query format and blamed the model. Then the results, where the same two systems swap places by sixty points and hybrid fusion — the thing everyone recommends — lands 33 to 47 points behind the better of its two parents. Then how a dense retriever is trained, where the hard negatives are the whole game. Then reranking, the approximate index, and the four metrics, one of which is a ceiling on everything downstream. And finally where to put the vectors, which is a smaller question than it is usually made to be.

19.2 The experiment, and why it has two conditions

Chapter 16 measured the sparse side on 7,165 Turkish documents: with queries whose terms are inflected differently from the document — the case where a user types the dictionary form — surface-form BM25 reaches 28.0% recall@1 and stemming takes it to 95.3%.

The obvious next step is to add a dense retriever to that setup and see. The obvious next step is also a trap, and it is worth spending a paragraph on because the same trap is available in any retrieval evaluation you build.

Those queries are three disconnected content words. That is the natural input for a lexical matcher and it is out of distribution for a bi-encoder trained to embed sentences. Running only that condition and reporting “dense retrieval is weak in Turkish” would have been measuring the query format and blaming the model.

So the experiment runs two query conditions against the same documents:

Two conditions: each family plays at home in one and away in the other

condition	the query is	who this favours
A. keyword, inflected	three content words, in a different inflection than the document uses	lexical matching, if it can handle morphology
B. cross-lingual	the document’s professional English translation	dense only — there is no shared vocabulary at all

And two dense models, for the same reason: paraphrase-multilingual-MiniLM is a similarity model, trained to score how alike two sentences are, and multilingual-e5-base (Wang et al. 2024) is a retrieval model, trained on query-document pairs with hard negatives. Reaching for the first because it is the famous one is a common mistake, and running only it would have understated dense retrieval.

One instruction from E5’s model card that this chapter should have recorded and did not. multilingual-e5-base is an asymmetric retrieval model: it is trained with a query: prefix on queries and a passage: prefix on documents, and the card is explicit that it behaves measurably worse without them. This is exactly the setting where the prefixes matter most — an asymmetric query-document task is what they exist for — and it is the most common silent error in an embedding pipeline; Chapter 11 makes the same point for the symmetric case, where the card prescribes query: on both sides. Our runs do not state which convention they used. If the prefixes were omitted, E5’s numbers below are measured under a setting its own card advises against — which is a reason to re-run rather than a guarantee that the scores would rise or that both of this chapter’s headline margins would shrink. Treat the E5 rows as pending that check, and record the prefixes beside the model name when you run this.

Two upstream decisions this experiment sidesteps, so that you do not miss them in your own system. First, what counts as a “document”: here every document is a single sentence, but real documents are longer and have to be cut into chunks before they are embedded, and the best chunk size depends on the kind of question (Section 20.3 measures it). Second, for Turkish BM25, lowercase with Turkish rules: Python’s default lower() turns I into i instead of ı and İ into i plus a combining dot, which mangles words before stemming ever runs (Section 16.2).

19.3 What the measurements say

Both conditions run against the same corpus, so the two tables below are directly comparable — and the point is how little they have in common. 800 queries per condition, 7,165 documents, and one relevant document per query by construction — each query is built from one target document and that document is treated as the only correct answer.

That last assumption is not verified, and it is not symmetric. On 7,165 sentences of news text, three content words will often describe more than one sentence, and the system that surfaces a near-paraphrase is precisely the dense one — so unlabelled relevant documents likely push dense recall@1 down more than BM25’s. The fix is the one this chapter’s own interview section prescribes: pool the top-10 of every system on a subsample and label the union by hand. We did not do it. Until someone does, read condition A’s dense numbers as a floor rather than an estimate.

Two things to know before reading the tables. First, the columns. R@1 and R@10 are recall@1 and recall@10; with one relevant document per query they simply ask “is the right document first?” and “is it anywhere in the top ten?”. MRR@10 and nDCG@10 also reward a high position, and all four are defined properly in Section 19.7. Second, the hybrid rows. They combine a BM25 ranking and a dense ranking with reciprocal rank fusion (RRF) (Cormack et al. 2009). Every document gets the score \(\sum_s 1/(k + \text{rank}_s(d))\), summed over the systems \(s\) that returned it, and the fused list is sorted by that score. A document a system did not return adds nothing from that system. \(k\) is a smoothing constant; the original paper used \(k = 60\), and it remains the common default.

reciprocal rank fusion (RRF) — merge several ranked lists by giving each document \(\sum 1/(k + \text{rank})\) over the lists and re-sorting. It uses only ranks, never scores, so it needs no calibration — and it cannot tell a strong system’s first place from a weak system’s. ↩︎

Condition A — keyword queries, inflected

This is the condition lexical matching should win, provided it can handle Turkish morphology:

system                                        R@1    R@10   MRR@10  nDCG@10
BM25 surface form                           27.5%   54.1%     34.9     39.4
BM25 stemmed                                96.0%   99.8%     97.7     98.2
dense: paraphrase-MiniLM (similarity model) 10.6%   28.2%     15.6     18.6
dense: multilingual-e5-base (retrieval)     35.9%   63.7%     44.5     49.1
hybrid RRF (unweighted), surface + e5       37.6%   71.6%     47.6     53.3
hybrid RRF (unweighted), stemmed + e5       63.2%   96.5%     73.7     79.2

Stemmed BM25 wins by sixty points. A few hundred lines of hand-written suffix-stripping rules beat a trained multilingual retriever 96.0% to 35.9% — and beat the similarity model, the one most people reach for first, 96.0% to 10.6%.

The surface-form and stemmed rows reproduce Chapter 16’s 28.0% and 95.3% on a slightly different query sample (800 queries per condition rather than 1,481), which is the continuity check that lets the rest of the table be read against it.

Condition B — cross-lingual queries

Now the same documents, queried in English. There is no shared vocabulary for a lexical matcher to work with:

system                                        R@1    R@10   MRR@10  nDCG@10
BM25 surface form                           23.2%   43.5%     28.9     32.3
BM25 stemmed                                23.8%   41.9%     29.1     32.1
dense: paraphrase-MiniLM (similarity model) 93.8%   98.1%     95.4     96.0
dense: multilingual-e5-base (retrieval)     96.2%   99.9%     97.8     98.3
hybrid RRF (unweighted), surface + e5       49.4%   99.6%     71.6     78.7
hybrid RRF (unweighted), stemmed + e5       48.1%   99.6%     70.9     78.2

Dense wins by seventy-two points. The English query and the Turkish document share almost no vocabulary, so BM25 is reduced to matching proper nouns and numbers — which is exactly why it still scores 23.8% rather than zero, and worth noticing, because those are precisely the cases dense retrieval is worst at.

But say what condition B is favourable to, the way we said it for condition A. The query here is a full professional translation of the document, and that is the most in-distribution input a multilingual bi-encoder can receive: both of these models were trained on parallel sentence pairs, so this task is closer to bitext mining (finding the sentence that is the translation of a given sentence) than to search. Real cross-lingual queries are short and question-shaped and do not cover the whole document. So read 96.2% as “dense retrieval solves translation matching” rather than “dense retrieval solves cross-lingual search”, and read the seventy-two-point margin as an upper bound on the dense advantage — exactly as condition A’s sixty-point margin is an upper bound on the lexical one. Each condition hands its winner a query drawn from the target document itself. The missing arm is condition B′: three content words from the English translation, which is condition A’s cross-lingual twin. It would put a floor under the dense margin, and it is the run we would add first.

Stemming is worth 68 points in condition A and 0.6 points in condition B. That is not a contradiction; it is the same fact seen twice. Stemming fixes vocabulary mismatch within a language, and in condition B the mismatch is total, so there is nothing for it to fix.

The finding that contradicts the received wisdom

Take the better of the two hybrid rows in each table (stemmed + e5 in condition A, surface + e5 in condition B) and read it against the best single system:

condition A     stemmed BM25 alone         96.0%  R@1
                hybrid stemmed + e5        63.2%  R@1      -32.8

condition B     dense e5 alone             96.2%  R@1
                hybrid surface + e5        49.4%  R@1      -46.8

Unweighted reciprocal rank fusion did worse than the better parent in both conditions, and badly. That is the opposite of the standard claim that hybrid retrieval beats both parents (Cormack et al. 2009), and the reason is visible in the mechanism.

Be precise about what was measured, because the claim is narrower than “hybrid retrieval is oversold”. These rows are unweighted, rank-based fusion at a constant \(k\) this chapter does not report — and the widget below exists to show you that \(k\) changes the answer, which makes not reporting it a real gap.

Two standard repairs went untried. Weighted RRF — \(\sum w_s/(k + \text{rank}_s)\) with a per-system weight fitted on held-out queries — is the obvious one, and this chapter defers it to an exercise rather than running it. Score-based fusion is the better one: Bruch et al. (2024) analyses exactly this question and finds that a convex combination of normalised scores beats RRF both in and out of domain, with a single parameter that needs very few examples to tune. So the finding to carry is this one: unweighted rank fusion of unequal parents can be harmful — it was in both cases measured here — because rank fusion throws away precisely the score-magnitude information that would tell it which parent to trust.

RRF assumes the systems it is fusing are comparably good. It scores each document by \(\sum 1/(k + \text{rank})\) across the lists, which treats a rank-1 hit from a useless system exactly like a rank-1 hit from an excellent one.

When one parent is at 96% and the other at 36%, the weak parent’s top-ranked irrelevant documents arrive with full weight and displace the strong parent’s correct answer. Note that “confidence” is not the word: rank fusion never sees a confidence score, which is exactly the information that would have told it which parent to trust.

Fuse two rankers by hand — and watch the right document lose

viewof rrfGoldStrong = Inputs.range([1, 10], {step: 1, value: 1,
  label: "gold document's rank in the strong system"})

viewof rrfGoldWeak = Inputs.range([1, 60], {step: 1, value: 40,
  label: "its rank in the weak system (60 = the bottom of a 60-deep list)"})

viewof rrfK = Inputs.range([1, 60], {step: 1, value: 60, label: "RRF constant k"})

// The weak system is confident about the wrong thing: it ranks a decoy first.
// The strong system has that decoy far down its own list.
rrfRows = {
  const score = (a, b) => 1 / (rrfK + a) + 1 / (rrfK + b);
  const rows = [
    {doc: "the right document", strong: rrfGoldStrong, weak: rrfGoldWeak},
    {doc: "decoy the weak system loves", strong: 20, weak: 1},
    {doc: "decoy the weak system likes", strong: 25, weak: 2},
    {doc: "decoy both systems dislike", strong: 15, weak: 15}
  ].map(r => ({...r, fused: score(r.strong, r.weak)}));
  const order = [...rows].sort((a, b) => b.fused - a.fused);
  return rows.map(r => ({...r, rank: order.indexOf(r) + 1}));
}

Inputs.table(rrfRows, {
  columns: ["doc", "strong", "weak", "fused", "rank"],
  header: {doc: "document", strong: "strong rank", weak: "weak rank",
           fused: "RRF score", rank: "fused rank"},
  format: {fused: v => v.toFixed(5)},
  width: {doc: 240, strong: 100, weak: 100, fused: 100, rank: 90},
  rows: 5})

md`**${rrfRows[0].rank === 1
  ? "The right document survives at rank 1."
  : `The right document lands at rank ${rrfRows[0].rank}.`}** The strong
system ranked it ${rrfGoldStrong === 1 ? "first" : `${rrfGoldStrong}${
  rrfGoldStrong === 2 ? "nd" : rrfGoldStrong === 3 ? "rd" : "th"}`}, above
all three decoys, and it is ${rrfRows[0].rank === 1
  ? "still first among these four, even though"
  : "no longer first among these four once"} a system that is wrong gets an
equal vote.`

Move the second slider and the mechanism becomes obvious: the weak system does not need to be right to change the outcome, only to put the wrong document at the top of its list. And notice what k does. A small k makes first place worth far more than tenth, so each system’s top hit dominates. Raising it flattens the differences between ranks, and at the limit the fused order is simply the order of the summed ranks — a document that sits in the middle of both lists can then beat one that is first in one list and deep in the other. Neither setting knows which system to trust.

The damage has a shape worth seeing, because it tells you when you can get away with it:

condition B     dense alone       R@1 96.2%    R@10  99.9%
                hybrid            R@1 49.4%    R@10  99.6%

Recall@10 barely moved; recall@1 halved. Fusion did not lose the document — it pushed it down the list. So if your consumer reads the top ten, the fusion cost you almost nothing; if it reads the top one, fusion cost you half your accuracy.

All three findings fit in one picture — the swap, the fusion loss, and the metric that hides it:

Recall@1 on one corpus: the winner swaps, and fusion loses to the better system in both conditions

Both panels are drawn at the same scale, so a bar of the same length means the same recall@1. The two families change places completely between the conditions, on one corpus with one set of documents. Each fusion bar is the better of the two hybrids in its table; in condition B that is the fusion with surface-form BM25 (23.2% alone), not the stemmed BM25 drawn above it. In each panel the dashed line marks the better parent and the rose bracket is what fusing cost against it. The bottom band is the same condition-B fusion seen at depth ten, where the two bars are indistinguishable — the right document was never lost, only pushed down, which is why a system can regress badly while the number you are watching does not move.

So when should you fuse?

The rule the measurement supports, which is narrower than the folklore:

• Fuse when the parents are comparable and fail differently. That is the case Cormack et al. (2009) studied, and where fusion earns its reputation.
• Do not fuse when one parent dominates. Check first, on your own queries. It costs one evaluation run and the alternative is shipping a 47-point regression that every blog post told you was an improvement.
• If your query mix contains both kinds — and most real systems do, with users typing part numbers and asking questions — the answer is usually to route rather than to fuse: classify the query, send it to the retriever that handles it, and keep fusion for the genuinely ambiguous middle.

Weighted fusion is the obvious repair, and it is not free: the weight is a fitted parameter, so it needs held-out queries and it will drift as your query mix changes. Routing has the same requirement more honestly, because a router is a model you can evaluate rather than a constant you tuned once.

And the choice of dense model matters exactly where the task is hard

One more comparison the two conditions make possible. Put the similarity model and the retrieval model next to each other in both:

                              condition A   condition B
paraphrase-MiniLM (similarity)      10.6%         93.8%
multilingual-e5-base (retrieval)    35.9%         96.2%
                                   ------        ------
                                   +25.3          +2.4

On the easy condition (B, for a dense model) the two are 2.4 points apart, so a casual benchmark would conclude the model choice does not matter. On the hard condition (A) the retrieval-trained model is more than three times better (35.9% against 10.6%).

The gap between a good and a mediocre component only appears on the inputs that are hard for it, which is an argument for building the difficult condition into your evaluation rather than reporting the average of an easy one.

19.4 Training a dense retriever

The models above were used off the shelf. When they are not good enough on your domain, the training recipe is short to state and has one part that matters far more than the rest.

A bi-encoder is trained contrastively: pull the embedding of a query toward its relevant document and push it away from irrelevant ones. The loss is usually InfoNCE over a batch. It is ordinary softmax cross-entropy in which the “classes” are the candidate documents: score the query against the relevant document and against several irrelevant ones, and ask the softmax to put its probability on the relevant one. The whole difficulty is in the word irrelevant — which documents you use as the wrong answers.

In-batch negatives are free: for a batch of query-document pairs, every other document in the batch is a negative. They cost nothing and they are mostly easy — a random document about football is not a plausible answer to a query about invoices, so the model learns to separate topics and stops there.

Hard negatives are documents that look right and are not. Mine them by retrieving with your current model and taking the top-ranked wrong answers.

hard negative — a wrong document that looks right to the current model, found by retrieving with it and keeping the top-ranked wrong answers. Training against these, rather than random documents, is most of what makes a dense retriever precise. ↩︎

This is the whole game. A retriever trained only on random negatives learns topical similarity; a retriever trained on hard negatives learns the distinction your users actually need, which is between two documents that are both about invoices and only one of which answers the question.

Two failure modes to plan for:

1. False negatives. Your “hard negative” may be an unlabelled relevant document, and training says it is wrong. On a corpus with sparse relevance labels this is common enough to hurt; the usual mitigation is to skip the very top of the retrieved list and mine from a little further down.
2. Staleness. Negatives mined with the initial model stop being hard once the model improves. Re-mine periodically, which is what makes this an iterative procedure rather than a preprocessing step.

If you have no labelled pairs at all, generate them: ask an LLM to write a question that each document answers, and train on those. The synthetic query is noisy on the input side and the document is genuine on the output side — the same asymmetry that makes back-translation work in Chapter 15.

19.5 Reranking

Every architecture above compares a query vector to a document vector that was computed without ever seeing the query. There is a more accurate thing to do, and a reason nobody does it at corpus scale.

A cross-encoder (Nogueira and Cho 2019) reads the query and the document together, in one forward pass with attention across both. That is strictly more informative than comparing two independently computed vectors, and it is why a cross-encoder is usually more accurate than a bi-encoder of similar size and training — usually, not always: a cross-encoder trained on another domain can still lose to a well-trained bi-encoder on yours.

cross-encoder — query and document concatenated and read in a single forward pass, so attention runs across both. Usually more accurate than a bi-encoder and impossible to precompute, which is why it reranks a shortlist instead of searching a corpus. ↩︎

It is also unusable as a retriever. Because the score depends on the pair, nothing can be precomputed: scoring a query against a million documents means a million forward passes. Hence the standard two-stage pattern:

Why the architecture is always two stages

The bi-encoder is fast because it never sees a pair; the cross-encoder is accurate because it only ever sees pairs. Neither property is negotiable, so production systems buy recall with the first and precision with the second.

Three consequences worth internalising:

• The first stage sets a ceiling the second cannot raise. A reranker can only reorder what it is given. If recall@50 is 80%, no reranker gets you past 80% — which is why Section 19.7 argues recall@k is the metric to watch on stage one.
• The cost is linear in the shortlist, so the shortlist size is your quality/latency dial, and it is an easier dial than the ANN parameter: its cost is predictable, and the recall it hands the reranker can only grow as the shortlist grows. Final accuracy is another matter — it usually plateaus and can even dip, as step 6 of Section 19.12 explains.
• Late interaction is the middle path. ColBERT (Khattab and Zaharia 2020) keeps a vector per token and scores by summed maximum similarity (each query token finds its most similar document token, and those best matches are added up), which recovers much of the cross-encoder’s accuracy while remaining precomputable. It costs an order of magnitude more storage, which is the trade.

19.6 Approximate nearest neighbour, and the dial nobody sets on purpose

Exhaustive search over a few thousand vectors is a matrix multiply and takes no time at all — the dense numbers above were computed that way. At millions of vectors it stops being free, and you move to an approximate index.

“Approximate” means the index may miss some of the true nearest neighbours in exchange for speed. Two families dominate in practice:

HNSW (Malkov and Yashunin 2018) builds a navigable graph and walks it greedily from a coarse layer down to a fine one. Fast and accurate, memory-hungry — the graph plus the full vectors sit in RAM.

IVF-PQ (Johnson et al. 2019) partitions the space into cells (an inverted file, IVF: k-means clusters, each with a list of the vectors inside it), searches only the nearest few cells, and stores compressed vectors via product quantisation (PQ). PQ cuts each vector into short pieces and replaces every piece by the index of its nearest entry in a small learned codebook. A 768-dimensional float32 vector takes 3,072 bytes; cut into 96 pieces with 256 codebook entries each, it takes 96 bytes, one per piece — 32 times smaller. The compression is lossy in a way that costs accuracy.

Both expose a knob — efSearch for HNSW (how many candidates the graph walk keeps), nprobe for IVF (how many cells it searches) — that trades recall against latency. The thing to remember is that the default is somebody else’s operating point, chosen for a benchmark that is not your corpus.

The discipline is the same one Chapter 12 applied to a decision threshold: sweep it, plot recall against p95 latency (the latency that 95% of queries stay under), state your latency SLO (the service-level objective, the latency target you have committed to), and pick the point that meets it.

And measure recall against exhaustive search on your own data, not against the library’s published curve — the loss depends on how your vectors are distributed, which is a property of your corpus and your embedding model.

There is a trap in evaluating an ANN index against the labels rather than against exhaustive search. If your recall@10 drops when you switch on the index, you want to know whether the index lost documents or whether the embedding was wrong to begin with. Exhaustive search is the control that separates the two, and it is cheap to run once on a sample.

19.7 Measuring retrieval

Everything above was reported in four columns — recall at two depths, MRR and nDCG — with only a one-line gloss of each. The table below adds precision@k, which the tables did not use. These metrics answer different questions, and picking the wrong one is how a retrieval regression stays invisible.

Four metrics, four questions

metric	asks	use it when
recall@k	what share of the relevant documents is in the top k? With one relevant document per query — this chapter’s setup — that reduces to is it there at all, which is also called hit rate or success@k	the consumer reads all k — which includes every RAG system
MRR	how high is the first right answer?	the user reads one result and stops
nDCG@k	are the relevant documents near the top, weighted by how relevant?	graded relevance, several relevant documents
precision@k	what share of the k is relevant?	the cost of a wrong result is per-result

recall@k — the share of a query’s relevant documents that appear in the top k. When there is exactly one relevant document, as here, it collapses to did the right document appear at all — the quantity often called hit rate or success@k. The number that bounds everything downstream: a generator cannot ground an answer in a document the retriever never returned. ↩︎

The two rank-sensitive metrics are short enough to write down. For a set of queries \(Q\), let \(\text{rank}_q\) be the position of the first relevant document for query \(q\):

\[ \text{MRR@}k = \frac{1}{|Q|}\sum_{q \in Q} \frac{1}{\text{rank}_q} \quad (\text{a term counts as } 0 \text{ if } \text{rank}_q > k) \]

\[ \text{DCG@}k = \sum_{i=1}^{k} \frac{\text{rel}_i}{\log_2(i+1)}, \qquad \text{nDCG@}k = \frac{\text{DCG@}k}{\text{IDCG@}k} \]

Here \(\text{rel}_i\) is the relevance grade of the document at position \(i\) (0 or 1 in this chapter), and IDCG@\(k\) is the DCG of the best possible ordering, so nDCG lies between 0 and 1. Some tools use the gain \(2^{\text{rel}_i}-1\) instead of \(\text{rel}_i\); with 0/1 labels the two agree.

A worked example with one relevant document, as in this chapter’s tables. Suppose it comes back at rank 3. Then recall@1 = 0, recall@10 = 1, the reciprocal rank is \(1/3 \approx 0.33\), and nDCG@10 is \(1/\log_2 4 = 0.5\) (the ideal list puts it at rank 1, so IDCG = 1). Both metrics punish rank 3, but nDCG punishes it more gently. That is why, in every row of the tables above, nDCG@10 sits between MRR@10 and recall@10.

If you report one number for the retrieval stage of a RAG system, report recall@k for the k you actually pass to the model. It is a ceiling on what grounding can deliver: Chapter 20’s generator can only ground in what arrives, so with one relevant document per question a retrieval recall@10 of 80% means 20% of answers cannot be grounded in retrieved evidence before generation begins — and no amount of prompting fixes them. (Intuition check 5 below states exactly when this ceiling holds.)

The corollary is that the two stages want different metrics. Stage one is judged on recall, because its job is not to miss. Stage two — a reranker, or the generator itself — is judged on precision and ordering, because its job is to choose. Optimising the first stage for precision@1 is a common and expensive mistake.

Two practical notes on measurement itself:

• You need relevance judgements, and building them is the real work. With no labels, the cheapest honest start is what this chapter did: take documents you have, construct queries whose answer you know, and accept that synthetic queries are a proxy. BEIR (Thakur et al. 2021) is the reference benchmark suite if you want to compare against published numbers rather than measure your own corpus.
• Report the query mix. Section 19.3 is one long argument that a retrieval number without its query distribution is not interpretable — the same two systems swap places by sixty points between two conditions on the same corpus.

19.8 Vector database, or not

The last decision is where the vectors live, and it is usually made too early and too expensively. The honest guide is by scale:

Where to put the vectors

documents	what to use	why
under ~100k	NumPy array plus exhaustive search	a matrix multiply; the dense numbers in this chapter were computed this way, and it is exact
~100k–10M	pgvector in the Postgres you already run, or FAISS in-process	you get transactions, backups, joins and access control for free, and no new service to operate
10M+, or multi-tenant	a dedicated vector store	when sharding, replication and filtered search stop being your problem to solve

The default answer for most projects is the boring one: keep the vectors next to the data they belong to.

The operational cost of a second datastore — consistency between it and your source of truth, its own backup story, its own access-control model — is usually larger than the query latency you were trying to save, and it arrives on a different team’s schedule.

The question that actually forces a dedicated store is rarely raw scale. It is filtered search at scale: “the top 10 documents this user is allowed to see, from the last 90 days, in Turkish.” Doing that well requires the filter and the vector search to cooperate, and that is a genuine engineering capability rather than a matter of size.

19.9 Intuition checks

Answer each one before you open it. Most are questions this chapter settled with a measurement rather than an argument.

Note1. Should you replace BM25 with a dense retriever?

Only after checking which queries you actually get. Measured on one corpus, the same two systems swapped places by sixty points between two query conditions: stemmed BM25 96.0% against dense 35.9% on keyword queries, and dense 96.2% against BM25 23.8% cross-lingually.

Neither family dominates. Lexical wins exact-match and rare-token queries; dense wins paraphrase, synonymy and anything crossing a language boundary. Sample a hundred real queries and classify them before choosing.

Note2. Everyone says hybrid retrieval beats both parents. Does it?

Not when the parents are unequal. Measured here, reciprocal rank fusion cost 32.8 points in one condition and 46.8 in the other, against the better parent each time.

RRF weights a rank-1 hit from a weak system exactly like a rank-1 hit from a strong one, so the weak parent’s top-ranked wrong answers displace the strong parent’s right ones. Fusion is a variance-reduction trick for comparable systems. Check that yours are comparable before fusing, which costs one evaluation run.

Note3. Your recall@10 barely changed but users complain. What happened?

Look at recall@1. In this chapter’s condition B, fusion took recall@10 from 99.9% to 99.6% — nothing — while recall@1 went from 96.2% to 49.4%. The document was still retrieved; it was pushed down the list.

Which metric matters depends on the consumer. A RAG pipeline passing ten chunks to a model cares about recall@10. A user reading the first result cares about recall@1. Reporting only the one that looks good is the easiest self-deception in retrieval.

Note4. Which embedding model should you use?

Not the famous one, and the difference only shows on hard inputs. A similarity model and a retrieval model were within 2.4 points on the easy condition here — and 25.3 points apart on the hard one, 10.6% against 35.9%.

Similarity models are trained to score how alike two sentences are; retrieval models are trained on query-document pairs with hard negatives. They are different tasks. And the practical consequence for evaluation: if your test set is easy, every model looks the same and you will pick by download count.

Note5. Why is recall@k the number that bounds a RAG system?

Because a generator cannot ground an answer in a document it never received. If recall@10 is 80% and you pass ten chunks, then 20% of questions are ungroundable before the model is invoked, and prompt engineering cannot reach them.

State the ceiling precisely, because it holds under conditions worth naming. The model may still answer some of that 20% from its parametric knowledge, which is not grounded in anything you retrieved. But a fact may also appear in a document your relevance judgements did not happen to mark — and an answer supported by that document is grounded in retrieved evidence, judgements or no judgements. So recall@k bounds grounded answers when a single marked document is the only place the fact appears and the judgements are complete. With redundant evidence, incomplete judgements, or questions needing two passages at once, document recall is not that bound in either direction. Measure end-to-end answer accuracy separately for the number you actually want.

This is why the two stages want different metrics. Stage one is judged on not missing; stage two — the reranker, or the generator — is judged on choosing well from what arrived. Tuning stage one for precision@1 throws away the recall the second stage needed.

Note6. Do you need a vector database?

Under about 100k documents, no — exhaustive search over a NumPy array is a matrix multiply and it is exact. Up to roughly ten million, pgvector in the Postgres you already run gives you transactions, backups, joins and access control for free.

The thing that genuinely forces a dedicated store is rarely scale; it is filtered search — the top ten documents this user may see, from the last 90 days, in this language — which needs the filter and the vector search to cooperate. Weigh that against operating a second datastore with its own consistency and backup story.

19.10 Interview questions

On sourcing — the same caveat as Chapter 2. These recur in the two credible open sources this book uses, plus the ones this chapter’s own measurements make answerable.

Testing whether you understand the mechanism

Q: Write down BM25 and explain what k1 and b are doing.

The scoring function is IDF times a saturating term frequency, with a length correction:

\[ \text{BM25}(q,d) = \sum_{t \in q} \text{IDF}(t) \cdot \frac{f_{t,d}\,(k_1+1)}{f_{t,d} + k_1\left(1 - b + b\,\frac{|d|}{\text{avgdl}}\right)} \]

Here \(f_{t,d}\) is how often term \(t\) occurs in document \(d\), \(|d|\) is the document’s length and avgdl the average length in the collection. The IDF is usually \(\log\frac{N - n_t + 0.5}{n_t + 0.5}\) for \(N\) documents of which \(n_t\) contain \(t\) (Lucene adds 1 inside the log so it never goes negative). Common defaults are \(k_1 \approx 1.2\)–\(2.0\) and \(b = 0.75\).

The part worth saying out loud is why it saturates. In Chapter 2’s TF-IDF with raw counts the tenth occurrence of a term counts exactly as much as the first; here the term-frequency factor can never exceed \(k_1+1\), and \(k_1\) sets how fast it approaches that ceiling. With \(k_1 = 1.2\) (and ignoring length), a word used three times contributes 1.57 × IDF and a word used twenty times only 2.08 × IDF — a third more, not seven times more — so repetition cannot swamp a document that matches more of the query’s other terms. And \(b\) interpolates length normalisation: \(b=0\) ignores document length, \(b=1\) divides it out fully. If you add that sublinear_tf in scikit-learn is the crude version of the same idea, you have connected the two chapters.

Q: Bi-encoder or cross-encoder? Why not use the better one everywhere?

A bi-encoder embeds query and document independently, so documents can be encoded once, offline, and search is a vector operation. A cross-encoder reads the pair jointly and scores it, which is strictly more informative and cannot be precomputed — the cost is one forward pass per candidate at query time. That is why the architecture is always both: retrieve a shortlist cheaply, rerank it expensively. The follow-up to volunteer is where the shortlist saturates, because reranking 1,000 candidates rarely buys more than reranking 100 and costs ten times as much.

Q: What is the quality dial in an approximate index, and what is it set to right now?

efSearch in HNSW, nprobe in IVF — how much of the graph or how many clusters get visited. The point of the question is the second half: it has a default, the default was chosen by the library author and not by you, and it is a recall setting wearing a performance costume. An ANN index that silently returns 92% of the true neighbours has thrown away 8% of what exact search would have found, before the model sees anything. Measure approximate against exhaustive on a sample and you will know the number instead of assuming it.

Testing whether you have shipped one

Q: You are handed a corpus and no queries. How do you build an evaluation set?

Collect a hundred real queries — from logs, from support tickets, from the people who will use it — because invented queries are systematically cleaner than real ones and will flatter every system you test. Then pool: run two or three different retrievers, label the union of their top-10, and you have relevance judgements without reading the whole corpus. Say explicitly that a hundred labelled queries beats a thousand unlabelled ones, and that the set is worth more than any model choice you will make afterwards.

Q: A stakeholder wants semantic search to replace keyword search. What do you tell them?

That the answer is an empirical question about their queries, and then show them the two conditions from this chapter: on inflected keyword queries stemmed BM25 scored 96.0% recall@1 against a dense retriever’s 35.9%; on cross-lingual queries the same dense model scored 96.2% against BM25’s 23.8%. Same corpus, opposite verdicts. So the real deliverable is a census of their query mix, and if it is genuinely mixed, routing by query type beats fusing — this chapter measured RRF costing 32.8 and 46.8 recall@1 points against the better single retriever.

Q: Retrieval looks good but the answers are bad. Whose problem is it?

Do not guess — attribute. Check whether the supporting document was in the top-k at all: if it was not, this is a retrieval problem and no prompting fixes it; if it was, the model had what it needed and did not use it, which is a grounding problem. Chapter 20 builds that decomposition properly. The one-sentence version, and the reason this chapter exists before that one: recall@k is a ceiling on the whole system, so it is the first thing to measure and the last thing you can fix by tuning a prompt.

19.11 The 2026 toolbox

Everything in this chapter ran on one laptop with the versions below.

The stack these numbers came from

library	version used	what it is for here
sentence-transformers	6.0.0	both dense models, and the encoding in Section 19.3
snowballstemmer	3.1.1	the Turkish stemmer that is worth 68 points in condition A
numpy	2.5.2	BM25, exhaustive dense search, and RRF — the whole retrieval stack in this chapter is a few hundred lines

Worth knowing about, not used above:

The wider ecosystem, versions as of writing

library	current version	when you would reach for it
faiss-cpu	1.15.0	HNSW and IVF-PQ indexes (Johnson et al. 2019); the reference implementation, and where to sweep the recall/latency dial of Section 19.6
pgvector	0.5.0	vectors in the database you already run — the default answer for most projects; this is the Python client’s version, the PostgreSQL extension is versioned separately (0.8.x)
qdrant-client / chromadb	1.19.0 / 1.5.9	dedicated stores, when filtered search at scale becomes the requirement
rank-bm25	0.2.2	BM25 without writing it yourself, for a quick baseline
ir-measures	0.4.3	the standard IR metrics computed the standard way, so your nDCG matches everyone else’s definition of nDCG

19.12 Lab: build and tune a hybrid retriever

Build both families on your own queries, then check the fusion before you ship it rather than after.

Step 1 — collect a hundred real queries

From logs if you have them, from users if you do not. This is the step that decides everything else, because Section 19.3 is one long demonstration that the query distribution determines which retriever wins.

Classify them: exact-match (part numbers, names, codes), natural-language questions, and anything crossing a language boundary.

Step 2 — label relevance, then build BM25

For each query, the document that answers it. Fifty labelled queries is a workable starting point for error analysis, but do not read it as enough to confirm a ten-point gap. Do the arithmetic: ten points on fifty queries is five extra successes, and even in the most favourable split — five gained, none lost — a two-sided exact McNemar gives \(p = 2/2^5 = 0.0625\), which does not clear 5%. What it actually takes depends on the baseline rate, on how correlated the two systems’ errors are (a paired test on shared queries needs far fewer than an unpaired one), and on how many comparisons you plan to make. If the gap you care about is five points rather than ten, or the systems agree on almost everything, budget for several hundred. Then build BM25 — and if your language is morphologically rich, build it twice, with and without stemming, because Chapter 16 says that one step can be worth more than sixty points.

Step 3 — add dense retrieval, with a retrieval model

Not a similarity model. Encode, search exhaustively — under 100k documents you do not need an index yet — and report recall@1 and recall@10 next to BM25.

Step 4 — check whether fusing helps before shipping it

Run RRF and compare against the better parent, not against the weaker one. If the parents differ by more than about twenty points, expect fusion to hurt and verify. The twenty is a rule of thumb, not a measured threshold: the gaps measured in this chapter were sixty points and more. This is a fifteen-minute check, and the common belief that hybrid always wins suggests it is often skipped.

Step 5 — route instead, if your query mix is mixed

Classify the query and send it to the retriever that handles it. Measure the router’s accuracy separately from the retrievers’, because a router error and a retrieval error look identical in the final number and have different fixes.

Step 6 — add a reranker and find where the shortlist saturates

Cross-encoder over the top 10, 25, 50, 100. Plot final accuracy against latency. The shortlist size is a cleaner dial than any ANN parameter — but do not assume the curve only goes up. A larger pool raises the recall the reranker gets to work with, which is monotone; final accuracy is not, because each extra candidate is also another chance for the reranker to score a distractor above the right answer. Plot it rather than extrapolating it, and expect a plateau or a slight decline past some size.

What to hand in

One table of recall@1, recall@10 and nDCG@10 for every system on your own queries, broken down by query type; the fusion check from step 4 with its verdict; and a sentence naming the ceiling your first stage imposes on everything downstream.

19.13 Exercises

1. Find the swap. Build BM25 and a dense retriever on one corpus. Find three queries where each beats the other by a wide margin, and state the pattern in one sentence.

2. Reproduce the fusion result. Take two retrievers that differ by at least twenty points on your data and fuse them with RRF. Report recall@1 and recall@10 for both parents and the fusion. Explain the shape of the damage.

3. Weight the fusion. Repeat exercise 2 with a weighted combination instead of plain RRF, tuning the weight on held-out queries. How much of the loss do you recover, and what did the tuning cost you in generalisation?

4. Similarity model against retrieval model. Run both on an easy query set and a hard one. Report the gap in each. Then say which set you would have used if you had only built one.

5. The ANN dial. Sweep efSearch or nprobe, and plot recall against p95 latency — measuring recall against exhaustive search, not against your labels, so you separate index loss from embedding error.

6. Recall as a ceiling. Measure recall@k on your retriever for k = 1, 5, 10, 25. For each, state the maximum accuracy a downstream generator could achieve. Compare against what your RAG system actually achieves, and say where the remaining gap lives.

7. Query-type census. Sample 100 production queries and classify them. Predict, from this chapter’s tables, which retriever should win your mix. Then measure and see whether you were right.

19.14 Where to go deeper

Do this first: take twenty real queries and check whether the right document is in the top ten. That number is the ceiling on everything you build afterwards, and most teams have never measured it.

Beyond the papers

resource	what it is
Introduction to Information Retrieval — Manning, Raghavan, Schütze	free, and still the best foundation; the chapters on indexing and evaluation have not aged
Sentence Transformers	the practical guide to bi-encoders, cross-encoders and when each is appropriate
Pinecone Learn	vendor-published but genuinely good explainers on ANN indexes and hybrid search
faiss	the reference ANN library; its wiki explains the recall/latency dial better than most papers
ColBERT	late interaction, the middle path between the two architectures in Section 19.5

The primary sources

topic	source
BM25, derived rather than asserted	Robertson and Zaragoza (2009)
dense retrieval as it became standard	Karpukhin et al. (2020)
the multilingual retrieval model used here	Wang et al. (2024)
reranking with a cross-encoder	Nogueira and Cho (2019)
late interaction	Khattab and Zaharia (2020)
the fusion this chapter measured as harmful	Cormack et al. (2009)
approximate search that works	Malkov and Yashunin (2018); Johnson et al. (2019)
the benchmark that showed generalisation is hard	Thakur et al. (2021)

19.15 What to remember

• Neither retrieval family dominates, and the query type decides. Same corpus, two query conditions: stemmed BM25 96.0% against dense 35.9% on keyword queries; dense 96.2% against BM25 23.8% cross-lingually. A retrieval number without its query distribution is not interpretable.

• Unweighted RRF is not free, and it is not always positive. It cost 32.8 and 46.8 recall@1 points against the better parent in the two conditions, because rank fusion weights a rank-1 hit from a weak system like a rank-1 hit from a strong one. Note the word unweighted: rank fusion discards score magnitude, which is the only signal that separates a confident-and-right system from a confident-and-wrong one. Weighted RRF does not recover it: it still fuses ranks, and only scales each system’s contribution by a fixed weight. Score-normalised fusion (Bruch et al. 2024) uses the scores themselves. Each needs one fitted parameter, and neither was measured here.

• The damage lands on recall@1, not recall@10. In condition B, fusion took R@10 from 99.9% to 99.6% and R@1 from 96.2% to 49.4%. Which metric you watch determines whether you notice.

• Fuse comparable systems; route mixed query loads. If one retriever dominates on your queries, use it. If your users send both kinds, classify the query and dispatch — and evaluate the router separately.

• A similarity model is not a retrieval model. The two were 2.4 points apart on the easy condition and 25.3 apart on the hard one. Component quality only becomes visible on the inputs that are hard for that component, which is an argument for keeping a difficult slice in your evaluation.

• Hard negatives are the whole game in dense training. In-batch negatives teach topical separation; mined hard negatives teach the distinction your users need. Re-mine as the model improves, and mine slightly below the top of the list to avoid training against unlabelled relevant documents.

• Retrieve broadly, rerank precisely. A cross-encoder is usually more accurate and cannot be precomputed, so it reorders a shortlist. It can only reorder what stage one returned — the first stage sets a ceiling the second cannot raise.

• Report recall@k for the k you actually pass downstream. It bounds what any RAG system can ground: a generator cannot ground an answer in a document it never received. With one relevant document per question, 80% recall@10 means 20% of questions cannot be grounded in retrieved evidence before generation starts (the model may still answer some from memory, ungrounded).

• Keep the vectors where the data is until scale forces otherwise. Under 100k documents, exhaustive search is exact and instant. What eventually forces a dedicated store is filtered search at scale, not raw volume.

Bruch, Sebastian, Siyu Gai, and Amir Ingber. 2024. “An Analysis of Fusion Functions for Hybrid Retrieval.” ACM Transactions on Information Systems 42 (1): 1–35. https://doi.org/10.1145/3596512.

Cormack, Gordon V., Charles L. A. Clarke, and Stefan Buettcher. 2009. “Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods.” Proceedings of SIGIR.

Johnson, Jeff, Matthijs Douze, and Hervé Jégou. 2019. “Billion-Scale Similarity Search with GPUs.” IEEE Transactions on Big Data.

Karpukhin, Vladimir, Barlas Oğuz, Sewon Min, et al. 2020. “Dense Passage Retrieval for Open-Domain Question Answering.” Proceedings of EMNLP.

Khattab, Omar, and Matei Zaharia. 2020. “ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT.” Proceedings of SIGIR.

Malkov, Yu A., and Dmitry A. Yashunin. 2018. “Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs.” IEEE Transactions on Pattern Analysis and Machine Intelligence.

Nogueira, Rodrigo, and Kyunghyun Cho. 2019. “Passage Re-Ranking with BERT.” arXiv Preprint arXiv:1901.04085.

Robertson, Stephen, and Hugo Zaragoza. 2009. “The Probabilistic Relevance Framework: BM25 and Beyond.” Foundations and Trends in Information Retrieval 3 (4): 333–89. https://doi.org/10.1561/1500000019.

Thakur, Nandan, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. “BEIR: A Heterogeneous Benchmark for Zero-Shot Evaluation of Information Retrieval Models.” NeurIPS Datasets and Benchmarks Track.

Wang, Liang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. “Multilingual E5 Text Embeddings: A Technical Report.” arXiv Preprint arXiv:2402.05672.