29 Agentic RAG
Learning objectives. After this chapter you can:
- turn retrieval from a fixed pipeline step into a tool the model chooses;
- recognise the question shape that classic RAG structurally cannot answer;
- bound an iterative retrieval loop so it terminates and stays affordable;
- measure agentic RAG against classic RAG rather than assuming it is better.
29.1 From pipeline to policy
Chapter 20’s pipeline retrieves once, then generates. Every design decision — what to retrieve, how much, from where — is made by you, at build time, and the same thing happens on every request. That predictability is a feature, and it is why the pipeline is the right default.
Agentic RAG moves one decision from build time to run time: retrieval becomes a tool the model calls, as many times as it judges necessary. The retriever stops being stage 5 of Chapter 20’s pipeline and, in Chapter 23’s terms, becomes an entry in the tool list.
What that buys, precisely:
- A second query the first one could not have contained. This is the whole case, and Section 29.2 measures it.
- A sufficiency judgement. Do I have enough to answer? becomes an explicit step rather than an assumption.
- A choice of source. With several indexes, which one to ask is a decision the query itself can inform.
What it costs is what run-time decisions always cost, and Chapter 23 priced it: predictability, debuggability, a variable bill, and a loop whose termination is a model’s judgement rather than a condition.
What we will look at. First the one question shape a fixed pipeline structurally cannot answer, and a measurement of how badly — with a control that shows what the loop costs when the pipeline was fine all along. Then the two ways to get a second query, the rule for choosing between them, and the named designs (Self-RAG and others) you will meet in the literature. Then what happens with several indexes, where the tempting answer is the one Chapter 19 already measured losing. Then how to stop a loop whose exit condition is a model’s opinion. And finally how to compare the two architectures without accidentally measuring the case you built for.
29.2 The ceiling classic RAG cannot climb
Here is the question shape that decides everything in this chapter.
[document 1] ZRG-14's successor project has been designated KVT-22.
[document 2] KVT-22's approved budget is 4,382 lira.
question What is the budget of ZRG-14's successor project?The question contains ZRG-14. It does not contain KVT-22, because the asker does not know it — that is what they are asking.
Some vocabulary before going on. Each step from one document to the next is a hop, so this is a two-hop question, and questions that need two or more such steps are called multi-hop. KVT-22, the name that links the two documents, is the bridge entity: it is the answer to the first hop and the search key for the second.
So a retriever given this query finds document 1 easily and has nothing in the query that singles out document 2 from the other budget documents: the term needed to retrieve the second document only exists in the first. That is a strong disadvantage rather than a proof of impossibility — document 2 might still surface on some other shared term, through a dense model that puts the two projects near each other, from a larger k, or from an index that had linked or merged the pair in advance. What the query cannot do is express the second need.
This is not a weak retriever. Chapter 19’s measurements do not apply, because the issue is not ranking. It is that one query cannot express the information need. That is a structural ceiling on one query.
Note the size of what that rules out, because it is smaller than it looks. It rules out a single retrieval, not a fixed pipeline. You can write a fixed pipeline that breaks this example with no model judgement in it at all: retrieve, extract the named entities from what came back, retrieve again with those entities, generate. Two retrievals, two model calls, deterministic cost, guaranteed termination, no loop. That is query decomposition by bridge entity, and it is a real third option the choice below does not offer — so read what follows as “one query is not enough”, not as “you need a loop”.
Measured
Thirty two-hop questions of exactly that shape, with the facts planted among 660 documents of real Turkish news. And — because a chapter that only measures the case its subject wins is not measuring — thirty single-hop controls, where the question does contain the key.
qwen3.5:9b, 30 questions per cell, 660 documents, top-3 retrieval
question system accuracy 95% CI tokens
1-hop classic 100.0% [88.6, 100.0] 155
1-hop agentic 100.0% [88.6, 100.0] 179
2-hop classic 0.0% [ 0.0, 11.4] 158
2-hop agentic 30.0% [16.7, 47.9] 1,043
A rounds column has been withdrawn from this table. A *round* is one
retrieve-then-judge cycle and the budget is three, yet the recorded mean for
the agentic two-hop cell was 3.43: either the cap applied to model calls
rather than retrieval rounds, or the counter was wrong, and the run logs that
would say which were not kept. A number that breaks its own cap is not a
measurement.The two-hop classic bar has no length at all — 0 of 30 — and its interval reaches only 11.4%, so whatever the true rate is, it is small. Note how wide the agentic two-hop interval is: thirty questions puts the true value somewhere between 16.7% and 47.9%, which is enough to establish that the ceiling moved and not enough to call the system usable.
Three readings, and the third is the one worth arguing about.
This pipeline answered none of them. Classic RAG got 0 of 30 two-hop questions, from a retriever that scored 100% on the single-hop versions of the same facts. Read that as this pipeline’s measured rate, not as a proven zero for classic RAG in general: 0/30 has a 95% Wilson upper bound of 11.4%, and a different retriever, a larger k, or an index that pre-linked related documents is a different experiment. When people say a RAG system “cannot handle multi-hop questions”, this is the shape of the failure.
One check this experiment owes you and does not have: the rank of the second gold document. A zero has two possible causes with different consequences. If document 2 never entered the top 3, the failure is retrieval and no reranker over the same top-3 candidates addresses it — the reading above. If it sometimes did, and the model still could not join it to document 1, the failure is aggregation, a reranker is irrelevant for a different reason, and the honest sentence is a different one. Exercise 2 asks the reader to run exactly this check; the chapter should run it too. Until it does, read “structural” as the likely explanation rather than the established one.
On single-hop questions the loop buys nothing and costs 1.15×. Identical accuracy, and 15% more tokens (179 against 155) for the sufficiency check that always came back sufficient. In most deployments the majority of questions are single-hop — step 1 of the lab measures yours — so this is usually the case that dominates a real workload, and it is the case where the architecture is pure overhead.
Breaking the ceiling is not the same as clearing it. Agentic retrieval went from 0% to 30% — a 30-percentage-point gain from an observed baseline of zero, where the relative version is undefined rather than infinite — and a system that is still wrong seven times in ten. It cost 6.6× the tokens on that cell. Per correct answer that is about 3,475 tokens, against a classic pipeline that produces none at any price.
Agentic RAG removed the ceiling. It did not deliver a working system. The loop can now express the second query; the model frequently fails to formulate it, or fails to use what came back — which is Chapter 23’s finding again, where tool use was perfect and aggregation was not.
Small measurement, one model, thirty questions per cell, intervals around ±15 points. What it establishes is the ordering and the shape: the one-shot pipeline scored 0/30 on two-hop questions, iteration lifts that, cost multiplies, and the single-hop control loses. A larger model would very likely score higher than 30%. The cost and single-hop findings would not change, because neither is about model capability; the zero would hold too if the unrun rank check shows document 2 never reached the top 3.
What this means for what you build
The honest recommendation is a router rather than an architecture:
- Default to the classic pipeline. It wins on the majority case, on cost, on latency, and on debuggability.
- Detect the questions that need iteration — ones whose answer depends on a key the question does not contain — and route only those into the loop.
- Measure your own mix, and blend the numbers before believing them. The 6.6× is the two-hop cell, not your bill. At 5% two-hop traffic an all-agentic architecture costs 0.95 × 179 + 0.05 × 1,043 ≈ 222 tokens per request against the classic pipeline’s ≈ 155 — a blended 1.43×, for +1.5 accuracy points overall. At 20% two-hop it is ≈ 352 tokens, a blended 2.26×, for +6 points. The per-cell multiplier is the number that sells the architecture; the blended one is the number you pay.
- Then check whether you need the router at all. In this measurement the loop largely routed itself: on single-hop questions it spent only 24 tokens more than the pipeline, which fits a loop that rarely or never issued a second query. (The withdrawn round counter recorded 1.00 rounds there, but that counter is not trusted.) The 24 tokens are what a perfect router could save — about 10% of the blended bill at a 5% mix (0.95 × 24 ≈ 23 of 222 tokens), for identical accuracy. So the case for routing here is not cost. It is latency, debuggability and tail variance. And on accuracy an imperfect classifier can only lose: every two-hop question it misroutes into the pipeline scores zero rather than 30%, so its errors can cost more than its small token saving is worth.
29.3 Planning, and when to skip it
Two ways to get the second query, and they behave differently.
Plan first. Decompose the question into sub-questions before retrieving anything: which project succeeded ZRG-14 and what is that project’s budget. Clean when the sub-queries can be written up front. The two-hop example is the awkward case: the two sub-questions can be written, but the second one’s search query cannot, because it needs KVT-22, which only the first answer supplies. A plan with a variable in it (budget of <answer 1>) handles that, if something fills the variable in at run time; a plan of fixed strings does not.
Retrieve, then decide. Retrieve, look, and issue another query if the answer is not there. This handles the dependent case, which is the case that motivated the architecture. It is also the loop that can fail to terminate.
The rule that falls out: plan when the sub-questions are independent, iterate when they are dependent. Independent sub-questions are also the ones worth running in parallel, which is Chapter 28’s read-heavy case and the one where fan-out genuinely pays.
Sufficiency is the interesting step
The loop needs an answer to do I have enough?, and this is where agentic RAG is most fragile. Chapter 23 measured the general version of the problem: the loop’s exit is a model’s prediction, not a checked condition, and in a text protocol it failed to terminate on 60% of one-step tasks, having already retrieved the answer.
Two design choices should make the judgement more reliable, and neither is a more carefully worded version of the same question. (Neither was measured in this chapter; the first rests on Chapter 23’s result.)
- Make “answer” and “search again” structurally different actions, per Chapter 18 and Chapter 23 — a tool call versus content, not two strings competing in the same output.
- Ask for the missing key, not for a judgement. What do you still need? produces a retrievable term. Do you have enough? tends to produce a yes. The first is also a form of query rewriting: the model turns the gap it sees into the next search query, instead of re-sending the user’s words.
Named designs you will meet
The literature has given names to most of these moves, and it helps to map them onto this chapter’s vocabulary before reading it.
- Self-RAG (Asai et al. 2024) trains the model itself to emit special reflection tokens: one that says whether to retrieve at all at this point, and others that grade whether a retrieved passage is relevant and whether the generated text is supported by it and useful. It adds the decision this chapter’s loop never made — skip retrieval entirely — and it needs a fine-tuned model, not a prompt.
- Corrective RAG (CRAG) (Yan et al., 2024) puts a small trained evaluator after the retriever. When the retrieved documents look wrong, it falls back to web search; it also filters the documents it keeps. That is a sufficiency check done by a separate model rather than by the generator.
- IRCoT (Trivedi et al., 2023) alternates one sentence of step-by-step reasoning with one retrieval, using each new sentence as the next query — the retrieve-then-decide loop at its finest grain, built for multi-hop questions.
- Adaptive-RAG (Jeong et al., 2024) trains a small classifier to predict question complexity and routes each question to no retrieval, single-step retrieval or an iterative loop — the router recommended above, as a published design.
None of them changes this chapter’s accounting: each still needs the per-type accuracy and tokens-per-correct-answer comparison in Section 29.6 before it earns a place on your traffic.
29.4 Routing across sources
Everything so far has assumed one index. The third thing the loop buys was a choice of source, and it only becomes a choice when there is more than one.
With several — a document store, a ticket system, a code repository, a database — which one to ask becomes a decision, and there are three ways to make it.
| approach | how it works | when it fits |
|---|---|---|
| Search everything, merge | query all sources, fuse the results | few sources, similar content. But Chapter 19 measured what naive fusion can cost: fusing two retrievers of unequal quality with unweighted reciprocal rank fusion (RRF) lost 32.8 and 46.8 recall@1 points against the better one, while at recall@10 the same fusion did almost no damage (99.9% against 99.6% in the condition where it was measured), so for a pipeline that hands the model several chunks the harm depends on how much the first rank matters. That was two retrievers over one corpus, not several sources, but the lesson carries over: equal-weight fusion hurts when one input is much better |
| Route once | a classifier or the model picks the source, then retrieves | predictable and cheap; the right default when sources are clearly distinct |
| Let the agent choose | each source is a tool with its own description | genuinely mixed needs, and the case where Chapter 24’s rules decide whether it works — the description is what tells the model when each source applies |
One thing that is easy to lose when sources are merged: they do not have equal trust. An internal policy document and a scraped web page are not interchangeable evidence, and once their contents are concatenated into one context the distinction is gone.
So keep provenance on every retrieved chunk, pass it to the model, and require it in the answer — this is Chapter 26’s provenance argument arriving in a different system.
29.5 Bounding the loop
A retrieval loop whose exit is a model’s judgement needs external bounds, and this is not a tuning detail — Chapter 23’s step cap is an architectural component precisely because the alternative is unbounded.
Four bounds, all cheap, and the fourth is the one people skip:
- A round cap. Two or three. If the answer has not appeared, more rounds rarely find it and reliably cost more.
- A token budget for the whole request, checked between rounds. This is the one that protects you from a pathological question rather than a pathological average.
- A repeat check. If a query is one the loop has already issued, stop — Chapter 23 watched a loop re-issue the same call until its budget ran out, having had the answer since step 0.
- A floor on the answer — a minimum of evidence below which the system does not answer. When the cap is reached without an answer, say so. A loop that exhausts its budget and then answers anyway has converted a detectable failure into an undetectable one.
29.6 Evaluating one against the other
The bounds above keep the loop affordable. This section is about the prior question — whether it should exist at all on your traffic — which is decided by a comparison, not by a demo.
That comparison has four columns, and reporting fewer is how agentic RAG gets adopted on the strength of its best case.
| what to report | why it is not optional |
|---|---|
| Accuracy, split by question type | the aggregate hides everything. Here it would have averaged a 100% single-hop tie with a 0%-versus-30% multi-hop gap into a number describing neither |
| Tokens per correct answer | not tokens per request. A system that costs 6.6× and answers 30% of the time costs about 3,475 tokens per right answer; the per-request figure makes it look cheaper than it is |
| Rounds, and how often the cap was hit | report both, and define a round. A mean tells you the typical spend; the cap-hit rate tells you whether the loop stops when satisfied or runs out of budget, and those are different systems. The mean alone cannot distinguish them — and a counter that breaks its own cap, as this chapter’s withdrawn round count did, cannot be interpreted at all |
| Variance | a fixed pipeline costs the same every time. A loop does not, and a p99 that is ten times the median is an operational fact your budget has to survive |
And the sampling discipline from Section 22.3 applies with force here. Your eval set has to contain both question types, with enough of each to measure each one: with a real mix of 5% multi-hop, a representative set of 100 questions holds about five multi-hop ones, far too few for an interval. So stratify, as this chapter did (30 of each), report per type, and then reweight to your real proportions for the headline — the blending in Section 29.2. An eval built only from the multi-hop questions that motivated the project, and reported without reweighting, will show agentic RAG winning enormously on a distribution no user produces.
29.7 Intuition checks
Answer each one before you open it. Three are settled by the measurement in this chapter — the measured zero, what iteration recovers, and what it costs on the questions that were fine already — and the other three are design decisions, two of which lean on measurements from earlier chapters rather than on anything argued here.
Because the term needed to retrieve the second document exists only in the first. What is the budget of ZRG-14’s successor? contains ZRG-14 and not KVT-22 — the asker does not know it, that is what they are asking. Measured here, a retriever that scored 100% on single-hop questions scored 0% on the two-hop versions of the same facts. For this one-shot top-3 pipeline the likely cause is structural rather than ranking — the term that finds the second document exists only in the first — though the second document’s rank was not recorded, so a failure to join the two is not ruled out. Either way a reranker over the same candidates does not touch it. A different retriever, a larger k or a different index might find the second document on the first query; this experiment did not test those.
It removes the ceiling without clearing it. From 0% to 30% — a 30-percentage-point gain from an observed zero, and still wrong seven times in ten — at 6.6× the tokens on that cell, which is a blended 1.43× if two-hop questions are 5% of your traffic. The loop can now express the second query; the model often fails to formulate it or to use what came back, which is Chapter 23’s aggregation failure again.
1.15× for nothing. On single-hop questions both systems scored 100%, and the agentic version spent 15% more tokens on a sufficiency check that always came back sufficient. Since single-hop questions are usually the majority in production, this is the case that dominates a real workload — which is the argument for routing rather than replacing.
Plan when they are independent, iterate when they are dependent. Be careful with “cannot be planned”: the two-hop example can be written down as a plan with a variable in it — s = find_successor("ZRG-14"); get_budget(s) — and a system whose plans support bindings can express that fine. What cannot be written in advance is the literal second query, so a planner that emits fixed strings has to become a loop, and a planner with variables needs a runtime that fills them in. Independent sub-questions are also the ones worth running in parallel, which is Chapter 28’s read-heavy case.
Four bounds, and the fourth is the one people skip: a round cap of two or three; a token budget checked between rounds; a repeat check, because Chapter 23 watched a loop re-issue a query it already had the answer to until its budget ran out; and a floor on the answer — when the cap is reached without one, say so. A loop that exhausts its budget and answers anyway has converted a detectable failure into an undetectable one.
Route once if the sources are clearly distinct — cheap and predictable. Give the agent one tool per source when the needs are genuinely mixed, in which case Chapter 24’s rules decide whether it works, since the description is what says when each source applies. Search-everything-and-merge is the tempting option and Chapter 19 measured its cost in a related setting: fusing two retrievers of unequal quality lost 32.8 and 46.8 recall@1 points against the better one, though at recall@10 the loss was almost nil, so the cost depends on how much the first rank matters. And keep provenance: sources do not have equal trust, and concatenation destroys the distinction.
29.8 Interview questions
On sourcing — the same caveat as Chapter 2. These recur in the two credible open sources this book uses, plus the ones this chapter’s own measurement makes answerable.
Testing whether you understand the problem
Q: When does a RAG pipeline structurally fail?
When the query cannot contain the term needed to retrieve the answer — the multi-hop case, where the key lives in a document you have not retrieved yet. Say why it is structural rather than a ranking problem, because that is what rules out the usual fixes: measured here, a retriever at 100% on single-hop questions scored 0% on two-hop versions of the same facts. That fits the structural explanation, but it is not proof: the rank check that would rule out a join failure was not recorded. Then name the remedy: a loop that can issue a second query the first could not have contained.
Q: What does agentic RAG cost?
Predictability, and tokens. On the multi-hop case it took accuracy from 0% to 30% at 6.6× the tokens; on single-hop questions it bought nothing and cost 1.15×. The answer that shows judgement is the conclusion: route rather than replace, because the majority case is the one that pays the overhead.
Testing whether you have shipped one
Q: How would you evaluate agentic RAG against classic RAG?
Four columns, split by question type: accuracy, tokens per correct answer rather than per request, rounds with cap-hit rate, and variance. And the eval set has to contain enough of both question types to measure each, with the headline reweighted to production proportions — an eval built only from the multi-hop questions that motivated the project will show a huge win on a distribution no user produces.
Q: Your retrieval loop occasionally costs fifty times the median. What do you do?
Look at those requests before changing anything; they are usually one of two things. Either the loop is re-issuing queries it has already run, which a repeat check fixes, or the question is unanswerable from the corpus and the loop is searching until its budget ends — which needs a floor on the answer, so that exhausting the cap produces an explicit “not found” rather than a confident guess. Then bound the request, not just the round count.
29.9 The 2026 toolbox
Both conditions ran on one laptop with the versions below. Sixty questions against one local model is small enough to fit there, which is also why the intervals in Section 29.2 are as wide as they are.
| library | version used | what it is for here |
|---|---|---|
ollama |
0.24.0 | serving qwen3.5:9b for both conditions |
snowballstemmer |
3.1.1 | Turkish stemming for the BM25 index |
numpy |
2.5.2 | the retriever and the scoring |
Worth knowing about, not used above:
| library | current version | when you would reach for it |
|---|---|---|
llama-index |
0.14.24 | sub-question engines and routers already implemented |
langgraph |
1.2.11 | the loop as a graph, with the caps as edges you can see |
ragas |
0.4.3 | scoring retrieval and generation separately |
openai-agents |
0.22.0 | retrieval as a tool in a small readable loop |
29.10 Lab: route, do not replace
The title is the finding. Steps 1 and 2 exist to tell you whether the rest is worth doing on your traffic.
Step 1 — classify fifty real questions
Single-hop or multi-hop? The proportion is the most important number in this lab, and almost nobody knows it before measuring.
Step 2 — run the classic pipeline on all fifty
Accuracy by type, tokens, latency. Note which failures are structural — the answer’s key was not in the question — and which are ranking failures.
Step 3 — add the loop, capped at three rounds
Retrieval as a tool, an explicit “what do you still need” step, and the four bounds from Section 29.5.
Step 4 — measure both, by question type
Four columns. Report the single-hop overhead honestly; it is the case that dominates your traffic.
Step 5 — compute tokens per correct answer
Per type. This is the number that decides, and it is usually the first time the loop looks expensive rather than clever.
Step 6 — build the router
Send only the questions that need it into the loop. Re-measure the blend. If routing does not beat both pure architectures on your mix, say so.
What to hand in
The question-type proportion from step 1, the four-column table by type, the tokens-per-correct-answer figures, and one sentence on whether the loop earned its place on your traffic.
29.11 Exercises
- Find your proportion. Classify fifty real questions as single- or multi-hop. Predict the split first, then measure it.
- Test the zero. Construct twenty questions whose answer key is absent from the question. Measure your classic pipeline’s accuracy on them and report it with its interval — with three questions, 0/3 would still leave a 95% upper bound of 56%; with twenty, 0/20 leaves 16%. If it scores zero, check the second gold document’s rank to tell a retrieval failure from an aggregation failure.
- Cost per correct answer. Compute it for both architectures on both question types. Which comparison changes your mind?
- Find the knee. Plot accuracy against the round cap, from one to five. Where do additional rounds stop paying?
- Break the loop deliberately. Ask a question your corpus cannot answer. Does the loop stop and say so, or exhaust its budget and answer anyway?
- Route it. Build a classifier from the labels of exercise 1, send only multi-hop questions into the loop, and measure the blend against both pure systems.
- Trust levels. Take two sources with different reliability and check whether your answers carry provenance. If they do not, construct the case where that matters and show it failing.
29.12 Where to go deeper
Before reading: classify fifty of your own questions as single- or multi-hop. The proportion decides which half of this literature is relevant to you.
| resource | what it is |
|---|---|
ragas |
the retrieval/generation split scored separately, which is what the four-column comparison needs |
llama-index documentation |
query decomposition, routers and sub-question engines with the abstractions named |
langgraph |
the retrieval loop as an explicit graph, which makes the bounds visible |
| Chip Huyen’s blog | on when a retrieval architecture is the right product decision at all |
| Eugene Yan on LLM patterns | RAG variants placed among the other patterns, with trade-offs stated |
| topic | source |
|---|---|
| the original retrieval-augmented formulation | Lewis et al. (2020) |
| deciding whether to retrieve at all | Asai et al. (2024) |
| the map of the variants | Gao et al. (2023) |
| reasoning interleaved with acting | Yao et al. (2023) |
| what irrelevant retrieved context costs | Shi et al. (2023) |
| why re-checking without new evidence rarely fixes a wrong answer | Huang et al. (2024) |
29.13 What to remember
- Our one-shot top-3 pipeline answered none of the 30 two-hop questions. A retriever scoring 100% on single-hop questions scored 0% on two-hop versions of the same facts, most likely because the term needed for the second document exists only in the first (the second document’s rank was not recorded). That is a result of the tested setup, not a universal zero-success bound; a reranker over the same candidates does not fix it.
- Agentic retrieval removes the ceiling without clearing it. 0% to 30%, at 6.6× the tokens on that cell — about 3,475 tokens per correct answer, against a pipeline that produces none at any price. Blend it before you quote it: at a 5% two-hop mix the whole architecture costs 1.43×, not 6.6×.
- On single-hop questions the loop costs 1.15× and buys nothing. That is the case that dominates real traffic, which makes routing — not replacing — the conclusion.
- Plan independent sub-questions; iterate on — or bind variables for — dependent ones. The literal second query of a two-hop question cannot be written until the first is answered, but a plan with a variable in it can be. A fixed two-stage pipeline is the comparison this chapter did not run.
- Ask for the missing key, not for a judgement. What do you still need? produces a retrievable term; do you have enough? produces a yes.
- Bound the loop four ways: round cap, token budget, repeat check, and a floor on the answer. Without the last, a loop that exhausts its budget answers anyway and turns a detectable failure into an undetectable one.
- Report four columns, split by question type. Accuracy, tokens per correct answer, rounds with cap-hit rate, and variance — measure each type on enough questions, then weight the headline by your real question mix, not by the multi-hop cases that motivated the project.