FINDING NEEDLES · CHAPTER 14
The basic pipeline retrieves once and trusts whatever comes back. Put a check between the search and the answer, and most of its quiet mistakes become visible.
Picture the help desk at a bus station. A rider asks a question. The clerk flips the FAQ binder to the page that looks right. A careless clerk reads that page aloud, whatever it says. A good clerk glances at it first.
If the page answers the question, the clerk reads it. If it is close, the clerk tries other words, or looks up the one missing fact and turns to a second page. If nothing fits, the clerk says so. And if the rider asks what 9 times 4 is, the clerk does not open the binder at all.
The basic RAG pipeline is the careless clerk. Everything in this chapter is one decision added around retrieval: should I search, is what I found good enough, and if not, what do I try next.
The whole knowledge base is seven lines. Each line is one chunk.
read first · multi-hop · agentic · cite
| Chunk | Text |
|---|---|
| 1 | A day pass costs 5 dollars and covers every bus and tram. |
| 2 | A bike goes on the front rack of any day bus. |
| 3 | An owl bus runs every 30 minutes from midnight to 5 am. |
| 4 | A folding bike may ride inside an owl bus. |
| 5 | Route 12 stops at the museum. |
| 6 | The last departure on route 7 is 22:15. |
| 7 | The last departure on route 12 is 23:40. |
The retriever is deliberately simple. Drop stop words, count how many query words each chunk contains, and return the top chunk. Ties go to the lower id. Like a vector index, it always returns a chunk, even when nothing matches. The basic pipeline then answers from that chunk, no questions asked.
| Question | Basic pipeline uses | Right? |
|---|---|---|
| How much is a day pass? | chunk 1 | yes |
| Can I take my bike on the night bus? | chunk 2 (day-bus rack) | no |
| Are dogs allowed on the tram? | chunk 1 (day pass) | no, the FAQ does not say |
| What is 9 times 4? | chunk 1 (day pass) | no search was needed |
| When is the last departure to the museum? | chunk 6 (route 7, 22:15) | no, it is 23:40 |
One right answer out of five. Worse, nothing in the output tells you which four are wrong. A generator handed chunk 6 will write “the last departure is 22:15” with total confidence, and cite it.
The first fix is a grader that looks at the query and the retrieved chunk and says how well they fit. Here the grader is coverage: the share of the query’s words that the chunk contains.
of the words the rider used, what fraction does this chunk actually mention.
Two thresholds turn the number into a verdict. At 0.7 or more the chunk is correct. Below 0.4 it is incorrect. In between it is ambiguous. Each verdict triggers a different action.
Answer from the chunk. “How much is a day pass?” keeps day and pass. Chunk 1 has both, coverage 1.000.
Something fits, something is missing. Retry with a rewritten query or a second hop. If the retry still does not reach correct, answer but hedge.
Retry. If nothing improves, abstain. “Dogs allowed on the tram” finds only tram in chunk 1, coverage 0.333.
This is the core move of Corrective RAG (CRAG). Its grader is a small trained model, not word overlap, and on an incorrect verdict it goes to web search instead of a glossary. The shape is the same: a cheap judge sits between retrieval and generation and decides whether the evidence is trusted, repaired, or replaced.
An ambiguous grade is useful because it says what is missing. Two retries use that.
“When is the last departure to the museum?” has no glossary fix. The question names a place. The timetable names routes. No single chunk links the museum to a time.
The question about dogs gains nothing from either retry. The glossary has no entry for dogs or allowed, so there is nothing to rewrite, and searching for those two missing words finds no chunk at all. The final grade stays incorrect, so the controller abstains: the FAQ does not say. That is a better answer than a confident paragraph about day passes.
“What is 9 times 4?” has the terms 9, times, 4. None of them appears anywhere in the FAQ. The basic pipeline still retrieves, gets chunk 1 at coverage 0, and stuffs a day-pass line into the prompt. At best that wastes time. At worst the generator tries to use it.
The controller checks first. If no query word is a word the FAQ uses, it skips retrieval and lets the model answer on its own. Greetings and thanks fall out the same way.
That rule is crude. Two published ideas do it properly. Self-RAG trains the model to emit a special token that means “retrieve now”, and more tokens that grade whether a passage is relevant and whether the answer is supported by it. Adaptive-RAG trains a small classifier that sends each question to no retrieval, one retrieval, or a multi-step loop, by how complex it looks. Both replace a fixed “always retrieve once” with a decision.
You can. Self-RAG does exactly that: the same model that writes the answer also writes the critique tokens. The cost is training data with those tokens in it. A separate small grader, as in CRAG, can be trained or swapped on its own, and it adds one cheap call per retrieval. Most teams start with a separate grader, often a reranker score with a threshold, because it is easier to test.
Interactive · Basic pipeline versus controller
Which step changes the answer? Watch the coverage before and after each retry.
| Question | Steps | Final coverage | Outcome |
|---|---|---|---|
| day pass | retrieve | 1.000 | answer, chunk 1 |
| bike on night bus | retrieve, rewrite | 0.500 → 0.750 | answer, chunk 4 |
| dogs on tram | retrieve, hop search finds nothing | 0.333 | abstain |
| 9 times 4 | none | — | no retrieval |
| last to museum | retrieve, hop, retrieve | 0.667 → 0.800 | answer, chunk 7 |
Four answers changed. Across the five questions the controller made four extra index calls and skipped one. That is the trade: a little latency on the hard questions, in exchange for not being confidently wrong on them.
"""A retrieval controller over a seven-line city bus FAQ.
The basic pipeline always retrieves one chunk and trusts it.
The controller grades what came back, then answers, rewrites and
retries, follows a missing piece to a second hop, or abstains.
It can also decide that a question needs no retrieval at all.
Every part is a toy rule so you can check it by hand.
"""
FAQ = {
1: "A day pass costs 5 dollars and covers every bus and tram.",
2: "A bike goes on the front rack of any day bus.",
3: "An owl bus runs every 30 minutes from midnight to 5 am.",
4: "A folding bike may ride inside an owl bus.",
5: "Route 12 stops at the museum.",
6: "The last departure on route 7 is 22:15.",
7: "The last departure on route 12 is 23:40.",
}
STOP = {
"a", "an", "the", "and", "of", "on", "at", "to", "is", "are", "can",
"i", "my", "what", "when", "how", "much", "does", "any", "from",
}
# Riders say "night bus". The FAQ says "owl bus". A real system asks a model to rewrite.
GLOSSARY = {"night": "owl", "ticket": "pass"}
CORRECT_AT = 0.7 # coverage at or above this: trust the chunk
INCORRECT_BELOW = 0.4 # coverage below this: the chunk is wrong
def terms(text):
"""Lowercase, keep letters, digits and ':', drop stop words."""
cleaned = "".join(ch if ch.isalnum() or ch == ":" else " " for ch in text.lower())
return [word for word in cleaned.split() if word not in STOP]
VOCAB = {word for text in FAQ.values() for word in terms(text)}
def retrieve(query_terms):
"""Top-1 chunk by how many query terms it contains. Ties go to the lower id.
Like a vector index, it always returns something, even at zero overlap.
Coverage is the share of query terms found in the chunk.
"""
best_id, best_hits = None, -1
for doc_id in sorted(FAQ):
words = set(terms(FAQ[doc_id]))
hits = sum(1 for term in query_terms if term in words)
if hits > best_hits:
best_id, best_hits = doc_id, hits
return best_id, best_hits / len(query_terms)
def grade(coverage):
"""The retrieval grader. A real one is a small trained model."""
if coverage >= CORRECT_AT:
return "correct"
if coverage < INCORRECT_BELOW:
return "incorrect"
return "ambiguous"
def needs_retrieval(query_terms):
"""Skip the index when no query term is a word the FAQ uses."""
return any(term in VOCAB for term in query_terms)
def rewrite(query_terms):
"""Swap rider words for FAQ words."""
return [GLOSSARY.get(term, term) for term in query_terms]
def basic_pipeline(question):
"""Always retrieve, always answer from the top chunk."""
doc_id, _ = retrieve(terms(question))
return doc_id
def controller(question):
"""Return (trace, outcome, doc_id). Outcome: answer, hedge, abstain, or no retrieval."""
query = terms(question)
if not needs_retrieval(query):
return [], "no retrieval", None
trace = []
doc_id, coverage = retrieve(query)
trace.append(("retrieve", query, doc_id, coverage))
if grade(coverage) != "correct":
new_query = rewrite(query)
if new_query != query:
new_id, new_cov = retrieve(new_query)
trace.append(("rewrite", new_query, new_id, new_cov))
if new_cov > coverage:
query, doc_id, coverage = new_query, new_id, new_cov
if grade(coverage) != "correct":
found = set(terms(FAQ[doc_id]))
covered = [term for term in query if term in found]
missing = [term for term in query if term not in found]
bridge_id, bridge_cov = retrieve(missing)
if bridge_cov > 0:
bridge = [w for w in terms(FAQ[bridge_id]) if w not in missing]
hop_query = covered + bridge
hop_id, hop_cov = retrieve(hop_query)
trace.append(("hop", missing, bridge_id, bridge_cov))
trace.append(("retrieve", hop_query, hop_id, hop_cov))
if hop_cov > coverage:
doc_id, coverage = hop_id, hop_cov
verdict = grade(coverage)
outcome = {"correct": "answer", "ambiguous": "hedge", "incorrect": "abstain"}[verdict]
return trace, outcome, (None if outcome == "abstain" else doc_id)
QUESTIONS = [
"How much is a day pass?",
"Can I take my bike on the night bus?",
"Are dogs allowed on the tram?",
"What is 9 times 4?",
"When is the last departure to the museum?",
]
Trusting top-k because it is top-k. A nearest-neighbour search always returns k results. Rank 1 can still be wrong. “9 times 4” got chunk 1 at coverage 0.
Retrying forever. Every retry is another index call and maybe another model call. Cap the retries, keep the best chunk seen so far, and accept “I don’t know” as a valid result.
Tuning thresholds by feel. 0.7 and 0.4 fit seven lines. On your data, pick them from a labelled set of questions, and measure how often each verdict is right. That is the next chapter.
Grading the retrieval and forgetting the answer. A correct chunk does not guarantee a faithful answer. The generator can still add facts the chunk never said. Grade both.
Chunks 2 and 4 both matched bike and bus, a tie at 0.500. The tie went to the lower id. The word that would split them, night, is not in the FAQ.
Nothing is misspelled or renamed. The answer genuinely lives in two chunks: one maps the museum to route 12, the other gives route 12’s last time. You need two retrievals.
It retrieves chunk 1 at 0.333, which is incorrect. Neither retry finds anything better, so it abstains.
None of its words is in the FAQ, so any retrieved chunk would be noise. The model can answer alone.