FINDING NEEDLES · CHAPTER 14

Beyond the basic pipeline

The basic pipeline retrieves once and trusts whatever comes back. Put a check between the search and the answer, and most of its quiet mistakes become visible.

~15 min read7 FAQ lines, 5 questionsTested Python

What you'll learn

  • Why a retriever that always returns something is dangerous
  • How a grader sorts retrieval into correct, ambiguous, and incorrect
  • Two ways to retry: rewrite the query, or follow the missing piece to a second hop
  • When to skip retrieval entirely, and when to say “I don’t know”

Before you start

The clerk who reads the page first

Picture the help desk at a bus station. A rider asks a question. The clerk flips the FAQ binder to the page that looks right. A careless clerk reads that page aloud, whatever it says. A good clerk glances at it first.

If the page answers the question, the clerk reads it. If it is close, the clerk tries other words, or looks up the one missing fact and turns to a second page. If nothing fits, the clerk says so. And if the rider asks what 9 times 4 is, the clerk does not open the binder at all.

The basic RAG pipeline is the careless clerk. Everything in this chapter is one decision added around retrieval: should I search, is what I found good enough, and if not, what do I try next.

A bus FAQ, and a pipeline that always answers

The whole knowledge base is seven lines. Each line is one chunk.

read first · multi-hop · agentic · cite

MANIM Advanced RAG: multi-hop, compress, rewrite, agentic loops.
ChunkText
1A day pass costs 5 dollars and covers every bus and tram.
2A bike goes on the front rack of any day bus.
3An owl bus runs every 30 minutes from midnight to 5 am.
4A folding bike may ride inside an owl bus.
5Route 12 stops at the museum.
6The last departure on route 7 is 22:15.
7The last departure on route 12 is 23:40.

The retriever is deliberately simple. Drop stop words, count how many query words each chunk contains, and return the top chunk. Ties go to the lower id. Like a vector index, it always returns a chunk, even when nothing matches. The basic pipeline then answers from that chunk, no questions asked.

QuestionBasic pipeline usesRight?
How much is a day pass?chunk 1yes
Can I take my bike on the night bus?chunk 2 (day-bus rack)no
Are dogs allowed on the tram?chunk 1 (day pass)no, the FAQ does not say
What is 9 times 4?chunk 1 (day pass)no search was needed
When is the last departure to the museum?chunk 6 (route 7, 22:15)no, it is 23:40

One right answer out of five. Worse, nothing in the output tells you which four are wrong. A generator handed chunk 6 will write “the last departure is 22:15” with total confidence, and cite it.

Correct, ambiguous, incorrect

The first fix is a grader that looks at the query and the retrieved chunk and says how well they fit. Here the grader is coverage: the share of the query’s words that the chunk contains.

of the words the rider used, what fraction does this chunk actually mention.

Two thresholds turn the number into a verdict. At 0.7 or more the chunk is correct. Below 0.4 it is incorrect. In between it is ambiguous. Each verdict triggers a different action.

Correct (≥ 0.7)

Answer from the chunk. “How much is a day pass?” keeps day and pass. Chunk 1 has both, coverage 1.000.

Ambiguous (0.4 to 0.7)

Something fits, something is missing. Retry with a rewritten query or a second hop. If the retry still does not reach correct, answer but hedge.

Incorrect (< 0.4)

Retry. If nothing improves, abstain. “Dogs allowed on the tram” finds only tram in chunk 1, coverage 0.333.

This is the core move of Corrective RAG (CRAG). Its grader is a small trained model, not word overlap, and on an incorrect verdict it goes to web search instead of a glossary. The shape is the same: a cheap judge sits between retrieval and generation and decides whether the evidence is trusted, repaired, or replaced.

Rewrite the words, or follow the missing piece

An ambiguous grade is useful because it says what is missing. Two retries use that.

Rewrite: the rider’s words are not the FAQ’s words

  1. “Can I take my bike on the night bus?” becomes take, bike, night, bus. Chunks 2 and 4 both contain bike and bus. Coverage 0.500, a tie, and chunk 2 wins on the lower id. Verdict: ambiguous.
  2. The FAQ never says night. It says owl. A glossary rewrites night to owl. Real systems ask a model to rewrite the query, as in chapter 11.
  3. Retrieve again with take, bike, owl, bus. Chunk 4 now has bike, owl and bus. Coverage 0.750, verdict correct. The answer is the folding-bike rule, not the day-bus rack.

Second hop: the question needs two chunks

“When is the last departure to the museum?” has no glossary fix. The question names a place. The timetable names routes. No single chunk links the museum to a time.

  1. Hop 1 retrieves last, departure, museum. Chunk 6 has last and departure, coverage 0.667, ambiguous. It is the route 7 line, so trusting it gives 22:15.
  2. The missing word is museum. Search for it alone. Chunk 5 answers: route 12 stops at the museum. That chunk is the bridge.
  3. Build the hop 2 query from what was covered plus the bridge: last, departure, route, 12, stops. Chunk 7 has four of the five. Coverage 0.800, correct. The answer is 23:40.
question → termsno FAQ word? skip retrieval
retrieve top chunk
grade coveragecorrect · ambiguous · incorrect
not correct: rewrite and retry, then follow the missing piecekeep whichever chunk grades higher
final grade: answer · hedge · abstain

The question about dogs gains nothing from either retry. The glossary has no entry for dogs or allowed, so there is nothing to rewrite, and searching for those two missing words finds no chunk at all. The final grade stays incorrect, so the controller abstains: the FAQ does not say. That is a better answer than a confident paragraph about day passes.

Some questions should never reach the index

“What is 9 times 4?” has the terms 9, times, 4. None of them appears anywhere in the FAQ. The basic pipeline still retrieves, gets chunk 1 at coverage 0, and stuffs a day-pass line into the prompt. At best that wastes time. At worst the generator tries to use it.

The controller checks first. If no query word is a word the FAQ uses, it skips retrieval and lets the model answer on its own. Greetings and thanks fall out the same way.

That rule is crude. Two published ideas do it properly. Self-RAG trains the model to emit a special token that means “retrieve now”, and more tokens that grade whether a passage is relevant and whether the answer is supported by it. Adaptive-RAG trains a small classifier that sends each question to no retrieval, one retrieval, or a multi-step loop, by how complex it looks. Both replace a fixed “always retrieve once” with a decision.

Why not grade with the generator itself?

You can. Self-RAG does exactly that: the same model that writes the answer also writes the critique tokens. The cost is training data with those tokens in it. A separate small grader, as in CRAG, can be trained or swapped on its own, and it adds one cheap call per retrieval. Most teams start with a separate grader, often a reranker score with a threshold, because it is easier to test.

Five questions, one trace each

Interactive · Basic pipeline versus controller

Which step changes the answer? Watch the coverage before and after each retry.

QuestionStepsFinal coverageOutcome
day passretrieve1.000answer, chunk 1
bike on night busretrieve, rewrite0.500 → 0.750answer, chunk 4
dogs on tramretrieve, hop search finds nothing0.333abstain
9 times 4none—no retrieval
last to museumretrieve, hop, retrieve0.667 → 0.800answer, chunk 7

Four answers changed. Across the five questions the controller made four extra index calls and skipped one. That is the trade: a little latency on the hard questions, in exchange for not being confidently wrong on them.

The controller you can re-run

corrective_rag.py4 of 5 answers fixed
"""A retrieval controller over a seven-line city bus FAQ.

The basic pipeline always retrieves one chunk and trusts it.
The controller grades what came back, then answers, rewrites and
retries, follows a missing piece to a second hop, or abstains.
It can also decide that a question needs no retrieval at all.
Every part is a toy rule so you can check it by hand.
"""

FAQ = {
    1: "A day pass costs 5 dollars and covers every bus and tram.",
    2: "A bike goes on the front rack of any day bus.",
    3: "An owl bus runs every 30 minutes from midnight to 5 am.",
    4: "A folding bike may ride inside an owl bus.",
    5: "Route 12 stops at the museum.",
    6: "The last departure on route 7 is 22:15.",
    7: "The last departure on route 12 is 23:40.",
}

STOP = {
    "a", "an", "the", "and", "of", "on", "at", "to", "is", "are", "can",
    "i", "my", "what", "when", "how", "much", "does", "any", "from",
}

# Riders say "night bus". The FAQ says "owl bus". A real system asks a model to rewrite.
GLOSSARY = {"night": "owl", "ticket": "pass"}

CORRECT_AT = 0.7     # coverage at or above this: trust the chunk
INCORRECT_BELOW = 0.4  # coverage below this: the chunk is wrong


def terms(text):
    """Lowercase, keep letters, digits and ':', drop stop words."""
    cleaned = "".join(ch if ch.isalnum() or ch == ":" else " " for ch in text.lower())
    return [word for word in cleaned.split() if word not in STOP]


VOCAB = {word for text in FAQ.values() for word in terms(text)}


def retrieve(query_terms):
    """Top-1 chunk by how many query terms it contains. Ties go to the lower id.

    Like a vector index, it always returns something, even at zero overlap.
    Coverage is the share of query terms found in the chunk.
    """
    best_id, best_hits = None, -1
    for doc_id in sorted(FAQ):
        words = set(terms(FAQ[doc_id]))
        hits = sum(1 for term in query_terms if term in words)
        if hits > best_hits:
            best_id, best_hits = doc_id, hits
    return best_id, best_hits / len(query_terms)


def grade(coverage):
    """The retrieval grader. A real one is a small trained model."""
    if coverage >= CORRECT_AT:
        return "correct"
    if coverage < INCORRECT_BELOW:
        return "incorrect"
    return "ambiguous"


def needs_retrieval(query_terms):
    """Skip the index when no query term is a word the FAQ uses."""
    return any(term in VOCAB for term in query_terms)


def rewrite(query_terms):
    """Swap rider words for FAQ words."""
    return [GLOSSARY.get(term, term) for term in query_terms]


def basic_pipeline(question):
    """Always retrieve, always answer from the top chunk."""
    doc_id, _ = retrieve(terms(question))
    return doc_id


def controller(question):
    """Return (trace, outcome, doc_id). Outcome: answer, hedge, abstain, or no retrieval."""
    query = terms(question)
    if not needs_retrieval(query):
        return [], "no retrieval", None

    trace = []
    doc_id, coverage = retrieve(query)
    trace.append(("retrieve", query, doc_id, coverage))

    if grade(coverage) != "correct":
        new_query = rewrite(query)
        if new_query != query:
            new_id, new_cov = retrieve(new_query)
            trace.append(("rewrite", new_query, new_id, new_cov))
            if new_cov > coverage:
                query, doc_id, coverage = new_query, new_id, new_cov

    if grade(coverage) != "correct":
        found = set(terms(FAQ[doc_id]))
        covered = [term for term in query if term in found]
        missing = [term for term in query if term not in found]
        bridge_id, bridge_cov = retrieve(missing)
        if bridge_cov > 0:
            bridge = [w for w in terms(FAQ[bridge_id]) if w not in missing]
            hop_query = covered + bridge
            hop_id, hop_cov = retrieve(hop_query)
            trace.append(("hop", missing, bridge_id, bridge_cov))
            trace.append(("retrieve", hop_query, hop_id, hop_cov))
            if hop_cov > coverage:
                doc_id, coverage = hop_id, hop_cov

    verdict = grade(coverage)
    outcome = {"correct": "answer", "ambiguous": "hedge", "incorrect": "abstain"}[verdict]
    return trace, outcome, (None if outcome == "abstain" else doc_id)


QUESTIONS = [
    "How much is a day pass?",
    "Can I take my bike on the night bus?",
    "Are dogs allowed on the tram?",
    "What is 9 times 4?",
    "When is the last departure to the museum?",
]

Mistakes learners make

Trusting top-k because it is top-k. A nearest-neighbour search always returns k results. Rank 1 can still be wrong. “9 times 4” got chunk 1 at coverage 0.

Retrying forever. Every retry is another index call and maybe another model call. Cap the retries, keep the best chunk seen so far, and accept “I don’t know” as a valid result.

Tuning thresholds by feel. 0.7 and 0.4 fit seven lines. On your data, pick them from a labelled set of questions, and measure how often each verdict is right. That is the next chapter.

Grading the retrieval and forgetting the answer. A correct chunk does not guarantee a faithful answer. The generator can still add facts the chunk never said. Grade both.

What to remember

  • The basic pipeline got 1 of 5 questions right, and gave no sign which ones were wrong.
  • A grader between retrieval and generation sorts each result: correct at 0.7 or more, incorrect below 0.4, ambiguous in between.
  • Rewriting night to owl lifted coverage from 0.500 to 0.750. A second hop through the museum chunk lifted it from 0.667 to 0.800 and fixed 22:15 to 23:40.
  • If no retry helps, abstain. If the question needs no documents, do not retrieve.

Check yourself

1. Why did the basic pipeline pick chunk 2 for the night-bus question?

Chunks 2 and 4 both matched bike and bus, a tie at 0.500. The tie went to the lower id. The word that would split them, night, is not in the FAQ.

2. Why can’t a rewrite fix the museum question?

Nothing is misspelled or renamed. The answer genuinely lives in two chunks: one maps the museum to route 12, the other gives route 12’s last time. You need two retrievals.

3. What does the controller do with “Are dogs allowed on the tram?”

It retrieves chunk 1 at 0.333, which is incorrect. Neither retry finds anything better, so it abstains.

4. Why skip retrieval for “What is 9 times 4?”

None of its words is in the FAQ, so any retrieved chunk would be noise. The model can answer alone.

Go to the source

Keep going