FINDING NEEDLES · CHAPTER 12

RAG fundamentals

A model cannot know the rules of your garden. Hand it the right page, tell it to answer from that page, and make it say which page it used.

~15 min read6 passagesTested Python

What you'll learn

  • The three steps of retrieval-augmented generation: retrieve, pack, generate
  • How a context window turns into a token budget for sources
  • How numbered sources let you check every citation
  • How to tell a retrieval miss from a generator that ignored its sources

Before you start

The binder by the shed

A community garden keeps its rules in a binder by the tool shed. When a new member asks when the water comes on, nobody recites the answer from memory. They open the binder, find the page, read it out, and point at it.

Retrieval-augmented generation (RAG) does the same with a language model. A search step finds a few passages. A packing step fits them into the prompt. The model writes an answer from those passages and cites them.

Retrievetop passages
Packfit the budget
Generateanswer + [n]

The model is the reader, not the library. Everything the answer is allowed to say must be in the packed sources. The citation is how you check that it was.

Six passages, one question

The handbook is already split into six short passages. (How to split it is the next chapter.) The question is “When do the water taps open?”

binder · chunks · ground · eval

MANIM RAG fundamentals: binder in context; cite evidence.
IdPassageTokensShared words
1Plot fees are 30 dollars per season, paid by April 1.110
2The water taps open on May 1 and close on October 15.123
3Compost bins sit by the north gate. Add leaves and peelings, never meat.130
4Tools live in the shed. Sign a tool out and return it by dusk.140
5Coil the hose after use and turn the water tap off at the wall.141
6New members wait about two weeks for a free plot.100

The toy retriever drops stop words and counts shared content words. The question keeps water, taps and open. Passage 2 shares all three. Passage 5 shares only water, because “tap” is not “taps”. The retriever keeps the top 3 with a score above 0, so the hit list is passage 2 (score 3), then passage 5 (score 1).

A real system puts BM25, a dense retriever, or both with a reranker here. Chapters 3 to 10 built those. RAG does not care which one you use. It only needs a short ranked list.

The window is a budget

The prompt has four parts that share one context window: the instructions, the sources, the question, and room for the answer. The instructions and the question are fixed. The answer needs room you reserve in advance. What is left is the source budget.

window, minus instruction tokens, minus question tokens, minus the answer reserve, leaves the tokens you may spend on sources.

The toy tokenizer counts words and numbers, so the window is 64 tokens. A real window holds many thousands, and a real tokenizer splits words into smaller pieces, so the true counts are higher. The subtraction is the same.

  1. Passage 2 costs 12 tokens. 12 ≤ 23, so it goes in. Used: 12.
  2. Passage 5 costs 14 tokens. 12 + 14 = 26 > 23, so it is skipped.
  3. The list is done. One passage packed, 12 of 23 tokens used.

Pack in rank order, so the best passage goes first. If a passage does not fit, skip it and try the next one. Never cut a passage in the middle to make it fit: half a rule is worse than no rule.

Numbered sources, cited answer

The prompt numbers each packed passage by its slot. The instructions say to use only those sources, to cite them, and to say so when they do not answer.

Answer using only the numbered sources. Cite a source like [1]. ... [1] The water taps open on May 1 and close on October 15. Question: When do the water taps open?

No model is called in this chapter. The generator is a stub: it copies the source sentence that shares the most content words with the question, and cites its slot. If no sentence shares at least 2 words, it says “The sources do not say.” A real model writes freer prose, but it is asked to do the same job.

The stub answers: The water taps open on May 1 and close on October 15. [1]

Now you can check it. Slot [1] is passage 2. Every word of the claim appears in passage 2. The answer is grounded: supported by the source it cites.

a toy support test: the claim may only use words that its cited source contains. Real checkers use a model to judge entailment, but they ask the same question.

Wrong answer: whose fault?

A RAG answer can go wrong in two places. Either the right passage never reached the prompt, or it did and the generator did not use it. The fix is different, so find out which.

Retrieval miss

Question: “When can I start watering my plot?” The content words are start, watering and plot. Passage 2 says “water”, not “watering”, so it scores 0. The hits are passages 1 and 6, which only share “plot”.

Both fit: 11 + 10 = 21 tokens of a 22-token budget. No sentence shares 2 words, so the stub says The sources do not say. That is the honest answer to a bad context.

Generation ignored the context

Same question as before, same packed passage 2. This generator answers from an old memory: The water taps open on April 15. [1]

The citation looks fine. The check is not: “april” is not in passage 2. The claim borrowed the number 15 from the source and the month from nowhere.

CasePassage 2 packed?Answer supported?Verdict
“water taps open”, stubyesyesgrounded
“start watering”, stubnosays it cannot answerretrieval miss
“water taps open”, from memoryyesnogeneration ignored the context

A retrieval miss is fixed upstream: rewrite or expand the query (chapter 11), add a dense retriever that knows watering means water (chapters 7 and 10), or retrieve more. A generation failure is fixed downstream: a stronger instruction, a better model, or a check that rejects unsupported answers. Chapter 14 builds that check into a retry loop.

The pipeline you can re-run

rag.pybudget 23 · grounded · retrieval miss
"""Retrieve, pack, generate: RAG on a community-garden handbook.

No model is called. The "generator" is a template that copies the
best-matching source sentence and cites it. That is enough to show the
pipeline and both ways it fails.
"""
import re

# Six handbook passages. Ids match the table on the page.
HANDBOOK = {
    1: "Plot fees are 30 dollars per season, paid by April 1.",
    2: "The water taps open on May 1 and close on October 15.",
    3: "Compost bins sit by the north gate. Add leaves and peelings, never meat.",
    4: "Tools live in the shed. Sign a tool out and return it by dusk.",
    5: "Coil the hose after use and turn the water tap off at the wall.",
    6: "New members wait about two weeks for a free plot.",
}

STOP = {"the", "a", "an", "do", "does", "when", "what", "how", "i", "my",
        "can", "to", "of", "on", "in", "and", "by", "at", "for", "is", "are"}

INSTRUCTIONS = ("Answer using only the numbered sources. Cite a source like [1]. "
                "If the sources do not answer, say so.")

WINDOW = 64          # toy context window, in tokens
ANSWER_RESERVE = 16  # tokens kept free for the answer
TOP_K = 3


def tokens(text):
    """Toy tokenizer: lowercase words and numbers. Real tokenizers split finer."""
    return re.findall(r"[a-z0-9]+", text.lower())


def content_words(text):
    return {t for t in tokens(text) if t not in STOP}


# ---- 1. Retrieve ---------------------------------------------------------

def retrieve(question, k=TOP_K):
    """Score = shared content words. Keep the top k with a score above 0."""
    query = content_words(question)
    scored = [(len(query & content_words(text)), pid) for pid, text in HANDBOOK.items()]
    scored = [(s, pid) for s, pid in scored if s > 0]
    scored.sort(key=lambda row: (-row[0], row[1]))
    return [(pid, s) for s, pid in scored[:k]]


# ---- 2. Pack -------------------------------------------------------------

def source_budget(question):
    """What is left for sources after instructions, question and answer."""
    used = len(tokens(INSTRUCTIONS)) + len(tokens(question)) + ANSWER_RESERVE
    return WINDOW - used


def pack(question, hits):
    """Add passages in rank order. Skip any passage that would overflow."""
    budget = source_budget(question)
    packed, used = [], 0
    for pid, _ in hits:
        cost = len(tokens(HANDBOOK[pid]))
        if used + cost <= budget:
            packed.append(pid)
            used += cost
    return packed, used


def build_prompt(question, packed):
    """Number the sources by slot. [1] means the first packed passage."""
    lines = [INSTRUCTIONS, ""]
    for slot, pid in enumerate(packed, start=1):
        lines.append(f"[{slot}] {HANDBOOK[pid]}")
    lines += ["", f"Question: {question}"]
    return "\n".join(lines)


# ---- 3. Generate (a stub, not a model) ----------------------------------

MIN_OVERLAP = 2  # below this, the stub says the sources do not answer


def generate(question, packed):
    """Copy the source sentence that shares the most words, and cite its slot."""
    query = content_words(question)
    best = (0, None, None)
    for slot, pid in enumerate(packed, start=1):
        for sentence in re.split(r"(?<=\.)\s+", HANDBOOK[pid]):
            overlap = len(query & content_words(sentence))
            if overlap > best[0]:
                best = (overlap, slot, sentence)
    overlap, slot, sentence = best
    if overlap < MIN_OVERLAP:
        return "The sources do not say."
    return f"{sentence} [{slot}]"


STALE_MEMORY = "The water taps open on April 15."


def generate_from_memory(question, packed):
    """A generator that ignores its sources and repeats an old fact."""
    return f"{STALE_MEMORY} [1]"


# ---- 4. Check the answer against what it cites --------------------------

def supported(answer, packed):
    """Every word of the claim must appear in the cited source."""
    found = re.search(r"\[(\d+)\]", answer)
    if not found:
        return False
    slot = int(found.group(1))
    if not 1 <= slot <= len(packed):
        return False
    claim = set(tokens(answer[:found.start()]))
    return claim <= set(tokens(HANDBOOK[packed[slot - 1]]))


def diagnose(question, gold, generator=generate):
    """Name the failure: was the right passage packed, and was it used?"""
    packed, _ = pack(question, retrieve(question))
    answer = generator(question, packed)
    if gold not in packed:
        verdict = "retrieval miss"
    elif supported(answer, packed):
        verdict = "grounded"
    else:
        verdict = "generation ignored the context"
    return answer, verdict

Mistakes learners make

Forgetting the answer reserve. If the sources fill the whole window, the model has no room to answer and stops mid-sentence. Subtract the reserve before you pack, not after.

Reading [n] as a passage id. The citation names a slot in this prompt. In the “start watering” prompt, [2] is passage 6. Keep the slot-to-id map, or your citation links point at the wrong page.

Burying the best passage. Models use the start and the end of a long context better than the middle. Pack the top hit first, and do not pad the prompt with weak hits just because the budget allows it.

Blaming the model for every wrong answer. Look at the packed context first. If the right passage is not there, no prompt wording will save the answer.

What to remember

  • RAG is retrieve, pack, generate. The retriever can be anything from the earlier chapters.
  • The source budget is the window minus instructions, question and answer reserve: 64 − 19 − 6 − 16 = 23.
  • Pack in rank order and skip what does not fit. Passage 2 (12 tokens) went in; passage 5 (14 tokens) did not.
  • Number the sources, ask for citations, and check each claim against the source it cites.
  • If the right passage is missing, it is a retrieval miss. If it is there and the answer disagrees, the generator ignored it.

Check yourself

1. Why is the source budget 22 for “When can I start watering my plot?” and 23 for the first question?

The second question has 7 tokens instead of 6. Every token of the question comes out of the same window.

2. The stub answered “The sources do not say.” Is that a failure?

It is a failure of retrieval, and the right behaviour from the generator. Making up an answer from passages about fees would be worse.

3. “The water taps open on April 15. [1]” cites a real source. Why is it rejected?

The word “april” is not in passage 2. A citation is only worth something if the claim is supported by what it cites.

4. Passage 5 was skipped. When would that matter?

When the question is about turning the tap off. Then passage 5 would rank first and be packed first. Packing only drops the weakest hits that do not fit.

Go to the source

Keep going