TRUST, BUT VERIFY · CHAPTER 02

Retrieval-Augmented Generation

The weights have a cutoff and a finite memory. RAG puts a small, cited stack of documents next to the prompt so the model can answer from evidence instead of from guesswork.

~16 min read4-doc corpusTested Python

What you'll learn

  • The retrieve-then-read loop, and why it is not fine-tuning
  • Sparse overlap vs dense cosine on the same mini corpus
  • How to turn the top chunks into an answer with citations
  • Why bad retrieval is worse than no retrieval

Before you start

  • Evaluating LLMs (FactScore needs a source)
  • Cosine similarity as a dot product of unit vectors

An open book, not a better brain

Closed-book QA is an exam with the notes locked away. RAG is the same exam with a binder on the desk. You still need a student who can read. You also need the right pages in the binder. Putting the wrong chapter in front of them is worse than an empty desk: they will quote it confidently.

RAG does not update weights. It updates the prompt. Parametric knowledge (what the model memorized) and non-parametric knowledge (what you fetched) can disagree. When they do, the fetched text should win, and the answer should say where it came from.

Four documents, one query

Query: What time should I leave the bakery to catch the Pine Island ferry?

miss · noise · contradict

MANIM RAG fails as miss, noise, or contradiction — not “bad IQ.”

book · pipeline · failures · cite

MANIM RAG opens the book; it does not rewrite the model’s weights.
IdDocument
AThe harbour lighthouse uses a 12-second flash pattern.
BThe harbour bakery opens at 06:30 and sells rye on Tuesdays.
CThe ferry to Pine Island leaves pier 2 at 07:15.
DThe walk from the bakery to pier 2 takes 8 minutes.

The answer is 07:07: ferry at 07:15, eight-minute walk. No single document holds it. Retrieval has to return C and D (B is useful colour; A is a decoy).

Count the overlapping words

A stand-in for BM25: how many query words appear in the document. Query words include ferry, Pine, Island, bakery, time.

DocOverlap count
C5
D3
B2
A1

Sparse search loves rare exact words. It ranks C first because ferry and Pine Island are in the query. It is weaker on paraphrase: “catch the boat” would miss C.

Cosine on a tiny embedding

Each document is a 3-D vector. The query is \((1.0,\ 0.2,\ 0.0)\). Cosine:

the angle between the query and the document. 1 is parallel, 0 is orthogonal.

DocVectorCosine
D(0.7, 0.2, 0.2)0.961
B(0.9, 0.1, 0.3)0.946
C(0.8, 0.0, 0.4)0.877
A(0.1, 1.0, 0.0)0.293

Dense search ranks D, then B, then C. It knows “leave the bakery” is about D even without the word ferry. Hybrid search (sparse + dense) is how production indexes cover both exact names and paraphrases. A cross-encoder reranker then reads the query with each surviving chunk together, which is more accurate and too slow to run on the whole corpus.

Stuff the top chunks, then generate

Ingestchunk, embed, index
Retrievetop-k by sparse, dense, or both
Generateprompt = query + chunks + “cite your sources”

With D, C and B in the prompt, a faithful model can write: leave the bakery by 07:07 (8 minutes to pier 2, ferry at 07:15). The bakery’s 06:30 opening is extra. The lighthouse is correctly ignored if retrieval dropped A.

Chunking is a real design choice. Too big, and D’s eight minutes drown in a page of harbour trivia. Too small, and C’s 07:15 splits from “Pine Island”. Overlap the windows. Keep the same embedder at query time that you used at index time.

The ranking you can re-run

retrieve.pycosine D > B > C > A; leave by 07:07
"""Mini-corpus retrieval for one harbour-town query.

Four short documents and a 3-D embedding each. Rank by cosine
similarity, then by overlapping words (a stand-in for sparse search).
"""
import math

# Own corpus. Three or four short docs and one query.
DOCS = {
    "A": "The harbour lighthouse uses a 12-second flash pattern.",
    "B": "The harbour bakery opens at 06:30 and sells rye on Tuesdays.",
    "C": "The ferry to Pine Island leaves pier 2 at 07:15.",
    "D": "The walk from the bakery to pier 2 takes 8 minutes.",
}

QUERY = "What time should I leave the bakery to catch the Pine Island ferry?"

# Tiny 3-D embeddings. Chosen so the ranking is D > B > C > A.
QUERY_VEC = [1.0, 0.2, 0.0]
DOC_VECS = {
    "A": [0.1, 1.0, 0.0],
    "B": [0.9, 0.1, 0.3],
    "C": [0.8, 0.0, 0.4],
    "D": [0.7, 0.2, 0.2],
}


def dot(a, b):
    return sum(x * y for x, y in zip(a, b))


def norm(a):
    return math.sqrt(sum(x * x for x in a))


def cosine(a, b):
    return dot(a, b) / (norm(a) * norm(b))


def dense_rank(query_vec=None, doc_vecs=None):
    query_vec = QUERY_VEC if query_vec is None else query_vec
    doc_vecs = DOC_VECS if doc_vecs is None else doc_vecs
    scored = [(name, cosine(query_vec, vec)) for name, vec in doc_vecs.items()]
    scored.sort(key=lambda row: row[1], reverse=True)
    return scored


def tokenize(text):
    """Lowercase word tokens, stripping the few punctuation marks we use."""
    cleaned = text.lower().replace("?", "").replace(".", "").replace(",", "")
    return cleaned.split()


def overlap_score(query, doc):
    """Count of query words that also appear in the document."""
    q = set(tokenize(query))
    d = set(tokenize(doc))
    return len(q & d)


def sparse_rank(query=QUERY, docs=None):
    docs = DOCS if docs is None else docs
    scored = [(name, overlap_score(query, text)) for name, text in docs.items()]
    scored.sort(key=lambda row: row[1], reverse=True)
    return scored

Mistakes learners make

Bad retrieval is worse than none. If the binder opens on the lighthouse, the model will discuss flash patterns. Grounded and wrong. Check recall@k on questions, not just “we added a vector store”.

Fine-tuning instead of fetching. Fine-tuning changes style and skills. It is a poor cache for facts that move (the 07:15 ferry). RAG is the cache.

Lost-in-the-middle. Models read the start and end of a long context more carefully than the middle. Put the best chunks first, or rerank down to a handful.

A different embedder at query time. Cosine is only meaningful in one space. The query vector and the document vectors must come from the same model.

What to remember

  • RAG fetches documents at inference time and puts them in the prompt. Weights stay frozen.
  • Sparse overlap ranked C first. Dense cosine ranked D first (0.961) and A last (0.293).
  • The harbour answer is 07:07, which no single document states.
  • Wrong chunks hurt more than an empty context. Hybrid search plus a reranker is the usual production stack.

Check yourself

1. Why is A a decoy for this query?

It shares the harbour setting and almost no words that matter. Dense cosine 0.293 puts it last; a sloppy keyword search on “harbour” would not.

2. Why does the answer need two documents?

C has 07:15. D has the 8 minutes. The leave time 07:07 is computed from both.

3. When would sparse beat dense here?

If the user said the exact name “Pine Island ferry”. Overlap 5 on C is a strong exact-match signal.

4. Does RAG train the generator?

Not in the basic loop. It changes the prompt. (You can later fine-tune the generator to use retrieved context better. That is extra.)

Go to the source

Keep going