TRUST, BUT VERIFY · CHAPTER 02
The weights have a cutoff and a finite memory. RAG puts a small, cited stack of documents next to the prompt so the model can answer from evidence instead of from guesswork.
Closed-book QA is an exam with the notes locked away. RAG is the same exam with a binder on the desk. You still need a student who can read. You also need the right pages in the binder. Putting the wrong chapter in front of them is worse than an empty desk: they will quote it confidently.
RAG does not update weights. It updates the prompt. Parametric knowledge (what the model memorized) and non-parametric knowledge (what you fetched) can disagree. When they do, the fetched text should win, and the answer should say where it came from.
Query: What time should I leave the bakery to catch the Pine Island ferry?
miss · noise · contradict
book · pipeline · failures · cite
| Id | Document |
|---|---|
| A | The harbour lighthouse uses a 12-second flash pattern. |
| B | The harbour bakery opens at 06:30 and sells rye on Tuesdays. |
| C | The ferry to Pine Island leaves pier 2 at 07:15. |
| D | The walk from the bakery to pier 2 takes 8 minutes. |
The answer is 07:07: ferry at 07:15, eight-minute walk. No single document holds it. Retrieval has to return C and D (B is useful colour; A is a decoy).
A stand-in for BM25: how many query words appear in the document. Query words include ferry, Pine, Island, bakery, time.
| Doc | Overlap count |
|---|---|
| C | 5 |
| D | 3 |
| B | 2 |
| A | 1 |
Sparse search loves rare exact words. It ranks C first because ferry and Pine Island are in the query. It is weaker on paraphrase: “catch the boat” would miss C.
Each document is a 3-D vector. The query is \((1.0,\ 0.2,\ 0.0)\). Cosine:
the angle between the query and the document. 1 is parallel, 0 is orthogonal.
| Doc | Vector | Cosine |
|---|---|---|
| D | (0.7, 0.2, 0.2) | 0.961 |
| B | (0.9, 0.1, 0.3) | 0.946 |
| C | (0.8, 0.0, 0.4) | 0.877 |
| A | (0.1, 1.0, 0.0) | 0.293 |
Dense search ranks D, then B, then C. It knows “leave the bakery” is about D even without the word ferry. Hybrid search (sparse + dense) is how production indexes cover both exact names and paraphrases. A cross-encoder reranker then reads the query with each surviving chunk together, which is more accurate and too slow to run on the whole corpus.
With D, C and B in the prompt, a faithful model can write: leave the bakery by 07:07 (8 minutes to pier 2, ferry at 07:15). The bakery’s 06:30 opening is extra. The lighthouse is correctly ignored if retrieval dropped A.
Chunking is a real design choice. Too big, and D’s eight minutes drown in a page of harbour trivia. Too small, and C’s 07:15 splits from “Pine Island”. Overlap the windows. Keep the same embedder at query time that you used at index time.
"""Mini-corpus retrieval for one harbour-town query. Four short documents and a 3-D embedding each. Rank by cosine similarity, then by overlapping words (a stand-in for sparse search). """ import math # Own corpus. Three or four short docs and one query. DOCS = { "A": "The harbour lighthouse uses a 12-second flash pattern.", "B": "The harbour bakery opens at 06:30 and sells rye on Tuesdays.", "C": "The ferry to Pine Island leaves pier 2 at 07:15.", "D": "The walk from the bakery to pier 2 takes 8 minutes.", } QUERY = "What time should I leave the bakery to catch the Pine Island ferry?" # Tiny 3-D embeddings. Chosen so the ranking is D > B > C > A. QUERY_VEC = [1.0, 0.2, 0.0] DOC_VECS = { "A": [0.1, 1.0, 0.0], "B": [0.9, 0.1, 0.3], "C": [0.8, 0.0, 0.4], "D": [0.7, 0.2, 0.2], } def dot(a, b): return sum(x * y for x, y in zip(a, b)) def norm(a): return math.sqrt(sum(x * x for x in a)) def cosine(a, b): return dot(a, b) / (norm(a) * norm(b)) def dense_rank(query_vec=None, doc_vecs=None): query_vec = QUERY_VEC if query_vec is None else query_vec doc_vecs = DOC_VECS if doc_vecs is None else doc_vecs scored = [(name, cosine(query_vec, vec)) for name, vec in doc_vecs.items()] scored.sort(key=lambda row: row[1], reverse=True) return scored def tokenize(text): """Lowercase word tokens, stripping the few punctuation marks we use.""" cleaned = text.lower().replace("?", "").replace(".", "").replace(",", "") return cleaned.split() def overlap_score(query, doc): """Count of query words that also appear in the document.""" q = set(tokenize(query)) d = set(tokenize(doc)) return len(q & d) def sparse_rank(query=QUERY, docs=None): docs = DOCS if docs is None else docs scored = [(name, overlap_score(query, text)) for name, text in docs.items()] scored.sort(key=lambda row: row[1], reverse=True) return scored
Bad retrieval is worse than none. If the binder opens on the lighthouse, the model will discuss flash patterns. Grounded and wrong. Check recall@k on questions, not just “we added a vector store”.
Fine-tuning instead of fetching. Fine-tuning changes style and skills. It is a poor cache for facts that move (the 07:15 ferry). RAG is the cache.
Lost-in-the-middle. Models read the start and end of a long context more carefully than the middle. Put the best chunks first, or rerank down to a handful.
A different embedder at query time. Cosine is only meaningful in one space. The query vector and the document vectors must come from the same model.
It shares the harbour setting and almost no words that matter. Dense cosine 0.293 puts it last; a sloppy keyword search on “harbour” would not.
C has 07:15. D has the 8 minutes. The leave time 07:07 is computed from both.
If the user said the exact name “Pine Island ferry”. Overlap 5 on C is a strong exact-match signal.
Not in the basic loop. It changes the prompt. (You can later fine-tune the generator to use retrieved context better. That is extra.)