FINDING NEEDLES · CHAPTER 13
The retriever never sees your document. It sees the pieces you cut it into. Cut a rule in half and no search can put it back together.
Picture a board-game rulebook cut into index cards so players can look up one rule at a time. If you cut every 16 words with scissors, some rules end on one card and finish on the next. A player who pulls only the first card reads “take” and never learns how many.
A search index is that box of cards. Each chunk gets its own vector or its own postings. The retriever returns chunks, not documents, and the generator in chapter 12 only reads what the retriever returns.
Chunking decides what can be found. A fact that is split across two chunks is in neither of them. Every strategy in this chapter is a way to keep the answer and its conditions on the same card.
The rulebook for a made-up farming game has four headed sections and 77 words. The question is “How many seed tokens do you take for a full harvest?” The answer is one 13-word rule: When you complete a full harvest take three seed tokens from the bank.
overlap · headers · size
cards · size · overlap · poison
The simplest chunker flattens the text to words and cuts every S words. With S = 16 and no overlap you get 5 chunks.
| Chunk | Words | Score |
|---|---|---|
| 1 | setup each player takes a farm board and four workers shuffle the weather deck and place | 0 |
| 2 | it face down turns on your turn place one worker on an empty field then draw | 0 |
| 3 | one weather card and resolve it at once harvest when you complete a full harvest take | 3 |
| 4 | three seed tokens from the bank a frost card cancels the harvest winning the game ends | 3 |
| 5 | when the seed bank is empty the player with the most seeds wins | 1 |
The score is the number of question words a chunk shares, after stop words (the toy retriever from chapter 12). Chunks 3 and 4 tie at 3 and the earlier one wins. Chunk 3 ends on “take”. The number three is in chunk 4. The top chunk cannot answer the question.
N words, windows of S words, each starting S − O words after the last. Here O, the overlap, is 0.
Overlap starts each window O words before the previous one ended. A sentence that was cut gets a second chance to land whole inside the next window.
With S = 16 and O = 6, each window starts 10 words after the last. That gives ⌈61 / 10⌉ + 1 = 8 chunks. Chunk 5 reads “harvest when you complete a full harvest take three seed tokens from the bank a frost”. It scores 5, wins, and holds the whole rule.
That was partly luck. Overlap moves the cuts. It only promises this:
any run of at most O + 1 words fits inside one window. The rule is 13 words, so only O ≥ 12 guarantees it.
With O = 12 each window starts only 4 words after the last: 17 chunks for a 77-word book, more than three times the 5 you started with. Overlap is paid for in index size, embedding calls, and near-duplicate results.
Interactive · Move the cuts
Which overlaps keep the harvest rule whole at size 16? Try 3, 4, 5 and 6. Then try other sizes.
Pink is a top chunk that holds the whole rule. Red is a top chunk that was cut. Yellow is another chunk that holds the rule but did not win. At size 16 the rule survives at overlap 3, is lost again at 4 and 5, and comes back at 6.
The rulebook already tells you where the joints are: headings and full stops. A structure-aware splitter uses them.
With a limit of 20 words, every section fits in one chunk, so you get 4 chunks. The top chunk is the whole Harvest section:
It scores 5 and holds the rule and its exception. The heading prefix is a small version of a bigger idea: give each chunk enough of its surroundings to stand alone. Some systems ask a model to write one situating sentence per chunk before indexing it.
Run the structure-aware splitter at three sizes and look at the top chunk each time.
| Split | Chunks | Top chunk words | Harvest rule | Frost exception |
|---|---|---|---|---|
| one sentence (limit 14) | 8 | 14 | yes | no |
| one section (limit 20) | 4 | 20 | yes | yes |
| whole rulebook | 1 | 77 | yes | yes |
The sentence chunk is precise: 13 of its 14 words are the rule. It also drops the next sentence. A player who drew a frost card would be told to take three seeds anyway.
The whole rulebook can never miss, but only 13 of its 77 words answer the question. It eats the context budget from chapter 12, and its single vector blurs four topics into one point, so it matches every question a little and none well.
The section is the middle: 13 of 20 words are the rule, and the exception rides along. There is no size that is right for every corpus. Pick the unit a reader would quote, then test it with real questions (chapter 15). For scale, the dense retriever DPR split Wikipedia into 100-word passages.
You do not have to pick one size. Index the small chunks, because a one-sentence vector matches a precise question well. When a sentence wins, hand the generator its parent section instead. Here the sentence chunk for the harvest rule wins, and the generator receives the full Harvest section, frost exception included.
A related trick, late chunking, runs a long-context embedding model over the whole document first and only then pools the token vectors into chunk vectors. Each chunk vector has then seen its neighbours.
"""Chunking a board-game rulebook three ways.
Fixed-size windows, fixed-size windows with overlap, and a splitter that
respects headings and sentences. A toy word-overlap retriever then asks
one question and we check whether the top chunk holds the whole answer.
"""
import re
# A short rulebook for a made-up game. "## " marks a heading.
RULEBOOK = """## Setup
Each player takes a farm board and four workers. Shuffle the weather deck and place it face down.
## Turns
On your turn place one worker on an empty field. Then draw one weather card and resolve it at once.
## Harvest
When you complete a full harvest take three seed tokens from the bank. A frost card cancels the harvest.
## Winning
The game ends when the seed bank is empty. The player with the most seeds wins."""
QUESTION = "How many seed tokens do you take for a full harvest?"
ANSWER = "When you complete a full harvest take three seed tokens from the bank."
CAVEAT = "A frost card cancels the harvest."
STOP = {"how", "do", "you", "for", "a", "the", "and", "it", "on", "of", "from", "your", "when", "with"}
def words(text):
"""Lowercase words. Punctuation and the ## marker are dropped."""
return re.findall(r"[a-z]+", text.lower())
# ---- 1. Fixed-size windows ----------------------------------------------
def fixed_chunks(size, overlap=0):
"""Cut the word stream into windows of `size`, each starting `size - overlap` later."""
stream = words(RULEBOOK)
step = size - overlap
chunks = []
for start in range(0, len(stream), step):
chunks.append(" ".join(stream[start:start + size]))
if start + size >= len(stream):
break
return chunks
def guaranteed_span(overlap):
"""Any run of overlap + 1 words lands whole inside some window."""
return overlap + 1
# ---- 2. Heading- and sentence-aware splitting ---------------------------
def sections():
"""(heading, [sentences]) for each '## ' section."""
out = []
for block in RULEBOOK.split("## ")[1:]:
heading, body = block.split("\n", 1)
sentences = re.split(r"(?<=\.)\s+", body.strip())
out.append((heading.strip(), sentences))
return out
def structured_chunks(max_words):
"""Pack whole sentences up to max_words. Never cross a heading.
Each chunk starts with its heading, so "Harvest:" travels with the rule.
A sentence longer than max_words still becomes its own chunk.
"""
chunks = []
for heading, sentences in sections():
current = []
for sentence in sentences:
if current and len(words(" ".join(current + [sentence]))) > max_words:
chunks.append(f"{heading}: " + " ".join(current))
current = []
current.append(sentence)
if current:
chunks.append(f"{heading}: " + " ".join(current))
return chunks
# ---- 3. Retrieve one chunk and check it ---------------------------------
def content(text):
return {w for w in words(text) if w not in STOP}
def top_chunk(chunks, question=QUESTION):
"""The chunk that shares the most content words. Ties go to the earlier chunk."""
query = content(question)
scores = [len(query & content(chunk)) for chunk in chunks]
best = max(range(len(chunks)), key=lambda i: (scores[i], -i))
return best, scores[best]
def holds(chunk, sentence):
"""True if every word of the sentence appears, in order, inside the chunk."""
return " ".join(words(sentence)) in " ".join(words(chunk))
def whole_in(chunks, sentence=ANSWER):
"""Numbers (from 1) of every chunk that holds the whole sentence."""
return [i + 1 for i, chunk in enumerate(chunks) if holds(chunk, sentence)]
def report(chunks):
"""What the generator would get from the top chunk."""
i, score = top_chunk(chunks)
chunk = chunks[i]
return {
"chunks": len(chunks),
"top": i + 1,
"score": score,
"words": len(words(chunk)),
"answer": holds(chunk, ANSWER),
"caveat": holds(chunk, CAVEAT),
}
# ---- Deep dive: small-to-big --------------------------------------------
def parent_section(sentence):
"""Search on one sentence, hand the generator its whole section."""
for heading, sentences in sections():
if any(holds(s, sentence) for s in sentences):
return f"{heading}: " + " ".join(sentences)
return None
Treating overlap as a guarantee. It only covers spans of O + 1 words or fewer. At size 16, overlap 4 loses the 13-word rule that overlap 3 kept.
Letting near-duplicates crowd the top. At O = 12, chunks 10, 11 and 12 all score 5. Only chunk 11 holds the whole rule, and the tie goes to chunk 10. Heavy overlap fills your top-k with copies of the same few lines.
Dropping the heading. “A frost card cancels the harvest” means little without knowing it is a harvest rule. Prefix the heading or the document title.
Making chunks longer than the embedding model reads. Many encoders have a maximum input length and silently drop the rest. A chunk past that limit is searched on its first part only.
Each window starts 10 words after the last. ⌈(77 − 16) / 10⌉ + 1 = 7 + 1 = 8.
Changing the overlap moves every cut after the first one. Below the guarantee (O ≥ 12 for a 13-word rule), whether the rule survives depends on where the cuts happen to fall.
The exception is in the next sentence. A generator that sees only this chunk will give seeds after a frost card.
It matches on the one-sentence chunk, then returns the whole Harvest section, with the frost exception.