FINDING NEEDLES · CHAPTER 13

Chunking

The retriever never sees your document. It sees the pieces you cut it into. Cut a rule in half and no search can put it back together.

~14 min read77-word rulebookTested Python

What you'll learn

  • How fixed-size chunks cut an answer across a boundary
  • What overlap buys, what it costs, and when it guarantees anything
  • Why splitting on headings and sentences keeps rules whole
  • The chunk-size trade-off: a precise match against enough context

Before you start

A rulebook cut into cards

Picture a board-game rulebook cut into index cards so players can look up one rule at a time. If you cut every 16 words with scissors, some rules end on one card and finish on the next. A player who pulls only the first card reads “take” and never learns how many.

A search index is that box of cards. Each chunk gets its own vector or its own postings. The retriever returns chunks, not documents, and the generator in chapter 12 only reads what the retriever returns.

Chunking decides what can be found. A fact that is split across two chunks is in neither of them. Every strategy in this chapter is a way to keep the answer and its conditions on the same card.

Every 16 words, no matter what

The rulebook for a made-up farming game has four headed sections and 77 words. The question is “How many seed tokens do you take for a full harvest?” The answer is one 13-word rule: When you complete a full harvest take three seed tokens from the bank.

overlap · headers · size

MANIM Overlap + headers keep meaning across chunk boundaries.

cards · size · overlap · poison

MANIM Chunking cuts the rulebook; size/overlap/headers matter.

The simplest chunker flattens the text to words and cuts every S words. With S = 16 and no overlap you get 5 chunks.

ChunkWordsScore
1setup each player takes a farm board and four workers shuffle the weather deck and place0
2it face down turns on your turn place one worker on an empty field then draw0
3one weather card and resolve it at once harvest when you complete a full harvest take3
4three seed tokens from the bank a frost card cancels the harvest winning the game ends3
5when the seed bank is empty the player with the most seeds wins1

The score is the number of question words a chunk shares, after stop words (the toy retriever from chapter 12). Chunks 3 and 4 tie at 3 and the earlier one wins. Chunk 3 ends on “take”. The number three is in chunk 4. The top chunk cannot answer the question.

N words, windows of S words, each starting S − O words after the last. Here O, the overlap, is 0.

Repeat the edge of each window

Overlap starts each window O words before the previous one ended. A sentence that was cut gets a second chance to land whole inside the next window.

With S = 16 and O = 6, each window starts 10 words after the last. That gives ⌈61 / 10⌉ + 1 = 8 chunks. Chunk 5 reads “harvest when you complete a full harvest take three seed tokens from the bank a frost”. It scores 5, wins, and holds the whole rule.

That was partly luck. Overlap moves the cuts. It only promises this:

any run of at most O + 1 words fits inside one window. The rule is 13 words, so only O ≥ 12 guarantees it.

With O = 12 each window starts only 4 words after the last: 17 chunks for a 77-word book, more than three times the 5 you started with. Overlap is paid for in index size, embedding calls, and near-duplicate results.

Interactive · Move the cuts

Which overlaps keep the harvest rule whole at size 16? Try 3, 4, 5 and 6. Then try other sizes.

Pink is a top chunk that holds the whole rule. Red is a top chunk that was cut. Yellow is another chunk that holds the rule but did not win. At size 16 the rule survives at overlap 3, is lost again at 4 and 5, and comes back at 6.

Cut where the author did

The rulebook already tells you where the joints are: headings and full stops. A structure-aware splitter uses them.

  1. Split at each heading. A chunk never crosses from “Harvest” into “Winning”.
  2. Inside a section, split into sentences.
  3. Pack whole sentences into a chunk until the next one would pass the size limit. Never cut a sentence.
  4. Start each chunk with its heading, so “Harvest:” travels with the rule.

With a limit of 20 words, every section fits in one chunk, so you get 4 chunks. The top chunk is the whole Harvest section:

Harvest: When you complete a full harvest take three seed tokens from the bank. A frost card cancels the harvest.

It scores 5 and holds the rule and its exception. The heading prefix is a small version of a bigger idea: give each chunk enough of its surroundings to stand alone. Some systems ask a model to write one situating sentence per chunk before indexing it.

Precise or complete

Run the structure-aware splitter at three sizes and look at the top chunk each time.

SplitChunksTop chunk wordsHarvest ruleFrost exception
one sentence (limit 14)814yesno
one section (limit 20)420yesyes
whole rulebook177yesyes

The sentence chunk is precise: 13 of its 14 words are the rule. It also drops the next sentence. A player who drew a frost card would be told to take three seeds anyway.

The whole rulebook can never miss, but only 13 of its 77 words answer the question. It eats the context budget from chapter 12, and its single vector blurs four topics into one point, so it matches every question a little and none well.

The section is the middle: 13 of 20 words are the rule, and the exception rides along. There is no size that is right for every corpus. Pick the unit a reader would quote, then test it with real questions (chapter 15). For scale, the dense retriever DPR split Wikipedia into 100-word passages.

Small to big: search the sentence, return the section

You do not have to pick one size. Index the small chunks, because a one-sentence vector matches a precise question well. When a sentence wins, hand the generator its parent section instead. Here the sentence chunk for the harvest rule wins, and the generator receives the full Harvest section, frost exception included.

A related trick, late chunking, runs a long-context embedding model over the whole document first and only then pools the token vectors into chunk vectors. Each chunk vector has then seen its neighbours.

The splitters you can re-run

chunking.py16/0 cut · 16/6 whole · sections 4
"""Chunking a board-game rulebook three ways.

Fixed-size windows, fixed-size windows with overlap, and a splitter that
respects headings and sentences. A toy word-overlap retriever then asks
one question and we check whether the top chunk holds the whole answer.
"""
import re

# A short rulebook for a made-up game. "## " marks a heading.
RULEBOOK = """## Setup
Each player takes a farm board and four workers. Shuffle the weather deck and place it face down.
## Turns
On your turn place one worker on an empty field. Then draw one weather card and resolve it at once.
## Harvest
When you complete a full harvest take three seed tokens from the bank. A frost card cancels the harvest.
## Winning
The game ends when the seed bank is empty. The player with the most seeds wins."""

QUESTION = "How many seed tokens do you take for a full harvest?"
ANSWER = "When you complete a full harvest take three seed tokens from the bank."
CAVEAT = "A frost card cancels the harvest."

STOP = {"how", "do", "you", "for", "a", "the", "and", "it", "on", "of", "from", "your", "when", "with"}


def words(text):
    """Lowercase words. Punctuation and the ## marker are dropped."""
    return re.findall(r"[a-z]+", text.lower())


# ---- 1. Fixed-size windows ----------------------------------------------

def fixed_chunks(size, overlap=0):
    """Cut the word stream into windows of `size`, each starting `size - overlap` later."""
    stream = words(RULEBOOK)
    step = size - overlap
    chunks = []
    for start in range(0, len(stream), step):
        chunks.append(" ".join(stream[start:start + size]))
        if start + size >= len(stream):
            break
    return chunks


def guaranteed_span(overlap):
    """Any run of overlap + 1 words lands whole inside some window."""
    return overlap + 1


# ---- 2. Heading- and sentence-aware splitting ---------------------------

def sections():
    """(heading, [sentences]) for each '## ' section."""
    out = []
    for block in RULEBOOK.split("## ")[1:]:
        heading, body = block.split("\n", 1)
        sentences = re.split(r"(?<=\.)\s+", body.strip())
        out.append((heading.strip(), sentences))
    return out


def structured_chunks(max_words):
    """Pack whole sentences up to max_words. Never cross a heading.

    Each chunk starts with its heading, so "Harvest:" travels with the rule.
    A sentence longer than max_words still becomes its own chunk.
    """
    chunks = []
    for heading, sentences in sections():
        current = []
        for sentence in sentences:
            if current and len(words(" ".join(current + [sentence]))) > max_words:
                chunks.append(f"{heading}: " + " ".join(current))
                current = []
            current.append(sentence)
        if current:
            chunks.append(f"{heading}: " + " ".join(current))
    return chunks


# ---- 3. Retrieve one chunk and check it ---------------------------------

def content(text):
    return {w for w in words(text) if w not in STOP}


def top_chunk(chunks, question=QUESTION):
    """The chunk that shares the most content words. Ties go to the earlier chunk."""
    query = content(question)
    scores = [len(query & content(chunk)) for chunk in chunks]
    best = max(range(len(chunks)), key=lambda i: (scores[i], -i))
    return best, scores[best]


def holds(chunk, sentence):
    """True if every word of the sentence appears, in order, inside the chunk."""
    return " ".join(words(sentence)) in " ".join(words(chunk))


def whole_in(chunks, sentence=ANSWER):
    """Numbers (from 1) of every chunk that holds the whole sentence."""
    return [i + 1 for i, chunk in enumerate(chunks) if holds(chunk, sentence)]


def report(chunks):
    """What the generator would get from the top chunk."""
    i, score = top_chunk(chunks)
    chunk = chunks[i]
    return {
        "chunks": len(chunks),
        "top": i + 1,
        "score": score,
        "words": len(words(chunk)),
        "answer": holds(chunk, ANSWER),
        "caveat": holds(chunk, CAVEAT),
    }


# ---- Deep dive: small-to-big --------------------------------------------

def parent_section(sentence):
    """Search on one sentence, hand the generator its whole section."""
    for heading, sentences in sections():
        if any(holds(s, sentence) for s in sentences):
            return f"{heading}: " + " ".join(sentences)
    return None

Mistakes learners make

Treating overlap as a guarantee. It only covers spans of O + 1 words or fewer. At size 16, overlap 4 loses the 13-word rule that overlap 3 kept.

Letting near-duplicates crowd the top. At O = 12, chunks 10, 11 and 12 all score 5. Only chunk 11 holds the whole rule, and the tie goes to chunk 10. Heavy overlap fills your top-k with copies of the same few lines.

Dropping the heading. “A frost card cancels the harvest” means little without knowing it is a harvest rule. Prefix the heading or the document title.

Making chunks longer than the embedding model reads. Many encoders have a maximum input length and silently drop the rest. A chunk past that limit is searched on its first part only.

What to remember

  • Fixed 16-word chunks give 5 chunks. The top chunk ends on “take”, and the number three lands in chunk 4.
  • Overlap 6 gives 8 chunks and keeps the rule whole in chunk 5. Only O ≥ 12 guarantees a 13-word rule, at 17 chunks.
  • Splitting on headings and sentences never cuts a rule. At a 20-word limit you get 4 chunks, one per section.
  • Small chunks are precise but drop context. Big chunks keep context but dilute the match and spend the budget.

Check yourself

1. How many 16-word chunks with overlap 6 does a 77-word text make?

Each window starts 10 words after the last. ⌈(77 − 16) / 10⌉ + 1 = 7 + 1 = 8.

2. Why did overlap 4 fail when overlap 3 worked?

Changing the overlap moves every cut after the first one. Below the guarantee (O ≥ 12 for a 13-word rule), whether the rule survives depends on where the cuts happen to fall.

3. The one-sentence chunk holds the full rule. What is wrong with it?

The exception is in the next sentence. A generator that sees only this chunk will give seeds after a frost card.

4. What does small-to-big return for the harvest question?

It matches on the one-sentence chunk, then returns the whole Harvest section, with the frost exception.

Go to the source

Keep going