TRUST, BUT VERIFY · CHAPTER 05

Memorization & Factuality

Some strings in a model are stored almost verbatim. Some facts are assembled. Extraction is how you tell the two apart. Factuality is whether the assembled ones are true.

~16 min read6-token postcardTested Python

What you'll learn

  • The difference between memorizing a string and knowing a fact
  • What k-extractable means, on a postcard you can count
  • A capacity figure: about 3.6 bits per parameter
  • Intrinsic vs extrinsic hallucination, and why FactScore still matters

Before you start

A poem versus a recipe

You can recite a poem you saw twenty times. You can also tell someone how to boil an egg without quoting any one page. The first is memorization. The second is a fact you can use. Language models do both, in the same weights, and the user cannot see which one just fired.

Extraction is a test, not a vibe. If a short prefix of a training string makes the model emit the rest, that string was stored. If no prefix does, the model may still “know” something in the egg-boiling sense. Privacy and copyright care about the first. Factuality cares about whether the second is true.

Six tokens, twenty copies

Toy corpus: 240 tokens. A postcard of six tokens, MEET AT PIER TWO SUN DUSK, is copied 20 times. That is 120 tokens, 50% of the corpus. The rest is unrelated harbour chatter. A tiny model trained on this pile will see the postcard more often than anything else.

poem · extract · fluent · check

MANIM Fluency is not factuality; poems and recipes memorize differently.

This is not a real leak and not a real secret. It is a counting exercise so the next definition has numbers attached.

A prefix that yields the rest

A string is k-extractable if there exists a length-k prefix that, given to the model, is completed as the remainder of the string. On the postcard, k = 2:

Prefix (k = 2)Completion
MEET ATPIER TWO SUN DUSK

That is extraction: the model replayed training text from a prompt that was itself a piece of that text. Memorization is the storage. Extraction is the recovery. A string can be memorized and still hard to extract if you do not know a working prefix. Membership inference tries to notice storage without recovering the string. Extraction tries to recover it.

Useful memorization is a quotation, a common API, a theorem. Harmful memorization is private data and copyrighted passages. The mechanism is the same. The policy is not.

Bits per parameter

Morris et al. estimate that a trained transformer can store on the order of 3.6 bits per parameter of extractable data. For Llama 3 8B that is roughly 29 billion bits — a lot of postcards, and still much less than the training crawl. Deduplication, fewer epochs, and more data per parameter all push the model toward using those bits for general features instead of verbatim copies. They do not make extraction impossible.

Scale cuts both ways. Larger models memorize more in absolute terms and also generalize better. Dedup is the cheap lever that actually reduces verbatim copies of high-frequency passages.

When the completion is new, is it true?

Hallucinations are completions that are not true. In the RAG setting the standard split is:

Closed-book, there is no source in the prompt, so the split collapses to “supported by the world” vs not. FactScore, from chapter 01, is the practical tool: break the text into atoms, check each against a knowledge source, report the fraction. The potter paragraph scored 3 / 4 = 0.75 because one atom had no support.

A model can abstain (“I do not know”) or hedge. That is calibration. Reward models that always prefer a confident paragraph will punish abstention and invent atoms. Factuality and reward hacking meet there.

The postcard you can re-run

extract.pyk = 2 yields PIER TWO SUN DUSK; copy fraction 50%
"""A tiny illustration of extraction versus memorization.

A six-token postcard is copied many times into a 200-token toy corpus.
A prefix of the postcard reconstructs the rest. That is k-extractability
on a scale you can count by hand. It is not a real attack.
"""

# Own postcard. Harmless meeting-place text, not a secret from a video.
POSTCARD = ["MEET", "AT", "PIER", "TWO", "SUN", "DUSK"]
PREFIX = ["MEET", "AT"]          # k = 2
CORPUS_LEN = 240                 # 20 copies × 6 tokens = half the corpus
N_COPIES = 20


def postcard_tokens():
    return list(POSTCARD)


def extract(prefix, postcard=None):
    """If `prefix` is the start of the postcard, return the rest.

    This stands in for a model that memorized the string. A real model
    would sample; here the completion is deterministic so the count is
    exact.
    """
    postcard = POSTCARD if postcard is None else postcard
    if prefix == postcard[:len(prefix)]:
        return postcard[len(prefix):]
    return []


def is_k_extractable(k, postcard=None):
    """A string is k-extractable if some length-k prefix yields the rest."""
    postcard = POSTCARD if postcard is None else postcard
    if k > len(postcard) or k < 0:
        return False
    return extract(postcard[:k], postcard) == postcard[k:]


def copy_fraction(n_copies=N_COPIES, postcard=None, corpus_len=CORPUS_LEN):
    """Share of the toy corpus taken up by repeated copies of the postcard."""
    postcard = POSTCARD if postcard is None else postcard
    return (n_copies * len(postcard)) / corpus_len

Mistakes learners make

Calling every wrong answer “memorization”. A wrong date can be a failed generalization. Memorization is storage of a training string. Extraction is recovering it.

Swapping intrinsic and extrinsic. Intrinsic contradicts the given source. Extrinsic is not checkable from that source. “False fact” vs “instruction violation” is a different pair.

“Just train on more data than capacity.” Capacity is a rough bit budget, not a switch that flips from memorizing to generalizing. Dupes and epochs still matter.

Treating extraction demos as attacks to copy. The postcard is a counting toy. Real extraction research is measured on public models with careful disclosure, not on private data.

What to remember

  • Memorization is storage. Extraction is recovery from a prefix. k-extractable means some length-k prefix yields the rest.
  • The postcard MEET AT PIER TWO SUN DUSK is 50% of a 240-token corpus at 20 copies. Prefix MEET AT reconstructs the rest.
  • Capacity is about 3.6 bits per parameter. Dedup beats “just scale” for cutting verbatim copies.
  • Intrinsic hallucination contradicts the source. Extrinsic is unsupported by it. FactScore counts atoms.

Check yourself

1. The prefix MEET AT produces PIER TWO SUN DUSK. What did we just show?

That the postcard is 2-extractable. The model (here, a lookup) stored the string and a short prefix recovers it.

2. Why is 50% an extreme toy?

Twenty copies of a six-token string eat half of 240 tokens. Real crawls are less concentrated after dedup, which is the point of dedup.

3. A RAG answer invents a prize the source never mentioned. Intrinsic or extrinsic?

Extrinsic relative to that source: the source does not support it. If the source had said a different prize, a contradiction would be intrinsic.

4. Does 3.6 bits/parameter mean Llama 3 8B stored the whole internet?

No. It is a capacity estimate, far below crawl size. The model cannot have stored everything verbatim. It can still extract some high-frequency strings.

Go to the source

Keep going