THE TRAINING GROUND · CHAPTER 01

The pre-training pipeline

A base model is next-token prediction on a packed stream of documents. The interesting work is deciding what those tokens are, how they sit in a batch, and which ones are allowed to see each other.

~20 min readOwn packing exampleTested Python

What you'll learn

  • How raw crawls become a packed training stream
  • The causal language-modeling objective, and what it is not
  • Why trainers pack many documents into one token row
  • How a document mask differs from a causal mask
  • How fill-in-the-middle rearranges a document so the model can infill

Before you start

A moving van, not a bookshelf

Imagine you are packing a van for a long trip. You do not give each book its own crate the size of the van. You stack books until the crate is full, slip a card between them, and close the lid. The crate is a fixed size because that is what fits on the truck.

Pre-training works the same way. The GPU wants a fixed-length row of tokens. Documents are the books. The card between them is an end-of-document token. The crate is one training sequence of length \(T\).

The model never sees “a document.” It sees tokens in a rectangle. Everything that makes two documents stay separate — positions, attention, the loss — is something you add on purpose.

From a crawl to a packed batch

Pre-training starts from random weights and a huge pile of text. The pile is not the batch. A lot of work happens before the first step.

van · stages · skill · gap

MANIM Pre-training buys next-token skill; it does not buy “do what I asked.”
Web crawl, books, coderaw documents
Filter and dedupdrop junk, near-copies, empty pages
Tokenizetext → integer IDs
Pack to length Tdocuments share one row
Trainnext-token loss on the packed stream

Quality filtering and near-duplicate removal matter more than they sound. A repeated license header or a garbled page can spike the loss. A passage the model sees too often is a passage it can recite.

The architecture does not change between “base” and “chat.” The same decoder, the same hidden size. What changes is the data and, later, the loss mask. Post-training — SFT, then preference methods — is a later chapter.

Pre-training buys general next-token skill. It does not buy “do what I asked.” That gap is why a base model often continues a request instead of answering it.

Predict the next token, almost everywhere

A decoder-only base model is trained with causal language modeling. At every position \(t\), it predicts token \(t+1\) from tokens \(1 \dots t\).

add up the surprise of each next token. Skip positions you have decided not to train on, such as padding.

That is different from masked language modeling, where random tokens inside the sequence are hidden and the model fills them in from both sides. BERT-style encoders use that objective. A causal decoder cannot, because it is not allowed to look right. Families and masks live in Transformer families.

One more distinction that trips people up. Hiding the loss on a prompt is not the same as hiding the attention on a prompt. Chat models are almost always causal. They still let the answer tokens attend to the prompt. They just do not train on predicting the prompt itself. That loss-mask story is SFT and chat templates.

Three notes in one crate

Here are three tiny documents, tokenized at word level so you can do this by hand.

DocumentTokensIDs (EOS = 1)
Aoak door shut11, 12, 13, 1
Brain fell21, 22, 1
Ctea is hot31, 32, 33, 1

That is 11 real tokens. A training row of length 16 has room for all three, plus 5 pads.

tokens oak door shut eos rain fell eos tea is hot eos position 0 1 2 3 0 1 2 0 1 2 3 doc id 0 1 2 pad 11 content tokens · 5 pads · length 16
Positions restart at 0 on every document. The model must not think “tea” sits at position 7 of a single story.

If each document sat in its own length-16 row you would store 48 slots and use 11 of them. Packing uses 16 slots for the same 11 tokens. That is the whole point: fewer pad tokens, more real predictions per GPU step.

pack_docs.pypacked IDs, reset positions, document mask, FIM order
"""Pack documents, reset positions, mask across docs, and rearrange for FIM.

The numbers on the page come from these three short documents.
"""

# Word-level IDs so a reader can follow by hand. Not a real BPE vocabulary.
EOS, PAD = 1, 0
FIM_PRE, FIM_SUF, FIM_MID = 2, 3, 4
VOCAB = {
    "oak": 11, "door": 12, "shut": 13,
    "rain": 21, "fell": 22,
    "tea": 31, "is": 32, "hot": 33,
    "the": 41, "red": 42, "kite": 43, "flew": 44, "high": 45,
}

# Three documents. Each keeps its own EOS so the model can learn where a
# document ends and the next one begins.
DOCS = [
    [VOCAB["oak"], VOCAB["door"], VOCAB["shut"], EOS],
    [VOCAB["rain"], VOCAB["fell"], EOS],
    [VOCAB["tea"], VOCAB["is"], VOCAB["hot"], EOS],
]

PACK_LEN = 16


def pack(docs, length=PACK_LEN):
    """Concatenate documents, then pad to a fixed training length."""
    tokens, positions, doc_ids = [], [], []
    for d, doc in enumerate(docs):
        tokens.extend(doc)
        positions.extend(range(len(doc)))  # position 0 at every document start
        doc_ids.extend([d] * len(doc))
    n = len(tokens)
    tokens += [PAD] * (length - n)
    positions += [0] * (length - n)
    doc_ids += [-1] * (length - n)
    return tokens, positions, doc_ids


def allowed_mask(doc_ids):
    """True where token i may attend to token j.

    Two rules: stay causal, and stay inside one document. Padding attends to
    nothing. This is the segmentation mask plus the usual causal triangle.
    """
    t = len(doc_ids)
    mask = [[False] * t for _ in range(t)]
    for i in range(t):
        if doc_ids[i] < 0:
            continue
        for j in range(i + 1):
            mask[i][j] = doc_ids[j] == doc_ids[i]
    return mask


def pack_waste(docs, length):
    """How many pad slots you pay if each document sits in its own row."""
    padded = length * len(docs)
    packed = length
    content = sum(len(d) for d in docs)
    return content, packed, padded


def fim_rearrange(tokens, split):
    """Split a document into prefix / middle / suffix and rearrange.

    Training order is: <pre> prefix <suf> suffix <mid> middle.
    The model then predicts the missing middle after seeing both sides.
    """
    prefix, middle, suffix = tokens[:split[0]], tokens[split[0]:split[1]], tokens[split[1]:]
    return [FIM_PRE, *prefix, FIM_SUF, *suffix, FIM_MID, *middle]

The batch is tokens. Documents are a story you tell the mask.

A batch of shape \((B, T)\) is \(B \times T\) tokens. It is not \(B\) documents. One row can hold three notes, as above. Another row can hold a single long article that was cut to length 16.

If you only apply a causal mask, “tea” can attend to “rain fell.” That leaks a document that was never part of the same text. The careful fix is a segmentation mask: token \(i\) may attend to token \(j\) only when \(j \le i\) and both IDs belong to the same document.

In the packed row, “tea” (index 7) may see itself. It may not see the EOS of “rain fell” (index 6), and it may not see “is” (index 8) yet. Padding rows are all blocked.

Causal only

Cheap. Some trainers accept the leak and rely on the EOS token to mark a reset. Positions still restart.

Causal + document

Cleaner. Attention cannot cross the card between books. This is what the allowed_mask function builds.

Offline, a cluster job (often Spark or a Rust/Python shard builder) writes these packed rows to disk. The training loop only reads integers. It never sees the original strings.

Hide the middle. Show both sides.

Next-token training only ever asks “what comes after this prefix?” Code editors also need the other question: “what belongs in this gap?” Fill-in-the-middle (FIM) teaches that by rearranging the document before packing.

Take one new sentence, still word-level: the red kite flew high. Split after “red” and after “kite.”

  1. Prefix = the red. Middle = kite. Suffix = flew high.
  2. Wrap with three special tokens that never appear in ordinary BPE.
  3. Train on the order FIM_PRE the red FIM_SUF flew high FIM_MID kite.

IDs: 2, 41, 42, 3, 44, 45, 4, 43. The model reads both sides, then has to emit the missing middle. That is the autocomplete used in an editor: you have the function header and the closing lines, and you want the body.

FIM is a data format, not a new architecture. The same causal decoder, the same packed rows. You just sprinkle rearranged documents into the mix so some of the next-token targets sit in a gap.

Mistakes learners make

Treating one batch row as one document. A row is a crate of tokens. Count documents with EOS and doc ids, not with \(B\).

Letting positions run across a pack. If “tea” is at position 7, rotary or learned positions think it is late in a single story. Reset to 0 at each document start.

Confusing loss masks with attention masks. SFT often zeros the loss on the prompt. The answer tokens still attend to that prompt. Prefix-LM attention is a different family.

Skipping the document mask and calling it free context. Cross-document attention is not “more data.” It is two unrelated texts glued together.

What to remember

  • Pre-training is next-token prediction on filtered, tokenized, packed text.
  • A batch is tokens. Documents share a row so the GPU does not sit on padding.
  • Reset positions per document. Optionally also block attention across documents.
  • FIM rearranges prefix / suffix / middle so a causal model can infill.

Check yourself

1. The packed row has 11 content tokens and length 16. How many pads? How many slots would three separate rows of length 16 use?

5 pads. Three separate rows would use 48 slots for the same 11 tokens.

2. Why does the position of “tea” go back to 0?

It is the first token of a new document. Position 7 would tell the model it is late in one long story.

3. May “tea” attend to “fell”?

Not under a document mask. They have different doc ids. A causal-only mask would allow it, which is the leak.

4. Write the FIM order for prefix = the red, middle = kite, suffix = flew high.

FIM_PRE the red FIM_SUF flew high FIM_MID kite.

5. Does FIM change the model architecture?

No. It changes how some documents are laid out before they are packed.

Go to the source

Keep going