THE TRAINING GROUND · CHAPTER 01
A base model is next-token prediction on a packed stream of documents. The interesting work is deciding what those tokens are, how they sit in a batch, and which ones are allowed to see each other.
Imagine you are packing a van for a long trip. You do not give each book its own crate the size of the van. You stack books until the crate is full, slip a card between them, and close the lid. The crate is a fixed size because that is what fits on the truck.
Pre-training works the same way. The GPU wants a fixed-length row of tokens. Documents are the books. The card between them is an end-of-document token. The crate is one training sequence of length \(T\).
The model never sees “a document.” It sees tokens in a rectangle. Everything that makes two documents stay separate — positions, attention, the loss — is something you add on purpose.
Pre-training starts from random weights and a huge pile of text. The pile is not the batch. A lot of work happens before the first step.
van · stages · skill · gap
Quality filtering and near-duplicate removal matter more than they sound. A repeated license header or a garbled page can spike the loss. A passage the model sees too often is a passage it can recite.
The architecture does not change between “base” and “chat.” The same decoder, the same hidden size. What changes is the data and, later, the loss mask. Post-training — SFT, then preference methods — is a later chapter.
Pre-training buys general next-token skill. It does not buy “do what I asked.” That gap is why a base model often continues a request instead of answering it.
A decoder-only base model is trained with causal language modeling. At every position \(t\), it predicts token \(t+1\) from tokens \(1 \dots t\).
add up the surprise of each next token. Skip positions you have decided not to train on, such as padding.
That is different from masked language modeling, where random tokens inside the sequence are hidden and the model fills them in from both sides. BERT-style encoders use that objective. A causal decoder cannot, because it is not allowed to look right. Families and masks live in Transformer families.
One more distinction that trips people up. Hiding the loss on a prompt is not the same as hiding the attention on a prompt. Chat models are almost always causal. They still let the answer tokens attend to the prompt. They just do not train on predicting the prompt itself. That loss-mask story is SFT and chat templates.
Here are three tiny documents, tokenized at word level so you can do this by hand.
| Document | Tokens | IDs (EOS = 1) |
|---|---|---|
| A | oak door shut | 11, 12, 13, 1 |
| B | rain fell | 21, 22, 1 |
| C | tea is hot | 31, 32, 33, 1 |
That is 11 real tokens. A training row of length 16 has room for all three, plus 5 pads.
If each document sat in its own length-16 row you would store 48 slots and use 11 of them. Packing uses 16 slots for the same 11 tokens. That is the whole point: fewer pad tokens, more real predictions per GPU step.
"""Pack documents, reset positions, mask across docs, and rearrange for FIM.
The numbers on the page come from these three short documents.
"""
# Word-level IDs so a reader can follow by hand. Not a real BPE vocabulary.
EOS, PAD = 1, 0
FIM_PRE, FIM_SUF, FIM_MID = 2, 3, 4
VOCAB = {
"oak": 11, "door": 12, "shut": 13,
"rain": 21, "fell": 22,
"tea": 31, "is": 32, "hot": 33,
"the": 41, "red": 42, "kite": 43, "flew": 44, "high": 45,
}
# Three documents. Each keeps its own EOS so the model can learn where a
# document ends and the next one begins.
DOCS = [
[VOCAB["oak"], VOCAB["door"], VOCAB["shut"], EOS],
[VOCAB["rain"], VOCAB["fell"], EOS],
[VOCAB["tea"], VOCAB["is"], VOCAB["hot"], EOS],
]
PACK_LEN = 16
def pack(docs, length=PACK_LEN):
"""Concatenate documents, then pad to a fixed training length."""
tokens, positions, doc_ids = [], [], []
for d, doc in enumerate(docs):
tokens.extend(doc)
positions.extend(range(len(doc))) # position 0 at every document start
doc_ids.extend([d] * len(doc))
n = len(tokens)
tokens += [PAD] * (length - n)
positions += [0] * (length - n)
doc_ids += [-1] * (length - n)
return tokens, positions, doc_ids
def allowed_mask(doc_ids):
"""True where token i may attend to token j.
Two rules: stay causal, and stay inside one document. Padding attends to
nothing. This is the segmentation mask plus the usual causal triangle.
"""
t = len(doc_ids)
mask = [[False] * t for _ in range(t)]
for i in range(t):
if doc_ids[i] < 0:
continue
for j in range(i + 1):
mask[i][j] = doc_ids[j] == doc_ids[i]
return mask
def pack_waste(docs, length):
"""How many pad slots you pay if each document sits in its own row."""
padded = length * len(docs)
packed = length
content = sum(len(d) for d in docs)
return content, packed, padded
def fim_rearrange(tokens, split):
"""Split a document into prefix / middle / suffix and rearrange.
Training order is: <pre> prefix <suf> suffix <mid> middle.
The model then predicts the missing middle after seeing both sides.
"""
prefix, middle, suffix = tokens[:split[0]], tokens[split[0]:split[1]], tokens[split[1]:]
return [FIM_PRE, *prefix, FIM_SUF, *suffix, FIM_MID, *middle]
A batch of shape \((B, T)\) is \(B \times T\) tokens. It is not \(B\) documents. One row can hold three notes, as above. Another row can hold a single long article that was cut to length 16.
If you only apply a causal mask, “tea” can attend to “rain fell.” That leaks a document that was never part of the same text. The careful fix is a segmentation mask: token \(i\) may attend to token \(j\) only when \(j \le i\) and both IDs belong to the same document.
In the packed row, “tea” (index 7) may see itself. It may not see the EOS of “rain fell” (index 6), and it may not see “is” (index 8) yet. Padding rows are all blocked.
Cheap. Some trainers accept the leak and rely on the EOS token to mark a reset. Positions still restart.
Cleaner. Attention cannot cross the card between books. This is what the allowed_mask function builds.
Offline, a cluster job (often Spark or a Rust/Python shard builder) writes these packed rows to disk. The training loop only reads integers. It never sees the original strings.
Next-token training only ever asks “what comes after this prefix?” Code editors also need the other question: “what belongs in this gap?” Fill-in-the-middle (FIM) teaches that by rearranging the document before packing.
Take one new sentence, still word-level: the red kite flew high. Split after “red” and after “kite.”
the red. Middle = kite. Suffix = flew high.FIM_PRE the red FIM_SUF flew high FIM_MID kite.IDs: 2, 41, 42, 3, 44, 45, 4, 43. The model reads both sides, then has to emit the missing middle. That is the autocomplete used in an editor: you have the function header and the closing lines, and you want the body.
FIM is a data format, not a new architecture. The same causal decoder, the same packed rows. You just sprinkle rearranged documents into the mix so some of the next-token targets sit in a gap.
Treating one batch row as one document. A row is a crate of tokens. Count documents with EOS and doc ids, not with \(B\).
Letting positions run across a pack. If “tea” is at position 7, rotary or learned positions think it is late in a single story. Reset to 0 at each document start.
Confusing loss masks with attention masks. SFT often zeros the loss on the prompt. The answer tokens still attend to that prompt. Prefix-LM attention is a different family.
Skipping the document mask and calling it free context. Cross-document attention is not “more data.” It is two unrelated texts glued together.
5 pads. Three separate rows would use 48 slots for the same 11 tokens.
It is the first token of a new document. Position 7 would tell the model it is late in one long story.
Not under a document mask. They have different doc ids. A causal-only mask would allow it, which is the leak.
FIM_PRE the red FIM_SUF flew high FIM_MID kite.
No. It changes how some documents are laid out before they are packed.