THE TRAINING GROUND · CHAPTER 05
A base model continues text. An instruct model answers. The jump is a small, carefully masked dataset and a template that names who is speaking.
Give a base model the line “When does the morning ferry leave?” and it may write another question, or a timetable fragment, or the rest of a blog post. It was trained to continue text, not to be a clerk.
Supervised fine-tuning shows it pairs of the form instruction → answer, written in a fixed wrapper so the model can see who is speaking. After that, the same decoder still predicts the next token. The tokens it is asked to predict are now answers.
SFT does not add a new head. It changes which tokens the loss is allowed to see. The architecture is the base model. The data is a stack of dialogues.
The objective is still next-token NLL. Two rules change.
clerk · template · mask · format
average surprise only on the answer positions A. The prompt is context, not a target.
That is why a book-length prompt with a one-line question does not drown the gradient. Without a mask, most of the loss would be “predict the next page of the book,” and the one-line answer would be a rounding error.
Attention stays causal. Answer tokens may look at the prompt. Prompt tokens may not look at the answer. Calling SFT a prefix-LM mask is the usual mix-up: prefix-LM is an attention pattern. SFT is a loss pattern.
A small, clean set can go a long way (the LIMA result: thousands of curated pairs, not millions). Style tics in the data — a favorite opener, a punctuation habit — become the model’s tics. Preference training (RLHF, DPO) comes later, in Reasoning from Scratch.
A template is a string recipe. It wraps each turn in special tokens so the model can tell a system note from a user question from an assistant answer. Families disagree on the spelling. Using the wrong wrapper at serve time is a common source of “the model ignores the system prompt.”
Our own tiny recipe, used only on this page:
open marker, role name, newline, the words, close marker. Repeat for every turn.
The harbor-clerk dialogue, fully wrapped:
| Role | Text |
|---|---|
| system | You are a harbor clerk. |
| user | When does the morning ferry leave? |
| assistant | It leaves at 06:40. |
Rendered as one training string (newlines shown as ⏎):
<|begin|>system⏎You are a harbor clerk.<|end|><|begin|>user⏎When does the morning ferry leave?<|end|><|begin|>assistant⏎It leaves at 06:40.<|end|>
Word-level tokenize that string and you get 27 tokens. The loss mask is 1 on exactly five of them: It, leaves, at, 06:40., and the closing <|end|>.
| Span | Tokens | Loss |
|---|---|---|
| system + user + assistant header | 22 | 0 |
| assistant answer + stop | 5 | 1 |
"""Render a tiny dialogue with a chat template and a response-only loss mask.
The harbor-clerk dialogue is the worked example on the page. Tokenization
is word-level so the mask is easy to check by hand.
"""
BOS, EOS = "<|begin|>", "<|end|>"
# One conversation. The system and user turns are the prompt. The assistant
# turn is the only span the SFT loss is allowed to train on.
DIALOGUE = [
("system", "You are a harbor clerk."),
("user", "When does the morning ferry leave?"),
("assistant", "It leaves at 06:40."),
]
def render(messages):
"""Turn role/text pairs into one training string.
Each turn is wrapped as: <|begin|>role\\ntext<|end|>
The newline after the role is part of the template, not the text.
"""
parts = []
for role, text in messages:
parts.append(f"{BOS}{role}\n{text}{EOS}")
return "".join(parts)
def tokenize(rendered):
"""Split on the template markers and whitespace. Keep the markers."""
tokens = []
rest = rendered
while rest:
if rest.startswith(BOS):
tokens.append(BOS)
rest = rest[len(BOS):]
continue
if rest.startswith(EOS):
tokens.append(EOS)
rest = rest[len(EOS):]
continue
if rest[0].isspace():
# Keep a single newline as its own token; skip other spaces.
if rest[0] == "\n":
tokens.append("\n")
rest = rest[1:]
continue
i = 1
while i < len(rest) and (not rest[i].isspace()) and not rest.startswith(BOS, i) and not rest.startswith(EOS, i):
i += 1
tokens.append(rest[:i])
rest = rest[i:]
return tokens
def loss_mask(tokens):
"""1 on assistant content (and its closing <|end|>), 0 everywhere else.
The opening <|begin|>assistant and the role newline stay untrained:
they are prompt, not answer. The closing <|end|> is part of the answer
so the model learns to stop.
"""
mask = [0] * len(tokens)
i = 0
while i < len(tokens):
if tokens[i] == BOS and i + 1 < len(tokens) and tokens[i + 1] == "assistant":
# Skip <|begin|>, "assistant", and the role newline.
i += 3
while i < len(tokens):
mask[i] = 1
if tokens[i] == EOS:
i += 1
break
i += 1
continue
i += 1
return mask
def nll_over_response(log_probs, mask):
"""Mean negative log-prob on the masked (assistant) positions only."""
picked = [lp for lp, m in zip(log_probs, mask) if m]
return -sum(picked) / len(picked)
Training on the prompt. Then the model gets better at repeating instructions, not at answering them.
Serving with a different template than you trained on. The special tokens are part of the language the model learned. Swap them and the clerk forgets the counter.
Calling the SFT mask a prefix-LM attention mask. Attention is still a causal triangle. Only the loss is masked.
Expecting SFT to add knowledge it never saw. SFT steers format and compliance. Facts still come from pre-training, retrieval, or later preference stages.
It, leaves, at, 06:40., and the closing <|end|>. Five tokens.
It is part of the prompt. The model should not have to “predict that it is the assistant.” It should predict the answer.
Yes. The mask is on the loss, not on attention.
<|begin|> and serve with <|user|>. What goes wrong?The model never saw those serve-time markers as role wrappers. It will not reliably treat the text as a user turn.
It is the first adaptation. Preferences and refusals usually need a later stage.