THE TRAINING GROUND · CHAPTER 05

Supervised fine-tuning & chat templates

A base model continues text. An instruct model answers. The jump is a small, carefully masked dataset and a template that names who is speaking.

~16 min readHarbor-clerk dialogueTested Python

What you'll learn

  • What SFT changes, and what it leaves alone
  • Why every model family ships a chat template
  • How to mask the loss so only the assistant turn is trained
  • Why a long prompt still uses causal attention

Before you start

A clerk, not a tape recorder

Give a base model the line “When does the morning ferry leave?” and it may write another question, or a timetable fragment, or the rest of a blog post. It was trained to continue text, not to be a clerk.

Supervised fine-tuning shows it pairs of the form instruction → answer, written in a fixed wrapper so the model can see who is speaking. After that, the same decoder still predicts the next token. The tokens it is asked to predict are now answers.

SFT does not add a new head. It changes which tokens the loss is allowed to see. The architecture is the base model. The data is a stack of dialogues.

Same loss, smaller pile, answers only

The objective is still next-token NLL. Two rules change.

clerk · template · mask · format

MANIM SFT teaches the clerk format; mask loss to assistant turns.
  1. The documents are instruction–response pairs (or multi-turn chats), not a web crawl.
  2. The loss is zero on the prompt. Only the assistant tokens — and their stop marker — receive gradient.

average surprise only on the answer positions A. The prompt is context, not a target.

That is why a book-length prompt with a one-line question does not drown the gradient. Without a mask, most of the loss would be “predict the next page of the book,” and the one-line answer would be a rounding error.

Attention stays causal. Answer tokens may look at the prompt. Prompt tokens may not look at the answer. Calling SFT a prefix-LM mask is the usual mix-up: prefix-LM is an attention pattern. SFT is a loss pattern.

A small, clean set can go a long way (the LIMA result: thousands of curated pairs, not millions). Style tics in the data — a favorite opener, a punctuation habit — become the model’s tics. Preference training (RLHF, DPO) comes later, in Reasoning from Scratch.

Name who is talking, every time

A template is a string recipe. It wraps each turn in special tokens so the model can tell a system note from a user question from an assistant answer. Families disagree on the spelling. Using the wrong wrapper at serve time is a common source of “the model ignores the system prompt.”

Our own tiny recipe, used only on this page:

open marker, role name, newline, the words, close marker. Repeat for every turn.

The harbor-clerk dialogue, fully wrapped:

RoleText
systemYou are a harbor clerk.
userWhen does the morning ferry leave?
assistantIt leaves at 06:40.

Rendered as one training string (newlines shown as ⏎):

<|begin|>system⏎You are a harbor clerk.<|end|><|begin|>user⏎When does the morning ferry leave?<|end|><|begin|>assistant⏎It leaves at 06:40.<|end|>

27 tokens, 5 of them trained

Word-level tokenize that string and you get 27 tokens. The loss mask is 1 on exactly five of them: It, leaves, at, 06:40., and the closing <|end|>.

SpanTokensLoss
system + user + assistant header220
assistant answer + stop51
The header <|begin|> assistant ⏎ is still prompt. The stop marker is part of the answer, so the model learns when to shut up.
chat_sft.pyrender, 27 tokens, 5-position response mask
"""Render a tiny dialogue with a chat template and a response-only loss mask.

The harbor-clerk dialogue is the worked example on the page. Tokenization
is word-level so the mask is easy to check by hand.
"""

BOS, EOS = "<|begin|>", "<|end|>"

# One conversation. The system and user turns are the prompt. The assistant
# turn is the only span the SFT loss is allowed to train on.
DIALOGUE = [
    ("system", "You are a harbor clerk."),
    ("user", "When does the morning ferry leave?"),
    ("assistant", "It leaves at 06:40."),
]


def render(messages):
    """Turn role/text pairs into one training string.

    Each turn is wrapped as: <|begin|>role\\ntext<|end|>
    The newline after the role is part of the template, not the text.
    """
    parts = []
    for role, text in messages:
        parts.append(f"{BOS}{role}\n{text}{EOS}")
    return "".join(parts)


def tokenize(rendered):
    """Split on the template markers and whitespace. Keep the markers."""
    tokens = []
    rest = rendered
    while rest:
        if rest.startswith(BOS):
            tokens.append(BOS)
            rest = rest[len(BOS):]
            continue
        if rest.startswith(EOS):
            tokens.append(EOS)
            rest = rest[len(EOS):]
            continue
        if rest[0].isspace():
            # Keep a single newline as its own token; skip other spaces.
            if rest[0] == "\n":
                tokens.append("\n")
            rest = rest[1:]
            continue
        i = 1
        while i < len(rest) and (not rest[i].isspace()) and not rest.startswith(BOS, i) and not rest.startswith(EOS, i):
            i += 1
        tokens.append(rest[:i])
        rest = rest[i:]
    return tokens


def loss_mask(tokens):
    """1 on assistant content (and its closing <|end|>), 0 everywhere else.

    The opening <|begin|>assistant and the role newline stay untrained:
    they are prompt, not answer. The closing <|end|> is part of the answer
    so the model learns to stop.
    """
    mask = [0] * len(tokens)
    i = 0
    while i < len(tokens):
        if tokens[i] == BOS and i + 1 < len(tokens) and tokens[i + 1] == "assistant":
            # Skip <|begin|>, "assistant", and the role newline.
            i += 3
            while i < len(tokens):
                mask[i] = 1
                if tokens[i] == EOS:
                    i += 1
                    break
                i += 1
            continue
        i += 1
    return mask


def nll_over_response(log_probs, mask):
    """Mean negative log-prob on the masked (assistant) positions only."""
    picked = [lp for lp, m in zip(log_probs, mask) if m]
    return -sum(picked) / len(picked)

Mistakes learners make

Training on the prompt. Then the model gets better at repeating instructions, not at answering them.

Serving with a different template than you trained on. The special tokens are part of the language the model learned. Swap them and the clerk forgets the counter.

Calling the SFT mask a prefix-LM attention mask. Attention is still a causal triangle. Only the loss is masked.

Expecting SFT to add knowledge it never saw. SFT steers format and compliance. Facts still come from pre-training, retrieval, or later preference stages.

What to remember

  • SFT is next-token training on dialogues, with the loss masked to the assistant span.
  • A chat template names who is talking. Train and serve with the same one.
  • In the harbor-clerk example, 27 tokens, 5 trained.
  • Attention stays causal. Preference methods come after SFT.

Check yourself

1. Which tokens in the harbor-clerk row have loss 1?

It, leaves, at, 06:40., and the closing <|end|>. Five tokens.

2. Why is the assistant role header untrained?

It is part of the prompt. The model should not have to “predict that it is the assistant.” It should predict the answer.

3. Does SFT let answer tokens attend to the user turn?

Yes. The mask is on the loss, not on attention.

4. You fine-tune with <|begin|> and serve with <|user|>. What goes wrong?

The model never saw those serve-time markers as role wrappers. It will not reliably treat the text as a user turn.

5. Is SFT enough for “helpful and harmless”?

It is the first adaptation. Preferences and refusals usually need a later stage.

Go to the source

Keep going