THE EVOLVING TRANSFORMER · STOP 03

The training loop

A five-step inner loop later stops can reuse. The corpus is a short harbour log — not a public play.

~10 min readNext-token CEAdamW

What you'll learn

  • Shifted windows: x is the prefix, y is the next character
  • The five-step step: zero_grad, forward, CE, clip, AdamW
  • How to swap build_model for a later stop

Before you start

A loop, not a new architecture

Stops 01 and 02 only forwarded random ids. This stop trains. Any module whose forward(x) returns (B, S, vocab) logits can sit in build_model. The default is a tiny GPT-2 — 32-wide, two layers — so a laptop finishes eight steps in a blink. Later stops keep that contract. Mixtral and DeepSeek also return an extra loss term; add it in the criterion when you swap them in.

A night clerk copies yesterday's harbour log one letter ahead of the cursor. The model sees “dawn watch: the har” and must name “b”. That shift is the whole objective.

The “why” of cross-entropy and Adam lives in Neurons and the Training Ground. Neural language models →

One batch of the harbour log

Sequence length 16, batch 4. Vocab is the number of distinct characters in the log (spaces and punctuation count).

harbour log · windows · train ↔ generate

MANIM Same model: cut windows, train_step, then generate.
TensorShapeNote
x(4, 16)characters 0…15 of each window
y(4, 16)characters 1…16 — the targets
logits(4, 16, 27)one row per position, 27-char vocab

A random model starts near ln(27) ≈ 3.30. After a few AdamW steps the loss should move.

Dataset, then the step

dataset.pywindows + chronological split
"""Tiny character-level corpus for the training-loop stop.

Own text — a short harbour log, not a public play. The tokenizer is
a sorted character map with no unknown token.
"""

import torch
from torch.utils.data import DataLoader, Dataset

# A few hundred characters. Enough to build windows and take a step.
CORPUS = (
    "dawn watch: the harbour bell rings twice. fog sits on the pier. "
    "a red lantern marks the channel. gulls lift, then settle. "
    "noon watch: ropes coil, crates stack, the tide turns east. "
    "dusk watch: lamps bloom along the quay. the log is closed.\n"
)


class CharTokenizer:
    """One id per unique character. encode / decode are inverses."""

    def __init__(self, text):
        chars = sorted(set(text))
        self.vocab_size = len(chars)
        self.char_to_id = {c: i for i, c in enumerate(chars)}
        self.id_to_char = {i: c for i, c in enumerate(chars)}

    def encode(self, text):
        return [self.char_to_id[c] for c in text]

    def decode(self, ids):
        return "".join(self.id_to_char[i] for i in ids)


class CharDataset(Dataset):
    """Sliding windows: x = tokens[i : i+S], y = tokens[i+1 : i+S+1]."""

    def __init__(self, token_ids, max_seq_len):
        self.data = torch.tensor(token_ids, dtype=torch.long)
        self.max_seq_len = max_seq_len

    def __len__(self):
        return len(self.data) - self.max_seq_len

    def __getitem__(self, idx):
        chunk = self.data[idx : idx + self.max_seq_len + 1]
        return chunk[:-1], chunk[1:]


def get_dataloaders(max_seq_len, batch_size, val_fraction=0.2, text=CORPUS):
    """Chronological split, then windowed (x, y) batches."""
    tokenizer = CharTokenizer(text)
    ids = tokenizer.encode(text)
    split = int(len(ids) * (1 - val_fraction))
    # keep a little room so both splits can form at least one window
    split = max(max_seq_len + 1, min(split, len(ids) - max_seq_len - 1))
    train_ds = CharDataset(ids[:split], max_seq_len)
    val_ds = CharDataset(ids[split:], max_seq_len)
    train_loader = DataLoader(train_ds, batch_size=batch_size, shuffle=True, drop_last=True)
    val_loader = DataLoader(val_ds, batch_size=batch_size, shuffle=False, drop_last=True)
    return train_loader, val_loader, tokenizer
train.pyone train_step + generate
"""A model-agnostic next-token loop later stops can reuse.

Any module whose forward(x) returns (B, S, vocab) logits works.
The default model is a tiny GPT-2 from stop 02 — not the 124M width.
"""

import sys
from pathlib import Path

import torch
import torch.nn as nn
from torch.optim import AdamW

from dataset import CORPUS, get_dataloaders

_GPT2_DIR = Path(__file__).resolve().parent.parent / "02-gpt2"
if str(_GPT2_DIR) not in sys.path:
    sys.path.insert(0, str(_GPT2_DIR))
from gpt2 import GPT, GPTConfig  # noqa: E402

MAX_SEQ_LEN = 16
BATCH_SIZE = 4
LR = 3e-4
N_STEPS = 8


def build_model(vocab_size, max_seq_len=MAX_SEQ_LEN):
    """Tiny decoder. Swap this to train a later stop's model."""
    config = GPTConfig(
        dmodel=32,
        vocabsize=vocab_size,
        max_seq_len=max_seq_len,
        N=2,
        H=4,
        dropout=0.0,
    )
    return GPT(config)


def train_step(model, optimizer, criterion, x, y):
    """zero_grad → forward → CE → backward → clip → step."""
    optimizer.zero_grad()
    logits = model(x)
    loss = criterion(logits.reshape(-1, logits.size(-1)), y.reshape(-1))
    loss.backward()
    nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
    optimizer.step()
    return loss.item(), logits


@torch.no_grad()
def generate(model, tokenizer, prompt, max_new_tokens=20, temperature=1.0):
    """Autoregressive sample from the last position."""
    model.eval()
    ids = torch.tensor([tokenizer.encode(prompt)], dtype=torch.long)
    for _ in range(max_new_tokens):
        logits = model(ids[:, -model.config.max_seq_len :])
        probs = torch.softmax(logits[:, -1, :] / temperature, dim=-1)
        nxt = torch.multinomial(probs, num_samples=1)
        ids = torch.cat([ids, nxt], dim=1)
    return tokenizer.decode(ids[0].tolist())


def run_loop(n_steps=N_STEPS):
    train_loader, _, tokenizer = get_dataloaders(MAX_SEQ_LEN, BATCH_SIZE)
    model = build_model(tokenizer.vocab_size)
    optimizer = AdamW(model.parameters(), lr=LR)
    criterion = nn.CrossEntropyLoss()
    model.train()
    iterator = iter(train_loader)
    losses = []
    for _ in range(n_steps):
        try:
            x, y = next(iterator)
        except StopIteration:
            iterator = iter(train_loader)
            x, y = next(iterator)
        loss, _ = train_step(model, optimizer, criterion, x, y)
        losses.append(loss)
    return model, tokenizer, losses


if __name__ == "__main__":
    model, tokenizer, losses = run_loop()
    print("vocab:", tokenizer.vocab_size, "corpus chars:", len(CORPUS))
    print("losses:", [round(v, 4) for v in losses])
    print(generate(model, tokenizer, "dawn"))

Test note. The check builds one batch, asserts x / y / logits, runs train_step, and checks the loss is finite and not identical after a step.

Mistakes this stop used to make

Training the 124M width on a laptop. build_model here is 32-wide. The published GPT-2 config stays on stop 02's page.

Assuming every later forward returns only logits. Mixtral returns (logits, aux). DeepSeek returns MTP logits too. Adapt the criterion.

Shuffling before the train/val split. The split is chronological so the val window is later in the log.

What to remember

  • y is x shifted by one. Cross-entropy is averaged over B×S.
  • Inner loop: zero_grad → forward → CE → clip 1.0 → AdamW.
  • Swap build_model to train a later stop. Keep a tiny config.

Check yourself

1. If x is “dawn”, what is the first target in y?

The second character, “a”.

2. What shape are logits for batch 4, length 16, vocab 27?

(4, 16, 27).

3. Why clip gradients at 1.0?

A single batch can spike the gradient. Clipping keeps AdamW from taking a wild step.

Go to the source

Keep going