THE EVOLVING TRANSFORMER · STOP 03
A five-step inner loop later stops can reuse. The corpus is a short harbour log — not a public play.
x is the prefix, y is the next characterbuild_model for a later stopStops 01 and 02 only forwarded random ids. This stop trains. Any module whose forward(x) returns (B, S, vocab) logits can sit in build_model. The default is a tiny GPT-2 — 32-wide, two layers — so a laptop finishes eight steps in a blink. Later stops keep that contract. Mixtral and DeepSeek also return an extra loss term; add it in the criterion when you swap them in.
A night clerk copies yesterday's harbour log one letter ahead of the cursor. The model sees “dawn watch: the har” and must name “b”. That shift is the whole objective.
The “why” of cross-entropy and Adam lives in Neurons and the Training Ground. Neural language models →
Sequence length 16, batch 4. Vocab is the number of distinct characters in the log (spaces and punctuation count).
harbour log · windows · train ↔ generate
| Tensor | Shape | Note |
|---|---|---|
x | (4, 16) | characters 0…15 of each window |
y | (4, 16) | characters 1…16 — the targets |
| logits | (4, 16, 27) | one row per position, 27-char vocab |
A random model starts near ln(27) ≈ 3.30. After a few AdamW steps the loss should move.
"""Tiny character-level corpus for the training-loop stop.
Own text — a short harbour log, not a public play. The tokenizer is
a sorted character map with no unknown token.
"""
import torch
from torch.utils.data import DataLoader, Dataset
# A few hundred characters. Enough to build windows and take a step.
CORPUS = (
"dawn watch: the harbour bell rings twice. fog sits on the pier. "
"a red lantern marks the channel. gulls lift, then settle. "
"noon watch: ropes coil, crates stack, the tide turns east. "
"dusk watch: lamps bloom along the quay. the log is closed.\n"
)
class CharTokenizer:
"""One id per unique character. encode / decode are inverses."""
def __init__(self, text):
chars = sorted(set(text))
self.vocab_size = len(chars)
self.char_to_id = {c: i for i, c in enumerate(chars)}
self.id_to_char = {i: c for i, c in enumerate(chars)}
def encode(self, text):
return [self.char_to_id[c] for c in text]
def decode(self, ids):
return "".join(self.id_to_char[i] for i in ids)
class CharDataset(Dataset):
"""Sliding windows: x = tokens[i : i+S], y = tokens[i+1 : i+S+1]."""
def __init__(self, token_ids, max_seq_len):
self.data = torch.tensor(token_ids, dtype=torch.long)
self.max_seq_len = max_seq_len
def __len__(self):
return len(self.data) - self.max_seq_len
def __getitem__(self, idx):
chunk = self.data[idx : idx + self.max_seq_len + 1]
return chunk[:-1], chunk[1:]
def get_dataloaders(max_seq_len, batch_size, val_fraction=0.2, text=CORPUS):
"""Chronological split, then windowed (x, y) batches."""
tokenizer = CharTokenizer(text)
ids = tokenizer.encode(text)
split = int(len(ids) * (1 - val_fraction))
# keep a little room so both splits can form at least one window
split = max(max_seq_len + 1, min(split, len(ids) - max_seq_len - 1))
train_ds = CharDataset(ids[:split], max_seq_len)
val_ds = CharDataset(ids[split:], max_seq_len)
train_loader = DataLoader(train_ds, batch_size=batch_size, shuffle=True, drop_last=True)
val_loader = DataLoader(val_ds, batch_size=batch_size, shuffle=False, drop_last=True)
return train_loader, val_loader, tokenizer
"""A model-agnostic next-token loop later stops can reuse.
Any module whose forward(x) returns (B, S, vocab) logits works.
The default model is a tiny GPT-2 from stop 02 — not the 124M width.
"""
import sys
from pathlib import Path
import torch
import torch.nn as nn
from torch.optim import AdamW
from dataset import CORPUS, get_dataloaders
_GPT2_DIR = Path(__file__).resolve().parent.parent / "02-gpt2"
if str(_GPT2_DIR) not in sys.path:
sys.path.insert(0, str(_GPT2_DIR))
from gpt2 import GPT, GPTConfig # noqa: E402
MAX_SEQ_LEN = 16
BATCH_SIZE = 4
LR = 3e-4
N_STEPS = 8
def build_model(vocab_size, max_seq_len=MAX_SEQ_LEN):
"""Tiny decoder. Swap this to train a later stop's model."""
config = GPTConfig(
dmodel=32,
vocabsize=vocab_size,
max_seq_len=max_seq_len,
N=2,
H=4,
dropout=0.0,
)
return GPT(config)
def train_step(model, optimizer, criterion, x, y):
"""zero_grad → forward → CE → backward → clip → step."""
optimizer.zero_grad()
logits = model(x)
loss = criterion(logits.reshape(-1, logits.size(-1)), y.reshape(-1))
loss.backward()
nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()
return loss.item(), logits
@torch.no_grad()
def generate(model, tokenizer, prompt, max_new_tokens=20, temperature=1.0):
"""Autoregressive sample from the last position."""
model.eval()
ids = torch.tensor([tokenizer.encode(prompt)], dtype=torch.long)
for _ in range(max_new_tokens):
logits = model(ids[:, -model.config.max_seq_len :])
probs = torch.softmax(logits[:, -1, :] / temperature, dim=-1)
nxt = torch.multinomial(probs, num_samples=1)
ids = torch.cat([ids, nxt], dim=1)
return tokenizer.decode(ids[0].tolist())
def run_loop(n_steps=N_STEPS):
train_loader, _, tokenizer = get_dataloaders(MAX_SEQ_LEN, BATCH_SIZE)
model = build_model(tokenizer.vocab_size)
optimizer = AdamW(model.parameters(), lr=LR)
criterion = nn.CrossEntropyLoss()
model.train()
iterator = iter(train_loader)
losses = []
for _ in range(n_steps):
try:
x, y = next(iterator)
except StopIteration:
iterator = iter(train_loader)
x, y = next(iterator)
loss, _ = train_step(model, optimizer, criterion, x, y)
losses.append(loss)
return model, tokenizer, losses
if __name__ == "__main__":
model, tokenizer, losses = run_loop()
print("vocab:", tokenizer.vocab_size, "corpus chars:", len(CORPUS))
print("losses:", [round(v, 4) for v in losses])
print(generate(model, tokenizer, "dawn"))
Test note. The check builds one batch, asserts x / y / logits, runs train_step, and checks the loss is finite and not identical after a step.
Training the 124M width on a laptop. build_model here is 32-wide. The published GPT-2 config stays on stop 02's page.
Assuming every later forward returns only logits. Mixtral returns (logits, aux). DeepSeek returns MTP logits too. Adapt the criterion.
Shuffling before the train/val split. The split is chronological so the val window is later in the log.
y is x shifted by one. Cross-entropy is averaged over B×S.build_model to train a later stop. Keep a tiny config.x is “dawn”, what is the first target in y?The second character, “a”.
(4, 16, 27).
A single batch can spike the gradient. Clipping keeps AdamW from taking a wild step.