THE TRANSFORMER DEEP CUTS · CHAPTER 03
A character-level decoder you can run. The corpus is a four-line harbour log. The model is two tiny blocks.
nn.Module and cross_entropyA keeper writes the same kind of line every dusk and every dawn. After a week of those lines, you can finish the next one yourself. A tiny GPT is that habit, stored in weights.
The lab is small on purpose. If the corpus is a patterned loop, a two-layer model on a laptop CPU can learn it in seconds. You are here to see the pieces click, not to beat a benchmark.
Two lines, repeated sixteen times:
dusk. fog on the reef. the lamp is lit. dawn. gulls on the roof. the lamp is out.
Character-level: each distinct symbol is a token. The alphabet is 21 symbols (letters, spaces, dots, a newline). A uniform guess at the next character has loss \(\ln 21 \approx\) 3.04. If your first forward pass lands near that, the loss is wired the right way around. If it lands far above, the logits are a mess. If it lands far below, you are already leaking the answer.
ln 21 → AdamW → loss falls → dusk.
Training slices the stream into windows of length 32. The target is the same window shifted one character left. Every position is a training case.
| Piece | Shape in / out | Lab default |
|---|---|---|
| token ids | (B, T) | B=2, T=32 |
| token + position embed | (B, T, n_embd) | (2, 32, 32) |
| attention scores, per head | (B, h, T, T) | (2, 4, 32, 32) |
| logits | (B, T, vocab) | (2, 32, 21) |
embed → 2 blocks → 21 logits → sample
Four heads of size \(32 / 4 = 8\). Two pre-norm blocks. A final LayerNorm, then a linear map to 21 logits. Position embeddings are learned and added, GPT-2 style. Rotary positions wait for chapter 04.
Generation crops the prefix to the context window, reads the last logit row, samples, and appends. That is the whole decode loop.
prefix grows
→
hit 32
→
[:, -32:]
→
sample · append
block_size or the position table breaks.Interactive · What the loss floor means
If the alphabet had \(V\) characters, where should a fresh model sit?
Cross-entropy on the shifted window. AdamW at \(3 \times 10^{-3}\) is plenty for this toy. After a handful of steps the loss should fall below 3.04. After a few dozen, samples start to look like the log: dusk. is a good prompt; the model has seen it 16 times.
This is not GPT-2. GPT-2 124M is 12 layers, 12 heads, width 768, context 1024, vocab 50,257, with tied embeddings. The next Evolving Transformer stop builds that one. This lab is the skeleton.
Attention includes \(W_O\). The block is pre-norm. generate crops with idx[:, -block_size:] so a long sample does not blow the position table.
"""A small character-level GPT you can run.
The corpus is an original four-line harbour log, repeated so a tiny
model has a pattern to copy. It is not Shakespeare and it is not a
video leftover. Train it, then sample from it.
"""
import torch
import torch.nn as nn
import torch.nn.functional as F
# Four lines, two times of day. The model only has to learn a short loop.
LINE_DUSK = "dusk. fog on the reef. the lamp is lit.\n"
LINE_DAWN = "dawn. gulls on the roof. the lamp is out.\n"
CORPUS = (LINE_DUSK + LINE_DAWN) * 16
# Character-level: every distinct symbol is one token. Tiny vocab, no BPE.
CHARS = sorted(set(CORPUS))
STOL, ITOS = {ch: i for i, ch in enumerate(CHARS)}, {i: ch for i, ch in enumerate(CHARS)}
VOCAB = len(CHARS)
def encode(text):
return torch.tensor([STOL[c] for c in text], dtype=torch.long)
def decode(ids):
return "".join(ITOS[int(i)] for i in ids)
class CausalSelfAttention(nn.Module):
"""One multi-head block. Same idea as chapter 02, sized for this lab."""
def __init__(self, n_embd, n_head, block_size):
super().__init__()
assert n_embd % n_head == 0
self.n_head = n_head
self.d_k = n_embd // n_head
self.qkv = nn.Linear(n_embd, 3 * n_embd, bias=False)
self.proj = nn.Linear(n_embd, n_embd, bias=False) # W_O
# Lower triangle: token i may look at 0..i, never i+1..
mask = torch.tril(torch.ones(block_size, block_size))
self.register_buffer("mask", mask.view(1, 1, block_size, block_size))
def forward(self, x):
b, t, c = x.shape
q, k, v = self.qkv(x).split(c, dim=-1)
q = q.view(b, t, self.n_head, self.d_k).transpose(1, 2)
k = k.view(b, t, self.n_head, self.d_k).transpose(1, 2)
v = v.view(b, t, self.n_head, self.d_k).transpose(1, 2)
scores = (q @ k.transpose(-2, -1)) / self.d_k**0.5
scores = scores.masked_fill(self.mask[:, :, :t, :t] == 0, float("-inf"))
return self.proj((F.softmax(scores, dim=-1) @ v).transpose(1, 2).contiguous().view(b, t, c))
class Block(nn.Module):
"""Pre-norm decoder block: norm, attend, residual, then norm, FFN, residual."""
def __init__(self, n_embd, n_head, block_size):
super().__init__()
self.ln1 = nn.LayerNorm(n_embd)
self.attn = CausalSelfAttention(n_embd, n_head, block_size)
self.ln2 = nn.LayerNorm(n_embd)
self.ffn = nn.Sequential(
nn.Linear(n_embd, 4 * n_embd),
nn.GELU(),
nn.Linear(4 * n_embd, n_embd),
)
def forward(self, x):
x = x + self.attn(self.ln1(x))
x = x + self.ffn(self.ln2(x))
return x
class TinyGPT(nn.Module):
"""Token embed + position embed + N blocks + language-model head."""
def __init__(self, vocab, n_embd, n_head, n_layer, block_size):
super().__init__()
self.block_size = block_size
self.tok = nn.Embedding(vocab, n_embd)
self.pos = nn.Embedding(block_size, n_embd)
self.blocks = nn.ModuleList(Block(n_embd, n_head, block_size) for _ in range(n_layer))
self.ln_f = nn.LayerNorm(n_embd)
self.head = nn.Linear(n_embd, vocab, bias=False)
def forward(self, idx):
b, t = idx.shape
x = self.tok(idx) + self.pos(torch.arange(t, device=idx.device))
for block in self.blocks:
x = block(x)
return self.head(self.ln_f(x)) # (B, T, vocab)
@torch.no_grad()
def generate(self, idx, n_new):
"""Grow a sequence one character at a time. Crop to the context window."""
for _ in range(n_new):
logits = self(idx[:, -self.block_size :])[:, -1, :]
idx = torch.cat([idx, torch.multinomial(F.softmax(logits, dim=-1), 1)], dim=1)
return idx
# Lab defaults. Small enough to train on a laptop CPU in seconds.
LAB = dict(n_embd=32, n_head=4, n_layer=2, block_size=32)
Reading a first loss above \(\ln V\) as "the wiring works." Above uniform means the init is peaked on the wrong characters. Near \(\ln V\) is the sanity check.
Forgetting to crop at generate time. Position embeddings only exist for block_size rows. A longer prefix must be sliced.
Skipping the scale inside the lab head. The scores still need \(\sqrt{d_k}\). The file divides by d_k ** 0.5.
\(\ln 21 \approx 3.04\) is a uniform guess over the 21-character alphabet. A fresh model should sit near it.
\(32 / 4 = 8\).
Attention is order-blind. Without positions, dusk and dawn would be bags of the same letters in a different arrangement. Chapter 04 is the longer answer.