THE TRANSFORMER DEEP CUTS · CHAPTER 03

Build a GPT
from Scratch

A character-level decoder you can run. The corpus is a four-line harbour log. The model is two tiny blocks.

~15 min read1 interactiveTested PyTorch code

What you'll learn

  • How to turn a small text into a character-level dataset
  • How token and position embeddings add, then feed a stack of blocks
  • How to train with next-character cross-entropy and sample from the result

Before you start

A log the model can actually finish

A keeper writes the same kind of line every dusk and every dawn. After a week of those lines, you can finish the next one yourself. A tiny GPT is that habit, stored in weights.

The lab is small on purpose. If the corpus is a patterned loop, a two-layer model on a laptop CPU can learn it in seconds. You are here to see the pieces click, not to beat a benchmark.

1,312 characters, 21 symbols

Two lines, repeated sixteen times:

harbour log
dusk. fog on the reef. the lamp is lit.
dawn. gulls on the roof. the lamp is out.

Character-level: each distinct symbol is a token. The alphabet is 21 symbols (letters, spaces, dots, a newline). A uniform guess at the next character has loss \(\ln 21 \approx\) 3.04. If your first forward pass lands near that, the loss is wired the right way around. If it lands far above, the logits are a mess. If it lands far below, you are already leaking the answer.

ln 21  →  AdamW  →  loss falls  →  dusk.

MANIM Fresh loss ≈ ln 21; train drops; samples look like the log.

Training slices the stream into windows of length 32. The target is the same window shifted one character left. Every position is a training case.

Embed, block, block, head

PieceShape in / outLab default
token ids(B, T)B=2, T=32
token + position embed(B, T, n_embd)(2, 32, 32)
attention scores, per head(B, h, T, T)(2, 4, 32, 32)
logits(B, T, vocab)(2, 32, 21)

embed  →  2 blocks  →  21 logits  →  sample

MANIM Embed → blocks → logits: the whole harbour stack once.

Four heads of size \(32 / 4 = 8\). Two pre-norm blocks. A final LayerNorm, then a linear map to 21 logits. Position embeddings are learned and added, GPT-2 style. Rotary positions wait for chapter 04.

Generation crops the prefix to the context window, reads the last logit row, samples, and appends. That is the whole decode loop.

prefix grows  →  hit 32  →  [:, -32:]  →  sample · append

MANIM Decode must crop to block_size or the position table breaks.

Interactive · What the loss floor means

If the alphabet had \(V\) characters, where should a fresh model sit?

21

AdamW, a few dozen steps

Cross-entropy on the shifted window. AdamW at \(3 \times 10^{-3}\) is plenty for this toy. After a handful of steps the loss should fall below 3.04. After a few dozen, samples start to look like the log: dusk. is a good prompt; the model has seen it 16 times.

This is not GPT-2. GPT-2 124M is 12 layers, 12 heads, width 768, context 1024, vocab 50,257, with tied embeddings. The next Evolving Transformer stop builds that one. This lab is the skeleton.

The whole model, one file

Attention includes \(W_O\). The block is pre-norm. generate crops with idx[:, -block_size:] so a long sample does not blow the position table.

tiny_gpt.pyforward (2, 32, 21); a few Adam steps drop the loss
"""A small character-level GPT you can run.

The corpus is an original four-line harbour log, repeated so a tiny
model has a pattern to copy. It is not Shakespeare and it is not a
video leftover. Train it, then sample from it.
"""
import torch
import torch.nn as nn
import torch.nn.functional as F

# Four lines, two times of day. The model only has to learn a short loop.
LINE_DUSK = "dusk. fog on the reef. the lamp is lit.\n"
LINE_DAWN = "dawn. gulls on the roof. the lamp is out.\n"
CORPUS = (LINE_DUSK + LINE_DAWN) * 16

# Character-level: every distinct symbol is one token. Tiny vocab, no BPE.
CHARS = sorted(set(CORPUS))
STOL, ITOS = {ch: i for i, ch in enumerate(CHARS)}, {i: ch for i, ch in enumerate(CHARS)}
VOCAB = len(CHARS)


def encode(text):
    return torch.tensor([STOL[c] for c in text], dtype=torch.long)


def decode(ids):
    return "".join(ITOS[int(i)] for i in ids)


class CausalSelfAttention(nn.Module):
    """One multi-head block. Same idea as chapter 02, sized for this lab."""

    def __init__(self, n_embd, n_head, block_size):
        super().__init__()
        assert n_embd % n_head == 0
        self.n_head = n_head
        self.d_k = n_embd // n_head
        self.qkv = nn.Linear(n_embd, 3 * n_embd, bias=False)
        self.proj = nn.Linear(n_embd, n_embd, bias=False)  # W_O
        # Lower triangle: token i may look at 0..i, never i+1..
        mask = torch.tril(torch.ones(block_size, block_size))
        self.register_buffer("mask", mask.view(1, 1, block_size, block_size))

    def forward(self, x):
        b, t, c = x.shape
        q, k, v = self.qkv(x).split(c, dim=-1)
        q = q.view(b, t, self.n_head, self.d_k).transpose(1, 2)
        k = k.view(b, t, self.n_head, self.d_k).transpose(1, 2)
        v = v.view(b, t, self.n_head, self.d_k).transpose(1, 2)
        scores = (q @ k.transpose(-2, -1)) / self.d_k**0.5
        scores = scores.masked_fill(self.mask[:, :, :t, :t] == 0, float("-inf"))
        return self.proj((F.softmax(scores, dim=-1) @ v).transpose(1, 2).contiguous().view(b, t, c))


class Block(nn.Module):
    """Pre-norm decoder block: norm, attend, residual, then norm, FFN, residual."""

    def __init__(self, n_embd, n_head, block_size):
        super().__init__()
        self.ln1 = nn.LayerNorm(n_embd)
        self.attn = CausalSelfAttention(n_embd, n_head, block_size)
        self.ln2 = nn.LayerNorm(n_embd)
        self.ffn = nn.Sequential(
            nn.Linear(n_embd, 4 * n_embd),
            nn.GELU(),
            nn.Linear(4 * n_embd, n_embd),
        )

    def forward(self, x):
        x = x + self.attn(self.ln1(x))
        x = x + self.ffn(self.ln2(x))
        return x


class TinyGPT(nn.Module):
    """Token embed + position embed + N blocks + language-model head."""

    def __init__(self, vocab, n_embd, n_head, n_layer, block_size):
        super().__init__()
        self.block_size = block_size
        self.tok = nn.Embedding(vocab, n_embd)
        self.pos = nn.Embedding(block_size, n_embd)
        self.blocks = nn.ModuleList(Block(n_embd, n_head, block_size) for _ in range(n_layer))
        self.ln_f = nn.LayerNorm(n_embd)
        self.head = nn.Linear(n_embd, vocab, bias=False)

    def forward(self, idx):
        b, t = idx.shape
        x = self.tok(idx) + self.pos(torch.arange(t, device=idx.device))
        for block in self.blocks:
            x = block(x)
        return self.head(self.ln_f(x))  # (B, T, vocab)

    @torch.no_grad()
    def generate(self, idx, n_new):
        """Grow a sequence one character at a time. Crop to the context window."""
        for _ in range(n_new):
            logits = self(idx[:, -self.block_size :])[:, -1, :]
            idx = torch.cat([idx, torch.multinomial(F.softmax(logits, dim=-1), 1)], dim=1)
        return idx


# Lab defaults. Small enough to train on a laptop CPU in seconds.
LAB = dict(n_embd=32, n_head=4, n_layer=2, block_size=32)

Mistakes learners make

Reading a first loss above \(\ln V\) as "the wiring works." Above uniform means the init is peaked on the wrong characters. Near \(\ln V\) is the sanity check.

Forgetting to crop at generate time. Position embeddings only exist for block_size rows. A longer prefix must be sliced.

Skipping the scale inside the lab head. The scores still need \(\sqrt{d_k}\). The file divides by d_k ** 0.5.

What to remember

  • A decoder is embeddings + a stack of (norm, attend, residual, norm, FFN, residual) + a vocab head.
  • Character-level keeps the tokenizer out of the way. This log has 1,312 characters and 21 symbols.
  • Train on a one-step shift. Sample from the last logit row. Crop to the window.

Check yourself

1. Why is the first-loss target 3.04, not 0?

\(\ln 21 \approx 3.04\) is a uniform guess over the 21-character alphabet. A fresh model should sit near it.

2. What is \(d_k\) in the lab config?

\(32 / 4 = 8\).

3. Why add a position embedding at all?

Attention is order-blind. Without positions, dusk and dawn would be bags of the same letters in a different arrangement. Chapter 04 is the longer answer.

Go to the source

Keep going