THE EVOLVING TRANSFORMER · STOP 02

GPT-2

Cut the encoder. Learn the positions. Flip the residual to pre-norm. Tie the head to the token table, and name every weight the way the public checkpoint does.

~12 min readDecoder-onlyWeight-tied

What you'll learn

  • The GPT-2 124M config: 12 / 12 / 64 / 768, context 1024, vocab 50,257
  • Pre-norm residuals, learned wpe, tanh-GELU, fused c_attn
  • Weight tying: lm_head.weight is the same storage as wte
  • How to copy a Hugging Face state_dict, including Conv1D transposes

Before you start

Decoder only, and the names are real

Stop 01 had an encoder, sinusoids, post-norm, ReLU, and a separate projection. GPT-2 drops the encoder and cross-attention. Positions become a learned wpe. Residuals flip to pre-norm. The FFN uses GPT-2's tanh GELU. The language-model head shares storage with the token embedding — that is weight tying, and it saves about 38.6M parameters, not “half the model”.

Imagine reprinting a dictionary as its own index. The same rows that turn an id into a vector turn a vector back into a score over ids. One table, two jobs.

Init is N(0, 0.02). Residual projections (c_proj in attention and MLP) scale that by 1/√(2N) so the residual stream does not blow up as layers stack. Module names follow the public checkpoint: transformer.wte, transformer.h.0.attn.c_attn, transformer.ln_f. Why a decoder-only GPT →

Published 124M, and a tiny batch

KnobGPT-2 124M
layers / heads / head dim / hidden12 / 12 / 64 / 768
context / vocab1024 / 50,257
weight tyingyes — lm_head is wte

Teaching batch: tokens (2, 8), hidden 32, 4 heads, vocab 50.

shared W · embed · lm_head

MANIM lm_head.weight is wte.weight — one matrix, two jobs.

drop encoder · absolute wpe · (2,8,50)

MANIM Decoder-only stack; learned absolute positions; HF names.
TensorShape
tokens(2, 8)
hidden states(2, 8, 32)
c_attn.weight(96, 32)
logits(2, 8, 50)

Fused QKV and a loader

load_hf_state_dict copies matching names and transposes the four Conv1D weights. The check builds a fake Hugging Face-shaped dict — it does not download.

gpt2.pytiny forward + tying + HF names
"""GPT-2 124M — decoder-only, pre-norm, learned positions, tied head.

Published config: 12 layers, 12 heads, head dim 64, hidden 768,
context 1024, vocab 50,257. Weight-tied.

Module names match the public Hugging Face checkpoint so a real
state_dict can be copied in (Conv1D weights are transposed).
"""

from dataclasses import dataclass
import math

import torch
import torch.nn as nn
import torch.nn.functional as F


@dataclass
class GPTConfig:
    # GPT-2 124M published numbers. Tests pass a tiny override.
    dmodel: int = 768
    vocabsize: int = 50257
    max_seq_len: int = 1024
    N: int = 12
    H: int = 12
    dropout: float = 0.0

    @property
    def dk(self):
        return self.dmodel // self.H  # 768 / 12 = 64

    @property
    def dff(self):
        return 4 * self.dmodel


# Hugging Face stores these as Conv1D: weight shape is (in, out).
# nn.Linear wants (out, in), so we transpose them on load.
HF_TRANSPOSE = ("attn.c_attn.weight", "attn.c_proj.weight", "mlp.c_fc.weight", "mlp.c_proj.weight")


class CausalSelfAttention(nn.Module):
    """Fused QKV projection named c_attn, matching the real GPT-2 weights."""

    def __init__(self, config):
        super().__init__()
        self.H = config.H
        self.dk = config.dk
        self.dmodel = config.dmodel
        # one Linear: dmodel → 3 * dmodel, then split into Q, K, V
        self.c_attn = nn.Linear(config.dmodel, 3 * config.dmodel)
        self.c_proj = nn.Linear(config.dmodel, config.dmodel)
        self.c_proj.NANOGPT_SCALE_INIT = 1  # residual-path scale at init
        self.attn_dropout = nn.Dropout(config.dropout)
        self.resid_dropout = nn.Dropout(config.dropout)

    def forward(self, x, mask):
        B, S, _ = x.shape
        qkv = self.c_attn(x)
        q, k, v = qkv.split(self.dmodel, dim=-1)
        q = q.view(B, S, self.H, self.dk).transpose(1, 2)
        k = k.view(B, S, self.H, self.dk).transpose(1, 2)
        v = v.view(B, S, self.H, self.dk).transpose(1, 2)
        scores = (q @ k.transpose(-1, -2)) / math.sqrt(self.dk)
        scores = scores.masked_fill(mask[:, :, :S, :S] == 0, float("-inf"))
        weights = self.attn_dropout(scores.softmax(dim=-1))
        out = (weights @ v).transpose(1, 2).contiguous().view(B, S, self.dmodel)
        return self.resid_dropout(self.c_proj(out))


class MLP(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.c_fc = nn.Linear(config.dmodel, config.dff)
        self.c_proj = nn.Linear(config.dff, config.dmodel)
        self.c_proj.NANOGPT_SCALE_INIT = 1
        self.dropout = nn.Dropout(config.dropout)

    def forward(self, x):
        # GPT-2's GELU is the tanh approximation, not the exact erf form
        return self.dropout(self.c_proj(F.gelu(self.c_fc(x), approximate="tanh")))


class Block(nn.Module):
    """Pre-norm block: x + attn(LN(x)), then x + mlp(LN(x))."""

    def __init__(self, config):
        super().__init__()
        self.ln_1 = nn.LayerNorm(config.dmodel)
        self.attn = CausalSelfAttention(config)
        self.ln_2 = nn.LayerNorm(config.dmodel)
        self.mlp = MLP(config)

    def forward(self, x, mask):
        x = x + self.attn(self.ln_1(x), mask)
        x = x + self.mlp(self.ln_2(x))
        return x


class GPT(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config
        # names match HF: transformer.wte, transformer.wpe, transformer.h, transformer.ln_f
        self.transformer = nn.ModuleDict(
            dict(
                wte=nn.Embedding(config.vocabsize, config.dmodel),
                wpe=nn.Embedding(config.max_seq_len, config.dmodel),
                h=nn.ModuleList([Block(config) for _ in range(config.N)]),
                ln_f=nn.LayerNorm(config.dmodel),
            )
        )
        self.lm_head = nn.Linear(config.dmodel, config.vocabsize, bias=False)
        # weight tying — same storage as the token embedding
        self.lm_head.weight = self.transformer.wte.weight
        self.drop = nn.Dropout(config.dropout)
        self.register_buffer(
            "mask",
            torch.tril(torch.ones(1, 1, config.max_seq_len, config.max_seq_len)),
        )
        self.apply(self._init_weights)

    def _init_weights(self, module):
        # OpenAI GPT-2: N(0, 0.02). Residual projections scale by 1/sqrt(2N).
        if isinstance(module, nn.Linear):
            std = 0.02
            if getattr(module, "NANOGPT_SCALE_INIT", None):
                std /= math.sqrt(2 * self.config.N)
            nn.init.normal_(module.weight, mean=0.0, std=std)
            if module.bias is not None:
                nn.init.zeros_(module.bias)
        elif isinstance(module, nn.Embedding):
            nn.init.normal_(module.weight, mean=0.0, std=0.02)

    def forward(self, idx):
        _, S = idx.shape
        pos = torch.arange(S, device=idx.device)
        x = self.drop(self.transformer.wte(idx) + self.transformer.wpe(pos))
        for block in self.transformer.h:
            x = block(x, self.mask)
        return self.lm_head(self.transformer.ln_f(x))

    def load_hf_state_dict(self, hf_state):
        """Copy a Hugging Face GPT-2 state_dict. Transposes Conv1D weights."""
        ours = self.state_dict()
        mapped = {}
        for name, tensor in hf_state.items():
            if name not in ours:
                continue
            if name.endswith(HF_TRANSPOSE):
                tensor = tensor.t().contiguous()
            mapped[name] = tensor
        # keep the tied pointer after the copy
        missing, unexpected = self.load_state_dict(mapped, strict=False)
        self.lm_head.weight = self.transformer.wte.weight
        return missing, unexpected

Test note. Forward (2, 8), assert logits (2, 8, 50), check data_ptr tying, and load a synthetic state_dict that uses the real GPT-2 key names.

Mistakes this stop used to make

“Tying halves the parameter count.” It saves 50,257 × 768 ≈ 38.6M of a ~163M untied model. The published 124M figure already includes tying.

Loading Conv1D weights into nn.Linear without a transpose. Hugging Face stores c_attn, c_proj, and c_fc as (in, out).

Using exact GELU. GPT-2's released code uses the tanh approximation.

What to remember

  • GPT-2 124M is 12 / 12 / 64 / 768, context 1024, vocab 50,257, weight-tied.
  • Pre-norm, learned positions, fused c_attn, tanh-GELU, N(0, 0.02) init.
  • Public weight names start with transformer. and use c_attn / c_fc.

Check yourself

1. What is GPT-2 124M's head dimension?

768 / 12 = 64.

2. Are the embeddings tied?

Yes. lm_head.weight is transformer.wte.weight.

3. Why transpose some weights on load?

Those layers were Conv1D in the original port. nn.Linear stores the opposite layout.

Go to the source

Keep going