THE EVOLVING TRANSFORMER · STOP 02
Cut the encoder. Learn the positions. Flip the residual to pre-norm. Tie the head to the token table, and name every weight the way the public checkpoint does.
c_attnlm_head.weight is the same storage as wteStop 01 had an encoder, sinusoids, post-norm, ReLU, and a separate projection. GPT-2 drops the encoder and cross-attention. Positions become a learned wpe. Residuals flip to pre-norm. The FFN uses GPT-2's tanh GELU. The language-model head shares storage with the token embedding — that is weight tying, and it saves about 38.6M parameters, not “half the model”.
Imagine reprinting a dictionary as its own index. The same rows that turn an id into a vector turn a vector back into a score over ids. One table, two jobs.
Init is N(0, 0.02). Residual projections (c_proj in attention and MLP) scale that by 1/√(2N) so the residual stream does not blow up as layers stack. Module names follow the public checkpoint: transformer.wte, transformer.h.0.attn.c_attn, transformer.ln_f. Why a decoder-only GPT →
| Knob | GPT-2 124M |
|---|---|
| layers / heads / head dim / hidden | 12 / 12 / 64 / 768 |
| context / vocab | 1024 / 50,257 |
| weight tying | yes — lm_head is wte |
Teaching batch: tokens (2, 8), hidden 32, 4 heads, vocab 50.
shared W · embed · lm_head
drop encoder · absolute wpe · (2,8,50)
| Tensor | Shape |
|---|---|
| tokens | (2, 8) |
| hidden states | (2, 8, 32) |
c_attn.weight | (96, 32) |
| logits | (2, 8, 50) |
load_hf_state_dict copies matching names and transposes the four Conv1D weights. The check builds a fake Hugging Face-shaped dict — it does not download.
"""GPT-2 124M — decoder-only, pre-norm, learned positions, tied head.
Published config: 12 layers, 12 heads, head dim 64, hidden 768,
context 1024, vocab 50,257. Weight-tied.
Module names match the public Hugging Face checkpoint so a real
state_dict can be copied in (Conv1D weights are transposed).
"""
from dataclasses import dataclass
import math
import torch
import torch.nn as nn
import torch.nn.functional as F
@dataclass
class GPTConfig:
# GPT-2 124M published numbers. Tests pass a tiny override.
dmodel: int = 768
vocabsize: int = 50257
max_seq_len: int = 1024
N: int = 12
H: int = 12
dropout: float = 0.0
@property
def dk(self):
return self.dmodel // self.H # 768 / 12 = 64
@property
def dff(self):
return 4 * self.dmodel
# Hugging Face stores these as Conv1D: weight shape is (in, out).
# nn.Linear wants (out, in), so we transpose them on load.
HF_TRANSPOSE = ("attn.c_attn.weight", "attn.c_proj.weight", "mlp.c_fc.weight", "mlp.c_proj.weight")
class CausalSelfAttention(nn.Module):
"""Fused QKV projection named c_attn, matching the real GPT-2 weights."""
def __init__(self, config):
super().__init__()
self.H = config.H
self.dk = config.dk
self.dmodel = config.dmodel
# one Linear: dmodel → 3 * dmodel, then split into Q, K, V
self.c_attn = nn.Linear(config.dmodel, 3 * config.dmodel)
self.c_proj = nn.Linear(config.dmodel, config.dmodel)
self.c_proj.NANOGPT_SCALE_INIT = 1 # residual-path scale at init
self.attn_dropout = nn.Dropout(config.dropout)
self.resid_dropout = nn.Dropout(config.dropout)
def forward(self, x, mask):
B, S, _ = x.shape
qkv = self.c_attn(x)
q, k, v = qkv.split(self.dmodel, dim=-1)
q = q.view(B, S, self.H, self.dk).transpose(1, 2)
k = k.view(B, S, self.H, self.dk).transpose(1, 2)
v = v.view(B, S, self.H, self.dk).transpose(1, 2)
scores = (q @ k.transpose(-1, -2)) / math.sqrt(self.dk)
scores = scores.masked_fill(mask[:, :, :S, :S] == 0, float("-inf"))
weights = self.attn_dropout(scores.softmax(dim=-1))
out = (weights @ v).transpose(1, 2).contiguous().view(B, S, self.dmodel)
return self.resid_dropout(self.c_proj(out))
class MLP(nn.Module):
def __init__(self, config):
super().__init__()
self.c_fc = nn.Linear(config.dmodel, config.dff)
self.c_proj = nn.Linear(config.dff, config.dmodel)
self.c_proj.NANOGPT_SCALE_INIT = 1
self.dropout = nn.Dropout(config.dropout)
def forward(self, x):
# GPT-2's GELU is the tanh approximation, not the exact erf form
return self.dropout(self.c_proj(F.gelu(self.c_fc(x), approximate="tanh")))
class Block(nn.Module):
"""Pre-norm block: x + attn(LN(x)), then x + mlp(LN(x))."""
def __init__(self, config):
super().__init__()
self.ln_1 = nn.LayerNorm(config.dmodel)
self.attn = CausalSelfAttention(config)
self.ln_2 = nn.LayerNorm(config.dmodel)
self.mlp = MLP(config)
def forward(self, x, mask):
x = x + self.attn(self.ln_1(x), mask)
x = x + self.mlp(self.ln_2(x))
return x
class GPT(nn.Module):
def __init__(self, config):
super().__init__()
self.config = config
# names match HF: transformer.wte, transformer.wpe, transformer.h, transformer.ln_f
self.transformer = nn.ModuleDict(
dict(
wte=nn.Embedding(config.vocabsize, config.dmodel),
wpe=nn.Embedding(config.max_seq_len, config.dmodel),
h=nn.ModuleList([Block(config) for _ in range(config.N)]),
ln_f=nn.LayerNorm(config.dmodel),
)
)
self.lm_head = nn.Linear(config.dmodel, config.vocabsize, bias=False)
# weight tying — same storage as the token embedding
self.lm_head.weight = self.transformer.wte.weight
self.drop = nn.Dropout(config.dropout)
self.register_buffer(
"mask",
torch.tril(torch.ones(1, 1, config.max_seq_len, config.max_seq_len)),
)
self.apply(self._init_weights)
def _init_weights(self, module):
# OpenAI GPT-2: N(0, 0.02). Residual projections scale by 1/sqrt(2N).
if isinstance(module, nn.Linear):
std = 0.02
if getattr(module, "NANOGPT_SCALE_INIT", None):
std /= math.sqrt(2 * self.config.N)
nn.init.normal_(module.weight, mean=0.0, std=std)
if module.bias is not None:
nn.init.zeros_(module.bias)
elif isinstance(module, nn.Embedding):
nn.init.normal_(module.weight, mean=0.0, std=0.02)
def forward(self, idx):
_, S = idx.shape
pos = torch.arange(S, device=idx.device)
x = self.drop(self.transformer.wte(idx) + self.transformer.wpe(pos))
for block in self.transformer.h:
x = block(x, self.mask)
return self.lm_head(self.transformer.ln_f(x))
def load_hf_state_dict(self, hf_state):
"""Copy a Hugging Face GPT-2 state_dict. Transposes Conv1D weights."""
ours = self.state_dict()
mapped = {}
for name, tensor in hf_state.items():
if name not in ours:
continue
if name.endswith(HF_TRANSPOSE):
tensor = tensor.t().contiguous()
mapped[name] = tensor
# keep the tied pointer after the copy
missing, unexpected = self.load_state_dict(mapped, strict=False)
self.lm_head.weight = self.transformer.wte.weight
return missing, unexpected
Test note. Forward (2, 8), assert logits (2, 8, 50), check data_ptr tying, and load a synthetic state_dict that uses the real GPT-2 key names.
“Tying halves the parameter count.” It saves 50,257 × 768 ≈ 38.6M of a ~163M untied model. The published 124M figure already includes tying.
Loading Conv1D weights into nn.Linear without a transpose. Hugging Face stores c_attn, c_proj, and c_fc as (in, out).
Using exact GELU. GPT-2's released code uses the tanh approximation.
c_attn, tanh-GELU, N(0, 0.02) init.transformer. and use c_attn / c_fc.768 / 12 = 64.
Yes. lm_head.weight is transformer.wte.weight.
Those layers were Conv1D in the original port. nn.Linear stores the opposite layout.