THE TRANSFORMER DEEP CUTS · CHAPTER 02
One lookup is a single opinion. Several lookups in parallel, then a learned mix, is the block every later chapter stacks.
A coffee shop runs three tasters on the same cup. One hunts for acidity. One hunts for roast. One hunts for the smell of the bag. Each writes a note. The head barista reads the three notes and writes a single line for the board.
The tasters are the heads. The barista's rewrite is \(W_O\). Without that rewrite you just have a pile of notes that nobody mixes.
Heads do not magically "parse syntax" or "track names." They are slices of the same vector, scored in parallel. What they specialise to is an empirical question, not a guarantee.
Split the model width \(d_{\text{model}}\) into \(h\) pieces of size \(d_k = d_{\text{model}} / h\). Each piece is one head: its own \(Q, K, V\), its own \(T \times T\) scores, its own mix of values.
More heads do not make a bigger score matrix. They make more of them. The cost of the matmuls stays about the same as one head of width \(d_{\text{model}}\), because \(h \times (T \times d_k)^2\) work equals one \((T \times d_{\text{model}})\) head.
run \(h\) scaled lookups on slices of the width, glue the results, then mix them with a square map \(W_O\).
Take GPT-2 124M: hidden 768, 12 heads, head dim 64. And Llama 3 8B: hidden 4096, 32 query heads, head dim 128.
| Step | Shape | GPT-2 124M | Llama 3 8B |
|---|---|---|---|
| input \(X\) | (B, T, d) | (B, T, 768) | (B, T, 4096) |
| \(XW_Q\), \(XW_K\), \(XW_V\) | (B, T, d) | (B, T, 768) | (B, T, 4096) |
| split into heads | (B, h, T, d_k) | (B, 12, T, 64) | (B, 32, T, 128) |
| scores | (B, h, T, T) | (B, 12, T, T) | (B, 32, T, T) |
| concat | (B, T, d) | (B, T, 768) | (B, T, 4096) |
| \(\times W_O\) | (B, T, d) | (B, T, 768) | (B, T, 4096) |
The four maps \(W_Q, W_K, W_V, W_O\) are each \(d \times d\). For GPT-2 that is \(4 \times 768 \times 768 = \) 2,359,296 weights, not counting biases (GPT-2 has them; LLaMA does not). The original FFN is two maps \(768 \to 3072 \to 768\): 4,718,592 weights. The FFN is the fat part of the block.
(B,T,768) → 12×64 → scores → W_O
Interactive · Split a width into heads
If you change the head count, what happens to \(d_k\) and to the score grid?
Several walkthroughs stop at "concat." That leaves the heads stacked side by side and never mixed. Every GPT, LLaMA, and DeepSeek attention layer has a fourth linear map \(W_O\).
Concat is just a reshape. \(W_O\) is where a head that found "the verb" can talk to a head that found "the subject." Later tricks (MQA, GQA, MLA) all still end with this map.
Parameter count for attention is \(4 d^2\), not \(3 d^2\). If a page quotes the smaller number, it forgot \(W_O\).
concat stacks → \(W_O\) → mix heads → \(4d^2\)
The original paper wraps each sublayer in a residual add and a LayerNorm, in that order (post-norm):
keep a clean copy of \(x\), add the sublayer, then re-centre and re-scale the sum.
MHA → add → LN → FFN path
The FFN is position-wise: the same two maps at every token, independently. Original width is \(4d\), with ReLU. GPT-2 uses GELU. LLaMA and Mistral switch the FFN for SwiGLU — that is chapter 05, along with pre-norm and RMSNorm.
LayerNorm subtracts the mean, divides by the standard deviation, then applies a learned gain \(\gamma\) and bias \(\beta\). It is not BatchNorm: the statistics are over the hidden axis of one token, not over the batch.
contiguous() is there because transpose breaks the memory layout that view wants. The block below is the 2017 order. Chapter 05 flips the norms to the input of each sublayer.
"""Multi-head attention with the output projection W_O, plus one block.
W_O is not optional. It is the map that mixes the heads back together.
Several older walkthroughs left it out. Do not.
"""
import torch
import torch.nn.functional as F
class MultiHeadAttention(torch.nn.Module):
"""n_heads in parallel, then a learned mix W_O.
Shapes, for one call:
x: (B, T, d_model)
q,k,v: (B, T, n_heads * d_k) after the input maps
heads: (B, n_heads, T, d_k) after the split
scores: (B, n_heads, T, T) still T by T, just more of them
concat: (B, T, n_heads * d_k)
out: (B, T, d_model) after W_O
"""
def __init__(self, d_model, n_heads):
super().__init__()
assert d_model % n_heads == 0, "d_model must split evenly across heads"
self.n_heads = n_heads
self.d_k = d_model // n_heads
# One matrix per role, then we slice it into heads. Same math as
# a separate Linear per head, fewer Python loops.
self.W_q = torch.nn.Linear(d_model, d_model, bias=False)
self.W_k = torch.nn.Linear(d_model, d_model, bias=False)
self.W_v = torch.nn.Linear(d_model, d_model, bias=False)
# W_O is the fourth map. Without it the heads cannot talk.
self.W_o = torch.nn.Linear(d_model, d_model, bias=False)
def _split(self, x):
# (B, T, d_model) -> (B, n_heads, T, d_k)
b, t, _ = x.shape
return x.view(b, t, self.n_heads, self.d_k).transpose(1, 2)
def forward(self, x, causal=True):
q, k, v = self._split(self.W_q(x)), self._split(self.W_k(x)), self._split(self.W_v(x))
scores = q @ k.transpose(-2, -1) / self.d_k**0.5
if causal:
t = x.shape[1]
future = torch.triu(torch.ones(t, t, dtype=torch.bool, device=x.device), 1)
scores = scores.masked_fill(future, float("-inf"))
heads = F.softmax(scores, dim=-1) @ v # (B, H, T, d_k)
concat = heads.transpose(1, 2).contiguous().view(x.shape)
return self.W_o(concat)
class TransformerBlock(torch.nn.Module):
"""Post-norm block from the 2017 paper: attention, residual, FFN, residual.
Modern decoders flip the norm to the *input* of each sublayer (chapter 05).
The residual add is the same idea either way.
"""
def __init__(self, d_model, n_heads, d_ff):
super().__init__()
self.attn = MultiHeadAttention(d_model, n_heads)
self.ffn = torch.nn.Sequential(
torch.nn.Linear(d_model, d_ff),
torch.nn.ReLU(),
torch.nn.Linear(d_ff, d_model),
)
self.n1 = torch.nn.LayerNorm(d_model)
self.n2 = torch.nn.LayerNorm(d_model)
def forward(self, x):
# Post-norm: add first, then normalise. Residual keeps a clean copy of x.
x = self.n1(x + self.attn(x))
x = self.n2(x + self.ffn(x))
return x
Leaving out \(W_O\). Concat is not a mix. The fourth map is part of the paper's formula and part of every trained checkpoint.
"More heads means a bigger score matrix." The scores stay \(T \times T\). You just have more of those grids.
Splitting along the sequence. Heads slice the width, not the tokens. Every head still sees every (allowed) token.
Reading attention maps as parses. A pretty head is a clue, not a proof of what the model "thinks."
\(4096 / 32 = 128\).
Four maps: \(W_Q, W_K, W_V, W_O\). The last one mixes heads.
No. Each grid is still \(T \times T\). You get more grids.
Before. Post-norm is \(\mathrm{LayerNorm}(x + \mathrm{sublayer}(x))\).