THE TRANSFORMER DEEP CUTS · CHAPTER 02

Multi-Head Attention &
the Transformer Block

One lookup is a single opinion. Several lookups in parallel, then a learned mix, is the block every later chapter stacks.

~18 min read1 interactiveTested PyTorch code

What you'll learn

  • Why we run several heads instead of one wide one
  • The shape walk: split, score, concat, \(W_O\)
  • Where the fourth map \(W_O\) sits, and why it is not optional
  • How residual, LayerNorm, and the FFN finish the 2017 block

Before you start

  • Self-attention (Q, K, V, the scale, the mask)
  • What a residual connection is

Several tasters, one blend

A coffee shop runs three tasters on the same cup. One hunts for acidity. One hunts for roast. One hunts for the smell of the bag. Each writes a note. The head barista reads the three notes and writes a single line for the board.

The tasters are the heads. The barista's rewrite is \(W_O\). Without that rewrite you just have a pile of notes that nobody mixes.

Heads do not magically "parse syntax" or "track names." They are slices of the same vector, scored in parallel. What they specialise to is an empirical question, not a guarantee.

Same tokens, more score grids

Split the model width \(d_{\text{model}}\) into \(h\) pieces of size \(d_k = d_{\text{model}} / h\). Each piece is one head: its own \(Q, K, V\), its own \(T \times T\) scores, its own mix of values.

More heads do not make a bigger score matrix. They make more of them. The cost of the matmuls stays about the same as one head of width \(d_{\text{model}}\), because \(h \times (T \times d_k)^2\) work equals one \((T \times d_{\text{model}})\) head.

run \(h\) scaled lookups on slices of the width, glue the results, then mix them with a square map \(W_O\).

Follow one tensor through the layer

Take GPT-2 124M: hidden 768, 12 heads, head dim 64. And Llama 3 8B: hidden 4096, 32 query heads, head dim 128.

StepShapeGPT-2 124MLlama 3 8B
input \(X\)(B, T, d)(B, T, 768)(B, T, 4096)
\(XW_Q\), \(XW_K\), \(XW_V\)(B, T, d)(B, T, 768)(B, T, 4096)
split into heads(B, h, T, d_k)(B, 12, T, 64)(B, 32, T, 128)
scores(B, h, T, T)(B, 12, T, T)(B, 32, T, T)
concat(B, T, d)(B, T, 768)(B, T, 4096)
\(\times W_O\)(B, T, d)(B, T, 768)(B, T, 4096)

The four maps \(W_Q, W_K, W_V, W_O\) are each \(d \times d\). For GPT-2 that is \(4 \times 768 \times 768 = \) 2,359,296 weights, not counting biases (GPT-2 has them; LLaMA does not). The original FFN is two maps \(768 \to 3072 \to 768\): 4,718,592 weights. The FFN is the fat part of the block.

(B,T,768)  →  12×64  →  scores  →  W_O

MANIM Follow one tensor through the GPT-2 layer.

Interactive · Split a width into heads

If you change the head count, what happens to \(d_k\) and to the score grid?

12

\(W_O\) mixes the heads

Several walkthroughs stop at "concat." That leaves the heads stacked side by side and never mixed. Every GPT, LLaMA, and DeepSeek attention layer has a fourth linear map \(W_O\).

Concat is just a reshape. \(W_O\) is where a head that found "the verb" can talk to a head that found "the subject." Later tricks (MQA, GQA, MLA) all still end with this map.

Parameter count for attention is \(4 d^2\), not \(3 d^2\). If a page quotes the smaller number, it forgot \(W_O\).

concat stacks  →  \(W_O\)  →  mix heads  →  \(4d^2\)

MANIM Concat stacks notes; W_O is the blend. Attention params are 4d², not 3d².

Attention is half the block

The original paper wraps each sublayer in a residual add and a LayerNorm, in that order (post-norm):

keep a clean copy of \(x\), add the sublayer, then re-centre and re-scale the sum.

MHA  →  add  →  LN  →  FFN path

MANIM Add, then LayerNorm — the 2017 residual order.

The FFN is position-wise: the same two maps at every token, independently. Original width is \(4d\), with ReLU. GPT-2 uses GELU. LLaMA and Mistral switch the FFN for SwiGLU — that is chapter 05, along with pre-norm and RMSNorm.

LayerNorm subtracts the mean, divides by the standard deviation, then applies a learned gain \(\gamma\) and bias \(\beta\). It is not BatchNorm: the statistics are over the hidden axis of one token, not over the batch.

Heads, \(W_O\), and a post-norm block

contiguous() is there because transpose breaks the memory layout that view wants. The block below is the 2017 order. Chapter 05 flips the norms to the input of each sublayer.

mha.pyW_O is present; residual block keeps (B, T, d)
"""Multi-head attention with the output projection W_O, plus one block.

W_O is not optional. It is the map that mixes the heads back together.
Several older walkthroughs left it out. Do not.
"""
import torch
import torch.nn.functional as F


class MultiHeadAttention(torch.nn.Module):
    """n_heads in parallel, then a learned mix W_O.

    Shapes, for one call:
      x:      (B, T, d_model)
      q,k,v:  (B, T, n_heads * d_k) after the input maps
      heads:  (B, n_heads, T, d_k) after the split
      scores: (B, n_heads, T, T)     still T by T, just more of them
      concat: (B, T, n_heads * d_k)
      out:    (B, T, d_model)        after W_O
    """

    def __init__(self, d_model, n_heads):
        super().__init__()
        assert d_model % n_heads == 0, "d_model must split evenly across heads"
        self.n_heads = n_heads
        self.d_k = d_model // n_heads
        # One matrix per role, then we slice it into heads. Same math as
        # a separate Linear per head, fewer Python loops.
        self.W_q = torch.nn.Linear(d_model, d_model, bias=False)
        self.W_k = torch.nn.Linear(d_model, d_model, bias=False)
        self.W_v = torch.nn.Linear(d_model, d_model, bias=False)
        # W_O is the fourth map. Without it the heads cannot talk.
        self.W_o = torch.nn.Linear(d_model, d_model, bias=False)

    def _split(self, x):
        # (B, T, d_model) -> (B, n_heads, T, d_k)
        b, t, _ = x.shape
        return x.view(b, t, self.n_heads, self.d_k).transpose(1, 2)

    def forward(self, x, causal=True):
        q, k, v = self._split(self.W_q(x)), self._split(self.W_k(x)), self._split(self.W_v(x))
        scores = q @ k.transpose(-2, -1) / self.d_k**0.5
        if causal:
            t = x.shape[1]
            future = torch.triu(torch.ones(t, t, dtype=torch.bool, device=x.device), 1)
            scores = scores.masked_fill(future, float("-inf"))
        heads = F.softmax(scores, dim=-1) @ v          # (B, H, T, d_k)
        concat = heads.transpose(1, 2).contiguous().view(x.shape)
        return self.W_o(concat)


class TransformerBlock(torch.nn.Module):
    """Post-norm block from the 2017 paper: attention, residual, FFN, residual.

    Modern decoders flip the norm to the *input* of each sublayer (chapter 05).
    The residual add is the same idea either way.
    """

    def __init__(self, d_model, n_heads, d_ff):
        super().__init__()
        self.attn = MultiHeadAttention(d_model, n_heads)
        self.ffn = torch.nn.Sequential(
            torch.nn.Linear(d_model, d_ff),
            torch.nn.ReLU(),
            torch.nn.Linear(d_ff, d_model),
        )
        self.n1 = torch.nn.LayerNorm(d_model)
        self.n2 = torch.nn.LayerNorm(d_model)

    def forward(self, x):
        # Post-norm: add first, then normalise. Residual keeps a clean copy of x.
        x = self.n1(x + self.attn(x))
        x = self.n2(x + self.ffn(x))
        return x

Mistakes learners make

Leaving out \(W_O\). Concat is not a mix. The fourth map is part of the paper's formula and part of every trained checkpoint.

"More heads means a bigger score matrix." The scores stay \(T \times T\). You just have more of those grids.

Splitting along the sequence. Heads slice the width, not the tokens. Every head still sees every (allowed) token.

Reading attention maps as parses. A pretty head is a clue, not a proof of what the model "thinks."

What to remember

  • Heads are parallel lookups on slices of \(d_{\text{model}}\). Scores stay \(T \times T\).
  • \(W_O\) mixes the concatenated heads. Count it. Code it.
  • The 2017 block is MHA + residual + LayerNorm, then FFN + residual + LayerNorm.
  • GPT-2: 12 heads of 64. Llama 3 8B: 32 heads of 128. Same recipe, different width.

Check yourself

1. Llama 3 8B has hidden 4096 and 32 heads. What is \(d_k\)?

\(4096 / 32 = 128\).

2. Why is attention \(4d^2\) weights, not \(3d^2\)?

Four maps: \(W_Q, W_K, W_V, W_O\). The last one mixes heads.

3. Does adding heads change the size of one score grid?

No. Each grid is still \(T \times T\). You get more grids.

4. In the 2017 block, is the residual added before or after LayerNorm?

Before. Post-norm is \(\mathrm{LayerNorm}(x + \mathrm{sublayer}(x))\).

Go to the source

Keep going