THE TRANSFORMER DEEP CUTS · CHAPTER 05
The 2017 block still works. The stack people train now moves the norm, drops the mean, and gates the FFN.
Picture a mountain path with a paved shoulder. In 2017 the hikers walk the path, then the crew resurfaces whatever they just walked (post-norm). Today the crew resurfaces the shoulder first, the hikers do their work, and they step back onto the original pavement (pre-norm). The pavement — the residual — never goes through the resurfacing. Gradients walk that pavement all the way down.
Pre-norm, RMSNorm, and SwiGLU are three small swaps. Attention itself is the same object as chapter 02, plus RoPE and later GQA. This chapter is only the swaps.
\(x \leftarrow \mathrm{LN}(x + F(x))\). The residual is filtered by the norm. Deep stacks can get a shaky gradient.
\(x \leftarrow x + F(\mathrm{Norm}(x))\). The residual is a clean copy of \(x\). Most modern decoders use this.
GPT-2 already flipped the order. LLaMA kept the flip and changed the norm.
post-norm → pre-norm → residual skips → norm · sublayer · add
LayerNorm re-centres and re-scales, then applies \(\gamma\) and \(\beta\). RMSNorm only re-scales:
divide by the root-mean-square of the hidden units, then stretch with a learned gain. No subtracted mean. No \(\beta\).
Worked vector \(x = [3,\, 0,\, 4]\):
| LayerNorm | RMSNorm | |
|---|---|---|
| reducer | mean \(7/3\), std \(\approx 1.700\) | RMS \(= \sqrt{25/3} \approx\) 2.887 |
| output (\(\gamma=1\)) | [0.392, −1.373, 0.981] | [1.039, 0, 1.386] |
RMSNorm is cheaper (one fewer reduction) and, in the LLaMA ablation, about as good. The hypothesis in the RMSNorm paper is that the scale is what matters, not the centre.
[3,0,4] → RMS → ÷RMS → no β
Interactive · One vector, two norms
Move the three entries. When do LayerNorm and RMSNorm disagree the most?
SiLU-gate a wide hidden layer, multiply by a second wide layer, then project back. SiLU\((z) = z\,\sigma(z)\).
W_gate ⊙ W_up → mul → W_down
Three matrices cost \(3 d h\). The original FFN cost \(2 d \cdot 4d = 8 d^2\). To keep the bill near \(8 d^2\), set \(h \approx (2/3)\cdot 4d = 8d/3\). At Llama 3 8B's width 4096 that is about 10,923. The published hidden size is 14,336 (exactly \(3.5 \times 4096\)), after their rounding rules.
Mistral 7B uses this SwiGLU, not a two-layer SiLU MLP. Mixtral's experts are the same three maps. A page that says "Mistral uses SiLU" skipped the gate and the third matrix.
| Original 2017 | GPT-2 124M | Llama 3 8B | Mistral 7B | |
|---|---|---|---|---|
| norm position | post | pre | pre | pre |
| norm type | LayerNorm | LayerNorm | RMSNorm | RMSNorm |
| FFN | ReLU, \(4d\) | GELU, \(4d\) | SwiGLU, 14,336 | SwiGLU, 14,336 |
| biases | yes | yes | no | no |
| position | sinusoid | learned 1024 | RoPE, base 500,000 | RoPE + window 4,096 |
Llama 3 8B: 32 layers, 32 query heads, 8 KV heads, head dim 128, hidden 4096, 8.03B parameters, vocab 128,256. Mistral 7B: 32 / 32 / 8 / 128, 7.24B, sliding window 4,096. The KV-head split is chapter 06.
The attention module is passed in. This file does not re-implement chapter 02. The FFN input is normalised once and reused by both the gate and the up-map.
"""Pre-norm, RMSNorm, and SwiGLU — the three LLaMA-era block changes.
Mistral 7B and Mixtral use the same SwiGLU FFN as LLaMA (three matrices,
SiLU on the gate). They do not use a plain two-layer SiLU MLP.
"""
import torch
import torch.nn.functional as F
class RMSNorm(torch.nn.Module):
"""Re-scale only. No mean subtraction, no bias. One vector of gains (gamma)."""
def __init__(self, dim, eps=1e-5):
super().__init__()
self.eps = eps
self.weight = torch.nn.Parameter(torch.ones(dim)) # gamma
def forward(self, x):
# Keep the reduction in float32 so small residuals do not vanish in bf16.
rms = x.float().pow(2).mean(dim=-1, keepdim=True).add(self.eps).rsqrt()
return (x.float() * rms).type_as(x) * self.weight
class SwiGLU(torch.nn.Module):
"""SwiGLU: SiLU(x W_gate) * (x W_up), then W_down.
SiLU(z) = z * sigmoid(z). The gate can shut a channel off smoothly.
Three matrices, not two. Hidden width is usually ~ (2/3)*4d so the
parameter count stays near the original 8 d^2 FFN.
"""
def __init__(self, d_model, d_ff):
super().__init__()
# LLaMA and Mistral drop every bias in the block.
self.w_gate = torch.nn.Linear(d_model, d_ff, bias=False)
self.w_up = torch.nn.Linear(d_model, d_ff, bias=False)
self.w_down = torch.nn.Linear(d_ff, d_model, bias=False)
def forward(self, x):
return self.w_down(F.silu(self.w_gate(x)) * self.w_up(x))
class ModernBlock(torch.nn.Module):
"""Pre-norm residual wrapper around an attention module and a SwiGLU.
Attention is the same object as chapter 02. What changed is *when*
we normalise (before the sublayer) and *what* the FFN is (SwiGLU).
"""
def __init__(self, d_model, attn, d_ff):
super().__init__()
self.n1 = RMSNorm(d_model)
self.attn = attn
self.n2 = RMSNorm(d_model)
self.ffn = SwiGLU(d_model, d_ff)
def forward(self, x):
# Pre-norm: the residual is a clean highway. Gradients skip the norm.
x = x + self.attn(self.n1(x))
x = x + self.ffn(self.n2(x))
return x
"Mistral 7B uses SiLU, LLaMA uses SwiGLU." Both use SwiGLU. SiLU is the gate function inside SwiGLU, not the whole FFN.
Normalising twice into the FFN. Compute \(\mathrm{RMSNorm}(x)\) once, then feed it to both \(W_{\mathrm{gate}}\) and \(W_{\mathrm{up}}\).
Keeping \(\beta\) in RMSNorm. There is no bias. Only \(\gamma\).
\(\sqrt{(9+0+16)/3} = \sqrt{25/3} \approx 2.887\). The normalised vector is [1.039, 0, 1.386].
10,923 is \((2/3)\cdot 4\cdot 4096\), the width that keeps 8 \(d^2\) parameters. 14,336 is what Llama 3 8B actually ships after rounding.
No. \(x\) is added after the sublayer. Only the sublayer's input is normalised.