THE TRANSFORMER DEEP CUTS · CHAPTER 05

The Modern Block:
Pre-Norm, RMSNorm, SwiGLU

The 2017 block still works. The stack people train now moves the norm, drops the mean, and gates the FFN.

~16 min read1 interactiveTested PyTorch code

What you'll learn

  • Why pre-norm is a cleaner gradient highway than post-norm
  • RMSNorm: re-scale only, no mean, no bias
  • SwiGLU: three matrices, SiLU on the gate, width \(\approx (2/3)\cdot 4d\)
  • Mistral 7B uses SwiGLU, the same FFN as LLaMA, not a plain SiLU MLP

Before you start

A clean shoulder, then the work

Picture a mountain path with a paved shoulder. In 2017 the hikers walk the path, then the crew resurfaces whatever they just walked (post-norm). Today the crew resurfaces the shoulder first, the hikers do their work, and they step back onto the original pavement (pre-norm). The pavement — the residual — never goes through the resurfacing. Gradients walk that pavement all the way down.

Pre-norm, RMSNorm, and SwiGLU are three small swaps. Attention itself is the same object as chapter 02, plus RoPE and later GQA. This chapter is only the swaps.

Normalise, then the sublayer, then add

Post-norm (2017)

\(x \leftarrow \mathrm{LN}(x + F(x))\). The residual is filtered by the norm. Deep stacks can get a shaky gradient.

Pre-norm (GPT-2, LLaMA)

\(x \leftarrow x + F(\mathrm{Norm}(x))\). The residual is a clean copy of \(x\). Most modern decoders use this.

GPT-2 already flipped the order. LLaMA kept the flip and changed the norm.

post-norm  →  pre-norm  →  residual skips  →  norm · sublayer · add

MANIM Resurface the shoulder first — the residual pavement never goes through the crew.

Drop the mean. Drop the bias.

LayerNorm re-centres and re-scales, then applies \(\gamma\) and \(\beta\). RMSNorm only re-scales:

divide by the root-mean-square of the hidden units, then stretch with a learned gain. No subtracted mean. No \(\beta\).

Worked vector \(x = [3,\, 0,\, 4]\):

LayerNormRMSNorm
reducermean \(7/3\), std \(\approx 1.700\)RMS \(= \sqrt{25/3} \approx\) 2.887
output (\(\gamma=1\))[0.392, −1.373, 0.981][1.039, 0, 1.386]

RMSNorm is cheaper (one fewer reduction) and, in the LLaMA ablation, about as good. The hypothesis in the RMSNorm paper is that the scale is what matters, not the centre.

[3,0,4]  →  RMS  →  ÷RMS  →  no β

MANIM No mean subtraction — only the RMS and γ.

Interactive · One vector, two norms

Move the three entries. When do LayerNorm and RMSNorm disagree the most?

3
0
4

Three maps, one gate

SiLU-gate a wide hidden layer, multiply by a second wide layer, then project back. SiLU\((z) = z\,\sigma(z)\).

W_gate  ⊙  W_up  →  mul  →  W_down

MANIM Three maps, one gate — not a plain SiLU MLP.

Three matrices cost \(3 d h\). The original FFN cost \(2 d \cdot 4d = 8 d^2\). To keep the bill near \(8 d^2\), set \(h \approx (2/3)\cdot 4d = 8d/3\). At Llama 3 8B's width 4096 that is about 10,923. The published hidden size is 14,336 (exactly \(3.5 \times 4096\)), after their rounding rules.

Mistral 7B uses this SwiGLU, not a two-layer SiLU MLP. Mixtral's experts are the same three maps. A page that says "Mistral uses SiLU" skipped the gate and the third matrix.

One table for the block

Original 2017GPT-2 124MLlama 3 8BMistral 7B
norm positionpostpreprepre
norm typeLayerNormLayerNormRMSNormRMSNorm
FFNReLU, \(4d\)GELU, \(4d\)SwiGLU, 14,336SwiGLU, 14,336
biasesyesyesnono
positionsinusoidlearned 1024RoPE, base 500,000RoPE + window 4,096

Llama 3 8B: 32 layers, 32 query heads, 8 KV heads, head dim 128, hidden 4096, 8.03B parameters, vocab 128,256. Mistral 7B: 32 / 32 / 8 / 128, 7.24B, sliding window 4,096. The KV-head split is chapter 06.

RMSNorm, SwiGLU, pre-norm wrapper

The attention module is passed in. This file does not re-implement chapter 02. The FFN input is normalised once and reused by both the gate and the up-map.

modern_block.pyRMSNorm matches [3,0,4]; SwiGLU has three bias-free maps
"""Pre-norm, RMSNorm, and SwiGLU — the three LLaMA-era block changes.

Mistral 7B and Mixtral use the same SwiGLU FFN as LLaMA (three matrices,
SiLU on the gate). They do not use a plain two-layer SiLU MLP.
"""
import torch
import torch.nn.functional as F


class RMSNorm(torch.nn.Module):
    """Re-scale only. No mean subtraction, no bias. One vector of gains (gamma)."""

    def __init__(self, dim, eps=1e-5):
        super().__init__()
        self.eps = eps
        self.weight = torch.nn.Parameter(torch.ones(dim))  # gamma

    def forward(self, x):
        # Keep the reduction in float32 so small residuals do not vanish in bf16.
        rms = x.float().pow(2).mean(dim=-1, keepdim=True).add(self.eps).rsqrt()
        return (x.float() * rms).type_as(x) * self.weight


class SwiGLU(torch.nn.Module):
    """SwiGLU: SiLU(x W_gate) * (x W_up), then W_down.

    SiLU(z) = z * sigmoid(z). The gate can shut a channel off smoothly.
    Three matrices, not two. Hidden width is usually ~ (2/3)*4d so the
    parameter count stays near the original 8 d^2 FFN.
    """

    def __init__(self, d_model, d_ff):
        super().__init__()
        # LLaMA and Mistral drop every bias in the block.
        self.w_gate = torch.nn.Linear(d_model, d_ff, bias=False)
        self.w_up = torch.nn.Linear(d_model, d_ff, bias=False)
        self.w_down = torch.nn.Linear(d_ff, d_model, bias=False)

    def forward(self, x):
        return self.w_down(F.silu(self.w_gate(x)) * self.w_up(x))


class ModernBlock(torch.nn.Module):
    """Pre-norm residual wrapper around an attention module and a SwiGLU.

    Attention is the same object as chapter 02. What changed is *when*
    we normalise (before the sublayer) and *what* the FFN is (SwiGLU).
    """

    def __init__(self, d_model, attn, d_ff):
        super().__init__()
        self.n1 = RMSNorm(d_model)
        self.attn = attn
        self.n2 = RMSNorm(d_model)
        self.ffn = SwiGLU(d_model, d_ff)

    def forward(self, x):
        # Pre-norm: the residual is a clean highway. Gradients skip the norm.
        x = x + self.attn(self.n1(x))
        x = x + self.ffn(self.n2(x))
        return x

Mistakes learners make

"Mistral 7B uses SiLU, LLaMA uses SwiGLU." Both use SwiGLU. SiLU is the gate function inside SwiGLU, not the whole FFN.

Normalising twice into the FFN. Compute \(\mathrm{RMSNorm}(x)\) once, then feed it to both \(W_{\mathrm{gate}}\) and \(W_{\mathrm{up}}\).

Keeping \(\beta\) in RMSNorm. There is no bias. Only \(\gamma\).

What to remember

  • Pre-norm: \(x + F(\mathrm{Norm}(x))\). The residual is a clean highway.
  • RMSNorm re-scales. It does not re-centre. One gain vector.
  • SwiGLU is three matrices. Width \(\approx (2/3)\cdot 4d\). Llama 3 8B publishes 14,336.
  • Mistral's FFN is SwiGLU, same as LLaMA.

Check yourself

1. RMS of [3, 0, 4]?

\(\sqrt{(9+0+16)/3} = \sqrt{25/3} \approx 2.887\). The normalised vector is [1.039, 0, 1.386].

2. Why 10,923 and 14,336 for the same model?

10,923 is \((2/3)\cdot 4\cdot 4096\), the width that keeps 8 \(d^2\) parameters. 14,336 is what Llama 3 8B actually ships after rounding.

3. Does a pre-norm residual pass through RMSNorm?

No. \(x\) is added after the sublayer. Only the sublayer's input is normalised.

Go to the source

Keep going