NEURONS ALL THE WAY DOWN · CHAPTER 04

Keeping Deep Nets Trainable

Random weights are not a strategy. Activations saturate, logits get too sharp, and gradients vanish. Scale the weights, normalize the activations, then watch the histograms.

~22 min read1 interactiveTested PyTorch

What you'll learn

  • Why a random head can start far above \(\ln V\), and how to shrink it
  • How fan-in scaling and Kaiming gains keep \(xW\) at std ≈ 1
  • BatchNorm: the forward, the \(\gamma,\beta\) affine, and a correct running EMA
  • LayerNorm and RMSNorm — the versions transformers actually use
  • The update-to-data ratio as a one-number health check

Before you start

A stack of volume knobs

Each layer multiplies by a weight matrix and then squashes. If the matrix is too loud, \(\tanh\) slams every hidden unit to \(\pm 1\) and the local gradient \(1-\tanh^2\) is 0. If it is too quiet, every unit sits near 0 and the net is almost linear. Depth multiplies those mistakes. Five layers of “std 3 in, std 9 out” do not give you a deeper thinker. They give you NaNs.

So we do two things. We pick the volume of \(W\) from the fan-in, once, at init. And we put a normalizer — BatchNorm, LayerNorm, or RMSNorm — after the multiply, so each layer’s output stays on a known scale even as the weights move.

Training is a conversation between activations going forward and gradients going back. If either side is tiny or huge, the conversation stops. This chapter is about keeping both sides audible.

Worse than uniform is a clue

A head that knows nothing about \(V\) classes should put \(1/V\) on each, so the NLL is \(\ln V\). For the 5-symbol trail alphabet that is \(\ln 5 \approx \mathbf{1.609}\). Chapter 03 started at 2.92. That is not “the loss is wired wrong.” That is a random \(W_2\) with std 1 writing logits of size \(\sqrt{n_{\mathrm{hidden}}}\), then softmax becoming a spike on a random class.

Push it further. Logits \((-1, 8, 0, 0, 0)\) with target 0 have NLL \(\approx \mathbf{9.00}\). The model is almost sure the answer is class 1. The first many steps of training are spent flattening that spike. Shrink \(W_2\) (and \(b_2\)) by 10× or 100× at init and the hockey-stick at the start of the loss curve goes away.

Read the first loss before you read anything else. Near \(\ln V\): the head is sane. Much larger: the logits are too sharp. Much smaller: you are leaking the label or evaluating on the training batch you just overfit.

When tanh stops passing gradient

\(\tanh\) is almost linear near 0 and flat near \(\pm 1\). Its derivative is \(1 - \tanh^2(n)\). At \(n = 0\) that is 1. At \(\tanh n = 0.99\) that is \(1 - 0.9801 = 0.02\). A unit stuck in the flat region is a dead-ish unit: it still outputs something, but it no longer tells its incoming weights how to change.

Draw pre-activations at std 4 (what you get from fan-in 16 and unscaled \(W\)). About 51% of the tanh outputs land outside \(\pm 0.99\). Draw them at std 1, and that fraction drops under 1%.

ReLU dies a different way: a unit that is negative stays at 0, and if the incoming weights never drag it back, it is gone for good. Leaky ReLU and GELU are later answers. The first answer is: do not feed the nonlinearity a wild pre-activation.

Interactive · Tanh saturation

How many hidden units freeze as you turn up the pre-activation scale?

One multiply should not change the volume

Let each row of \(X\) be \(N(0,1)\) of length \(n\), and each column of \(W\) be \(N(0, \sigma_w^2)\). Then each output is a sum of \(n\) independent products, and

the output standard deviation grows with the square root of the fan-in, unless you shrink the weights to match.

With \(n = 16\) and \(\sigma_w = 1\), a 4000-row sample comes out at std 3.81, next to the predicted 4. Set \(\sigma_w = 1/\sqrt{16} = \mathbf{0.25}\) and the same sample comes out at std 0.95, next to the predicted 1.

σw=1 → std≈4  →  σw=0.25  →  std≈0.95

MANIM Shrink σw by 1/√n or the volume balloons. Use the beat buttons to jump.

That \(1/\sqrt{n}\) is the linear / “Xavier” scale. Two common gains sit on top of it:

NonlinearityGain\(\sigma_w\) for \(n=16\)Why
linear / identity10.25preserve variance through the multiply
ReLU√20.354ReLU zeroes half the mass, so boost by √2
tanh5/30.417tanh shrinks variance; 5/3 is the usual restore

Kaiming init is \(\sigma_w = \mathrm{gain} / \sqrt{\mathrm{fan\_in}}\). PyTorch’s nn.Linear actually uses a uniform box of the same variance, not a Gaussian, and its default gain is for a linear layer. If you then put \(\tanh\) on top, you are slightly quiet. Either set the gain, or let a normalizer (next section) absorb the mismatch.

Standardize the batch, then let the net restretch it

BatchNorm, per channel, over the batch:

subtract the batch mean, divide by the batch std, then scale and shift with two learned numbers per channel.

Worked channel, four activations \([1, 3, 5, 7]\):

  1. \(\mu = 4\). Population variance \(\sigma^2 = (9+1+1+9)/4 = 5\). (The forward pass divides by \(N\), not \(N-1\).)
  2. \(\hat x = (x-4)/\sqrt{5} \approx (-1.342,\, -0.447,\, 0.447,\, 1.342)\).
  3. With \(\gamma = 2\), \(\beta = -1\): \(y \approx (-3.68,\, -1.89,\, -0.11,\, 1.68)\).

\(\gamma\) and \(\beta\) are there so the layer can still learn a scale and a shift. If the right answer really is “all large and negative,” the net can get that by driving \(\gamma\) and \(\beta\), instead of fighting the normalizer.

A bias sitting immediately before BatchNorm is wasted. The mean-subtract deletes it. Set bias=False on that linear layer.

The running average, written correctly

At test time there is no batch. So during training we keep an exponential moving average of \(\mu\) and \(\sigma^2\). PyTorch’s default momentum is \(m = 0.1\):

90% the old running value, 10% this batch. The two weights add to 1, so the running value is always a convex combination of the batches it has seen.

Start at 0. See batches with means 4, then 6, then 5:

After batchCorrect EMABroken \(0.99\cdot\mathrm{run}+0.001\cdot\mathrm{batch}\)
40.40.004
60.960.010
51.3640.015

The broken mix — which an older version of this lesson wrote down — has weights that add to 0.991. Every step leaks 0.9% of the mass toward zero. After three batches the “running mean” is 0.015, not 1.364. Leave it running for an epoch and it is still near 0 while the data lives near 5. Eval then looks like a different model.

→ 1.364  ·  0.99r+0.001b  ·  ends ≈ 0.015

MANIM Broken EMA drains to 0. Use the beat buttons to jump.

In model.eval() the layer uses the running stats and does not update them. A forgotten eval() is why a trained net “gets worse” the moment you start reporting validation loss.

Does BatchNorm “reduce internal covariate shift”?

That was the original motivation. Later analysis (Santurkar et al., 2018) argues the real gift is a smoother loss landscape — the gradients change less when the weights move. Either story agrees on the engineering: put BN between the linear map and the nonlinearity, keep a correct EMA, switch to eval stats at test time.

Normalize one example, not the batch

BatchNorm couples every example in the minibatch. That is a regularizer, and a nuisance: the forward pass of sentence A depends on sentence B, the running stats must be warmed up, and batch size 1 is unhappy. Transformers switched to LayerNorm: the same formula, but the mean and variance run across the features of one example.

On the vector \([2, 4, 6]\):

three features, one example. No batch. No train/eval split. No running EMA.

A 2×3 toy makes the axis difference concrete. Rows are examples, columns are features:

f0f1f2LN mean (across features)
example 01232
example 13454
BN mean (across batch)234

RMSNorm drops the mean subtraction and the shift \(\beta\). It only rescales by the root-mean-square:

no centering. One learned gain per channel, no bias. Cheaper, and what Llama-style blocks use.

On \([1, 2, 2]\), \(\mathrm{rms} = \sqrt{3} \approx 1.732\), so \(\hat x \approx (0.577,\, 1.155,\, 1.155)\). Modern decoder blocks (see Deep Cuts, “the modern block”) put RMSNorm before attention and the MLP, not after. That is a placement choice. The formula is the one above.

2×3 toy  ·  BN μ=[2,3,4]  ·  LN μ=[2,4]  ·  RMS · no mean

MANIM Same grid, different axes: BN down the batch, LN across features. Use the beat buttons to jump.

Histograms, then one ratio

When a deep stack misbehaves, plot three things per layer, once after a forward and once after a backward:

A healthy step moves each weight by about a thousandth of its size. Worked: \(w = 0.8\), \(\partial\mathcal{L}/\partial w = 0.15\), \(\eta = 0.01\). The update is 0.0015, and the ratio is \(0.0015 / 0.8 = \mathbf{0.001875}\). Near \(10^{-3}\). If the ratio is \(10^{-6}\), raise the learning rate or you will wait forever. If it is \(10^{-1}\), you are jumping out of the basin.

BatchNorm makes the stack much less sensitive to the Kaiming gain — that is one of its jobs. Residuals (a later chapter) make it less sensitive still, by giving every layer a path that does not go through a squash.

Mistakes learners make

EMA weights that do not add to 1. 0.99 * running + 0.001 * batch leaks mass toward zero. Always (1-m)*running + m*batch.

Leaving BatchNorm in train mode at eval. Then the reported loss depends on the validation batch size, and the running stats freeze wherever you left them.

Keeping a bias in the linear layer right before BN. The mean-subtract deletes it. The extra parameter does no work.

Using BatchNorm inside a transformer. Sequences are variable length, batches mix documents, and decode is batch-size-1. Use LayerNorm or RMSNorm, over features.

Treating \(\lvert h\rvert>0.99\) as the only saturation test. Look at the gradient histogram too. A unit can have a moderate activation and a vanishing incoming grad from the layer above.

What to remember

  • First loss should sit near \(\ln V\). Much higher means a too-sharp random head.
  • \(\mathrm{std}(XW) \approx \sigma_w\sqrt{n}\). Kaiming picks \(\sigma_w = \mathrm{gain}/\sqrt{n}\).
  • BatchNorm standardizes the batch, then \(\gamma,\beta\) restretch. The running EMA is \((1-m)r + m\cdot\mathrm{batch}\).
  • LayerNorm / RMSNorm standardize one example across features. Transformers use these.
  • A healthy update-to-data ratio is about \(10^{-3}\).
init_norm.pyBN / LN / RMSNorm match torch; EMA weights sum to 1
"""Init, activations, BatchNorm, LayerNorm, RMSNorm.

A deep net trains only if activations and gradients stay a reasonable
size. This file is the worked arithmetic from the chapter: the fan-in
rule, a 4-number BatchNorm, a 3-number LayerNorm / RMSNorm, and the
EMA whose two weights actually add to 1.
"""
import math
import torch
import torch.nn.functional as F

# --- Expected loss at a uniform head. Same V=5 as the trail-word LM. ---
V = 5
UNIFORM_NLL = math.log(V)  # 1.609437...

# A "confidently wrong" head: one huge logit on the wrong class.
# Target is class 0; the peak is on class 1.
CONFIDENT_LOGITS = torch.tensor([[-1.0, 8.0, 0.0, 0.0, 0.0]])
CONFIDENT_NLL = F.cross_entropy(CONFIDENT_LOGITS, torch.tensor([0])).item()


def matmul_std(fan_in, w_std, n_rows=4000, seed=0):
    """Empirical std of Y = X @ W, with X ~ N(0,1) and W ~ N(0, w_std^2)."""
    g = torch.Generator().manual_seed(seed)
    X = torch.randn(n_rows, fan_in, generator=g)
    W = torch.randn(fan_in, fan_in, generator=g) * w_std
    return (X @ W).std().item()


FAN_IN = 16
STD_UNSCALED = matmul_std(FAN_IN, w_std=1.0)          # ≈ sqrt(16) = 4
STD_SCALED = matmul_std(FAN_IN, w_std=1 / math.sqrt(FAN_IN))  # ≈ 1
KAIMING_LINEAR = 1 / math.sqrt(FAN_IN)                # 0.25
KAIMING_RELU = math.sqrt(2 / FAN_IN)                  # 0.353...
TANH_GAIN = 5 / 3
KAIMING_TANH = TANH_GAIN / math.sqrt(FAN_IN)          # (5/3)/4 = 0.416...


def tanh_saturation(pre, thresh=0.99):
    """Fraction of tanh activations with |h| > thresh (a dead-ish unit)."""
    h = torch.tanh(pre)
    return (h.abs() > thresh).float().mean().item(), h.std().item()


# A badly scaled hidden pre-activation vs a 1/sqrt(fan_in) one.
_sat = torch.Generator().manual_seed(1)
BAD_PRE = torch.randn(2000, 64, generator=_sat) * 4.0     # std 4, tanh saturates
GOOD_PRE = torch.randn(2000, 64, generator=_sat) * 1.0    # std 1
BAD_SAT, BAD_H_STD = tanh_saturation(BAD_PRE)
GOOD_SAT, GOOD_H_STD = tanh_saturation(GOOD_PRE)


# --- BatchNorm on four numbers. Population /N, one channel. ---
BN_X = torch.tensor([1.0, 3.0, 5.0, 7.0])
BN_MEAN = BN_X.mean()                     # 4
BN_VAR = BN_X.var(unbiased=False)         # 5  (divide by N, not N-1)
BN_EPS = 1e-5
BN_XHAT = (BN_X - BN_MEAN) / torch.sqrt(BN_VAR + BN_EPS)
BN_GAMMA, BN_BETA = 2.0, -1.0
BN_Y = BN_GAMMA * BN_XHAT + BN_BETA


def batch_norm(x, gamma, beta, eps=1e-5):
    """BN over the leading axis. x is (N,) or (N, C) with C=1."""
    mean = x.mean(dim=0)
    var = x.var(dim=0, unbiased=False)
    xhat = (x - mean) / torch.sqrt(var + eps)
    return gamma * xhat + beta, mean, var, xhat


# --- EMA: the two coefficients must sum to 1. ---
# PyTorch BatchNorm: running = (1 - m) * running + m * batch, default m=0.1
EMA_M = 0.1
EMA_START = 0.0
EMA_BATCHES = [4.0, 6.0, 5.0]


def ema(batches, m=EMA_M, start=EMA_START):
    running = start
    history = [running]
    for batch in batches:
        running = (1 - m) * running + m * batch
        history.append(running)
    return history


EMA_HISTORY = ema(EMA_BATCHES)
# Wrong formula from the old chapter: 0.99 * running + 0.001 * batch.
# The weights add to 0.991, so the average silently decays toward 0.


def ema_wrong(batches, start=EMA_START):
    running = start
    for batch in batches:
        running = 0.99 * running + 0.001 * batch
    return running


EMA_WRONG_END = ema_wrong(EMA_BATCHES)


# --- LayerNorm: same three numbers, normalize over features, not batch. ---
LN_X = torch.tensor([2.0, 4.0, 6.0])
LN_MEAN = LN_X.mean()                     # 4
LN_VAR = LN_X.var(unbiased=False)         # ((4+0+4)/3) = 8/3
LN_XHAT = (LN_X - LN_MEAN) / torch.sqrt(LN_VAR + BN_EPS)
LN_Y = F.layer_norm(LN_X, (3,), eps=BN_EPS)  # gamma=1, beta=0


# --- RMSNorm: no mean subtract, no beta. Llama-style. ---
RMS_X = torch.tensor([1.0, 2.0, 2.0])
RMS = torch.sqrt((RMS_X ** 2).mean())     # sqrt(3)
RMS_Y = RMS_X / RMS                       # gamma = 1


def rms_norm(x, gamma=1.0, eps=1e-5):
    """y = gamma * x / rms(x). No mean, no beta."""
    rms = torch.sqrt((x ** 2).mean() + eps)
    return gamma * x / rms


# Healthy step size: |lr * grad| / |weight| sits near 1e-3.
UPD_W, UPD_GRAD, UPD_LR = 0.8, 0.15, 0.01
UPD_RATIO = abs(UPD_LR * UPD_GRAD) / abs(UPD_W)  # 0.001875

# A 2×3 matrix showing BN-over-rows vs LN-over-cols.
AXIS_X = torch.tensor([[1.0, 2.0, 3.0],
                       [3.0, 4.0, 5.0]])
# BN on dim 0 (per column, across the batch of 2):
AXIS_BN_MEAN = AXIS_X.mean(dim=0)         # [2, 3, 4]
# LN on dim 1 (per row, across 3 features):
AXIS_LN_MEAN = AXIS_X.mean(dim=1)         # [2, 4]

Check yourself

1. A 5-way head that is truly uniform has what NLL? What does NLL ≈ 9.00 tell you?

\(\ln 5 \approx 1.609\). 9.00 means a huge logit on the wrong class — confidently wrong, not uninformed.

2. Why does fan-in 16 with \(\sigma_w=1\) produce outputs of std ≈ 4?

Each output is a sum of 16 independent products. Variances add, so std grows as \(\sqrt{16}=4\).

3. After batches 4, 6, 5 and \(m=0.1\), what is the running mean if you started at 0?

\(0.4\), then \(0.96\), then \(1.364\). The broken 0.99/0.001 mix ends at 0.015.

4. What does RMSNorm drop, compared to LayerNorm?

The mean subtraction and the shift \(\beta\). It only divides by the rms and multiplies by \(\gamma\).

5. \(w=0.8\), grad \(=0.15\), \(\eta=0.01\). Is the step healthy?

Update/data \(= 0.001875 \approx 10^{-3}\). Yes.

Go to the source

Keep going