NEURONS ALL THE WAY DOWN · CHAPTER 04
Random weights are not a strategy. Activations saturate, logits get too sharp, and gradients vanish. Scale the weights, normalize the activations, then watch the histograms.
Each layer multiplies by a weight matrix and then squashes. If the matrix is too loud, \(\tanh\) slams every hidden unit to \(\pm 1\) and the local gradient \(1-\tanh^2\) is 0. If it is too quiet, every unit sits near 0 and the net is almost linear. Depth multiplies those mistakes. Five layers of “std 3 in, std 9 out” do not give you a deeper thinker. They give you NaNs.
So we do two things. We pick the volume of \(W\) from the fan-in, once, at init. And we put a normalizer — BatchNorm, LayerNorm, or RMSNorm — after the multiply, so each layer’s output stays on a known scale even as the weights move.
Training is a conversation between activations going forward and gradients going back. If either side is tiny or huge, the conversation stops. This chapter is about keeping both sides audible.
A head that knows nothing about \(V\) classes should put \(1/V\) on each, so the NLL is \(\ln V\). For the 5-symbol trail alphabet that is \(\ln 5 \approx \mathbf{1.609}\). Chapter 03 started at 2.92. That is not “the loss is wired wrong.” That is a random \(W_2\) with std 1 writing logits of size \(\sqrt{n_{\mathrm{hidden}}}\), then softmax becoming a spike on a random class.
Push it further. Logits \((-1, 8, 0, 0, 0)\) with target 0 have NLL \(\approx \mathbf{9.00}\). The model is almost sure the answer is class 1. The first many steps of training are spent flattening that spike. Shrink \(W_2\) (and \(b_2\)) by 10× or 100× at init and the hockey-stick at the start of the loss curve goes away.
Read the first loss before you read anything else. Near \(\ln V\): the head is sane. Much larger: the logits are too sharp. Much smaller: you are leaking the label or evaluating on the training batch you just overfit.
\(\tanh\) is almost linear near 0 and flat near \(\pm 1\). Its derivative is \(1 - \tanh^2(n)\). At \(n = 0\) that is 1. At \(\tanh n = 0.99\) that is \(1 - 0.9801 = 0.02\). A unit stuck in the flat region is a dead-ish unit: it still outputs something, but it no longer tells its incoming weights how to change.
Draw pre-activations at std 4 (what you get from fan-in 16 and unscaled \(W\)). About 51% of the tanh outputs land outside \(\pm 0.99\). Draw them at std 1, and that fraction drops under 1%.
ReLU dies a different way: a unit that is negative stays at 0, and if the incoming weights never drag it back, it is gone for good. Leaky ReLU and GELU are later answers. The first answer is: do not feed the nonlinearity a wild pre-activation.
Interactive · Tanh saturation
How many hidden units freeze as you turn up the pre-activation scale?
Let each row of \(X\) be \(N(0,1)\) of length \(n\), and each column of \(W\) be \(N(0, \sigma_w^2)\). Then each output is a sum of \(n\) independent products, and
the output standard deviation grows with the square root of the fan-in, unless you shrink the weights to match.
With \(n = 16\) and \(\sigma_w = 1\), a 4000-row sample comes out at std 3.81, next to the predicted 4. Set \(\sigma_w = 1/\sqrt{16} = \mathbf{0.25}\) and the same sample comes out at std 0.95, next to the predicted 1.
σw=1 → std≈4 → σw=0.25 → std≈0.95
That \(1/\sqrt{n}\) is the linear / “Xavier” scale. Two common gains sit on top of it:
| Nonlinearity | Gain | \(\sigma_w\) for \(n=16\) | Why |
|---|---|---|---|
| linear / identity | 1 | 0.25 | preserve variance through the multiply |
| ReLU | √2 | 0.354 | ReLU zeroes half the mass, so boost by √2 |
| tanh | 5/3 | 0.417 | tanh shrinks variance; 5/3 is the usual restore |
Kaiming init is \(\sigma_w = \mathrm{gain} / \sqrt{\mathrm{fan\_in}}\). PyTorch’s nn.Linear actually uses a uniform box of the same variance, not a Gaussian, and its default gain is for a linear layer. If you then put \(\tanh\) on top, you are slightly quiet. Either set the gain, or let a normalizer (next section) absorb the mismatch.
BatchNorm, per channel, over the batch:
subtract the batch mean, divide by the batch std, then scale and shift with two learned numbers per channel.
Worked channel, four activations \([1, 3, 5, 7]\):
\(\gamma\) and \(\beta\) are there so the layer can still learn a scale and a shift. If the right answer really is “all large and negative,” the net can get that by driving \(\gamma\) and \(\beta\), instead of fighting the normalizer.
A bias sitting immediately before BatchNorm is wasted. The mean-subtract deletes it. Set bias=False on that linear layer.
At test time there is no batch. So during training we keep an exponential moving average of \(\mu\) and \(\sigma^2\). PyTorch’s default momentum is \(m = 0.1\):
90% the old running value, 10% this batch. The two weights add to 1, so the running value is always a convex combination of the batches it has seen.
Start at 0. See batches with means 4, then 6, then 5:
| After batch | Correct EMA | Broken \(0.99\cdot\mathrm{run}+0.001\cdot\mathrm{batch}\) |
|---|---|---|
| 4 | 0.4 | 0.004 |
| 6 | 0.96 | 0.010 |
| 5 | 1.364 | 0.015 |
The broken mix — which an older version of this lesson wrote down — has weights that add to 0.991. Every step leaks 0.9% of the mass toward zero. After three batches the “running mean” is 0.015, not 1.364. Leave it running for an epoch and it is still near 0 while the data lives near 5. Eval then looks like a different model.
→ 1.364 · 0.99r+0.001b · ends ≈ 0.015
In model.eval() the layer uses the running stats and does not update them. A forgotten eval() is why a trained net “gets worse” the moment you start reporting validation loss.
That was the original motivation. Later analysis (Santurkar et al., 2018) argues the real gift is a smoother loss landscape — the gradients change less when the weights move. Either story agrees on the engineering: put BN between the linear map and the nonlinearity, keep a correct EMA, switch to eval stats at test time.
BatchNorm couples every example in the minibatch. That is a regularizer, and a nuisance: the forward pass of sentence A depends on sentence B, the running stats must be warmed up, and batch size 1 is unhappy. Transformers switched to LayerNorm: the same formula, but the mean and variance run across the features of one example.
On the vector \([2, 4, 6]\):
three features, one example. No batch. No train/eval split. No running EMA.
A 2×3 toy makes the axis difference concrete. Rows are examples, columns are features:
| f0 | f1 | f2 | LN mean (across features) | |
|---|---|---|---|---|
| example 0 | 1 | 2 | 3 | 2 |
| example 1 | 3 | 4 | 5 | 4 |
| BN mean (across batch) | 2 | 3 | 4 |
RMSNorm drops the mean subtraction and the shift \(\beta\). It only rescales by the root-mean-square:
no centering. One learned gain per channel, no bias. Cheaper, and what Llama-style blocks use.
On \([1, 2, 2]\), \(\mathrm{rms} = \sqrt{3} \approx 1.732\), so \(\hat x \approx (0.577,\, 1.155,\, 1.155)\). Modern decoder blocks (see Deep Cuts, “the modern block”) put RMSNorm before attention and the MLP, not after. That is a placement choice. The formula is the one above.
2×3 toy · BN μ=[2,3,4] · LN μ=[2,4] · RMS · no mean
When a deep stack misbehaves, plot three things per layer, once after a forward and once after a backward:
A healthy step moves each weight by about a thousandth of its size. Worked: \(w = 0.8\), \(\partial\mathcal{L}/\partial w = 0.15\), \(\eta = 0.01\). The update is 0.0015, and the ratio is \(0.0015 / 0.8 = \mathbf{0.001875}\). Near \(10^{-3}\). If the ratio is \(10^{-6}\), raise the learning rate or you will wait forever. If it is \(10^{-1}\), you are jumping out of the basin.
BatchNorm makes the stack much less sensitive to the Kaiming gain — that is one of its jobs. Residuals (a later chapter) make it less sensitive still, by giving every layer a path that does not go through a squash.
EMA weights that do not add to 1. 0.99 * running + 0.001 * batch leaks mass toward zero. Always (1-m)*running + m*batch.
Leaving BatchNorm in train mode at eval. Then the reported loss depends on the validation batch size, and the running stats freeze wherever you left them.
Keeping a bias in the linear layer right before BN. The mean-subtract deletes it. The extra parameter does no work.
Using BatchNorm inside a transformer. Sequences are variable length, batches mix documents, and decode is batch-size-1. Use LayerNorm or RMSNorm, over features.
Treating \(\lvert h\rvert>0.99\) as the only saturation test. Look at the gradient histogram too. A unit can have a moderate activation and a vanishing incoming grad from the layer above.
"""Init, activations, BatchNorm, LayerNorm, RMSNorm.
A deep net trains only if activations and gradients stay a reasonable
size. This file is the worked arithmetic from the chapter: the fan-in
rule, a 4-number BatchNorm, a 3-number LayerNorm / RMSNorm, and the
EMA whose two weights actually add to 1.
"""
import math
import torch
import torch.nn.functional as F
# --- Expected loss at a uniform head. Same V=5 as the trail-word LM. ---
V = 5
UNIFORM_NLL = math.log(V) # 1.609437...
# A "confidently wrong" head: one huge logit on the wrong class.
# Target is class 0; the peak is on class 1.
CONFIDENT_LOGITS = torch.tensor([[-1.0, 8.0, 0.0, 0.0, 0.0]])
CONFIDENT_NLL = F.cross_entropy(CONFIDENT_LOGITS, torch.tensor([0])).item()
def matmul_std(fan_in, w_std, n_rows=4000, seed=0):
"""Empirical std of Y = X @ W, with X ~ N(0,1) and W ~ N(0, w_std^2)."""
g = torch.Generator().manual_seed(seed)
X = torch.randn(n_rows, fan_in, generator=g)
W = torch.randn(fan_in, fan_in, generator=g) * w_std
return (X @ W).std().item()
FAN_IN = 16
STD_UNSCALED = matmul_std(FAN_IN, w_std=1.0) # ≈ sqrt(16) = 4
STD_SCALED = matmul_std(FAN_IN, w_std=1 / math.sqrt(FAN_IN)) # ≈ 1
KAIMING_LINEAR = 1 / math.sqrt(FAN_IN) # 0.25
KAIMING_RELU = math.sqrt(2 / FAN_IN) # 0.353...
TANH_GAIN = 5 / 3
KAIMING_TANH = TANH_GAIN / math.sqrt(FAN_IN) # (5/3)/4 = 0.416...
def tanh_saturation(pre, thresh=0.99):
"""Fraction of tanh activations with |h| > thresh (a dead-ish unit)."""
h = torch.tanh(pre)
return (h.abs() > thresh).float().mean().item(), h.std().item()
# A badly scaled hidden pre-activation vs a 1/sqrt(fan_in) one.
_sat = torch.Generator().manual_seed(1)
BAD_PRE = torch.randn(2000, 64, generator=_sat) * 4.0 # std 4, tanh saturates
GOOD_PRE = torch.randn(2000, 64, generator=_sat) * 1.0 # std 1
BAD_SAT, BAD_H_STD = tanh_saturation(BAD_PRE)
GOOD_SAT, GOOD_H_STD = tanh_saturation(GOOD_PRE)
# --- BatchNorm on four numbers. Population /N, one channel. ---
BN_X = torch.tensor([1.0, 3.0, 5.0, 7.0])
BN_MEAN = BN_X.mean() # 4
BN_VAR = BN_X.var(unbiased=False) # 5 (divide by N, not N-1)
BN_EPS = 1e-5
BN_XHAT = (BN_X - BN_MEAN) / torch.sqrt(BN_VAR + BN_EPS)
BN_GAMMA, BN_BETA = 2.0, -1.0
BN_Y = BN_GAMMA * BN_XHAT + BN_BETA
def batch_norm(x, gamma, beta, eps=1e-5):
"""BN over the leading axis. x is (N,) or (N, C) with C=1."""
mean = x.mean(dim=0)
var = x.var(dim=0, unbiased=False)
xhat = (x - mean) / torch.sqrt(var + eps)
return gamma * xhat + beta, mean, var, xhat
# --- EMA: the two coefficients must sum to 1. ---
# PyTorch BatchNorm: running = (1 - m) * running + m * batch, default m=0.1
EMA_M = 0.1
EMA_START = 0.0
EMA_BATCHES = [4.0, 6.0, 5.0]
def ema(batches, m=EMA_M, start=EMA_START):
running = start
history = [running]
for batch in batches:
running = (1 - m) * running + m * batch
history.append(running)
return history
EMA_HISTORY = ema(EMA_BATCHES)
# Wrong formula from the old chapter: 0.99 * running + 0.001 * batch.
# The weights add to 0.991, so the average silently decays toward 0.
def ema_wrong(batches, start=EMA_START):
running = start
for batch in batches:
running = 0.99 * running + 0.001 * batch
return running
EMA_WRONG_END = ema_wrong(EMA_BATCHES)
# --- LayerNorm: same three numbers, normalize over features, not batch. ---
LN_X = torch.tensor([2.0, 4.0, 6.0])
LN_MEAN = LN_X.mean() # 4
LN_VAR = LN_X.var(unbiased=False) # ((4+0+4)/3) = 8/3
LN_XHAT = (LN_X - LN_MEAN) / torch.sqrt(LN_VAR + BN_EPS)
LN_Y = F.layer_norm(LN_X, (3,), eps=BN_EPS) # gamma=1, beta=0
# --- RMSNorm: no mean subtract, no beta. Llama-style. ---
RMS_X = torch.tensor([1.0, 2.0, 2.0])
RMS = torch.sqrt((RMS_X ** 2).mean()) # sqrt(3)
RMS_Y = RMS_X / RMS # gamma = 1
def rms_norm(x, gamma=1.0, eps=1e-5):
"""y = gamma * x / rms(x). No mean, no beta."""
rms = torch.sqrt((x ** 2).mean() + eps)
return gamma * x / rms
# Healthy step size: |lr * grad| / |weight| sits near 1e-3.
UPD_W, UPD_GRAD, UPD_LR = 0.8, 0.15, 0.01
UPD_RATIO = abs(UPD_LR * UPD_GRAD) / abs(UPD_W) # 0.001875
# A 2×3 matrix showing BN-over-rows vs LN-over-cols.
AXIS_X = torch.tensor([[1.0, 2.0, 3.0],
[3.0, 4.0, 5.0]])
# BN on dim 0 (per column, across the batch of 2):
AXIS_BN_MEAN = AXIS_X.mean(dim=0) # [2, 3, 4]
# LN on dim 1 (per row, across 3 features):
AXIS_LN_MEAN = AXIS_X.mean(dim=1) # [2, 4]
\(\ln 5 \approx 1.609\). 9.00 means a huge logit on the wrong class — confidently wrong, not uninformed.
Each output is a sum of 16 independent products. Variances add, so std grows as \(\sqrt{16}=4\).
\(0.4\), then \(0.96\), then \(1.364\). The broken 0.99/0.001 mix ends at 0.015.
The mean subtraction and the shift \(\beta\). It only divides by the rms and multiplies by \(\gamma\).
Update/data \(= 0.001875 \approx 10^{-3}\). Yes.