NEURONS ALL THE WAY DOWN · CHAPTER 05
Scalar autograd was the chain rule. This chapter is the same rule with shapes. Every tensor gradient of a tiny MLP, including a BatchNorm backward that is easy to get wrong.
+=In chapter 02, every node was one number. Now a node is a tensor. The local rule is the same — multiply the incoming gradient by the local Jacobian — but you never write the Jacobian. You write a handful of sums, transposes, and multiplies that implement the vector-Jacobian product, and you check the output shape.
Three habits get you through any layer:
+= as \(a + a\).If the shapes do not line up, the formula is wrong. Matching shapes is not a style point. It is how you find the missing transpose.
Let \(A = [1,\,2]\) (shape \(1\times 2\)) and \(B = \begin{bmatrix}3\\4\end{bmatrix}\) (shape \(2\times 1\)). Then \(C = AB = [11]\). Seed \(\partial L/\partial C = [1]\). The two parent gradients are
put the incoming gradient on the left or the right so the remaining axis matches the parent. The transpose is whichever one makes the shapes legal.
In code: dA = dC @ B.T and dB = A.T @ dC. If you write them the other way around, PyTorch will refuse the multiply, which is the kindest possible error.
Broadcast is the dual of a sum. Forward: add a bias of shape \((C,)\) to a batch of shape \((B, C)\). The bias is copied down the batch. Backward: that copy is \(B\) paths into the same bias, so
sum the incoming gradient over every axis the forward pass broadcasted.
Worked: \(x + b\) with \(b = (0.5, -0.5)\) and \(\partial L/\partial y = \mathbf{1}_{2\times 2}\). Then \(\partial L/\partial b = (2, 2)\). Two rows, each contributing 1.
C=[11], ∂C=1 · dA = dC @ Bᵀ · dB = Aᵀ @ dC · ∂b = Σ_batch
Mean cross-entropy is \(-\frac{1}{B}\sum_i \log P_{i, y_i}\). Writing the softmax Jacobian and then the log Jacobian is a page of algebra. They collapse:
at the true class, subtract 1 from the probability, then divide by the batch. Everywhere else, the gradient is just the probability, over B.
Read it as a force. Softmax already put mass \(P\) on each class. The one-hot wanted mass 1 on class \(y\). The residual \(P - e_y\) is how far the distribution still has to move. The loss pulls the correct logit up (negative entry) and the others down (positive entries).
P = softmax(z) · e_y → (P−e_y)/B
Same first row as chapter 03: \(z = (2.0,\,0.5,\,-1.0)\), \(P \approx (0.786,\,0.175,\,0.039)\), target 0. For a batch of two,
half of (0.786 − 1, 0.175, 0.039). The other row of the batch contributes the other half of the mean.
A useful check: the row of \(\partial\mathcal{L}/\partial z\) sums to 0. Softmax lives on a simplex; a legal tangent vector is mean-zero.
Reuse the four numbers from chapter 04. \(x = (1,3,5,7)\), \(\mu=4\), \(\sigma^2=5\), \(\hat x \approx (-1.342,\,-0.447,\,0.447,\,1.342)\), \(\gamma=2\), \(\beta=-1\). Call \(\mathrm{bndiff} = x-\mu\) and \(\mathrm{bnraw} = \hat x = \mathrm{bndiff}\cdot(\sigma^2+\varepsilon)^{-1/2}\).
Upstream we take \(\partial L/\partial y = (0.2,\,-0.1,\,0.4,\,-0.3)\). Walking backwards, in order:
The compact form, once the dust settles:
a scaled residual: take N copies of the incoming grad, subtract its sum, subtract the projection onto xhat. The result is orthogonal to the all-ones vector and to xhat — BatchNorm’s two constraints.
On these numbers the input gradient is about \((0,\,-0.179,\,0.358,\,-0.179)\). The fused line and the step-by-step line agree with F.batch_norm’s autograd to machine precision.
The version that used to sit in this lesson swapped two lines. It set d_bnraw = inv * d_y (skipping \(\gamma\), and applying \(\mathrm{inv}\) one step too early) and d_inv = (bnraw * d_y).sum() (using \(\hat x\) where \(x-\mu\) belongs). The result is a tidy, wrong tensor — here \((0,\,-0.040,\,0.080,\,-0.040)\) — that will not match a gradient check. The algebra looks almost right, which is why the swap survives.
Forward, C[ids] copies rows. Backward, two copies of the same row must add. That is scatter-add:
every position that plucked row k sends its incoming gradient back to row k. Same row twice, two terms.
Worked table \(C \in \mathbb{R}^{3\times 2}\), ids \(= (0, 2, 0)\), incoming \(E\)-grad \(\bigl[(1,0),\,(0,1),\,(1,1)\bigr]\). Row 0 is used at positions 0 and 2, so it receives \((1,0)+(1,1) = \mathbf{[2, 1]}\). Row 2 receives \((0,1)\). Row 1 is unused and stays 0. In PyTorch: dC.index_add_(0, ids, dE).
ids=(0,2,0) → index_add_ → row0=[2,1]
\(\tanh\) you already know. At \(z=1.2\), \(h\approx 0.834\) and the local grad is \(\mathbf{0.305}\) — the same neuron as chapter 02. A view or reshape is the identity on the numbers; the backward is a view the other way, no arithmetic.
"""Tensor-level backward pass of a tiny MLP, written by hand.
Shapes stay small so every gradient on the page is a handful of
numbers. The check script compares each piece to PyTorch autograd,
and checks that the *correct* BatchNorm backward matches — the
older lesson swapped bndiff and bnraw in two lines.
Notation on every gradient: dZ means ∂L/∂Z.
"""
import torch
import torch.nn.functional as F
torch.set_default_dtype(torch.float64)
# --- 1. Matmul: C = A @ B. Shapes tell you the transposes. ---
A = torch.tensor([[1.0, 2.0]]) # (1, 2)
B = torch.tensor([[3.0], [4.0]]) # (2, 1)
C = A @ B # (1, 1) = [[11]]
dC = torch.tensor([[1.0]])
dA = dC @ B.T # (1, 2) = [[3, 4]]
dB = A.T @ dC # (2, 1) = [[1], [2]]
# --- 2. Broadcast ⇔ sum. y = x + b, b is (C,), x is (B, C). ---
X_BCAST = torch.tensor([[1.0, 2.0],
[3.0, 4.0]]) # (2, 2)
B_VEC = torch.tensor([0.5, -0.5]) # (2,)
Y_BCAST = X_BCAST + B_VEC
dY_BCAST = torch.tensor([[1.0, 1.0],
[1.0, 1.0]])
dX_BCAST = dY_BCAST # same shape
dB_VEC = dY_BCAST.sum(0) # (2,) = [2, 2]
# --- 3. Softmax + mean CE. Fused: d_logits = (P - onehot) / B. ---
LOGITS = torch.tensor([[2.0, 0.5, -1.0],
[0.0, 1.0, 0.0]]) # (2, 3)
TARGETS = torch.tensor([0, 1])
PROBS = F.softmax(LOGITS, dim=-1)
ONEHOT = F.one_hot(TARGETS, 3).to(LOGITS.dtype)
D_LOGITS = (PROBS - ONEHOT) / LOGITS.shape[0]
def fused_d_logits(logits, targets):
"""Mean-CE backward in one line. The 'forces' picture: pull p toward 1 at y."""
probs = F.softmax(logits, dim=-1)
onehot = F.one_hot(targets, logits.shape[-1]).to(logits.dtype)
return (probs - onehot) / logits.shape[0]
# --- 4. tanh. h = tanh(z), dh/dz = 1 - h^2. ---
Z_TANH = torch.tensor([1.2, -0.5])
H_TANH = torch.tanh(Z_TANH)
dH_TANH = torch.tensor([1.0, 1.0])
dZ_TANH = (1 - H_TANH ** 2) * dH_TANH
# --- 5. BatchNorm backward, written in the *correct* order. ---
# Forward, one channel, N=4 (the same four numbers as chapter 04):
# mu, var over the batch; xhat = (x-mu)/sqrt(var+eps); y = gamma*xhat + beta
BN_X = torch.tensor([1.0, 3.0, 5.0, 7.0])
BN_GAMMA = torch.tensor(2.0)
BN_BETA = torch.tensor(-1.0)
BN_EPS = 1e-5
BN_N = BN_X.numel()
BN_MU = BN_X.mean()
BN_VAR = BN_X.var(unbiased=False)
BN_INV = 1.0 / torch.sqrt(BN_VAR + BN_EPS)
BN_DIFF = BN_X - BN_MU # "bndiff"
BN_RAW = BN_DIFF * BN_INV # "bnraw" = xhat
BN_Y = BN_GAMMA * BN_RAW + BN_BETA
# Upstream grad from the next layer (say tanh). Chosen so the numbers move.
dBN_Y = torch.tensor([0.2, -0.1, 0.4, -0.3])
def bn_backward_steps(x, gamma, beta, eps, dy):
"""Step-by-step BN backward. Order matters. Do not swap bndiff and bnraw."""
n = x.numel()
mu = x.mean()
var = x.var(unbiased=False)
inv = 1.0 / torch.sqrt(var + eps)
diff = x - mu
raw = diff * inv # xhat
# y = gamma * raw + beta
d_raw = gamma * dy # NOT inv * dy (that was the old bug)
d_gamma = (raw * dy).sum()
d_beta = dy.sum()
# raw = diff * inv
d_diff = inv * d_raw
d_inv = (diff * d_raw).sum() # NOT (raw * dy).sum()
# inv = (var + eps)^(-1/2)
d_var = d_inv * (-0.5) * inv ** 3
# mu is used in `diff` and inside `var`. Both paths, so += .
d_mu = -d_diff.sum() + d_var * (-2.0 * diff.sum() / n)
# x is used in `diff` and inside `var`. Both paths, so += .
dx = d_diff + d_var * (2.0 * diff / n) + d_mu / n
return {
"d_raw": d_raw, "d_gamma": d_gamma, "d_beta": d_beta,
"d_diff": d_diff, "d_inv": d_inv, "d_var": d_var, "d_mu": d_mu, "dx": dx,
"raw": raw, "inv": inv, "mu": mu, "var": var,
}
def bn_backward_fused(x, gamma, eps, dy):
"""The compact form, one channel:
dx = (gamma / (N * sigma)) * (N*dy - sum(dy) - xhat * sum(dy * xhat))
"""
n = x.numel()
mu = x.mean()
var = x.var(unbiased=False)
sigma = torch.sqrt(var + eps)
xhat = (x - mu) / sigma
return (gamma / (n * sigma)) * (
n * dy - dy.sum() - xhat * (dy * xhat).sum()
)
BN_STEPS = bn_backward_steps(BN_X, BN_GAMMA, BN_BETA, BN_EPS, dBN_Y)
BN_FUSED_DX = bn_backward_fused(BN_X, BN_GAMMA, BN_EPS, dBN_Y)
def bn_backward_buggy(x, gamma, eps, dy):
"""The two swapped lines from the old lesson. Kept only so the check
can prove they do *not* match PyTorch."""
n = x.numel()
mu = x.mean()
var = x.var(unbiased=False)
inv = 1.0 / torch.sqrt(var + eps)
diff = x - mu
raw = diff * inv
# BUG: treats d_hpreact as if it were already d_raw, and multiplies
# by inv a second time. Then uses raw where diff belongs.
d_raw = inv * dy
d_inv = (raw * dy).sum()
d_var = d_inv * (-0.5) * inv ** 3
d_diff = inv * d_raw
d_mu = -d_diff.sum() + d_var * (-2.0 * diff.sum() / n)
dx = d_diff + d_var * (2.0 * diff / n) + d_mu / n
return dx
# --- 6. Embedding scatter-add. Two uses of row 0 must add. ---
EMB_C = torch.tensor([[0.1, 0.2],
[0.3, 0.4],
[0.5, 0.6]]) # (3, 2)
EMB_IDS = torch.tensor([0, 2, 0])
EMB_OUT = EMB_C[EMB_IDS] # (3, 2)
dEMB_OUT = torch.tensor([[1.0, 0.0],
[0.0, 1.0],
[1.0, 1.0]])
dEMB_C = torch.zeros_like(EMB_C)
dEMB_C.index_add_(0, EMB_IDS, dEMB_OUT) # row 0 gets [1,0] + [1,1] = [2, 1]
def torch_bn_dx(x, gamma, beta, eps, dy):
"""Autograd reference for the BN input gradient."""
x = x.detach().requires_grad_(True)
y = F.batch_norm(
x.view(-1, 1),
running_mean=None, running_var=None,
weight=gamma.view(1), bias=beta.view(1),
training=True, eps=eps,
).view(-1)
y.backward(dy)
return x.grad
Forgetting += on a reused tensor. \(\mu\) in BatchNorm, an embedding row, a weight used in two heads. Two paths, one parameter, add the grads.
Swapping bndiff and bnraw. They are the same vector only when \(\sigma=1\). The backward of a product needs the other factor: \(\mathrm{inv}\) with \(\mathrm{bndiff}\), not with \(\hat x\).
Dropping the \(1/B\). F.cross_entropy is a mean. The fused gradient divides by \(B\). A sum-reduction would not. Check the reduction before you compare to autograd.
Clipping the loss instead of the gradient. loss.clamp changes the objective. clip_grad_norm_ changes the step. They are not interchangeable.
dA = dC @ B.T, dB = A.T @ dC. Shapes pick the transpose.\([3, 4]\). Each entry of \(A\) is multiplied by the matching entry of \(B\), and \(C=11\).
The forward pass copied the bias to every row. Each copy is a path back to the same vector, so the incoming grads add.
\((0.786 - 1)/2 \approx -0.107\).
The sum, \(0.2\). \(\beta\) is added after the normalize, so its grad is just \(\sum \partial L/\partial y\).
\([2, 1]\). Positions 0 and 2 both pointed at row 0.