NEURONS ALL THE WAY DOWN · CHAPTER 05

Backprop by Hand

Scalar autograd was the chain rule. This chapter is the same rule with shapes. Every tensor gradient of a tiny MLP, including a BatchNorm backward that is easy to get wrong.

~20 min readWorked tensorsTested PyTorch

What you'll learn

  • How to read matmul transposes off the shapes
  • Why a broadcast in the forward pass is a sum in the backward pass
  • The fused softmax–cross-entropy gradient \(P - e_y\)
  • The correct BatchNorm backward — and the two-line swap that silently breaks it
  • Embedding gradients as scatter-add

Before you start

Shapes are the other half of the chain rule

In chapter 02, every node was one number. Now a node is a tensor. The local rule is the same — multiply the incoming gradient by the local Jacobian — but you never write the Jacobian. You write a handful of sums, transposes, and multiplies that implement the vector-Jacobian product, and you check the output shape.

Three habits get you through any layer:

  1. The incoming gradient has the shape of the layer’s output.
  2. Each parent’s gradient has the shape of that parent. If a forward broadcast stretched a vector, the backward sums the extra axis.
  3. A tensor used twice gets two paths. Add them. Same += as \(a + a\).

If the shapes do not line up, the formula is wrong. Matching shapes is not a style point. It is how you find the missing transpose.

Transposes you can see

Let \(A = [1,\,2]\) (shape \(1\times 2\)) and \(B = \begin{bmatrix}3\\4\end{bmatrix}\) (shape \(2\times 1\)). Then \(C = AB = [11]\). Seed \(\partial L/\partial C = [1]\). The two parent gradients are

put the incoming gradient on the left or the right so the remaining axis matches the parent. The transpose is whichever one makes the shapes legal.

In code: dA = dC @ B.T and dB = A.T @ dC. If you write them the other way around, PyTorch will refuse the multiply, which is the kindest possible error.

Broadcast is the dual of a sum. Forward: add a bias of shape \((C,)\) to a batch of shape \((B, C)\). The bias is copied down the batch. Backward: that copy is \(B\) paths into the same bias, so

sum the incoming gradient over every axis the forward pass broadcasted.

Worked: \(x + b\) with \(b = (0.5, -0.5)\) and \(\partial L/\partial y = \mathbf{1}_{2\times 2}\). Then \(\partial L/\partial b = (2, 2)\). Two rows, each contributing 1.

C=[11], ∂C=1  ·  dA = dC @ Bᵀ  ·  dB = Aᵀ @ dC  ·  ∂b = Σ_batch

MANIM Transpose so shapes match; a forward broadcast is a backward sum. Use the beat buttons to jump.

One fused line

Mean cross-entropy is \(-\frac{1}{B}\sum_i \log P_{i, y_i}\). Writing the softmax Jacobian and then the log Jacobian is a page of algebra. They collapse:

at the true class, subtract 1 from the probability, then divide by the batch. Everywhere else, the gradient is just the probability, over B.

Read it as a force. Softmax already put mass \(P\) on each class. The one-hot wanted mass 1 on class \(y\). The residual \(P - e_y\) is how far the distribution still has to move. The loss pulls the correct logit up (negative entry) and the others down (positive entries).

P = softmax(z)  ·  e_y  →  (P−e_y)/B

MANIM Softmax+CE residual is (P − ey)/B. Use the beat buttons to jump.

Same first row as chapter 03: \(z = (2.0,\,0.5,\,-1.0)\), \(P \approx (0.786,\,0.175,\,0.039)\), target 0. For a batch of two,

half of (0.786 − 1, 0.175, 0.039). The other row of the batch contributes the other half of the mean.

A useful check: the row of \(\partial\mathcal{L}/\partial z\) sums to 0. Softmax lives on a simplex; a legal tangent vector is mean-zero.

Do not swap bndiff and bnraw

Reuse the four numbers from chapter 04. \(x = (1,3,5,7)\), \(\mu=4\), \(\sigma^2=5\), \(\hat x \approx (-1.342,\,-0.447,\,0.447,\,1.342)\), \(\gamma=2\), \(\beta=-1\). Call \(\mathrm{bndiff} = x-\mu\) and \(\mathrm{bnraw} = \hat x = \mathrm{bndiff}\cdot(\sigma^2+\varepsilon)^{-1/2}\).

Upstream we take \(\partial L/\partial y = (0.2,\,-0.1,\,0.4,\,-0.3)\). Walking backwards, in order:

  1. \(y = \gamma\cdot\mathrm{bnraw} + \beta\), so \(\partial L/\partial\mathrm{bnraw} = \gamma\cdot \partial L/\partial y\) and \(\partial L/\partial\gamma = \sum \hat x \odot \partial L/\partial y \approx -0.447\), \(\partial L/\partial\beta = \sum \partial L/\partial y = 0.2\).
  2. \(\mathrm{bnraw} = \mathrm{bndiff}\cdot\mathrm{inv}\), so \(\partial L/\partial\mathrm{bndiff} = \mathrm{inv}\cdot\partial L/\partial\mathrm{bnraw}\) and \(\partial L/\partial\mathrm{inv} = \sum \mathrm{bndiff}\odot\partial L/\partial\mathrm{bnraw}\).
  3. \(\mathrm{inv} = (\sigma^2+\varepsilon)^{-1/2}\), so \(\partial L/\partial\sigma^2 = (\partial L/\partial\mathrm{inv})\cdot(-\tfrac12)\mathrm{inv}^{3}\).
  4. \(\mu\) and \(x\) are each used twice (inside \(\mathrm{bndiff}\) and inside \(\sigma^2\)). Add both paths.

The compact form, once the dust settles:

a scaled residual: take N copies of the incoming grad, subtract its sum, subtract the projection onto xhat. The result is orthogonal to the all-ones vector and to xhat — BatchNorm’s two constraints.

On these numbers the input gradient is about \((0,\,-0.179,\,0.358,\,-0.179)\). The fused line and the step-by-step line agree with F.batch_norm’s autograd to machine precision.

The version that used to sit in this lesson swapped two lines. It set d_bnraw = inv * d_y (skipping \(\gamma\), and applying \(\mathrm{inv}\) one step too early) and d_inv = (bnraw * d_y).sum() (using \(\hat x\) where \(x-\mu\) belongs). The result is a tidy, wrong tensor — here \((0,\,-0.040,\,0.080,\,-0.040)\) — that will not match a gradient check. The algebra looks almost right, which is why the swap survives.

Gather forward, scatter-add backward

Forward, C[ids] copies rows. Backward, two copies of the same row must add. That is scatter-add:

every position that plucked row k sends its incoming gradient back to row k. Same row twice, two terms.

Worked table \(C \in \mathbb{R}^{3\times 2}\), ids \(= (0, 2, 0)\), incoming \(E\)-grad \(\bigl[(1,0),\,(0,1),\,(1,1)\bigr]\). Row 0 is used at positions 0 and 2, so it receives \((1,0)+(1,1) = \mathbf{[2, 1]}\). Row 2 receives \((0,1)\). Row 1 is unused and stays 0. In PyTorch: dC.index_add_(0, ids, dE).

ids=(0,2,0)  →  index_add_  →  row0=[2,1]

MANIM Two gathers of row 0 add on the way back. Use the beat buttons to jump.

\(\tanh\) you already know. At \(z=1.2\), \(h\approx 0.834\) and the local grad is \(\mathbf{0.305}\) — the same neuron as chapter 02. A view or reshape is the identity on the numbers; the backward is a view the other way, no arithmetic.

manual_backprop.pyevery piece matches autograd; the old BN swap does not
"""Tensor-level backward pass of a tiny MLP, written by hand.

Shapes stay small so every gradient on the page is a handful of
numbers. The check script compares each piece to PyTorch autograd,
and checks that the *correct* BatchNorm backward matches — the
older lesson swapped bndiff and bnraw in two lines.

Notation on every gradient: dZ means ∂L/∂Z.
"""
import torch
import torch.nn.functional as F

torch.set_default_dtype(torch.float64)


# --- 1. Matmul: C = A @ B.  Shapes tell you the transposes. ---
A = torch.tensor([[1.0, 2.0]])          # (1, 2)
B = torch.tensor([[3.0], [4.0]])        # (2, 1)
C = A @ B                               # (1, 1) = [[11]]
dC = torch.tensor([[1.0]])
dA = dC @ B.T                           # (1, 2) = [[3, 4]]
dB = A.T @ dC                           # (2, 1) = [[1], [2]]


# --- 2. Broadcast ⇔ sum.  y = x + b, b is (C,), x is (B, C). ---
X_BCAST = torch.tensor([[1.0, 2.0],
                        [3.0, 4.0]])    # (2, 2)
B_VEC = torch.tensor([0.5, -0.5])       # (2,)
Y_BCAST = X_BCAST + B_VEC
dY_BCAST = torch.tensor([[1.0, 1.0],
                         [1.0, 1.0]])
dX_BCAST = dY_BCAST                     # same shape
dB_VEC = dY_BCAST.sum(0)                # (2,) = [2, 2]


# --- 3. Softmax + mean CE.  Fused: d_logits = (P - onehot) / B. ---
LOGITS = torch.tensor([[2.0, 0.5, -1.0],
                       [0.0, 1.0,  0.0]])   # (2, 3)
TARGETS = torch.tensor([0, 1])
PROBS = F.softmax(LOGITS, dim=-1)
ONEHOT = F.one_hot(TARGETS, 3).to(LOGITS.dtype)
D_LOGITS = (PROBS - ONEHOT) / LOGITS.shape[0]


def fused_d_logits(logits, targets):
    """Mean-CE backward in one line. The 'forces' picture: pull p toward 1 at y."""
    probs = F.softmax(logits, dim=-1)
    onehot = F.one_hot(targets, logits.shape[-1]).to(logits.dtype)
    return (probs - onehot) / logits.shape[0]


# --- 4. tanh.  h = tanh(z),  dh/dz = 1 - h^2. ---
Z_TANH = torch.tensor([1.2, -0.5])
H_TANH = torch.tanh(Z_TANH)
dH_TANH = torch.tensor([1.0, 1.0])
dZ_TANH = (1 - H_TANH ** 2) * dH_TANH


# --- 5. BatchNorm backward, written in the *correct* order. ---
# Forward, one channel, N=4 (the same four numbers as chapter 04):
#   mu, var over the batch; xhat = (x-mu)/sqrt(var+eps); y = gamma*xhat + beta
BN_X = torch.tensor([1.0, 3.0, 5.0, 7.0])
BN_GAMMA = torch.tensor(2.0)
BN_BETA = torch.tensor(-1.0)
BN_EPS = 1e-5
BN_N = BN_X.numel()
BN_MU = BN_X.mean()
BN_VAR = BN_X.var(unbiased=False)
BN_INV = 1.0 / torch.sqrt(BN_VAR + BN_EPS)
BN_DIFF = BN_X - BN_MU                  # "bndiff"
BN_RAW = BN_DIFF * BN_INV               # "bnraw" = xhat
BN_Y = BN_GAMMA * BN_RAW + BN_BETA

# Upstream grad from the next layer (say tanh). Chosen so the numbers move.
dBN_Y = torch.tensor([0.2, -0.1, 0.4, -0.3])


def bn_backward_steps(x, gamma, beta, eps, dy):
    """Step-by-step BN backward. Order matters. Do not swap bndiff and bnraw."""
    n = x.numel()
    mu = x.mean()
    var = x.var(unbiased=False)
    inv = 1.0 / torch.sqrt(var + eps)
    diff = x - mu
    raw = diff * inv                    # xhat
    # y = gamma * raw + beta
    d_raw = gamma * dy                  # NOT inv * dy  (that was the old bug)
    d_gamma = (raw * dy).sum()
    d_beta = dy.sum()
    # raw = diff * inv
    d_diff = inv * d_raw
    d_inv = (diff * d_raw).sum()        # NOT (raw * dy).sum()
    # inv = (var + eps)^(-1/2)
    d_var = d_inv * (-0.5) * inv ** 3
    # mu is used in `diff` and inside `var`. Both paths, so += .
    d_mu = -d_diff.sum() + d_var * (-2.0 * diff.sum() / n)
    # x is used in `diff` and inside `var`. Both paths, so += .
    dx = d_diff + d_var * (2.0 * diff / n) + d_mu / n
    return {
        "d_raw": d_raw, "d_gamma": d_gamma, "d_beta": d_beta,
        "d_diff": d_diff, "d_inv": d_inv, "d_var": d_var, "d_mu": d_mu, "dx": dx,
        "raw": raw, "inv": inv, "mu": mu, "var": var,
    }


def bn_backward_fused(x, gamma, eps, dy):
    """The compact form, one channel:

    dx = (gamma / (N * sigma)) * (N*dy - sum(dy) - xhat * sum(dy * xhat))
    """
    n = x.numel()
    mu = x.mean()
    var = x.var(unbiased=False)
    sigma = torch.sqrt(var + eps)
    xhat = (x - mu) / sigma
    return (gamma / (n * sigma)) * (
        n * dy - dy.sum() - xhat * (dy * xhat).sum()
    )


BN_STEPS = bn_backward_steps(BN_X, BN_GAMMA, BN_BETA, BN_EPS, dBN_Y)
BN_FUSED_DX = bn_backward_fused(BN_X, BN_GAMMA, BN_EPS, dBN_Y)


def bn_backward_buggy(x, gamma, eps, dy):
    """The two swapped lines from the old lesson. Kept only so the check
    can prove they do *not* match PyTorch."""
    n = x.numel()
    mu = x.mean()
    var = x.var(unbiased=False)
    inv = 1.0 / torch.sqrt(var + eps)
    diff = x - mu
    raw = diff * inv
    # BUG: treats d_hpreact as if it were already d_raw, and multiplies
    # by inv a second time. Then uses raw where diff belongs.
    d_raw = inv * dy
    d_inv = (raw * dy).sum()
    d_var = d_inv * (-0.5) * inv ** 3
    d_diff = inv * d_raw
    d_mu = -d_diff.sum() + d_var * (-2.0 * diff.sum() / n)
    dx = d_diff + d_var * (2.0 * diff / n) + d_mu / n
    return dx


# --- 6. Embedding scatter-add.  Two uses of row 0 must add. ---
EMB_C = torch.tensor([[0.1, 0.2],
                      [0.3, 0.4],
                      [0.5, 0.6]])      # (3, 2)
EMB_IDS = torch.tensor([0, 2, 0])
EMB_OUT = EMB_C[EMB_IDS]                # (3, 2)
dEMB_OUT = torch.tensor([[1.0, 0.0],
                         [0.0, 1.0],
                         [1.0, 1.0]])
dEMB_C = torch.zeros_like(EMB_C)
dEMB_C.index_add_(0, EMB_IDS, dEMB_OUT)  # row 0 gets [1,0] + [1,1] = [2, 1]


def torch_bn_dx(x, gamma, beta, eps, dy):
    """Autograd reference for the BN input gradient."""
    x = x.detach().requires_grad_(True)
    y = F.batch_norm(
        x.view(-1, 1),
        running_mean=None, running_var=None,
        weight=gamma.view(1), bias=beta.view(1),
        training=True, eps=eps,
    ).view(-1)
    y.backward(dy)
    return x.grad

Mistakes learners make

Forgetting += on a reused tensor. \(\mu\) in BatchNorm, an embedding row, a weight used in two heads. Two paths, one parameter, add the grads.

Swapping bndiff and bnraw. They are the same vector only when \(\sigma=1\). The backward of a product needs the other factor: \(\mathrm{inv}\) with \(\mathrm{bndiff}\), not with \(\hat x\).

Dropping the \(1/B\). F.cross_entropy is a mean. The fused gradient divides by \(B\). A sum-reduction would not. Check the reduction before you compare to autograd.

Clipping the loss instead of the gradient. loss.clamp changes the objective. clip_grad_norm_ changes the step. They are not interchangeable.

What to remember

  • Matmul: dA = dC @ B.T, dB = A.T @ dC. Shapes pick the transpose.
  • Broadcast forward ⇔ sum backward.
  • \(\partial\mathcal{L}/\partial z = (P - e_y)/B\). Rows of that gradient sum to 0.
  • BN backward has two paths through \(\mu\) and \(\sigma^2\). Do not swap \(\mathrm{bndiff}\) and \(\hat x\).
  • Embedding backward is scatter-add. A row used twice adds twice.

Check yourself

1. \(A=[1,2]\), \(B=[3,4]^\top\), \(\partial L/\partial C=1\). What is \(\partial L/\partial A\)?

\([3, 4]\). Each entry of \(A\) is multiplied by the matching entry of \(B\), and \(C=11\).

2. Why does a bias gradient sum over the batch?

The forward pass copied the bias to every row. Each copy is a path back to the same vector, so the incoming grads add.

3. Softmax probabilities \((0.786, 0.175, 0.039)\), target 0, batch 2. What is the first entry of \(\partial\mathcal{L}/\partial z\)?

\((0.786 - 1)/2 \approx -0.107\).

4. In BN, what is \(\partial L/\partial\beta\) for incoming \((0.2, -0.1, 0.4, -0.3)\)?

The sum, \(0.2\). \(\beta\) is added after the normalize, so its grad is just \(\sum \partial L/\partial y\).

5. ids \(=(0,2,0)\), incoming row-grads \((1,0)\), \((0,1)\), \((1,1)\). What lands on row 0 of \(C\)?

\([2, 1]\). Positions 0 and 2 both pointed at row 0.

Go to the source

Keep going