NEURONS ALL THE WAY DOWN · CHAPTER 02
A neural net is a big algebraic expression. Backprop is the chain rule run backwards over that expression. A hundred lines of Python are enough to train one.
+ routes a gradient and * swaps the parents+=, not =__add__ / __mul__Imagine a bakery that prices a cake by a chain of small steps. Flour costs \(a\). Eggs cost \(b\). The batter is \(a \times b\). Icing is \(c\). The cake is batter plus icing. A discount \(d\) multiplies the total.
If the final price should go down, who gets the complaint? The bakery does not re-bake a thousand cakes with slightly different flour prices. It keeps the receipt. Each line says “I was made from these two things, by this operation.” Walking the receipt backwards, with the chain rule, tells every ingredient how much it pushed the total.
That receipt is a computational graph. Walking it backwards is reverse-mode automatic differentiation — autograd, backprop, same idea. Deep learning is this idea at scale. This chapter builds it for scalars, then wraps scalars into a 2-layer MLP.
Forward: compute the number and remember the recipe. Backward: the final output seeds gradient 1, and each node multiplies that incoming gradient by its own local derivative. You never re-run the whole expression from scratch for each input.
Take \(f(x) = x^3 - 3x\) at \(x = 2\). The value is \(8 - 6 = 2\). The analytic derivative is \(3x^2 - 3 = 12 - 3 = \mathbf{9}\).
You can also poke the input and watch the output. The symmetric finite difference is
nudge x a tiny step both ways and divide the change in f by the distance you travelled.
With \(h = 10^{-4}\) this returns 9.00000001. Good enough to check a backward pass. Push \(h\) to \(10^{-16}\) and the subtraction cancels every useful digit. Finite differences are a test, not an engine. An engine wants the analytic local derivative at every node.
Interactive · Finite differences
For \(f(x)=x^3-3x\), when does the numerical derivative stop matching 9?
Now a graph you can label by hand. Let \(a=4\), \(b=-2\), \(c=3\), \(d=0.5\), and
multiply a and b, add c, then multiply by d. L is the “loss” we differentiate.
Forward: \(a\cdot b = -8\), \(p = -8 + 3 = -5\), \(L = -5 \times 0.5 = \mathbf{-2.5}\).
Backward starts at \(L\) with \(\partial L / \partial L = 1\). The last node is a product, so each parent gets the other parent:
Two local rules take you all the way to the leaves:
\(\partial(u+v)/\partial u = 1\). The upstream gradient is copied to both parents. Addition does not rescale, it only delivers.
\(\partial(u\cdot v)/\partial u = v\). Each parent is multiplied by the other parent’s data, then by the upstream gradient.
p = a·b+c → L · ∂L = 1 · × swaps · + copies · ∂a=−1 · ∂b=2
Every fancier op is the same pattern. \(\tanh\) has local derivative \(1 - \tanh^2\). A power \(x^k\) has local derivative \(k x^{k-1}\). You cache the forward value so the backward step is a multiply, not a re-compute of the whole expression.
A Value stores the forward number, a grad that starts at 0, the parents that produced it, and a closure _backward that knows the local rule. Operators return new Values. backward() topological-sorts the graph (children before parent), seeds the end node with gradient 1, and walks the list in reverse.
"""Tiny scalar autograd, then a 2-layer MLP.
A neural net is a big algebraic expression. Each number remembers how it
was made. Backward walks that graph in reverse and applies the chain
rule. About a hundred lines is enough to train a small MLP.
Every worked number on the chapter page is computed here so the check
script can import it instead of re-deriving it.
"""
import math
import random
class Value:
"""One scalar in a computational graph.
data — the forward number
grad — d(final loss) / d(this value), filled by backward()
_prev — the Values that produced this one
_op — a label for drawing / debugging ('+', '*', 'tanh', ...)
"""
def __init__(self, data, _children=(), _op=""):
self.data = float(data)
self.grad = 0.0
# children of the *forward* arrow. Backward visits them after us.
self._prev = tuple(_children)
self._op = _op
# Local chain-rule step. Assigned by the operator that created us.
self._backward = lambda: None
def __repr__(self):
return f"Value({self.data:.4f}, grad={self.grad:.4f})"
def __add__(self, other):
other = other if isinstance(other, Value) else Value(other)
out = Value(self.data + other.data, (self, other), "+")
def _backward():
# Addition routes the same upstream grad to both parents.
# += is required: a Value can be used more than once.
self.grad += out.grad
other.grad += out.grad
out._backward = _backward
return out
def __radd__(self, other):
return self + other
def __mul__(self, other):
other = other if isinstance(other, Value) else Value(other)
out = Value(self.data * other.data, (self, other), "*")
def _backward():
# Product: each parent gets the other parent times upstream.
self.grad += other.data * out.grad
other.grad += self.data * out.grad
out._backward = _backward
return out
def __rmul__(self, other):
return self * other
def __neg__(self):
return self * -1
def __sub__(self, other):
return self + (-other)
def __rsub__(self, other):
return Value(other) + (-self)
def __pow__(self, exp):
# Exponent is a Python float, not a Value. Enough for squares.
out = Value(self.data ** exp, (self,), f"**{exp}")
def _backward():
self.grad += exp * (self.data ** (exp - 1)) * out.grad
out._backward = _backward
return out
def tanh(self):
t = math.tanh(self.data)
out = Value(t, (self,), "tanh")
def _backward():
# d/dx tanh(x) = 1 - tanh(x)^2. Use the cached forward value.
self.grad += (1 - t ** 2) * out.grad
out._backward = _backward
return out
def backward(self):
"""Reverse-mode AD: topo-sort, seed grad=1, walk backwards."""
topo = []
seen = set()
def build(v):
if v not in seen:
seen.add(v)
for child in v._prev:
build(child)
topo.append(v) # post-order: children before parent
build(self)
self.grad = 1.0
for v in reversed(topo):
v._backward()
class Neuron:
"""Weighted sum plus bias, then tanh. The smallest useful unit."""
def __init__(self, n_in, weights, bias):
# weights / bias are passed in so the page can use fixed numbers.
self.w = [Value(w) for w in weights]
self.b = Value(bias)
def __call__(self, x):
# sum(..., start=bias) is the affine layer. tanh is the squash.
act = sum((wi * xi for wi, xi in zip(self.w, x)), self.b)
return act.tanh()
def parameters(self):
return self.w + [self.b]
class Layer:
def __init__(self, n_in, specs):
# specs is a list of (weights, bias) one per neuron.
self.neurons = [Neuron(n_in, w, b) for w, b in specs]
def __call__(self, x):
out = [n(x) for n in self.neurons]
return out[0] if len(out) == 1 else out
def parameters(self):
return [p for n in self.neurons for p in n.parameters()]
class MLP:
"""A stack of layers. MLP(2, ...) means two inputs."""
def __init__(self, layers):
self.layers = layers
def __call__(self, x):
for layer in self.layers:
x = layer(x)
# A 1-neuron layer returns a scalar Value. Wrap it so the
# next layer can still zip over "inputs".
if isinstance(x, Value):
x = [x]
return x[0] if len(x) == 1 else x
def parameters(self):
return [p for layer in self.layers for p in layer.parameters()]
def finite_diff(f, x, h=1e-4):
"""Symmetric difference (f(x+h) - f(x-h)) / (2h)."""
return (f(x + h) - f(x - h)) / (2 * h)
# --- Worked example 1: a cubic you can differentiate by hand. ---
# f(x) = x^3 - 3x. At x = 2: f = 2, f' = 9.
def cubic(x):
return x ** 3 - 3 * x
CUBIC_X = 2.0
CUBIC_F = cubic(CUBIC_X) # 8 - 6 = 2
CUBIC_ANALYTIC = 3 * CUBIC_X ** 2 - 3 # 12 - 3 = 9
CUBIC_NUMERICAL = finite_diff(cubic, CUBIC_X)
# --- Worked example 2: a 4-node graph. ---
# p = a*b + c, L = p*d
# a=4, b=-2, c=3, d=0.5 → p=-5, L=-2.5
# grads: a=-1, b=2, c=0.5, d=-5
def worked_graph():
a, b, c, d = Value(4), Value(-2), Value(3), Value(0.5)
p = a * b + c
L = p * d
L.backward()
return a, b, c, d, p, L
A, B, C, D, P, L = worked_graph()
# --- Worked example 3: one neuron. ---
# n = 0.5*0.8 + (-1)*(-0.6) + 0.2 = 1.2
# o = tanh(1.2)
NEURON_X = [0.5, -1.0]
NEURON_W = [0.8, -0.6]
NEURON_BIAS = 0.2
NEURON_PRE = NEURON_X[0] * NEURON_W[0] + NEURON_X[1] * NEURON_W[1] + NEURON_BIAS
NEURON_OUT = math.tanh(NEURON_PRE)
NEURON_LOCAL = 1 - NEURON_OUT ** 2 # do/dn
def worked_neuron():
x = [Value(v) for v in NEURON_X]
n = Neuron(2, NEURON_W, NEURON_BIAS)
o = n(x)
o.backward()
return n, o
NEURON, NEURON_O = worked_neuron()
# --- Worked example 4: the += bug. L = a + a must give a.grad = 2. ---
def double_use(a_data=3.0):
a = Value(a_data)
out = a + a
out.backward()
return a, out
DOUBLE_A, DOUBLE_OUT = double_use()
# --- Worked example 5: a 2-layer MLP (2 → 2 → 1) on two points. ---
def make_toy_mlp():
# Hidden layer: two tanh neurons. Output: one tanh neuron.
hidden = Layer(2, [
([0.4, -0.3], 0.1),
([-0.2, 0.5], -0.1),
])
output = Layer(2, [([0.6, -0.4], 0.0)])
return MLP([hidden, output])
def mse(pred, target):
return (pred - target) ** 2
TOY_DATA = [([1.0, -1.0], 1.0), ([-1.0, 1.0], -1.0)]
def train_step(model, lr=0.1):
"""One full-batch step. Returns the mean square loss *before* the step."""
loss = sum(mse(model(x), y) for x, y in TOY_DATA)
# Mean over the two points. sum of Values starts at 0 (int), so wrap.
if not isinstance(loss, Value):
loss = Value(loss) # pragma: no cover — sum of two Values is a Value
loss = loss * (1 / len(TOY_DATA))
for p in model.parameters():
p.grad = 0.0
loss.backward()
for p in model.parameters():
p.data -= lr * p.grad
return loss.data
def train(steps=20, lr=0.1, seed=0):
"""Run a few steps. Seed is unused (weights are fixed) but kept for API."""
random.seed(seed)
model = make_toy_mlp()
losses = [train_step(model, lr=lr) for _ in range(steps)]
return model, losses
TOY_MODEL, TOY_LOSSES = train()
TOY_LOSS_START = TOY_LOSSES[0]
TOY_LOSS_END = TOY_LOSSES[-1]
+= on grad — a node used twice must accumulate, not overwrite. See the pitfall below.build is a DFS post-order. reversed(topo) is then a valid backward order.Value objects, nested.PyTorch does this with tensors instead of scalars, and fuses the walk into C++ / CUDA. The mapping is word-for-word: data is the tensor, grad is .grad, backward() is .backward(), and you still have to zero gradients between steps.
A neuron is \(o = \tanh(w \cdot x + b)\). Use two inputs so the arithmetic stays on one line: \(x = (0.5,\,-1.0)\), \(w = (0.8,\,-0.6)\), \(b = 0.2\).
the pre-activation is 1.2. tanh squishes it into (−1, 1). About 0.834.
The local derivative of \(\tanh\) is \(1 - o^2 \approx \mathbf{0.305}\). That single number fans out:
n = 1.2 → o ≈ 0.834 → 1−o² ≈ 0.305
| Parameter | Local rule | Gradient |
|---|---|---|
| w1 = 0.8 | \(\times x_1\) | 0.305 × 0.5 = 0.153 |
| w2 = −0.6 | \(\times x_2\) | 0.305 × (−1) = −0.305 |
| b = 0.2 | \(\times 1\) | 0.305 |
If \(x_1\) had been 0, \(w_1\) would get gradient 0. A weight on a dead input does not learn. That fact will matter when we look at saturated \(\tanh\) and dead ReLUs in chapter 04.
The backward of a neuron is three multiplies. The backward of a deep net is the same three multiplies, once per neuron, in reverse order. Depth is bookkeeping, not new math.
A layer is a list of neurons that share an input. An MLP is a list of layers. Ours is \(2 \rightarrow 2 \rightarrow 1\): two hidden tanh units, one tanh output. Nine parameters. The data is two labelled points, \((1,-1)\mapsto 1\) and \((-1,1)\mapsto -1\). Loss is mean squared error.
One training step is four lines: forward, zero_grad, backward, p.data -= lr * p.grad. With learning rate 0.1 the loss starts at 0.217 and after 20 steps is 0.024. The same loop, on tensors, is how every later chapter trains.
Two stacked linear maps collapse. \(W_2(W_1 x) = (W_2 W_1)x\), which is one linear map. A nonlinearity between them makes the composition actually deeper. That is why every layer here ends in \(\tanh\). Chapter 04 looks at which nonlinearity, and what happens when it saturates.
Assigning to grad instead of adding. If \(L = a + a\) and the + backward does self.grad = out.grad, the second write overwrites the first and \(a\) gets 1 instead of 2. The multivariate chain rule says: sum every path. Always +=.
L = a+a · grad=1 wrong · grad=2 right
Forgetting to zero gradients. The next backward will add to the previous one. The step is then too large, in the right direction by accident. It can look like training “still works.” Zero every step.
Calling _backward instead of assigning it. out._backward = _backward stores the closure. out._backward = _backward() runs it now, while out.grad is still 0, and stores None.
Using finite differences as the optimizer. One check against a numerical gradient is healthy. Doing it for every parameter of a real net is one forward per parameter. Reverse mode is one forward and one backward, for all parameters at once.
+=. Topological sort plus that rule is the whole algorithm.\(L = p \cdot d\) so \(\partial L/\partial p = 0.5\). \(p = a\cdot b + c\) so the product gives \(\partial L/\partial a = b \cdot 0.5 = -2 \cdot 0.5 = -1\).
\(1 - \tanh(1.2)^2 \approx 1 - 0.834^2 \approx 0.305\).
grad instead of adding, what do you get?\(a.\mathrm{grad} = 1\) instead of 2. The second write overwrites the first. The two paths through \(L\) were not summed.
Finite differences need a forward pass per parameter. Reverse mode needs one forward and one backward, and produces every parameter’s gradient.
They collapse to one linear map, \(W_2 W_1\). Depth does nothing until you put a squash in the middle.