NEURONS ALL THE WAY DOWN · CHAPTER 02

Autograd from Scratch

A neural net is a big algebraic expression. Backprop is the chain rule run backwards over that expression. A hundred lines of Python are enough to train one.

~20 min read1 interactiveTested Python + PyTorch

What you'll learn

  • How a scalar computational graph stores the forward recipe
  • How reverse-mode autograd applies the chain rule once per node
  • Why + routes a gradient and * swaps the parents
  • Why a Value used twice needs +=, not =
  • How Neuron → Layer → MLP is just nested applications of the same object

Before you start

  • High-school derivatives: power rule, product rule, chain rule
  • Python classes and __add__ / __mul__
  • What is a language model? — optional. This chapter does not need it.

A receipt for every number

Imagine a bakery that prices a cake by a chain of small steps. Flour costs \(a\). Eggs cost \(b\). The batter is \(a \times b\). Icing is \(c\). The cake is batter plus icing. A discount \(d\) multiplies the total.

If the final price should go down, who gets the complaint? The bakery does not re-bake a thousand cakes with slightly different flour prices. It keeps the receipt. Each line says “I was made from these two things, by this operation.” Walking the receipt backwards, with the chain rule, tells every ingredient how much it pushed the total.

That receipt is a computational graph. Walking it backwards is reverse-mode automatic differentiation — autograd, backprop, same idea. Deep learning is this idea at scale. This chapter builds it for scalars, then wraps scalars into a 2-layer MLP.

Forward: compute the number and remember the recipe. Backward: the final output seeds gradient 1, and each node multiplies that incoming gradient by its own local derivative. You never re-run the whole expression from scratch for each input.

Rise over run, then stop doing it

Take \(f(x) = x^3 - 3x\) at \(x = 2\). The value is \(8 - 6 = 2\). The analytic derivative is \(3x^2 - 3 = 12 - 3 = \mathbf{9}\).

You can also poke the input and watch the output. The symmetric finite difference is

nudge x a tiny step both ways and divide the change in f by the distance you travelled.

With \(h = 10^{-4}\) this returns 9.00000001. Good enough to check a backward pass. Push \(h\) to \(10^{-16}\) and the subtraction cancels every useful digit. Finite differences are a test, not an engine. An engine wants the analytic local derivative at every node.

Interactive · Finite differences

For \(f(x)=x^3-3x\), when does the numerical derivative stop matching 9?

Plus routes, times swaps

Now a graph you can label by hand. Let \(a=4\), \(b=-2\), \(c=3\), \(d=0.5\), and

multiply a and b, add c, then multiply by d. L is the “loss” we differentiate.

Forward: \(a\cdot b = -8\), \(p = -8 + 3 = -5\), \(L = -5 \times 0.5 = \mathbf{-2.5}\).

Backward starts at \(L\) with \(\partial L / \partial L = 1\). The last node is a product, so each parent gets the other parent:

  1. \(\partial L / \partial p = d = 0.5\), and \(\partial L / \partial d = p = -5\).
  2. \(p = u + c\) with \(u = a\cdot b\). Addition copies the incoming 0.5 to both \(u\) and \(c\). So \(\partial L / \partial c = 0.5\).
  3. \(u = a\cdot b\). Product swaps: \(\partial L / \partial a = b \cdot 0.5 = \mathbf{-1}\), \(\partial L / \partial b = a \cdot 0.5 = \mathbf{2}\).

Two local rules take you all the way to the leaves:

Addition routes

\(\partial(u+v)/\partial u = 1\). The upstream gradient is copied to both parents. Addition does not rescale, it only delivers.

Multiplication swaps

\(\partial(u\cdot v)/\partial u = v\). Each parent is multiplied by the other parent’s data, then by the upstream gradient.

p = a·b+c → L  ·  ∂L = 1  ·  × swaps  ·  + copies  ·  ∂a=−1 · ∂b=2

MANIM The receipt walks backward: × swaps, + copies. Use the beat buttons to jump.

Every fancier op is the same pattern. \(\tanh\) has local derivative \(1 - \tanh^2\). A power \(x^k\) has local derivative \(k x^{k-1}\). You cache the forward value so the backward step is a multiply, not a re-compute of the whole expression.

One object that remembers its recipe

A Value stores the forward number, a grad that starts at 0, the parents that produced it, and a closure _backward that knows the local rule. Operators return new Values. backward() topological-sorts the graph (children before parent), seeds the end node with gradient 1, and walks the list in reverse.

autograd.pygraph grads match PyTorch; a+a gives grad 2
"""Tiny scalar autograd, then a 2-layer MLP.

A neural net is a big algebraic expression. Each number remembers how it
was made. Backward walks that graph in reverse and applies the chain
rule. About a hundred lines is enough to train a small MLP.

Every worked number on the chapter page is computed here so the check
script can import it instead of re-deriving it.
"""
import math
import random


class Value:
    """One scalar in a computational graph.

    data   — the forward number
    grad   — d(final loss) / d(this value), filled by backward()
    _prev  — the Values that produced this one
    _op    — a label for drawing / debugging ('+', '*', 'tanh', ...)
    """

    def __init__(self, data, _children=(), _op=""):
        self.data = float(data)
        self.grad = 0.0
        # children of the *forward* arrow. Backward visits them after us.
        self._prev = tuple(_children)
        self._op = _op
        # Local chain-rule step. Assigned by the operator that created us.
        self._backward = lambda: None

    def __repr__(self):
        return f"Value({self.data:.4f}, grad={self.grad:.4f})"

    def __add__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data + other.data, (self, other), "+")

        def _backward():
            # Addition routes the same upstream grad to both parents.
            # += is required: a Value can be used more than once.
            self.grad += out.grad
            other.grad += out.grad

        out._backward = _backward
        return out

    def __radd__(self, other):
        return self + other

    def __mul__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data * other.data, (self, other), "*")

        def _backward():
            # Product: each parent gets the other parent times upstream.
            self.grad += other.data * out.grad
            other.grad += self.data * out.grad

        out._backward = _backward
        return out

    def __rmul__(self, other):
        return self * other

    def __neg__(self):
        return self * -1

    def __sub__(self, other):
        return self + (-other)

    def __rsub__(self, other):
        return Value(other) + (-self)

    def __pow__(self, exp):
        # Exponent is a Python float, not a Value. Enough for squares.
        out = Value(self.data ** exp, (self,), f"**{exp}")

        def _backward():
            self.grad += exp * (self.data ** (exp - 1)) * out.grad

        out._backward = _backward
        return out

    def tanh(self):
        t = math.tanh(self.data)
        out = Value(t, (self,), "tanh")

        def _backward():
            # d/dx tanh(x) = 1 - tanh(x)^2. Use the cached forward value.
            self.grad += (1 - t ** 2) * out.grad

        out._backward = _backward
        return out

    def backward(self):
        """Reverse-mode AD: topo-sort, seed grad=1, walk backwards."""
        topo = []
        seen = set()

        def build(v):
            if v not in seen:
                seen.add(v)
                for child in v._prev:
                    build(child)
                topo.append(v)  # post-order: children before parent

        build(self)
        self.grad = 1.0
        for v in reversed(topo):
            v._backward()


class Neuron:
    """Weighted sum plus bias, then tanh. The smallest useful unit."""

    def __init__(self, n_in, weights, bias):
        # weights / bias are passed in so the page can use fixed numbers.
        self.w = [Value(w) for w in weights]
        self.b = Value(bias)

    def __call__(self, x):
        # sum(..., start=bias) is the affine layer. tanh is the squash.
        act = sum((wi * xi for wi, xi in zip(self.w, x)), self.b)
        return act.tanh()

    def parameters(self):
        return self.w + [self.b]


class Layer:
    def __init__(self, n_in, specs):
        # specs is a list of (weights, bias) one per neuron.
        self.neurons = [Neuron(n_in, w, b) for w, b in specs]

    def __call__(self, x):
        out = [n(x) for n in self.neurons]
        return out[0] if len(out) == 1 else out

    def parameters(self):
        return [p for n in self.neurons for p in n.parameters()]


class MLP:
    """A stack of layers. MLP(2, ...) means two inputs."""

    def __init__(self, layers):
        self.layers = layers

    def __call__(self, x):
        for layer in self.layers:
            x = layer(x)
            # A 1-neuron layer returns a scalar Value. Wrap it so the
            # next layer can still zip over "inputs".
            if isinstance(x, Value):
                x = [x]
        return x[0] if len(x) == 1 else x

    def parameters(self):
        return [p for layer in self.layers for p in layer.parameters()]


def finite_diff(f, x, h=1e-4):
    """Symmetric difference (f(x+h) - f(x-h)) / (2h)."""
    return (f(x + h) - f(x - h)) / (2 * h)


# --- Worked example 1: a cubic you can differentiate by hand. ---
# f(x) = x^3 - 3x.  At x = 2: f = 2, f' = 9.

def cubic(x):
    return x ** 3 - 3 * x


CUBIC_X = 2.0
CUBIC_F = cubic(CUBIC_X)                 # 8 - 6 = 2
CUBIC_ANALYTIC = 3 * CUBIC_X ** 2 - 3    # 12 - 3 = 9
CUBIC_NUMERICAL = finite_diff(cubic, CUBIC_X)


# --- Worked example 2: a 4-node graph. ---
# p = a*b + c,  L = p*d
# a=4, b=-2, c=3, d=0.5  →  p=-5, L=-2.5
# grads: a=-1, b=2, c=0.5, d=-5

def worked_graph():
    a, b, c, d = Value(4), Value(-2), Value(3), Value(0.5)
    p = a * b + c
    L = p * d
    L.backward()
    return a, b, c, d, p, L


A, B, C, D, P, L = worked_graph()


# --- Worked example 3: one neuron. ---
# n = 0.5*0.8 + (-1)*(-0.6) + 0.2 = 1.2
# o = tanh(1.2)

NEURON_X = [0.5, -1.0]
NEURON_W = [0.8, -0.6]
NEURON_BIAS = 0.2
NEURON_PRE = NEURON_X[0] * NEURON_W[0] + NEURON_X[1] * NEURON_W[1] + NEURON_BIAS
NEURON_OUT = math.tanh(NEURON_PRE)
NEURON_LOCAL = 1 - NEURON_OUT ** 2  # do/dn


def worked_neuron():
    x = [Value(v) for v in NEURON_X]
    n = Neuron(2, NEURON_W, NEURON_BIAS)
    o = n(x)
    o.backward()
    return n, o


NEURON, NEURON_O = worked_neuron()


# --- Worked example 4: the += bug. L = a + a must give a.grad = 2. ---

def double_use(a_data=3.0):
    a = Value(a_data)
    out = a + a
    out.backward()
    return a, out


DOUBLE_A, DOUBLE_OUT = double_use()


# --- Worked example 5: a 2-layer MLP (2 → 2 → 1) on two points. ---

def make_toy_mlp():
    # Hidden layer: two tanh neurons. Output: one tanh neuron.
    hidden = Layer(2, [
        ([0.4, -0.3], 0.1),
        ([-0.2, 0.5], -0.1),
    ])
    output = Layer(2, [([0.6, -0.4], 0.0)])
    return MLP([hidden, output])


def mse(pred, target):
    return (pred - target) ** 2


TOY_DATA = [([1.0, -1.0], 1.0), ([-1.0, 1.0], -1.0)]


def train_step(model, lr=0.1):
    """One full-batch step. Returns the mean square loss *before* the step."""
    loss = sum(mse(model(x), y) for x, y in TOY_DATA)
    # Mean over the two points. sum of Values starts at 0 (int), so wrap.
    if not isinstance(loss, Value):
        loss = Value(loss)  # pragma: no cover — sum of two Values is a Value
    loss = loss * (1 / len(TOY_DATA))
    for p in model.parameters():
        p.grad = 0.0
    loss.backward()
    for p in model.parameters():
        p.data -= lr * p.grad
    return loss.data


def train(steps=20, lr=0.1, seed=0):
    """Run a few steps. Seed is unused (weights are fixed) but kept for API."""
    random.seed(seed)
    model = make_toy_mlp()
    losses = [train_step(model, lr=lr) for _ in range(steps)]
    return model, losses


TOY_MODEL, TOY_LOSSES = train()
TOY_LOSS_START = TOY_LOSSES[0]
TOY_LOSS_END = TOY_LOSSES[-1]
  • += on grad — a node used twice must accumulate, not overwrite. See the pitfall below.
  • build is a DFS post-order. reversed(topo) is then a valid backward order.
  • The MLP at the bottom of the file is the same Value objects, nested.

PyTorch does this with tensors instead of scalars, and fuses the walk into C++ / CUDA. The mapping is word-for-word: data is the tensor, grad is .grad, backward() is .backward(), and you still have to zero gradients between steps.

A weighted sum, then a squash

A neuron is \(o = \tanh(w \cdot x + b)\). Use two inputs so the arithmetic stays on one line: \(x = (0.5,\,-1.0)\), \(w = (0.8,\,-0.6)\), \(b = 0.2\).

the pre-activation is 1.2. tanh squishes it into (−1, 1). About 0.834.

The local derivative of \(\tanh\) is \(1 - o^2 \approx \mathbf{0.305}\). That single number fans out:

n = 1.2  →  o ≈ 0.834  →  1−o² ≈ 0.305

MANIM Local grad of tanh is 1−tanh² on the curve. Use the beat buttons to jump.
ParameterLocal ruleGradient
w1 = 0.8\(\times x_1\)0.305 × 0.5 = 0.153
w2 = −0.6\(\times x_2\)0.305 × (−1) = −0.305
b = 0.2\(\times 1\)0.305

If \(x_1\) had been 0, \(w_1\) would get gradient 0. A weight on a dead input does not learn. That fact will matter when we look at saturated \(\tanh\) and dead ReLUs in chapter 04.

The backward of a neuron is three multiplies. The backward of a deep net is the same three multiplies, once per neuron, in reverse order. Depth is bookkeeping, not new math.

Stack neurons, train on two points

A layer is a list of neurons that share an input. An MLP is a list of layers. Ours is \(2 \rightarrow 2 \rightarrow 1\): two hidden tanh units, one tanh output. Nine parameters. The data is two labelled points, \((1,-1)\mapsto 1\) and \((-1,1)\mapsto -1\). Loss is mean squared error.

x ∈ ℝ²two numbers in
hidden, 2 × tanh6 parameters
output tanh3 parameters
(pred − y)²mean over 2 points

One training step is four lines: forward, zero_grad, backward, p.data -= lr * p.grad. With learning rate 0.1 the loss starts at 0.217 and after 20 steps is 0.024. The same loop, on tensors, is how every later chapter trains.

Why the tanh is not optional

Two stacked linear maps collapse. \(W_2(W_1 x) = (W_2 W_1)x\), which is one linear map. A nonlinearity between them makes the composition actually deeper. That is why every layer here ends in \(\tanh\). Chapter 04 looks at which nonlinearity, and what happens when it saturates.

Mistakes learners make

Assigning to grad instead of adding. If \(L = a + a\) and the + backward does self.grad = out.grad, the second write overwrites the first and \(a\) gets 1 instead of 2. The multivariate chain rule says: sum every path. Always +=.

L = a+a  ·  grad=1 wrong  ·  grad=2 right

MANIM a+a must +=, not overwrite. Use the beat buttons to jump.

Forgetting to zero gradients. The next backward will add to the previous one. The step is then too large, in the right direction by accident. It can look like training “still works.” Zero every step.

Calling _backward instead of assigning it. out._backward = _backward stores the closure. out._backward = _backward() runs it now, while out.grad is still 0, and stores None.

Using finite differences as the optimizer. One check against a numerical gradient is healthy. Doing it for every parameter of a real net is one forward per parameter. Reverse mode is one forward and one backward, for all parameters at once.

What to remember

  • Forward builds a graph of Values. Backward is the chain rule in reverse, one local multiply per node.
  • Addition copies the upstream gradient. Multiplication multiplies by the other parent.
  • A Value used twice needs +=. Topological sort plus that rule is the whole algorithm.
  • Neuron / Layer / MLP are nested loops over the same object. Training is forward → zero → backward → step.

Check yourself

1. In the four-node graph, why is \(\partial L/\partial a = -1\)?

\(L = p \cdot d\) so \(\partial L/\partial p = 0.5\). \(p = a\cdot b + c\) so the product gives \(\partial L/\partial a = b \cdot 0.5 = -2 \cdot 0.5 = -1\).

2. What is \(\partial \tanh(n) / \partial n\) at \(n = 1.2\)?

\(1 - \tanh(1.2)^2 \approx 1 - 0.834^2 \approx 0.305\).

3. If \(L = a + a\) and you assign to grad instead of adding, what do you get?

\(a.\mathrm{grad} = 1\) instead of 2. The second write overwrites the first. The two paths through \(L\) were not summed.

4. Why is reverse mode cheaper than poking every input?

Finite differences need a forward pass per parameter. Reverse mode needs one forward and one backward, and produces every parameter’s gradient.

5. What happens if two linear layers have no nonlinearity between them?

They collapse to one linear map, \(W_2 W_1\). Depth does nothing until you put a squash in the middle.

Go to the source

Keep going