NEURONS ALL THE WAY DOWN · CHAPTER 06

Optimizers

You have the gradient. The optimizer decides how to walk. Four ways to take a step, then a learning-rate schedule that almost every modern run actually uses.

~16 min read1 interactiveTested against torch.optim

What you'll learn

  • SGD on a 1-D bowl, in numbers you can do by hand
  • How momentum keeps a velocity and why it overshoots
  • Adam’s two moments and the bias-correction that makes step 1 legal
  • Why AdamW decays the weight, not the gradient
  • Linear warmup plus cosine decay, with a worked 10-step schedule

Before you start

A marble and a bowl

Take the smallest interesting loss: \(L(w) = \tfrac12 (w-4)^2\). The minimum is at \(w=4\). The gradient is \(L'(w) = w-4\). Start at \(w=0\), so the first gradient is \(-4\): “walk right.”

Every optimizer in this chapter is a rule for turning that number into a new \(w\). SGD follows it. Momentum remembers the last few followings. Adam also remembers how noisy the followings have been, and steps less in noisy directions. AdamW adds a separate pull toward the origin. The schedule changes how long each step is allowed to be.

On this bowl the “right answer” is obvious. That is the point. If an optimizer cannot walk from 0 to 4 in a way you can explain, it will not get clearer in 8 billion dimensions.

Subtract the gradient

take a step of size η, opposite the gradient. That is the whole algorithm.

With \(\eta = 0.5\), starting at 0:

Step\(w\)\(\nabla L\)new \(w\)
10−40 − 0.5·(−4) = 2
22−23
33−13.5
43.5−0.53.75

Each step halves the remaining gap. After four steps we are at 3.75, closing in on 4. In many dimensions, “SGD” usually means this update on a minibatch: an unbiased, noisy estimate of the full gradient. The noise is a regularizer. The slowness in a narrow valley is the problem momentum is for.

Keep a velocity

PyTorch’s SGD with momentum (dampening 0) stores a velocity \(v\):

blend the new gradient into a running velocity, then step along the velocity. μ is usually 0.9.

Same bowl, \(\eta=0.5\), \(\mu=0.9\). Step 1 matches SGD: \(v=-4\), \(w=2\). Step 2 does not. The new gradient is \(-2\), but the velocity is \(0.9\cdot(-4)+(-2)=-5.6\), so \(w=2-0.5\cdot(-5.6)=\mathbf{4.8}\). Past the minimum. Step 4 sits at 7.37, still going the wrong way, until the velocity finally flips.

That overshoot is the feature in a ravine: the component of the gradient that keeps pointing the same way builds up, and the component that chatters left-right averages out. On a 1-D bowl there is no ravine, so you just see the overshoot.

A step size for every coordinate

Adam keeps an exponential average of the gradient (\(m\), first moment) and of its square (\(v\), second moment). The update is a momentum step, divided by the recent rms of the gradient:

m is a smoothed gradient. v is a smoothed g², so √v is a smoothed |g|.

divide by 1 − βᵗ because m and v started at 0. Without that, step 1 would be 10× too small (β1=0.9) and 1000× too small in v (β2=0.999).

Step 1 on the bowl, \(\eta=0.5\), \(\beta_1=0.9\), \(\beta_2=0.999\), \(g=-4\):

  1. \(m = 0.1 \cdot (-4) = -0.4\), \(v = 0.001 \cdot 16 = 0.016\).
  2. \(\hat m = -0.4 / 0.1 = -4\), \(\hat v = 0.016 / 0.001 = 16\).
  3. \(w = 0 - 0.5 \cdot (-4) / 4 = \mathbf{0.5}\).

Bias correction restored the true gradient, then \(\sqrt{\hat v}=4\) cancelled \(\lvert g\rvert\), and the step collapsed to \(\eta\) in the sign of \(-g\). On the first step of a 1-D problem, Adam walks a distance \(\eta\), not \(\eta\lvert g\rvert\). That is why people quote Adam learning rates like \(3\cdot 10^{-4}\) instead of \(0.5\): the update is already scale-free.

m = 0.1 g  →  m̂ = g  →  first step 0.5

MANIM Bias correction undoes “started at 0”. Use the beat buttons to jump.

Decay the weight, not the gradient

Classical “Adam + L2” adds \(\lambda w\) to the gradient before the second-moment divide. Large weights then look like large, stable gradients, and the decay is shrunk by \(\sqrt{v}\). AdamW takes the decay out of that fraction:

the Adam step, plus a separate pull of size ηλ toward the origin. λ is the weight decay, not the L2 coefficient in the loss.

On our bowl, \(w_0=0\), so step 1 of AdamW equals step 1 of Adam (\(0.5\)). After that the extra \(-\eta\lambda w\) tugs toward 0, away from the minimum at 4. After four steps Adam is at 1.972 and AdamW (\(\lambda=0.1\)) is at 1.829. That is not a bug. Weight decay is a regularizer. It is supposed to bias the solution toward small weights. On a real net you apply it to the 2-D weight matrices and skip the 1-D biases and norms — those should not be pulled to 0.

decay inside √v  ·  −ηλw outside  ·  1.972 vs 1.829

MANIM AdamW pulls w outside √v. Use the beat buttons to jump.

GPT-2 124M-style training uses AdamW with \(\beta = (0.9, 0.95)\), \(\varepsilon=10^{-8}\), \(\lambda=0.1\). The slightly smaller \(\beta_2\) forgets the second moment faster, which helps when the gradient scale changes during a long run.

Warmup, then cosine

A constant \(\eta\) is rarely what a long run wants. Two pieces show up everywhere:

climb linearly from 0 to η_max over the warmup, then coast down a cosine to η_min at step T.

Worked: \(\eta_{\max}=10^{-3}\), \(\eta_{\min}=10^{-4}\), warmup \(=2\), \(T=10\). Steps are 1-based.

StepPhaseLearning rate
1warmup0.0005
2peak0.001
6halfway cosine0.00055
10end0.0001

Step 6 is exactly the midpoint of the cosine, so the rate is the midpoint of \(10^{-3}\) and \(10^{-4}\), which is \(\mathbf{0.00055}\). Warmup exists because the first updates, especially with Adam, are biased and because BatchNorm / LayerNorm stats are still moving. A large step then can kick the net into a bad basin.

Interactive · One-dimensional bowl

Can you see momentum overshoot 4, and Adam take equal-sized first steps?

optimizers.pypaths match torch.optim.SGD / Adam / AdamW
"""SGD, momentum, Adam, AdamW, and a warmup-cosine schedule.

The toy loss is a 1-D bowl: L(w) = 1/2 (w - 4)^2.  Then L'(w) = w - 4.
Every number on the chapter page is one of these steps, so you can
redo it with a pencil.

Two PyTorch conventions we follow:
  SGD+momentum  v ← μ v + g ;  w ← w − η v
  AdamW         decay is applied to w, not to g  (decoupled)
"""
import math


def loss(w):
    """L(w) = 1/2 (w - 4)^2. Minimum at w = 4, L = 0."""
    return 0.5 * (w - 4.0) ** 2


def grad(w):
    """L'(w) = w - 4."""
    return w - 4.0


def sgd(w0, eta, steps):
    """Plain gradient descent. Returns the list of w, including w0."""
    w = w0
    hist = [w]
    for _ in range(steps):
        w = w - eta * grad(w)
        hist.append(w)
    return hist


def sgd_momentum(w0, eta, mu, steps):
    """Heavy ball as implemented by torch.optim.SGD (dampening = 0)."""
    w, v = w0, 0.0
    hist = [w]
    for _ in range(steps):
        g = grad(w)
        v = mu * v + g
        w = w - eta * v
        hist.append(w)
    return hist


def adam(w0, eta, beta1, beta2, eps, steps):
    """Kingma & Ba Adam, including bias correction."""
    w, m, v = w0, 0.0, 0.0
    hist = [w]
    for t in range(1, steps + 1):
        g = grad(w)
        m = beta1 * m + (1.0 - beta1) * g
        v = beta2 * v + (1.0 - beta2) * (g * g)
        # Bias correction: m and v start at 0, so early steps are too small.
        mhat = m / (1.0 - beta1 ** t)
        vhat = v / (1.0 - beta2 ** t)
        w = w - eta * mhat / (math.sqrt(vhat) + eps)
        hist.append(w)
    return hist


def adamw(w0, eta, beta1, beta2, eps, wd, steps):
    """AdamW: the same moments, plus a decay term on w itself.

    w ← w − η ( mhat/sqrt(vhat) + λ w )
    The λ w term does not go through the second-moment denominator.
    """
    w, m, v = w0, 0.0, 0.0
    hist = [w]
    for t in range(1, steps + 1):
        g = grad(w)
        m = beta1 * m + (1.0 - beta1) * g
        v = beta2 * v + (1.0 - beta2) * (g * g)
        mhat = m / (1.0 - beta1 ** t)
        vhat = v / (1.0 - beta2 ** t)
        w = w - eta * (mhat / (math.sqrt(vhat) + eps) + wd * w)
        hist.append(w)
    return hist


def cosine_warmup(step, total, warmup, lr_max, lr_min):
    """Linear warmup for `warmup` steps, then cosine down to lr_min.

    step is 1-based: the first call is step=1.  step=0 would be lr=0
    and is not used.
    """
    if step <= warmup:
        return lr_max * step / warmup
    progress = (step - warmup) / (total - warmup)  # in (0, 1]
    return lr_min + 0.5 * (lr_max - lr_min) * (1.0 + math.cos(math.pi * progress))


# --- Numbers quoted on the page. ---

W0 = 0.0
ETA = 0.5
MU = 0.9
BETA1, BETA2, EPS = 0.9, 0.999, 1e-8
WD = 0.1

SGD_PATH = sgd(W0, ETA, steps=4)                 # 0, 2, 3, 3.5, 3.75
MOM_PATH = sgd_momentum(W0, ETA, MU, steps=4)    # 0, 2, 4.8, ...
ADAM_PATH = adam(W0, ETA, BETA1, BETA2, EPS, steps=4)
ADAMW_PATH = adamw(W0, ETA, BETA1, BETA2, EPS, WD, steps=4)

# Schedule: 10 steps, 2 of warmup, 1e-3 → 1e-4.
LR_MAX, LR_MIN, WARMUP, TOTAL = 1e-3, 1e-4, 2, 10
LR_STEPS = [cosine_warmup(s, TOTAL, WARMUP, LR_MAX, LR_MIN)
            for s in range(1, TOTAL + 1)]
# step 1 (mid-warmup), step 2 (peak), step 6 (halfway through cosine), step 10 (end)
LR_AT = {s: LR_STEPS[s - 1] for s in (1, 2, 6, 10)}

Mistakes learners make

Using an SGD learning rate with Adam. Adam’s update is already divided by \(\sqrt{v}\). \(\eta=0.5\) is a toy. On a transformer, start near \(3\cdot 10^{-4}\) or the model’s paper, not \(0.1\).

Adding L2 to the loss and calling it AdamW. That is Adam+L2. The decay then sits inside \(\sqrt{v}\). AdamW applies \(\lambda w\) after the adaptive scale.

Decaying the biases and LayerNorm gains. A common rule: decay only parameters with \(\mathrm{ndim} \ge 2\). 1-D tensors are scales and shifts; pulling them to 0 fights the normalizer.

Skipping warmup on a large-batch run. The first Adam steps have a half-baked \(v\). Warmup is cheap insurance. Cosine without warmup is fine on a tiny toy; it is not how GPT-3-style runs start.

What to remember

  • SGD: \(w \leftarrow w - \eta g\). On the bowl, \(0\to 2\to 3\to 3.5\to 3.75\).
  • Momentum stores a velocity. It helps in ravines and overshoots in bowls (\(2\to 4.8\)).
  • Adam: two moments, bias correction, scale-free steps. First step here lands at \(0.5\).
  • AdamW decays \(w\), not \(g\). Apply it to matrices, not to 1-D gains.
  • Warmup linearly, then cosine to a floor. Mid-cosine is the midpoint of \(\eta_{\max}\) and \(\eta_{\min}\).

Check yourself

1. \(L=\tfrac12(w-4)^2\), \(w=0\), \(\eta=0.5\). Where is SGD after one step?

\(g=-4\), so \(w=0-0.5\cdot(-4)=2\).

2. Why does momentum land at 4.8 on step 2?

Velocity is \(0.9\cdot(-4)+(-2)=-5.6\). Then \(2-0.5\cdot(-5.6)=4.8\). The old step is still 90% alive.

3. What does Adam’s bias correction do on step 1 with \(\beta_1=0.9\)?

It divides \(m\) by \(1-0.9=0.1\), undoing the “started at 0” shrink. \(\hat m\) equals \(g\).

4. Name the one extra term in AdamW versus Adam.

\(-\eta\lambda w\). A pull toward the origin that does not go through \(\sqrt{v}\).

5. Warmup 2, \(T=10\), \(\eta_{\max}=10^{-3}\), \(\eta_{\min}=10^{-4}\). What is \(\eta(6)\)?

\(0.00055\). Step 6 is halfway through the cosine, so the rate is halfway between the peak and the floor.

Go to the source

Keep going