NEURONS ALL THE WAY DOWN · CHAPTER 06
You have the gradient. The optimizer decides how to walk. Four ways to take a step, then a learning-rate schedule that almost every modern run actually uses.
Take the smallest interesting loss: \(L(w) = \tfrac12 (w-4)^2\). The minimum is at \(w=4\). The gradient is \(L'(w) = w-4\). Start at \(w=0\), so the first gradient is \(-4\): “walk right.”
Every optimizer in this chapter is a rule for turning that number into a new \(w\). SGD follows it. Momentum remembers the last few followings. Adam also remembers how noisy the followings have been, and steps less in noisy directions. AdamW adds a separate pull toward the origin. The schedule changes how long each step is allowed to be.
On this bowl the “right answer” is obvious. That is the point. If an optimizer cannot walk from 0 to 4 in a way you can explain, it will not get clearer in 8 billion dimensions.
take a step of size η, opposite the gradient. That is the whole algorithm.
With \(\eta = 0.5\), starting at 0:
| Step | \(w\) | \(\nabla L\) | new \(w\) |
|---|---|---|---|
| 1 | 0 | −4 | 0 − 0.5·(−4) = 2 |
| 2 | 2 | −2 | 3 |
| 3 | 3 | −1 | 3.5 |
| 4 | 3.5 | −0.5 | 3.75 |
Each step halves the remaining gap. After four steps we are at 3.75, closing in on 4. In many dimensions, “SGD” usually means this update on a minibatch: an unbiased, noisy estimate of the full gradient. The noise is a regularizer. The slowness in a narrow valley is the problem momentum is for.
PyTorch’s SGD with momentum (dampening 0) stores a velocity \(v\):
blend the new gradient into a running velocity, then step along the velocity. μ is usually 0.9.
Same bowl, \(\eta=0.5\), \(\mu=0.9\). Step 1 matches SGD: \(v=-4\), \(w=2\). Step 2 does not. The new gradient is \(-2\), but the velocity is \(0.9\cdot(-4)+(-2)=-5.6\), so \(w=2-0.5\cdot(-5.6)=\mathbf{4.8}\). Past the minimum. Step 4 sits at 7.37, still going the wrong way, until the velocity finally flips.
That overshoot is the feature in a ravine: the component of the gradient that keeps pointing the same way builds up, and the component that chatters left-right averages out. On a 1-D bowl there is no ravine, so you just see the overshoot.
Adam keeps an exponential average of the gradient (\(m\), first moment) and of its square (\(v\), second moment). The update is a momentum step, divided by the recent rms of the gradient:
m is a smoothed gradient. v is a smoothed g², so √v is a smoothed |g|.
divide by 1 − βᵗ because m and v started at 0. Without that, step 1 would be 10× too small (β1=0.9) and 1000× too small in v (β2=0.999).
Step 1 on the bowl, \(\eta=0.5\), \(\beta_1=0.9\), \(\beta_2=0.999\), \(g=-4\):
Bias correction restored the true gradient, then \(\sqrt{\hat v}=4\) cancelled \(\lvert g\rvert\), and the step collapsed to \(\eta\) in the sign of \(-g\). On the first step of a 1-D problem, Adam walks a distance \(\eta\), not \(\eta\lvert g\rvert\). That is why people quote Adam learning rates like \(3\cdot 10^{-4}\) instead of \(0.5\): the update is already scale-free.
m = 0.1 g → m̂ = g → first step 0.5
Classical “Adam + L2” adds \(\lambda w\) to the gradient before the second-moment divide. Large weights then look like large, stable gradients, and the decay is shrunk by \(\sqrt{v}\). AdamW takes the decay out of that fraction:
the Adam step, plus a separate pull of size ηλ toward the origin. λ is the weight decay, not the L2 coefficient in the loss.
On our bowl, \(w_0=0\), so step 1 of AdamW equals step 1 of Adam (\(0.5\)). After that the extra \(-\eta\lambda w\) tugs toward 0, away from the minimum at 4. After four steps Adam is at 1.972 and AdamW (\(\lambda=0.1\)) is at 1.829. That is not a bug. Weight decay is a regularizer. It is supposed to bias the solution toward small weights. On a real net you apply it to the 2-D weight matrices and skip the 1-D biases and norms — those should not be pulled to 0.
decay inside √v · −ηλw outside · 1.972 vs 1.829
GPT-2 124M-style training uses AdamW with \(\beta = (0.9, 0.95)\), \(\varepsilon=10^{-8}\), \(\lambda=0.1\). The slightly smaller \(\beta_2\) forgets the second moment faster, which helps when the gradient scale changes during a long run.
A constant \(\eta\) is rarely what a long run wants. Two pieces show up everywhere:
climb linearly from 0 to η_max over the warmup, then coast down a cosine to η_min at step T.
Worked: \(\eta_{\max}=10^{-3}\), \(\eta_{\min}=10^{-4}\), warmup \(=2\), \(T=10\). Steps are 1-based.
| Step | Phase | Learning rate |
|---|---|---|
| 1 | warmup | 0.0005 |
| 2 | peak | 0.001 |
| 6 | halfway cosine | 0.00055 |
| 10 | end | 0.0001 |
Step 6 is exactly the midpoint of the cosine, so the rate is the midpoint of \(10^{-3}\) and \(10^{-4}\), which is \(\mathbf{0.00055}\). Warmup exists because the first updates, especially with Adam, are biased and because BatchNorm / LayerNorm stats are still moving. A large step then can kick the net into a bad basin.
Interactive · One-dimensional bowl
Can you see momentum overshoot 4, and Adam take equal-sized first steps?
"""SGD, momentum, Adam, AdamW, and a warmup-cosine schedule.
The toy loss is a 1-D bowl: L(w) = 1/2 (w - 4)^2. Then L'(w) = w - 4.
Every number on the chapter page is one of these steps, so you can
redo it with a pencil.
Two PyTorch conventions we follow:
SGD+momentum v ← μ v + g ; w ← w − η v
AdamW decay is applied to w, not to g (decoupled)
"""
import math
def loss(w):
"""L(w) = 1/2 (w - 4)^2. Minimum at w = 4, L = 0."""
return 0.5 * (w - 4.0) ** 2
def grad(w):
"""L'(w) = w - 4."""
return w - 4.0
def sgd(w0, eta, steps):
"""Plain gradient descent. Returns the list of w, including w0."""
w = w0
hist = [w]
for _ in range(steps):
w = w - eta * grad(w)
hist.append(w)
return hist
def sgd_momentum(w0, eta, mu, steps):
"""Heavy ball as implemented by torch.optim.SGD (dampening = 0)."""
w, v = w0, 0.0
hist = [w]
for _ in range(steps):
g = grad(w)
v = mu * v + g
w = w - eta * v
hist.append(w)
return hist
def adam(w0, eta, beta1, beta2, eps, steps):
"""Kingma & Ba Adam, including bias correction."""
w, m, v = w0, 0.0, 0.0
hist = [w]
for t in range(1, steps + 1):
g = grad(w)
m = beta1 * m + (1.0 - beta1) * g
v = beta2 * v + (1.0 - beta2) * (g * g)
# Bias correction: m and v start at 0, so early steps are too small.
mhat = m / (1.0 - beta1 ** t)
vhat = v / (1.0 - beta2 ** t)
w = w - eta * mhat / (math.sqrt(vhat) + eps)
hist.append(w)
return hist
def adamw(w0, eta, beta1, beta2, eps, wd, steps):
"""AdamW: the same moments, plus a decay term on w itself.
w ← w − η ( mhat/sqrt(vhat) + λ w )
The λ w term does not go through the second-moment denominator.
"""
w, m, v = w0, 0.0, 0.0
hist = [w]
for t in range(1, steps + 1):
g = grad(w)
m = beta1 * m + (1.0 - beta1) * g
v = beta2 * v + (1.0 - beta2) * (g * g)
mhat = m / (1.0 - beta1 ** t)
vhat = v / (1.0 - beta2 ** t)
w = w - eta * (mhat / (math.sqrt(vhat) + eps) + wd * w)
hist.append(w)
return hist
def cosine_warmup(step, total, warmup, lr_max, lr_min):
"""Linear warmup for `warmup` steps, then cosine down to lr_min.
step is 1-based: the first call is step=1. step=0 would be lr=0
and is not used.
"""
if step <= warmup:
return lr_max * step / warmup
progress = (step - warmup) / (total - warmup) # in (0, 1]
return lr_min + 0.5 * (lr_max - lr_min) * (1.0 + math.cos(math.pi * progress))
# --- Numbers quoted on the page. ---
W0 = 0.0
ETA = 0.5
MU = 0.9
BETA1, BETA2, EPS = 0.9, 0.999, 1e-8
WD = 0.1
SGD_PATH = sgd(W0, ETA, steps=4) # 0, 2, 3, 3.5, 3.75
MOM_PATH = sgd_momentum(W0, ETA, MU, steps=4) # 0, 2, 4.8, ...
ADAM_PATH = adam(W0, ETA, BETA1, BETA2, EPS, steps=4)
ADAMW_PATH = adamw(W0, ETA, BETA1, BETA2, EPS, WD, steps=4)
# Schedule: 10 steps, 2 of warmup, 1e-3 → 1e-4.
LR_MAX, LR_MIN, WARMUP, TOTAL = 1e-3, 1e-4, 2, 10
LR_STEPS = [cosine_warmup(s, TOTAL, WARMUP, LR_MAX, LR_MIN)
for s in range(1, TOTAL + 1)]
# step 1 (mid-warmup), step 2 (peak), step 6 (halfway through cosine), step 10 (end)
LR_AT = {s: LR_STEPS[s - 1] for s in (1, 2, 6, 10)}
Using an SGD learning rate with Adam. Adam’s update is already divided by \(\sqrt{v}\). \(\eta=0.5\) is a toy. On a transformer, start near \(3\cdot 10^{-4}\) or the model’s paper, not \(0.1\).
Adding L2 to the loss and calling it AdamW. That is Adam+L2. The decay then sits inside \(\sqrt{v}\). AdamW applies \(\lambda w\) after the adaptive scale.
Decaying the biases and LayerNorm gains. A common rule: decay only parameters with \(\mathrm{ndim} \ge 2\). 1-D tensors are scales and shifts; pulling them to 0 fights the normalizer.
Skipping warmup on a large-batch run. The first Adam steps have a half-baked \(v\). Warmup is cheap insurance. Cosine without warmup is fine on a tiny toy; it is not how GPT-3-style runs start.
\(g=-4\), so \(w=0-0.5\cdot(-4)=2\).
Velocity is \(0.9\cdot(-4)+(-2)=-5.6\). Then \(2-0.5\cdot(-5.6)=4.8\). The old step is still 90% alive.
It divides \(m\) by \(1-0.9=0.1\), undoing the “started at 0” shrink. \(\hat m\) equals \(g\).
\(-\eta\lambda w\). A pull toward the origin that does not go through \(\sqrt{v}\).
\(0.00055\). Step 6 is halfway through the cosine, so the rate is halfway between the peak and the floor.