REASONING FROM SCRATCH · CHAPTER 07
REINFORCE takes whatever step the sample suggests. PPO first asks how far the new policy has moved, then clips. The easy mistake is the sign when the advantage is negative.
A harbor clerk who updates too hard after one lucky vault run will never show the safe door again. PPO’s clip is a railing: once the probability of an action has already moved by about \(1\pm\varepsilon\), further pushing that action does no more work.
We maximize \(L = \min(r\hat{A}, \operatorname{clip}(r, 1-\varepsilon, 1+\varepsilon)\hat{A})\). The min is pessimistic. The ratio \(r\) must be sampled under \(\pi_{\mathrm{old}}\), not under \(\pi_\theta\). Importance sampling is what makes that legal.
KL. \(\mathrm{KL}(\pi_{\mathrm{ref}} \,\|\, \pi) = \mathbb{E}_{x\sim\pi_{\mathrm{ref}}}[\log\pi_{\mathrm{ref}}(x) - \log\pi(x)]\). It is zero only when the two distributions match. It is not symmetric. PPO and GRPO use it as a leash so the policy cannot flee the reference model. \(\varepsilon\) in the clip is not a KL. It bounds the probability ratio, not the KL.
ratio · clip · min
dock · clip · trust · A/C
Importance sampling. You have samples from \(\pi_{\mathrm{old}}\) and you want an expectation under \(\pi_\theta\). Multiply each sample by the ratio \(r_t(\theta) = \pi_\theta(a_t\mid s_t)/\pi_{\mathrm{old}}(a_t\mid s_t)\). That is unbiased if \(\pi_{\mathrm{old}}\) covers the support of \(\pi_\theta\). PPO samples \(a \sim \pi_{\mathrm{old}}\), never \(a \sim \pi_\theta\).
take the worse of the raw ratio-times-advantage and the clipped version. Maximize L.
| Case | \(r\) | \(\hat{A}\) | \(r\hat{A}\) | clip \(r\) | clip\(\cdot\hat{A}\) | \(L\) | Gradient? |
|---|---|---|---|---|---|---|---|
| Positive, too far | 1.40 | +1.5 | 2.10 | 1.20 | 1.80 | 1.80 | Zero if \(r\) grows further |
| Negative, too far | 0.60 | −1.5 | −0.90 | 0.80 | −1.20 | −1.20 | Zero if \(r\) shrinks further |
| Inside the band | 1.10 | +1.5 | 1.65 | 1.10 | 1.65 | 1.65 | Flows |
Old notes said the clipped term is “less negative” than \(r\hat{A}\). That is false.
Take \(r = 0.60\), \(\hat{A} = -1.5\), \(\varepsilon = 0.2\). Unclipped: \(0.60 \times -1.5 = -0.90\). The clip lifts the ratio up to 0.80. Then \(0.80 \times -1.5 = -1.20\). −1.20 is more negative than −0.90. The min picks −1.20. That value does not depend on \(r\) any more, so the gradient is zero. PPO stops pushing the probability down.
The symmetric case: \(\hat{A} > 0\) and \(r > 1+\varepsilon\). The clip holds the ratio at 1.20. The min picks the smaller (clipped) product. Further increases of \(r\) do nothing. PPO stops pushing the probability up.
Clipping is one-sided. It does not keep KL near \(\varepsilon\). It zeros the gradient once that action has already moved enough in the direction the advantage wanted.
The TD residual is \(\delta_t = R_{t+1} + \gamma V(s_{t+1}) - V(s_t)\). GAE is the discounted sum \(\hat{A}_t = \sum_{\ell\ge 0}(\gamma\lambda)^\ell \delta_{t+\ell}\). \(\lambda = 0\) is one-step TD. \(\lambda = 1\) is Monte Carlo (with a value bootstrap at the end of the horizon). Two-step numbers: \(\delta = (1.0, 0.5)\), \(\gamma = 0.9\), \(\lambda = 0.95\). Then \(\hat{A}_0 = 1.0 + 0.9\times 0.95\times 0.5 = 1.4275\) and \(\hat{A}_1 = 0.5\).
TRPO maximizes a surrogate \(\mathbb{E}_{a\sim\pi_{\mathrm{old}}}[r_\theta \hat{A}]\) subject to \(\mathrm{KL}(\pi_{\mathrm{old}} \,\|\, \pi_\theta) \le \delta\). The samples come from \(\pi_{\mathrm{old}}\). Practically this needs a Fisher-vector product, conjugate gradient, and a line search. PPO keeps the “don’t move too far” idea and throws away the hard constraint. That is why PPO, not TRPO, is what InstructGPT and most LLM stacks run.
The full PPO loss also has a value-head squared error and an entropy bonus. We will meet the KL again as a per-token add-on to the reward in the next chapter.
“For \(r < 1-\varepsilon\) the clipped term is less negative.” It is more negative. Then the min freezes the objective and the gradient dies.
Sampling \(a \sim \pi_\theta\) in the surrogate. The ratio is an importance weight. The action is drawn from \(\pi_{\mathrm{old}}\).
“Clipping keeps KL near \(\varepsilon\).” \(\varepsilon\) bounds the ratio. KL can still grow if many tokens each move a little.
−0.90 and −1.20. The clipped product is more negative.
The min has selected the constant \((1-\varepsilon)\hat{A}\). Changing \(r\) further does not change \(L\).
\(\pi_{\mathrm{old}}\). The ratio does the reweighting.
1.4275.
"""PPO clipping and a tiny GAE backup. Every number is on the page.
The important sign: when the advantage is negative and the ratio has
already dropped below 1−ε, the clipped term is *more* negative than
r·A. The min then picks that constant, so the gradient is zero. PPO
stops pushing the probability down.
"""
def clip(r, eps):
return min(1 + eps, max(1 - eps, r))
def ppo_surrogate(r, adv, eps=0.2):
"""L = min(r A, clip(r, 1−ε, 1+ε) A). We *maximize* this."""
unclipped = r * adv
clipped = clip(r, eps) * adv
return min(unclipped, clipped), unclipped, clipped, clip(r, eps)
EPS = 0.2
# Three cases used on the page.
POS = ppo_surrogate(1.40, +1.5, EPS) # clipping active, limits the rise
NEG = ppo_surrogate(0.60, -1.5, EPS) # clipping active, gradient is 0
IN_ = ppo_surrogate(1.10, +1.5, EPS) # inside the band, no clip
def gae(deltas, gamma=0.9, lam=0.95):
"""Â_t = δ_t + (γλ) δ_{t+1} + (γλ)² δ_{t+2} + … backward recurrence."""
adv, out = 0.0, [0.0] * len(deltas)
for t in range(len(deltas) - 1, -1, -1):
adv = deltas[t] + gamma * lam * adv
out[t] = adv
return out
# Two-step GAE: δ = (1.0, 0.5). λ = 0.95, γ = 0.9.
DELTAS = (1.0, 0.5)
GAE_ADV = gae(DELTAS)
if __name__ == "__main__":
L, u, c, r_clip = POS
assert abs(u - 2.10) < 1e-12 and abs(c - 1.80) < 1e-12 and abs(L - 1.80) < 1e-12
L, u, c, r_clip = NEG
# unclipped = 0.60 * −1.5 = −0.90
# clipped ratio = 0.80, clipped = −1.20 ← more negative
# min(−0.90, −1.20) = −1.20
assert abs(u - (-0.90)) < 1e-12
assert abs(c - (-1.20)) < 1e-12
assert abs(L - (-1.20)) < 1e-12
assert c < u, "clipped term must be more negative when A < 0 and r < 1-eps"
L, u, c, _ = IN_
assert abs(u - 1.65) < 1e-12 and L == u
assert abs(GAE_ADV[0] - (1.0 + 0.9 * 0.95 * 0.5)) < 1e-12
assert abs(GAE_ADV[1] - 0.5) < 1e-12
print("ppo clip ok")