REASONING FROM SCRATCH · CHAPTER 07

PPO

REINFORCE takes whatever step the sample suggests. PPO first asks how far the new policy has moved, then clips. The easy mistake is the sign when the advantage is negative.

~22 min readTRPO & GAE as deep divesTested Python

What you'll learn

  • A one-screen KL and importance-sampling primer
  • The clipped surrogate, and where the gradient is actually zero
  • The negative-advantage case: the clipped term is more negative
  • GAE’s \(\lambda\) dial and TRPO’s trust region, as optional depth

Before you start

Do not step off the dock

A harbor clerk who updates too hard after one lucky vault run will never show the safe door again. PPO’s clip is a railing: once the probability of an action has already moved by about \(1\pm\varepsilon\), further pushing that action does no more work.

We maximize \(L = \min(r\hat{A}, \operatorname{clip}(r, 1-\varepsilon, 1+\varepsilon)\hat{A})\). The min is pessimistic. The ratio \(r\) must be sampled under \(\pi_{\mathrm{old}}\), not under \(\pi_\theta\). Importance sampling is what makes that legal.

KL and importance sampling

KL. \(\mathrm{KL}(\pi_{\mathrm{ref}} \,\|\, \pi) = \mathbb{E}_{x\sim\pi_{\mathrm{ref}}}[\log\pi_{\mathrm{ref}}(x) - \log\pi(x)]\). It is zero only when the two distributions match. It is not symmetric. PPO and GRPO use it as a leash so the policy cannot flee the reference model. \(\varepsilon\) in the clip is not a KL. It bounds the probability ratio, not the KL.

ratio · clip · min

MANIM PPO objective uses min(unclipped, clipped) advantage term.

dock · clip · trust · A/C

MANIM PPO clips the ratio so updates stay on the dock.

Importance sampling. You have samples from \(\pi_{\mathrm{old}}\) and you want an expectation under \(\pi_\theta\). Multiply each sample by the ratio \(r_t(\theta) = \pi_\theta(a_t\mid s_t)/\pi_{\mathrm{old}}(a_t\mid s_t)\). That is unbiased if \(\pi_{\mathrm{old}}\) covers the support of \(\pi_\theta\). PPO samples \(a \sim \pi_{\mathrm{old}}\), never \(a \sim \pi_\theta\).

Three numbers, \(\varepsilon = 0.2\)

take the worse of the raw ratio-times-advantage and the clipped version. Maximize L.

Case\(r\)\(\hat{A}\)\(r\hat{A}\)clip \(r\)clip\(\cdot\hat{A}\)\(L\)Gradient?
Positive, too far1.40+1.52.101.201.801.80Zero if \(r\) grows further
Negative, too far0.60−1.5−0.900.80−1.20−1.20Zero if \(r\) shrinks further
Inside the band1.10+1.51.651.101.651.65Flows

When \(\hat{A} < 0\) and \(r < 1-\varepsilon\)

Old notes said the clipped term is “less negative” than \(r\hat{A}\). That is false.

Take \(r = 0.60\), \(\hat{A} = -1.5\), \(\varepsilon = 0.2\). Unclipped: \(0.60 \times -1.5 = -0.90\). The clip lifts the ratio up to 0.80. Then \(0.80 \times -1.5 = -1.20\). −1.20 is more negative than −0.90. The min picks −1.20. That value does not depend on \(r\) any more, so the gradient is zero. PPO stops pushing the probability down.

The symmetric case: \(\hat{A} > 0\) and \(r > 1+\varepsilon\). The clip holds the ratio at 1.20. The min picks the smaller (clipped) product. Further increases of \(r\) do nothing. PPO stops pushing the probability up.

Clipping is one-sided. It does not keep KL near \(\varepsilon\). It zeros the gradient once that action has already moved enough in the direction the advantage wanted.

GAE and TRPO

GAE: a λ dial between TD and Monte Carlo

The TD residual is \(\delta_t = R_{t+1} + \gamma V(s_{t+1}) - V(s_t)\). GAE is the discounted sum \(\hat{A}_t = \sum_{\ell\ge 0}(\gamma\lambda)^\ell \delta_{t+\ell}\). \(\lambda = 0\) is one-step TD. \(\lambda = 1\) is Monte Carlo (with a value bootstrap at the end of the horizon). Two-step numbers: \(\delta = (1.0, 0.5)\), \(\gamma = 0.9\), \(\lambda = 0.95\). Then \(\hat{A}_0 = 1.0 + 0.9\times 0.95\times 0.5 = 1.4275\) and \(\hat{A}_1 = 0.5\).

TRPO: a real KL trust region

TRPO maximizes a surrogate \(\mathbb{E}_{a\sim\pi_{\mathrm{old}}}[r_\theta \hat{A}]\) subject to \(\mathrm{KL}(\pi_{\mathrm{old}} \,\|\, \pi_\theta) \le \delta\). The samples come from \(\pi_{\mathrm{old}}\). Practically this needs a Fisher-vector product, conjugate gradient, and a line search. PPO keeps the “don’t move too far” idea and throws away the hard constraint. That is why PPO, not TRPO, is what InstructGPT and most LLM stacks run.

The full PPO loss also has a value-head squared error and an entropy bonus. We will meet the KL again as a per-token add-on to the reward in the next chapter.

Mistakes learners make

“For \(r < 1-\varepsilon\) the clipped term is less negative.” It is more negative. Then the min freezes the objective and the gradient dies.

Sampling \(a \sim \pi_\theta\) in the surrogate. The ratio is an importance weight. The action is drawn from \(\pi_{\mathrm{old}}\).

“Clipping keeps KL near \(\varepsilon\).” \(\varepsilon\) bounds the ratio. KL can still grow if many tokens each move a little.

What to remember

  • Importance sampling lets you reuse \(\pi_{\mathrm{old}}\) rollouts. KL leashes \(\pi\) to a reference.
  • \(r=1.40\), \(\hat{A}=+1.5\): \(L=1.80\). \(r=0.60\), \(\hat{A}=-1.5\): clipped \(=-1.20\), more negative, gradient 0.
  • GAE mixes n-step advantages with \(\lambda\). TRPO is the constrained parent of PPO.

Check yourself

1. \(r=0.60\), \(\hat{A}=-1.5\), \(\varepsilon=0.2\). What is \(r\hat{A}\)? The clipped product? Which is more negative?

−0.90 and −1.20. The clipped product is more negative.

2. Why is the gradient zero in that region?

The min has selected the constant \((1-\varepsilon)\hat{A}\). Changing \(r\) further does not change \(L\).

3. Who do we sample the action from?

\(\pi_{\mathrm{old}}\). The ratio does the reweighting.

4. GAE with \(\delta=(1.0, 0.5)\), \(\gamma=0.9\), \(\lambda=0.95\). What is \(\hat{A}_0\)

1.4275.

The code

ppo_clip.py1.80 / −1.20 (more negative) / 1.65
"""PPO clipping and a tiny GAE backup. Every number is on the page.

The important sign: when the advantage is negative and the ratio has
already dropped below 1−ε, the clipped term is *more* negative than
r·A. The min then picks that constant, so the gradient is zero. PPO
stops pushing the probability down.
"""


def clip(r, eps):
    return min(1 + eps, max(1 - eps, r))


def ppo_surrogate(r, adv, eps=0.2):
    """L = min(r A, clip(r, 1−ε, 1+ε) A). We *maximize* this."""
    unclipped = r * adv
    clipped = clip(r, eps) * adv
    return min(unclipped, clipped), unclipped, clipped, clip(r, eps)


EPS = 0.2

# Three cases used on the page.
POS = ppo_surrogate(1.40, +1.5, EPS)     # clipping active, limits the rise
NEG = ppo_surrogate(0.60, -1.5, EPS)     # clipping active, gradient is 0
IN_ = ppo_surrogate(1.10, +1.5, EPS)     # inside the band, no clip


def gae(deltas, gamma=0.9, lam=0.95):
    """Â_t = δ_t + (γλ) δ_{t+1} + (γλ)² δ_{t+2} + …  backward recurrence."""
    adv, out = 0.0, [0.0] * len(deltas)
    for t in range(len(deltas) - 1, -1, -1):
        adv = deltas[t] + gamma * lam * adv
        out[t] = adv
    return out


# Two-step GAE: δ = (1.0, 0.5). λ = 0.95, γ = 0.9.
DELTAS = (1.0, 0.5)
GAE_ADV = gae(DELTAS)


if __name__ == "__main__":
    L, u, c, r_clip = POS
    assert abs(u - 2.10) < 1e-12 and abs(c - 1.80) < 1e-12 and abs(L - 1.80) < 1e-12
    L, u, c, r_clip = NEG
    # unclipped = 0.60 * −1.5 = −0.90
    # clipped ratio = 0.80, clipped = −1.20  ← more negative
    # min(−0.90, −1.20) = −1.20
    assert abs(u - (-0.90)) < 1e-12
    assert abs(c - (-1.20)) < 1e-12
    assert abs(L - (-1.20)) < 1e-12
    assert c < u, "clipped term must be more negative when A < 0 and r < 1-eps"
    L, u, c, _ = IN_
    assert abs(u - 1.65) < 1e-12 and L == u
    assert abs(GAE_ADV[0] - (1.0 + 0.9 * 0.95 * 0.5)) < 1e-12
    assert abs(GAE_ADV[1] - 0.5) < 1e-12
    print("ppo clip ok")

Go to the source

Keep going