REASONING FROM SCRATCH · CHAPTER 08

RLHF end to end

Preferences become a reward model. The reward model becomes a PPO signal with a KL leash. That is the 2022 chat stack, in one path.

~20 min readHarbor gale pairTested Python

What you'll learn

  • Why SFT is not enough: one gold answer, no negatives, exposure bias
  • Bradley–Terry: a pair of scores becomes a probability of winning
  • The three-stage pipeline: SFT → reward model → PPO with KL
  • Where the KL sits in InstructGPT (inside the reward) and what hacking looks like

Before you start

Two gale warnings

Prompt: “Weather for skippers leaving at dusk.”

Winner \(y_w\)

Gale tonight. Stay in port. Check the barometer at 04:00.

Loser \(y_l\)

The sea is a poem tonight, wild and free, a dance of foam.

A human (or a panel) prefers the first. RLHF’s job is to turn many such pairs into a scalar that PPO can maximize, without the policy fleeing the helpful SFT model.

A completion is one action in a contextual bandit. The tokens inside it are an MDP with a sparse terminal reward. Both views are true. The reward model scores the finished reply. PPO then broadcasts that score back through the tokens, with a KL tax.

Imitation cannot see its own mistakes

SFT maximizes log-prob of one gold reply. It never sees a worse reply. It never scores “this is better than that.” Teacher forcing trains on gold prefixes, then at generation the model conditions on its own tokens (exposure bias). Preferences fix that: you sample from the current policy, rank the samples, and learn a relative score.

KL term · hacking · beta

MANIM KL to SFT is the leash against reward hacking.

gales · pipeline · RM · KL

MANIM RLHF: SFT → RM → PPO, with KL to the SFT harbor.

Christiano et al. (2017) and Stiennon et al. (2020) built the loop. InstructGPT (Ouyang et al. 2022) applied it to chat. The RM in InstructGPT was trained on about 33k prompts, each with \(K = 4\)–9 ranked replies — many more pairwise comparisons than 33k.

A difference of scores is a win probability

the chance the winner beats the loser is a sigmoid of the score gap.

Our pair: \(r_w = 1.5\), \(r_l = -0.5\), so \(\Delta = 2.0\). Then \(\sigma(2) = 0.881\) and the negative-log loss is \(-\log 0.881 = 0.127\).

The architecture is an SFT trunk plus a scalar head on the last token. Two forwards per pair. The loss only cares about the difference, so an absolute “this reply is a 7” is not identified — only “this one is 2 above that one.”

Four models in memory

SFT policy
Reward model
PPO (policy + value)

PPO for LLMs keeps a frozen reference \(\pi_{\mathrm{ref}}\) (usually the SFT model) and a value head. That is four networks if the RM is separate: policy, reference, reward, value. InstructGPT folds the KL into the reward, then runs the usual clip on that shaped return:

tax the last token (or every token) by how far this policy has walked from the reference. β is the leash.

Worked token: \(\log\pi = -2.0\), \(\log\pi_{\mathrm{ref}} = -2.4\), so the KL piece is 0.40. With \(\beta = 0.10\) and \(r_{\mathrm{RM}} = 1.5\), the shaped reward is \(1.5 - 0.04 = 1.46\). γ is 1. The environment is deterministic. This formula is the common simplification; some stacks put the KL in the loss instead (GRPO will).

The RM is not the thing you wanted

If the RM likes length, the policy writes longer gale warnings. If it likes the word “certainly,” the policy stuffs it. Sycophancy and over-refusal are the same bug: a proxy that can be gamed. Defenses: the KL leash, RM ensembles, human spot-checks, and — when the task allows it — a program that checks the answer. That last door is RLVR, two chapters from now.

DPO, next, skips the RM and the PPO loop. It is the preference-only sibling of “no learned reward.”

Mistakes learners make

“RLHF was invented by InstructGPT.” The loop is older (Christiano 2017, Stiennon 2020). InstructGPT applied it to chat models.

“The RM saw 33k comparisons.” About 33k prompts, each with several ranked replies.

Putting the KL only in the clip and calling it InstructGPT. InstructGPT shaped the reward. GRPO later moves KL into the loss. Say which.

What to remember

  • SFT imitates one answer. Preferences teach “this, not that.”
  • \(\Delta = 2\) gives \(\sigma = 0.881\) and NLL 0.127 on the gale pair.
  • PPO + KL: four models. Shaped reward 1.46 in the worked token.
  • The RM can be hacked. A verifier is the way out when the answer is checkable.

Check yourself

1. \(r_w=1.5\), \(r_l=-0.5\). What is \(P(y_w \succ y_l)\)

\(\sigma(2) = 0.881\).

2. Name the four models in PPO-RLHF.

Policy, reference, reward model, value head.

3. \(\log\pi=-2.0\), \(\log\pi_{\mathrm{ref}}=-2.4\), \(\beta=0.10\), \(r_{\mathrm{RM}}=1.5\). Shaped reward?

1.46.

4. Why is length hacking so common?

Human raters and RMs often prefer longer, more “complete” text. The policy notices.

The code

bradley_terry.pyσ(2)=0.881, NLL=0.127, shaped=1.46
"""Bradley–Terry reward-model loss on one harbor-weather preference.

Winner y_w: a short, useful gale warning.
Loser  y_l: a pretty sentence that does not help a skipper.
"""
import math


def sigmoid(x):
    return 1.0 / (1.0 + math.exp(-x))


# Reward-model scores for the two replies. Difference Δ = r_w − r_l.
R_WIN, R_LOSE = 1.5, -0.5
DELTA = R_WIN - R_LOSE                    # 2.0
P_WIN = sigmoid(DELTA)                    # σ(2) ≈ 0.8808
NLL = -math.log(P_WIN)                    # ≈ 0.1269

# InstructGPT-style KL-shaped reward at the last token.
# r_shaped = r_RM − β (log π − log π_ref)
LOG_PI, LOG_REF, BETA = -2.0, -2.4, 0.10
KL_TOKEN = LOG_PI - LOG_REF               # 0.40
R_SHAPED = R_WIN - BETA * KL_TOKEN        # 1.46


if __name__ == "__main__":
    assert abs(DELTA - 2.0) < 1e-12
    assert abs(P_WIN - 0.88079707797) < 1e-9
    assert abs(NLL - 0.12692801104) < 1e-9
    assert abs(KL_TOKEN - 0.40) < 1e-12
    assert abs(R_SHAPED - 1.46) < 1e-12
    print("bradley-terry ok")

Go to the source

Keep going