REASONING FROM SCRATCH · CHAPTER 08
Preferences become a reward model. The reward model becomes a PPO signal with a KL leash. That is the 2022 chat stack, in one path.
Prompt: “Weather for skippers leaving at dusk.”
Gale tonight. Stay in port. Check the barometer at 04:00.
The sea is a poem tonight, wild and free, a dance of foam.
A human (or a panel) prefers the first. RLHF’s job is to turn many such pairs into a scalar that PPO can maximize, without the policy fleeing the helpful SFT model.
A completion is one action in a contextual bandit. The tokens inside it are an MDP with a sparse terminal reward. Both views are true. The reward model scores the finished reply. PPO then broadcasts that score back through the tokens, with a KL tax.
SFT maximizes log-prob of one gold reply. It never sees a worse reply. It never scores “this is better than that.” Teacher forcing trains on gold prefixes, then at generation the model conditions on its own tokens (exposure bias). Preferences fix that: you sample from the current policy, rank the samples, and learn a relative score.
KL term · hacking · beta
gales · pipeline · RM · KL
Christiano et al. (2017) and Stiennon et al. (2020) built the loop. InstructGPT (Ouyang et al. 2022) applied it to chat. The RM in InstructGPT was trained on about 33k prompts, each with \(K = 4\)–9 ranked replies — many more pairwise comparisons than 33k.
the chance the winner beats the loser is a sigmoid of the score gap.
Our pair: \(r_w = 1.5\), \(r_l = -0.5\), so \(\Delta = 2.0\). Then \(\sigma(2) = 0.881\) and the negative-log loss is \(-\log 0.881 = 0.127\).
The architecture is an SFT trunk plus a scalar head on the last token. Two forwards per pair. The loss only cares about the difference, so an absolute “this reply is a 7” is not identified — only “this one is 2 above that one.”
PPO for LLMs keeps a frozen reference \(\pi_{\mathrm{ref}}\) (usually the SFT model) and a value head. That is four networks if the RM is separate: policy, reference, reward, value. InstructGPT folds the KL into the reward, then runs the usual clip on that shaped return:
tax the last token (or every token) by how far this policy has walked from the reference. β is the leash.
Worked token: \(\log\pi = -2.0\), \(\log\pi_{\mathrm{ref}} = -2.4\), so the KL piece is 0.40. With \(\beta = 0.10\) and \(r_{\mathrm{RM}} = 1.5\), the shaped reward is \(1.5 - 0.04 = 1.46\). γ is 1. The environment is deterministic. This formula is the common simplification; some stacks put the KL in the loss instead (GRPO will).
If the RM likes length, the policy writes longer gale warnings. If it likes the word “certainly,” the policy stuffs it. Sycophancy and over-refusal are the same bug: a proxy that can be gamed. Defenses: the KL leash, RM ensembles, human spot-checks, and — when the task allows it — a program that checks the answer. That last door is RLVR, two chapters from now.
DPO, next, skips the RM and the PPO loop. It is the preference-only sibling of “no learned reward.”
“RLHF was invented by InstructGPT.” The loop is older (Christiano 2017, Stiennon 2020). InstructGPT applied it to chat models.
“The RM saw 33k comparisons.” About 33k prompts, each with several ranked replies.
Putting the KL only in the clip and calling it InstructGPT. InstructGPT shaped the reward. GRPO later moves KL into the loss. Say which.
\(\sigma(2) = 0.881\).
Policy, reference, reward model, value head.
1.46.
Human raters and RMs often prefer longer, more “complete” text. The policy notices.
"""Bradley–Terry reward-model loss on one harbor-weather preference.
Winner y_w: a short, useful gale warning.
Loser y_l: a pretty sentence that does not help a skipper.
"""
import math
def sigmoid(x):
return 1.0 / (1.0 + math.exp(-x))
# Reward-model scores for the two replies. Difference Δ = r_w − r_l.
R_WIN, R_LOSE = 1.5, -0.5
DELTA = R_WIN - R_LOSE # 2.0
P_WIN = sigmoid(DELTA) # σ(2) ≈ 0.8808
NLL = -math.log(P_WIN) # ≈ 0.1269
# InstructGPT-style KL-shaped reward at the last token.
# r_shaped = r_RM − β (log π − log π_ref)
LOG_PI, LOG_REF, BETA = -2.0, -2.4, 0.10
KL_TOKEN = LOG_PI - LOG_REF # 0.40
R_SHAPED = R_WIN - BETA * KL_TOKEN # 1.46
if __name__ == "__main__":
assert abs(DELTA - 2.0) < 1e-12
assert abs(P_WIN - 0.88079707797) < 1e-9
assert abs(NLL - 0.12692801104) < 1e-9
assert abs(KL_TOKEN - 0.40) < 1e-12
assert abs(R_SHAPED - 1.46) < 1e-12
print("bradley-terry ok")