REASONING FROM SCRATCH · CHAPTER 09
The KL-regularized reward has a closed-form best policy. Plug that form into Bradley–Terry and the partition function cancels. What remains is a classification loss on log-prob ratios.
Prompt \(x\): “Two sentences for a ridge hike. Weather is turning.”
Take a shell, 1 L water, and a headlamp. Storms build after noon.
Wear whatever feels authentic to your journey.
RLHF would fit a reward model, then run PPO. DPO notices that the reward you want is already a function of the policy you are fitting. Change variables, and the pair becomes a logistic loss. No RM. No sampler in the inner loop.
raise the winner’s log-ratio over the reference, lower the loser’s. β sets how hard we push.
pair · ref · loss
packing · no RM · prefer · BT
\(\log\pi(y\mid x)\) is the sum of token log-probs, with the prompt tokens masked so they do not enter the sum. Same gather you already do for SFT, twice.
| \(\log\pi_\theta\) | \(\log\pi_{\mathrm{ref}}\) | implicit \(\hat{r} = \beta\log(\pi/\pi_{\mathrm{ref}})\) | |
|---|---|---|---|
| Winner | −12.0 | −11.0 | −0.10 |
| Loser | −10.0 | −11.5 | +0.15 |
Margin: \(\beta[(-12-(-11)) - (-10-(-11.5))] = 0.1 \times (-1 - 1.5) = -0.25\). Then \(\sigma(-0.25) = 0.438\) and \(\mathcal{L} = -\log 0.438 = 0.826\). The gradient weights the pair by \(\sigma(-\mathrm{margin})\): a pair the model already ranks right gets a small step; this one, ranked backwards, gets a large one.
Both log-probs can fall together if that widens the margin (likelihood displacement). Off-policy preference data from an old policy is stale. Overfitting a small pair set is easy — start from SFT. Variants (IPO, KTO, ORPO, SimPO) change the regularizer. Online or iterative DPO re-samples from the current \(\pi_\theta\). DPO is alignment without an RL loop. It is not alignment without assumptions.
Forgetting to mask the prompt when you sum token log-probs. Then a long prompt dominates both \(y_w\) and \(y_l\) and the margin is noise.
Skipping the \(Z(x)\) cancellation. That cancellation is why you do not need to sample the whole vocabulary to the power of length.
“Sampling is not differentiable, therefore PPO.” DPO exists because the optimal-policy change of variables removes the need to sample in the inner loop.
It depends only on \(x\). Bradley–Terry uses \(r_w - r_l\), so the two \(+\beta\log Z(x)\) terms cancel.
−0.25.
+0.15. That is why the current policy is wrong.
"""DPO loss on one preference pair of our own.
Prompt x: "Two sentences for a ridge hike. Weather is turning."
Winner y_w: a concrete packing list.
Loser y_l: a vague pep talk.
The model currently likes the loser more. DPO should pull the log-odds
the other way. β = 0.1.
"""
import math
def sigmoid(x):
return 1.0 / (1.0 + math.exp(-x))
# Sequence log-probs (sum of token log-probs, prompt tokens already masked).
LOG_PI_W, LOG_PI_L = -12.0, -10.0
LOG_REF_W, LOG_REF_L = -11.0, -11.5
BETA = 0.1
def dpo_margin(log_pi_w, log_pi_l, log_ref_w, log_ref_l, beta=BETA):
"""β [ (log π_θ(y_w) − log π_ref(y_w)) − (log π_θ(y_l) − log π_ref(y_l)) ]."""
return beta * ((log_pi_w - log_ref_w) - (log_pi_l - log_ref_l))
MARGIN = dpo_margin(LOG_PI_W, LOG_PI_L, LOG_REF_W, LOG_REF_L)
LOSS = -math.log(sigmoid(MARGIN))
# Implicit reward used in the derivation: r̂(y) = β log(π_θ(y)/π_ref(y)).
IMPLICIT_W = BETA * (LOG_PI_W - LOG_REF_W) # 0.1 * (−1.0) = −0.10
IMPLICIT_L = BETA * (LOG_PI_L - LOG_REF_L) # 0.1 * (1.5) = 0.15
if __name__ == "__main__":
# ( −12 − −11 ) − ( −10 − −11.5 ) = (−1) − (1.5) = −2.5
# margin = 0.1 * −2.5 = −0.25
assert abs(MARGIN - (-0.25)) < 1e-12
assert abs(IMPLICIT_W - (-0.10)) < 1e-12
assert abs(IMPLICIT_L - 0.15) < 1e-12
assert abs(LOSS - (-math.log(sigmoid(-0.25)))) < 1e-12
print("dpo loss ok")