REASONING FROM SCRATCH · CHAPTER 09

DPO

The KL-regularized reward has a closed-form best policy. Plug that form into Bradley–Terry and the partition function cancels. What remains is a classification loss on log-prob ratios.

~16 min readRidge-hike pairTested Python

What you'll learn

  • Why the optimal policy under a KL leash is \(\pi^\star \propto \pi_{\mathrm{ref}} e^{r/\beta}\)
  • How \(Z(x)\) cancels in a preference pair
  • The DPO loss, and a worked margin of −0.25
  • What still goes wrong when both log-probs fall

Before you start

  • RLHF — Bradley–Terry and the KL objective

A packing list versus a pep talk

Prompt \(x\): “Two sentences for a ridge hike. Weather is turning.”

Winner \(y_w\)

Take a shell, 1 L water, and a headlamp. Storms build after noon.

Loser \(y_l\)

Wear whatever feels authentic to your journey.

RLHF would fit a reward model, then run PPO. DPO notices that the reward you want is already a function of the policy you are fitting. Change variables, and the pair becomes a logistic loss. No RM. No sampler in the inner loop.

From a KL-regularized reward to a loss

  1. The RLHF objective is \(\mathbb{E}[r] - \beta\,\mathrm{KL}(\pi \,\|\, \pi_{\mathrm{ref}})\). Under standard regularity, the maximizer is \(\pi^\star(y\mid x) = \frac{1}{Z(x)}\,\pi_{\mathrm{ref}}(y\mid x)\,\exp(r(x,y)/\beta)\). \(Z(x)\) is the sum over every possible \(y\) — intractable.
  2. Invert: \(r(x,y) = \beta\log\frac{\pi^\star(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)} + \beta\log Z(x)\). The same \(Z(x)\) sits on every reply to \(x\).
  3. Bradley–Terry only uses \(r_w - r_l\). The two \(\beta\log Z(x)\) terms cancel.
  4. Replace \(\pi^\star\) with the policy you are fitting, \(\pi_\theta\). The loss is binary cross-entropy on that margin.

raise the winner’s log-ratio over the reference, lower the loser’s. β sets how hard we push.

pair · ref · loss

MANIM DPO pair: chosen ↑, rejected ↓, relative to reference.

packing · no RM · prefer · BT

MANIM DPO: preference loss without a separate reward model.

\(\log\pi(y\mid x)\) is the sum of token log-probs, with the prompt tokens masked so they do not enter the sum. Same gather you already do for SFT, twice.

The model currently likes the pep talk

\(\log\pi_\theta\)\(\log\pi_{\mathrm{ref}}\)implicit \(\hat{r} = \beta\log(\pi/\pi_{\mathrm{ref}})\)
Winner−12.0−11.0−0.10
Loser−10.0−11.5+0.15

Margin: \(\beta[(-12-(-11)) - (-10-(-11.5))] = 0.1 \times (-1 - 1.5) = -0.25\). Then \(\sigma(-0.25) = 0.438\) and \(\mathcal{L} = -\log 0.438 = 0.826\). The gradient weights the pair by \(\sigma(-\mathrm{margin})\): a pair the model already ranks right gets a small step; this one, ranked backwards, gets a large one.

No RM does not mean no failure

Both log-probs can fall together if that widens the margin (likelihood displacement). Off-policy preference data from an old policy is stale. Overfitting a small pair set is easy — start from SFT. Variants (IPO, KTO, ORPO, SimPO) change the regularizer. Online or iterative DPO re-samples from the current \(\pi_\theta\). DPO is alignment without an RL loop. It is not alignment without assumptions.

Mistakes learners make

Forgetting to mask the prompt when you sum token log-probs. Then a long prompt dominates both \(y_w\) and \(y_l\) and the margin is noise.

Skipping the \(Z(x)\) cancellation. That cancellation is why you do not need to sample the whole vocabulary to the power of length.

“Sampling is not differentiable, therefore PPO.” DPO exists because the optimal-policy change of variables removes the need to sample in the inner loop.

What to remember

  • \(\pi^\star \propto \pi_{\mathrm{ref}} e^{r/\beta}\). Invert, subtract two rewards, \(Z(x)\) cancels.
  • Ridge-hike pair: margin −0.25, loss 0.826. The model liked the loser; DPO pushes the other way.
  • Still a supervised loss. Watch likelihood displacement and stale pairs.

Check yourself

1. Why does \(Z(x)\) disappear?

It depends only on \(x\). Bradley–Terry uses \(r_w - r_l\), so the two \(+\beta\log Z(x)\) terms cancel.

2. Compute the margin for the table above, \(\beta = 0.1\).

−0.25.

3. What is the implicit reward of the loser?

+0.15. That is why the current policy is wrong.

The code

dpo_loss.pymargin −0.25, implicit −0.10 / +0.15
"""DPO loss on one preference pair of our own.

Prompt x:  "Two sentences for a ridge hike. Weather is turning."
Winner y_w: a concrete packing list.
Loser  y_l: a vague pep talk.

The model currently likes the loser more. DPO should pull the log-odds
the other way. β = 0.1.
"""
import math


def sigmoid(x):
    return 1.0 / (1.0 + math.exp(-x))


# Sequence log-probs (sum of token log-probs, prompt tokens already masked).
LOG_PI_W, LOG_PI_L = -12.0, -10.0
LOG_REF_W, LOG_REF_L = -11.0, -11.5
BETA = 0.1


def dpo_margin(log_pi_w, log_pi_l, log_ref_w, log_ref_l, beta=BETA):
    """β [ (log π_θ(y_w) − log π_ref(y_w)) − (log π_θ(y_l) − log π_ref(y_l)) ]."""
    return beta * ((log_pi_w - log_ref_w) - (log_pi_l - log_ref_l))


MARGIN = dpo_margin(LOG_PI_W, LOG_PI_L, LOG_REF_W, LOG_REF_L)
LOSS = -math.log(sigmoid(MARGIN))

# Implicit reward used in the derivation: r̂(y) = β log(π_θ(y)/π_ref(y)).
IMPLICIT_W = BETA * (LOG_PI_W - LOG_REF_W)    # 0.1 * (−1.0) = −0.10
IMPLICIT_L = BETA * (LOG_PI_L - LOG_REF_L)    # 0.1 * (1.5)  =  0.15


if __name__ == "__main__":
    # ( −12 − −11 ) − ( −10 − −11.5 ) = (−1) − (1.5) = −2.5
    # margin = 0.1 * −2.5 = −0.25
    assert abs(MARGIN - (-0.25)) < 1e-12
    assert abs(IMPLICIT_W - (-0.10)) < 1e-12
    assert abs(IMPLICIT_L - 0.15) < 1e-12
    assert abs(LOSS - (-math.log(sigmoid(-0.25)))) < 1e-12
    print("dpo loss ok")

Go to the source

Keep going