REASONING FROM SCRATCH · CHAPTER 10

GRPO &
verifiable rewards

Drop the value head. Sample a group. Use the group mean as the baseline. The reward can still be a model — or a checker. Those are different knobs.

~20 min readDeepSeekMath 2024Tested Python

What you'll learn

  • GRPO’s origin: DeepSeekMath (Shao et al., February 2024), not R1
  • What is dropped: the critic / value head, not “the reward”
  • Group z-score advantages, all-zero groups, and the k3 KL estimator
  • RLVR: a verifier as the reward, the same checker you met in chapter 02

Before you start

Four slips, one mean

Back to the harbor puzzle. Sample four totals: 38, 28, 38, 42. A verifier marks them 1, 0, 1, 0. The group mean is 0.5. The two correct slips sit above the mean; the two wrong ones sit below. That mean is a Monte-Carlo estimate of \(V(x)\). You do not need a value network to compute it.

GRPO is PPO with the critic replaced by the group. It still clips. It still has a KL to a reference, in the loss (DeepSeekMath uses the unbiased k3 estimator). The reward source is a separate choice: a reward model, or a program.

Count the models, carefully

StackNetworksWhat scores the reply
PPO-RLHF4 — policy, reference, RM, valueReward model
GRPO + RM3 — policy, reference, RMReward model (no critic)
GRPO + verifier2 — policy, referenceA program (RLVR)

A common slip is “GRPO drops the reward model.” DeepSeekMath still had rewards; it dropped the value head. R1-Zero then chose a rule-based reward (accuracy + format), which is RLVR. Full R1 later put reward models back in for helpfulness. Algorithm and reward source are orthogonal.

slips · group · RLVR · no critic

MANIM GRPO: group-relative advantages; RLVR uses verifiers.

History: Shao et al., DeepSeekMath, February 2024, introduced GRPO with group size \(G = 64\). R1 (January 2025) popularized it and used \(G = 16\). Open recipes often use 8–16.

Mean, then scale

subtract the group mean, divide by the group std. On (1,0,1,0) that is exactly +1, −1, +1, −1.

\(\mu_G = 0.5\), \(\sigma_G = 0.5\). The mean is the baseline (chapter 06). The std is a scale: a rare success in a hard group becomes a large \(+\hat{A}\). A rubric scored 0–6 z-scores the same way — the units cancel. Then PPO’s clip runs on a token-level ratio, with a \(1/|o_i|\) length average in the real DeepSeekMath objective. A sequence-level ratio \(\pi(Y\mid X)/\pi_{\mathrm{old}}(Y\mid X)\) is numerically dead for long chains. Do not write the pseudocode that way.

The k3 KL term, per token: \(\frac{\pi_{\mathrm{ref}}}{\pi} - \log\frac{\pi_{\mathrm{ref}}}{\pi} - 1\). For \(\pi=0.20\), \(\pi_{\mathrm{ref}}=0.25\), that is 0.0269. Later recipes (DAPO and friends) sometimes drop KL. The original does not.

When the group has no contrast

All four slips wrong: rewards (0,0,0,0). Std is 0. Advantages are 0. No update. The “least bad” wrong answer does not get a boost under binary rewards. All four correct: same story. That is why a tiny random policy never learns the pier total in chapter 12 — almost every group is all zeros until a cold start creates some hits.

A rare success in a group of failures is amplified by the z-score. That is useful on hard prompts and dangerous if the success was a format fluke. Curriculum (easier prompts first) is how you keep mixed groups around.

The chapter-02 verifier, now as a reward

RL with verifiable rewards: \(r = \mathbf{1}[\mathrm{verify}(x,y)]\), plus maybe a format bonus for wrapping the working in <think> tags. Math match, unit tests, a compiler. The same checker can pick among samples at test time (chapter 02) and grade rollouts in training. That is the double duty.

Outcome-only rewards are unfaithful: a right total with a wrong East still scores 1. Process rewards try to fix that and are expensive to label. R1 tried them and walked away. Format hacking (valid tags, wrong math) is the new length hacking.

Length bias, Dr. GRPO, DAPO

Averaging the token surrogate by \(1/|o_i|\) makes longer traces cheaper per token. Models then ramble. Dr. GRPO and DAPO change the normalization and sometimes drop KL. Mention them; do not treat DeepSeekMath as those later patches.

Mistakes learners make

“GRPO arrived with R1 in 2025.” DeepSeekMath, February 2024.

“GRPO drops the reward model.” It drops the critic. Reward can still be an RM. RLVR is the special case that uses a verifier instead.

Reversed model counts (hero stats that say GRPO has 4 and PPO has 3). PPO is 4. GRPO+RM is 3. GRPO+verifier is 2.

An all-zero group boosts the “least bad” reply. Std is 0. Advantages are 0.

What to remember

  • GRPO: DeepSeekMath 2024. Critic gone. Reward optional as an RM or a verifier.
  • Harbor group (1,0,1,0) → advantages (+1, −1, +1, −1). All-zero → no update.
  • PPO 4 models, GRPO+RM 3, GRPO+verifier 2.
  • KL lives in the loss (k3). Token-level ratios, not one sequence ratio.

Check yourself

1. Who introduced GRPO, and when?

Shao et al., DeepSeekMath, February 2024.

2. Rewards (1,0,1,0). What are \(\mu\), \(\sigma\), and the four advantages?

0.5, 0.5, and +1, −1, +1, −1.

3. Does GRPO require dropping the reward model?

No. It requires dropping the value head. RLVR is the extra choice to use a checker.

4. What happens if every sample in the group is wrong?

Std = 0, advantages = 0, no gradient.

The code

grpo_advantage.py = ±1; all-zero → 0; k3 on (0.20, 0.25)
"""GRPO group advantages on the harbor crate puzzle.

Four sampled totals: two correct (38), two wrong (28 and 42).
Binary verifier rewards. The group mean is a Monte-Carlo baseline.
GRPO does *not* drop the reward. It drops the learned critic. A
programmatic verifier can still score each completion.
"""
import math


REWARDS = (1.0, 0.0, 1.0, 0.0)   # 38, 28, 38, 42
EPS = 1e-8                       # stabilizer when the group std is 0


def group_mean(rewards):
    return sum(rewards) / len(rewards)


def group_std(rewards):
    """Population std of the group. Many GRPO recipes use this."""
    mu = group_mean(rewards)
    var = sum((r - mu) ** 2 for r in rewards) / len(rewards)
    return math.sqrt(var)


def advantages(rewards, eps=EPS):
    """Â_i = (r_i − mean) / (std + ε). All-equal groups become zeros."""
    mu, sd = group_mean(rewards), group_std(rewards)
    denom = sd if sd > eps else eps
    return tuple((r - mu) / denom for r in rewards)


MU = group_mean(REWARDS)          # 0.5
STD = group_std(REWARDS)          # 0.5
ADV = advantages(REWARDS)         # (+1, −1, +1, −1)

# Edge case from test-time / RLVR practice: every sample is wrong.
ZERO_GROUP = (0.0, 0.0, 0.0, 0.0)
ZERO_ADV = advantages(ZERO_GROUP)  # near 0, 0, 0, 0


# Model counts. Reward source is independent of the critic.
PPO_MODELS = 4          # policy, reference, reward model, value head
GRPO_WITH_RM = 3        # policy, reference, reward model  (no critic)
GRPO_WITH_VERIFIER = 2  # policy, reference  (reward is a checker)


def k3_kl(pi, pi_ref):
    """Unbiased 'k3' KL estimator used in DeepSeekMath:  π_ref/π − log(π_ref/π) − 1."""
    ratio = pi_ref / pi
    return ratio - math.log(ratio) - 1.0


# One token: π = 0.20, π_ref = 0.25.
K3 = k3_kl(0.20, 0.25)


if __name__ == "__main__":
    assert abs(MU - 0.5) < 1e-12
    assert abs(STD - 0.5) < 1e-12
    assert all(abs(a - t) < 1e-8 for a, t in zip(ADV, (1.0, -1.0, 1.0, -1.0)))
    assert all(abs(a) < 1e-6 for a in ZERO_ADV)
    assert PPO_MODELS == 4 and GRPO_WITH_RM == 3 and GRPO_WITH_VERIFIER == 2
    # k3(0.20, 0.25) = 1.25 − log(1.25) − 1 ≈ 0.02686
    assert abs(K3 - (1.25 - math.log(1.25) - 1.0)) < 1e-12
    print("grpo advantage ok")

Go to the source

Keep going