REASONING FROM SCRATCH · CHAPTER 11

DeepSeek-R1

A short telling of one paper: start from a base model, reward checkable answers, and watch long thoughts appear. Then add a cold start, because pure RL is a messy talker.

~16 min readAIME 15.6% → 71.0%Tested facts

What you'll learn

  • R1-Zero: pure GRPO, rule-based rewards, no reasoning SFT
  • What it trained on (DeepSeek’s math/code/logic prompts — not GSM8K)
  • The four stages of full R1, ending in all-scenario RL
  • Distillation is SFT on teacher traces, and it beat small-model RL

Before you start

Leave the format alone

Most recipes begin by cloning a teacher’s handwriting. R1-Zero did the opposite. Take DeepSeek-V3-base. Sample groups. Score the final answer with a checker and score the wrapper with a regex. Do not show any gold chain of thought. The hope: if the only way to get the reward is to think longer, the model will invent the thinking.

The wrapper they asked for is two literal tags: <think>…</think> around the working, then the answer. Not a placeholder, not a redacted name. Ordinary tokens in the vocabulary.

Pure RL, and what that actually used

The training set is DeepSeek’s own automatically collected math, code, and logic prompts. It is not GSM8K. GSM8K shows up in reproduction notebooks because it is public and small. The paper’s headline is AIME 2024 pass@1: 15.6% on the base model, 71.0% after R1-Zero. Full R1 later reaches 79.8%.

format · long CoT · cold start · V3

MANIM R1: RL for reasoning traces; V3 block unchanged.

Rewards are rule-based: did the boxed answer match, did the code pass, did the tags appear. They tried process reward models and MCTS and dropped both. Response length grew with accuracy. An “aha” style of backtracking showed up in the traces — the model started writing that a first attempt was wrong and trying another. Language mixing (English working on a Chinese prompt, and the reverse) showed up too. That is why full R1 adds a language-consistency reward.

Group size here is \(G = 16\). DeepSeekMath had used 64. Same algorithm, cheaper groups.

Four stages, in order

1. Cold-start SFTa few thousand long, readable traces
2. Reasoning RLGRPO + accuracy + format + language consistency
3. Rejection-sampling SFT~800,000 traces: 600k reasoning + 200k general
4. All-scenario RLhelpfulness and harmlessness, plus reasoning

Stage 4 is not “final SFT.” It is a second RL pass over mixed prompts, and this time a reward model is allowed back in for the subjective ones. R1-Zero is the existence proof that long CoT can emerge from RLVR. Full R1 is the product that talks like a grown-up.

Small students, teacher traces

They distilled into Qwen and Llama by ordinary SFT on about 800k teacher samples. Not logit-level knowledge distillation. The finding that mattered: a small student trained on R1 traces beat a small model trained with RL of its own. If you only have one GPU, steal the traces. That is the honest reading, and it is why the next chapter starts with a cold start.

Mistakes learners make

Replacing the think tag with a sanitizer placeholder. The paper uses <think>. A filter that rewrites the tag breaks every regex.

“R1-Zero trained on GSM8K.” It trained on DeepSeek’s prompts and reported AIME 2024. GSM8K is a convenient reproduction set, not the paper.

Stage 4 as final SFT. The last stage is RL for all scenarios.

“R1 uses a PRM.” They tried PRMs and MCTS. They stopped.

What to remember

  • R1-Zero: V3-base + GRPO + rule-based rewards. No reasoning SFT. AIME 15.6% → 71.0%.
  • Tags are <think> and </think>. Training data is not GSM8K.
  • Full R1: cold-start SFT → reasoning RL → 800k rejection-sampling SFT → all-scenario RL.
  • Distillation was SFT on teacher traces, and it beat small-model RL.

Check yourself

1. What AIME 2024 pass@1 did R1-Zero report, from what base?

71.0%, up from 15.6% on V3-base.

2. Did R1-Zero train on GSM8K?

No. DeepSeek’s own math/code/logic prompts.

3. What are the four stages of full R1?

Cold-start SFT, reasoning RL, rejection-sampling SFT (~800k), all-scenario RL.

4. Write the opening think tag, exactly.

<think>

The code

Facts used on this page, so they cannot drift.

r1_facts.py15.6 → 71.0; 800,000; think tags
"""Checked facts for the DeepSeek-R1 case study.

R1-Zero trained on DeepSeek's own math, code, and logic prompts.
It did *not* train on GSM8K. The headline number is AIME 2024.
Think tags are the literal strings <think> and </think>.
"""

# R1-Zero, AIME 2024 pass@1, from the R1 paper.
AIME_BASE = 15.6          # DeepSeek-V3-base, no reasoning RL
AIME_R1_ZERO = 71.0       # after pure RL
AIME_R1 = 79.8            # full R1 (cold-start + multi-stage)

# Rejection-sampling SFT corpus used by full R1 (stage 3).
SFT_REASONING = 600_000
SFT_GENERAL = 200_000
SFT_TOTAL = SFT_REASONING + SFT_GENERAL   # 800,000

# GRPO group sizes reported in the two papers.
DEEPSEEKMATH_G = 64       # Shao et al., 2024
R1_G = 16

THINK_OPEN = "<think>"
THINK_CLOSE = "</think>"
ANSWER_OPEN = "<answer>"
ANSWER_CLOSE = "</answer>"

# The four stages of full R1, in order.
STAGES = (
    "cold-start SFT",
    "reasoning RL",
    "rejection-sampling SFT",
    "all-scenario RL",
)


if __name__ == "__main__":
    assert abs(AIME_R1_ZERO - AIME_BASE - 55.4) < 1e-9
    assert SFT_TOTAL == 800_000
    assert THINK_OPEN == "<think>" and "redacted" not in THINK_OPEN
    assert STAGES[3] == "all-scenario RL"
    print("r1 facts ok")

Go to the source

Keep going