REASONING FROM SCRATCH · CHAPTER 11
A short telling of one paper: start from a base model, reward checkable answers, and watch long thoughts appear. Then add a cold start, because pure RL is a messy talker.
Most recipes begin by cloning a teacher’s handwriting. R1-Zero did the opposite. Take DeepSeek-V3-base. Sample groups. Score the final answer with a checker and score the wrapper with a regex. Do not show any gold chain of thought. The hope: if the only way to get the reward is to think longer, the model will invent the thinking.
The wrapper they asked for is two literal tags: <think>…</think> around the working, then the answer. Not a placeholder, not a redacted name. Ordinary tokens in the vocabulary.
The training set is DeepSeek’s own automatically collected math, code, and logic prompts. It is not GSM8K. GSM8K shows up in reproduction notebooks because it is public and small. The paper’s headline is AIME 2024 pass@1: 15.6% on the base model, 71.0% after R1-Zero. Full R1 later reaches 79.8%.
format · long CoT · cold start · V3
Rewards are rule-based: did the boxed answer match, did the code pass, did the tags appear. They tried process reward models and MCTS and dropped both. Response length grew with accuracy. An “aha” style of backtracking showed up in the traces — the model started writing that a first attempt was wrong and trying another. Language mixing (English working on a Chinese prompt, and the reverse) showed up too. That is why full R1 adds a language-consistency reward.
Group size here is \(G = 16\). DeepSeekMath had used 64. Same algorithm, cheaper groups.
Stage 4 is not “final SFT.” It is a second RL pass over mixed prompts, and this time a reward model is allowed back in for the subjective ones. R1-Zero is the existence proof that long CoT can emerge from RLVR. Full R1 is the product that talks like a grown-up.
They distilled into Qwen and Llama by ordinary SFT on about 800k teacher samples. Not logit-level knowledge distillation. The finding that mattered: a small student trained on R1 traces beat a small model trained with RL of its own. If you only have one GPU, steal the traces. That is the honest reading, and it is why the next chapter starts with a cold start.
Replacing the think tag with a sanitizer placeholder. The paper uses <think>. A filter that rewrites the tag breaks every regex.
“R1-Zero trained on GSM8K.” It trained on DeepSeek’s prompts and reported AIME 2024. GSM8K is a convenient reproduction set, not the paper.
Stage 4 as final SFT. The last stage is RL for all scenarios.
“R1 uses a PRM.” They tried PRMs and MCTS. They stopped.
<think> and </think>. Training data is not GSM8K.71.0%, up from 15.6% on V3-base.
No. DeepSeek’s own math/code/logic prompts.
Cold-start SFT, reasoning RL, rejection-sampling SFT (~800k), all-scenario RL.
<think>
Facts used on this page, so they cannot drift.
"""Checked facts for the DeepSeek-R1 case study.
R1-Zero trained on DeepSeek's own math, code, and logic prompts.
It did *not* train on GSM8K. The headline number is AIME 2024.
Think tags are the literal strings <think> and </think>.
"""
# R1-Zero, AIME 2024 pass@1, from the R1 paper.
AIME_BASE = 15.6 # DeepSeek-V3-base, no reasoning RL
AIME_R1_ZERO = 71.0 # after pure RL
AIME_R1 = 79.8 # full R1 (cold-start + multi-stage)
# Rejection-sampling SFT corpus used by full R1 (stage 3).
SFT_REASONING = 600_000
SFT_GENERAL = 200_000
SFT_TOTAL = SFT_REASONING + SFT_GENERAL # 800,000
# GRPO group sizes reported in the two papers.
DEEPSEEKMATH_G = 64 # Shao et al., 2024
R1_G = 16
THINK_OPEN = "<think>"
THINK_CLOSE = "</think>"
ANSWER_OPEN = "<answer>"
ANSWER_CLOSE = "</answer>"
# The four stages of full R1, in order.
STAGES = (
"cold-start SFT",
"reasoning RL",
"rejection-sampling SFT",
"all-scenario RL",
)
if __name__ == "__main__":
assert abs(AIME_R1_ZERO - AIME_BASE - 55.4) < 1e-9
assert SFT_TOTAL == 800_000
assert THINK_OPEN == "<think>" and "redacted" not in THINK_OPEN
assert STAGES[3] == "all-scenario RL"
print("r1 facts ok")