REASONING FROM SCRATCH · CHAPTER 12

Build your first
reasoning model

A one-file GRPO loop on the pier-total task. No GPU. No critic. A verifier, a tiny cold start, and a group of eight samples.

~15 min readRuns in about a secondTested Python

What you'll learn

  • How to write a verifier for \(4n - 3d\)
  • Why a uniform policy never learns (all-zero groups)
  • A tiny SFT cold start, then GRPO, in one file
  • The numbers this run actually hits: 0.00 → 0.57 → 0.92

Before you start

Shrink the harbor until it fits in RAM

Chapter 01’s harbor puzzle is North \(n\), East \(n-d\), West \(2(n-d)\). The total is \(4n-3d\). For \(n=14\), \(d=6\) that is still 38. Here \(n\) lives in \(0..5\) and \(d\) in \(0..2\), so a 25-way softmax over totals 0..24 can represent every answer.

This is not a language model. It is the algorithm. A tiny MLP, a binary verifier, group size 8. The same shape as R1-Zero, plus the cold start full R1 needed so the first groups are not all zeros.

A checker instead of a reward model

Prompt: the pair \((n, d)\). Action: an integer total. Reward: 1 if the integer equals \(4n-3d\), else 0. For \((5, 2)\) the target is 14. No format tag, no language mix, no RM. That is RLVR at toy scale.

harbor · tiny · loop · measure

MANIM Build path: tiny model, verifiers, measure solve rate.

A uniform 25-way policy is right about 4% of the time. A group of 8 is then almost always all zeros. Chapter 10 already told you what happens: no gradient. So the file does 80 supervised steps on the correct total first. Accuracy jumps to about 0.57. Then 250 GRPO steps push it to about 0.92. Seed 7, so those are the numbers the check expects.

Cold start, then group-relative updates

Random MLPaccuracy 0.00 on the eval seed
80 SFT stepscross-entropy on the true total
250 GRPO stepsG = 8, group mean baseline, no critic

Each GRPO step: draw \((n,d)\), sample 8 totals, score them, z-score the rewards, take a softmax step scaled by \(\hat{A}/G\). If the group is all right or all wrong, skip. That is the whole of pier_reasoner.py.

One file, one second

From this folder: python pier_reasoner.py. You should see accuracy 0.00 → SFT 0.57 → GRPO 0.92. Change SFT_STEPS to 0 and watch GRPO stall. That is the all-zero lesson, not a bug.

pier_reasoner.py0.00 → 0.57 → 0.92 on seed 7; total(14, 6) = 38
"""A tiny GRPO reasoner for the pier-total task.

Prompt: two integers (n, d). North has n crates. East has n − d.
West has twice East. The total is 4n − 3d.

The policy is a small MLP over one-hot digits. It samples an integer
in 0..24. A verifier returns 1 if the sample equals 4n − 3d.

A few supervised warm-start steps (a tiny cold start) make the first
correct samples likely. GRPO then uses the group mean as the baseline.
No critic. No learned reward model. This file trains in about a second.
"""
import math
import random

N_MAX, D_MAX = 5, 2
N_ANSWERS = 25                       # integers 0..24 cover every 4n − 3d here
H = 16
G = 8                                # group size
LR = 0.25
SFT_STEPS = 80                       # cold start so some rollouts are correct
GRPO_STEPS = 250
SEED = 7


def total(n, d):
    """4n − 3d. Same closed form as the harbor puzzle in chapter 01."""
    return 4 * n - 3 * d


def one_hot(i, k):
    v = [0.0] * k
    v[i] = 1.0
    return v


def new_params(rng):
    def mat(a, b):
        scale = math.sqrt(2.0 / (a + b))
        return [[rng.uniform(-scale, scale) for _ in range(b)] for _ in range(a)]
    return {
        "w1": mat(N_MAX + 1 + D_MAX + 1, H),
        "b1": [0.0] * H,
        "w2": mat(H, N_ANSWERS),
        "b2": [0.0] * N_ANSWERS,
    }


def forward(params, n, d):
    x = one_hot(n, N_MAX + 1) + one_hot(d, D_MAX + 1)
    h = []
    for j in range(H):
        s = params["b1"][j] + sum(x[i] * params["w1"][i][j] for i in range(len(x)))
        h.append(math.tanh(s))
    logits = []
    for k in range(N_ANSWERS):
        s = params["b2"][k] + sum(h[j] * params["w2"][j][k] for j in range(H))
        logits.append(s)
    m = max(logits)
    exps = [math.exp(z - m) for z in logits]
    z = sum(exps)
    return x, h, [e / z for e in exps]


def sample(probs, rng):
    u, acc = rng.random(), 0.0
    for i, p in enumerate(probs):
        acc += p
        if u <= acc:
            return i
    return len(probs) - 1


def advantages(rewards):
    """Group-relative. All-equal groups (std = 0) give a zero update."""
    mu = sum(rewards) / len(rewards)
    var = sum((r - mu) ** 2 for r in rewards) / len(rewards)
    sd = math.sqrt(var)
    if sd < 1e-8:
        return [0.0] * len(rewards)
    return [(r - mu) / sd for r in rewards]


def sgd_softmax(params, n, d, answer, scale):
    """Raise the sampled class, lower the others. scale is  or +1 (SFT)."""
    x, h, probs = forward(params, n, d)
    dlogits = [scale * ((1.0 if k == answer else 0.0) - probs[k]) for k in range(N_ANSWERS)]
    dh = [0.0] * H
    for j in range(H):
        dh[j] = (1.0 - h[j] * h[j]) * sum(dlogits[k] * params["w2"][j][k] for k in range(N_ANSWERS))
    for j in range(H):
        for k in range(N_ANSWERS):
            params["w2"][j][k] += LR * h[j] * dlogits[k]
    for k in range(N_ANSWERS):
        params["b2"][k] += LR * dlogits[k]
    for i, xi in enumerate(x):
        for j in range(H):
            params["w1"][i][j] += LR * xi * dh[j]
    for j in range(H):
        params["b1"][j] += LR * dh[j]


def draw_prompt(rng):
    n = rng.randint(0, N_MAX)
    d = rng.randint(0, min(D_MAX, n))
    return n, d


def accuracy(params, rng, n_eval=72):
    hits = 0
    for _ in range(n_eval):
        n, d = draw_prompt(rng)
        _, _, probs = forward(params, n, d)
        pred = max(range(N_ANSWERS), key=lambda k: probs[k])
        hits += int(pred == total(n, d))
    return hits / n_eval


def train(sft_steps=SFT_STEPS, grpo_steps=GRPO_STEPS, seed=SEED):
    rng = random.Random(seed)
    params = new_params(rng)
    acc0 = accuracy(params, random.Random(0))

    # Cold start: a few supervised steps on the correct total.
    for _ in range(sft_steps):
        n, d = draw_prompt(rng)
        sgd_softmax(params, n, d, total(n, d), scale=1.0)
    acc_sft = accuracy(params, random.Random(0))

    # GRPO: sample a group, score with the verifier, step on Â.
    for _ in range(grpo_steps):
        n, d = draw_prompt(rng)
        _, _, probs = forward(params, n, d)
        answers = [sample(probs, rng) for _ in range(G)]
        rewards = [1.0 if a == total(n, d) else 0.0 for a in answers]
        advs = advantages(rewards)
        for a, adv in zip(answers, advs):
            if adv != 0.0:
                sgd_softmax(params, n, d, a, scale=adv / G)
    acc1 = accuracy(params, random.Random(0))
    return acc0, acc_sft, acc1


if __name__ == "__main__":
    assert total(14, 6) == 38          # the chapter-01 harbor numbers
    assert total(5, 2) == 14
    acc0, acc_sft, acc1 = train()
    print(f"accuracy {acc0:.2f} → SFT {acc_sft:.2f} → GRPO {acc1:.2f}")
    assert acc_sft > acc0
    assert acc1 > 0.70
    print("pier reasoner ok")

Mistakes learners make

Starting GRPO from a uniform policy and blaming the algorithm. Binary rewards plus a large action space yield all-zero groups. Cold-start first.

Calling this “R1-Zero on GSM8K.” The task is \(4n-3d\). R1-Zero did not train on GSM8K. This file is a shape-copy, not a reproduction.

Expecting format tags to appear. The action is an integer. There is no <think> here. Put those tags in a language model, not in this MLP.

What to remember

  • The pier total is \(4n-3d\). Harbor (14, 6) is still 38. Toy range: n≤5, d≤2.
  • Verifier reward, group of 8, no critic. That is GRPO + RLVR.
  • Seed 7: accuracy 0.00 → SFT 0.57 → GRPO 0.92.
  • The series started with a frozen model writing 38. It ends with a trained policy that computes 38’s little cousins.

Check yourself

1. Why is \(4\cdot 14 - 3\cdot 6 = 38\)

North 14, East 8, West 16. 14+8+16 = 38. Same identity.

2. What accuracy does this file print after SFT, after GRPO?

0.57 then 0.92 (seed 7).

3. What happens if you set SFT_STEPS = 0?

Most groups are all zeros. GRPO barely moves.

4. Where is the value head?

There isn’t one. The group mean is the baseline.

Go to the source

Keep going