REASONING FROM SCRATCH · CHAPTER 12
A one-file GRPO loop on the pier-total task. No GPU. No critic. A verifier, a tiny cold start, and a group of eight samples.
Chapter 01’s harbor puzzle is North \(n\), East \(n-d\), West \(2(n-d)\). The total is \(4n-3d\). For \(n=14\), \(d=6\) that is still 38. Here \(n\) lives in \(0..5\) and \(d\) in \(0..2\), so a 25-way softmax over totals 0..24 can represent every answer.
This is not a language model. It is the algorithm. A tiny MLP, a binary verifier, group size 8. The same shape as R1-Zero, plus the cold start full R1 needed so the first groups are not all zeros.
Prompt: the pair \((n, d)\). Action: an integer total. Reward: 1 if the integer equals \(4n-3d\), else 0. For \((5, 2)\) the target is 14. No format tag, no language mix, no RM. That is RLVR at toy scale.
harbor · tiny · loop · measure
A uniform 25-way policy is right about 4% of the time. A group of 8 is then almost always all zeros. Chapter 10 already told you what happens: no gradient. So the file does 80 supervised steps on the correct total first. Accuracy jumps to about 0.57. Then 250 GRPO steps push it to about 0.92. Seed 7, so those are the numbers the check expects.
Each GRPO step: draw \((n,d)\), sample 8 totals, score them, z-score the rewards, take a softmax step scaled by \(\hat{A}/G\). If the group is all right or all wrong, skip. That is the whole of pier_reasoner.py.
From this folder: python pier_reasoner.py. You should see accuracy 0.00 → SFT 0.57 → GRPO 0.92. Change SFT_STEPS to 0 and watch GRPO stall. That is the all-zero lesson, not a bug.
"""A tiny GRPO reasoner for the pier-total task.
Prompt: two integers (n, d). North has n crates. East has n − d.
West has twice East. The total is 4n − 3d.
The policy is a small MLP over one-hot digits. It samples an integer
in 0..24. A verifier returns 1 if the sample equals 4n − 3d.
A few supervised warm-start steps (a tiny cold start) make the first
correct samples likely. GRPO then uses the group mean as the baseline.
No critic. No learned reward model. This file trains in about a second.
"""
import math
import random
N_MAX, D_MAX = 5, 2
N_ANSWERS = 25 # integers 0..24 cover every 4n − 3d here
H = 16
G = 8 # group size
LR = 0.25
SFT_STEPS = 80 # cold start so some rollouts are correct
GRPO_STEPS = 250
SEED = 7
def total(n, d):
"""4n − 3d. Same closed form as the harbor puzzle in chapter 01."""
return 4 * n - 3 * d
def one_hot(i, k):
v = [0.0] * k
v[i] = 1.0
return v
def new_params(rng):
def mat(a, b):
scale = math.sqrt(2.0 / (a + b))
return [[rng.uniform(-scale, scale) for _ in range(b)] for _ in range(a)]
return {
"w1": mat(N_MAX + 1 + D_MAX + 1, H),
"b1": [0.0] * H,
"w2": mat(H, N_ANSWERS),
"b2": [0.0] * N_ANSWERS,
}
def forward(params, n, d):
x = one_hot(n, N_MAX + 1) + one_hot(d, D_MAX + 1)
h = []
for j in range(H):
s = params["b1"][j] + sum(x[i] * params["w1"][i][j] for i in range(len(x)))
h.append(math.tanh(s))
logits = []
for k in range(N_ANSWERS):
s = params["b2"][k] + sum(h[j] * params["w2"][j][k] for j in range(H))
logits.append(s)
m = max(logits)
exps = [math.exp(z - m) for z in logits]
z = sum(exps)
return x, h, [e / z for e in exps]
def sample(probs, rng):
u, acc = rng.random(), 0.0
for i, p in enumerate(probs):
acc += p
if u <= acc:
return i
return len(probs) - 1
def advantages(rewards):
"""Group-relative. All-equal groups (std = 0) give a zero update."""
mu = sum(rewards) / len(rewards)
var = sum((r - mu) ** 2 for r in rewards) / len(rewards)
sd = math.sqrt(var)
if sd < 1e-8:
return [0.0] * len(rewards)
return [(r - mu) / sd for r in rewards]
def sgd_softmax(params, n, d, answer, scale):
"""Raise the sampled class, lower the others. scale is  or +1 (SFT)."""
x, h, probs = forward(params, n, d)
dlogits = [scale * ((1.0 if k == answer else 0.0) - probs[k]) for k in range(N_ANSWERS)]
dh = [0.0] * H
for j in range(H):
dh[j] = (1.0 - h[j] * h[j]) * sum(dlogits[k] * params["w2"][j][k] for k in range(N_ANSWERS))
for j in range(H):
for k in range(N_ANSWERS):
params["w2"][j][k] += LR * h[j] * dlogits[k]
for k in range(N_ANSWERS):
params["b2"][k] += LR * dlogits[k]
for i, xi in enumerate(x):
for j in range(H):
params["w1"][i][j] += LR * xi * dh[j]
for j in range(H):
params["b1"][j] += LR * dh[j]
def draw_prompt(rng):
n = rng.randint(0, N_MAX)
d = rng.randint(0, min(D_MAX, n))
return n, d
def accuracy(params, rng, n_eval=72):
hits = 0
for _ in range(n_eval):
n, d = draw_prompt(rng)
_, _, probs = forward(params, n, d)
pred = max(range(N_ANSWERS), key=lambda k: probs[k])
hits += int(pred == total(n, d))
return hits / n_eval
def train(sft_steps=SFT_STEPS, grpo_steps=GRPO_STEPS, seed=SEED):
rng = random.Random(seed)
params = new_params(rng)
acc0 = accuracy(params, random.Random(0))
# Cold start: a few supervised steps on the correct total.
for _ in range(sft_steps):
n, d = draw_prompt(rng)
sgd_softmax(params, n, d, total(n, d), scale=1.0)
acc_sft = accuracy(params, random.Random(0))
# GRPO: sample a group, score with the verifier, step on Â.
for _ in range(grpo_steps):
n, d = draw_prompt(rng)
_, _, probs = forward(params, n, d)
answers = [sample(probs, rng) for _ in range(G)]
rewards = [1.0 if a == total(n, d) else 0.0 for a in answers]
advs = advantages(rewards)
for a, adv in zip(answers, advs):
if adv != 0.0:
sgd_softmax(params, n, d, a, scale=adv / G)
acc1 = accuracy(params, random.Random(0))
return acc0, acc_sft, acc1
if __name__ == "__main__":
assert total(14, 6) == 38 # the chapter-01 harbor numbers
assert total(5, 2) == 14
acc0, acc_sft, acc1 = train()
print(f"accuracy {acc0:.2f} → SFT {acc_sft:.2f} → GRPO {acc1:.2f}")
assert acc_sft > acc0
assert acc1 > 0.70
print("pier reasoner ok")
Starting GRPO from a uniform policy and blaming the algorithm. Binary rewards plus a large action space yield all-zero groups. Cold-start first.
Calling this “R1-Zero on GSM8K.” The task is \(4n-3d\). R1-Zero did not train on GSM8K. This file is a shape-copy, not a reproduction.
Expecting format tags to appear. The action is an integer. There is no <think> here. Put those tags in a language model, not in this MLP.
North 14, East 8, West 16. 14+8+16 = 38. Same identity.
0.57 then 0.92 (seed 7).
Most groups are all zeros. GRPO barely moves.
There isn’t one. The group mean is the baseline.
sgd_softmax.