Reasoning from Scratch

RL, alignment, and reasoning models — starting with reasoning that needs no training.

Reasoning without training

01

In-context learning & chain of thought

Reasoning that needs no extra training.

02

Test-time search & verifiers

Voting, best-of-N, process rewards, budget forcing.

RL foundations

03

RL vocabulary & bandits

Actions, rewards, and a one-armed lesson.

04

MDPs & Bellman

States, values, and the backup equation.

05

Learning values from experience

DP, Monte Carlo, TD, and Q-learning.

06

Policy gradients & REINFORCE

Push up the actions that paid off.

07

PPO

A clipped update, with TRPO and GAE as deep dives.

Learning from preferences

08

RLHF end to end

Preferences, a reward model, PPO with a KL leash.

09

DPO

Alignment without a reward model or an RL loop.

Training reasoners

10

GRPO & verifiable rewards

Group-relative advantages and a checker as the reward.

11

DeepSeek-R1

A case study of a model trained to think.

12

Build your first reasoning model

A small, runnable capstone.

Inspired by the work of Vizuara, Mohit Iyyer, Umar Jamil.