Beyond the Forward Pass

Decoding, inference, and serving.

01

Decoding strategies

Greedy, temperature, top-k, top-p, and beam.

02

Life of a prompt

Paged attention, continuous batching, chunked prefill.

03

Number formats & quantization

FP16 to FP8 to INT8 to INT4.

04

Speculative decoding

A cheap draft, a careful check.

Inspired by the work of Umar Jamil, Mohit Iyyer.