Decoding, inference, and serving.
Greedy, temperature, top-k, top-p, and beam.
Paged attention, continuous batching, chunked prefill.
FP16 to FP8 to INT8 to INT4.
A cheap draft, a careful check.
Inspired by the work of Umar Jamil, Mohit Iyyer.