THE EVOLVING TRANSFORMER · STOP 04
The modern decoder: RoPE instead of wpe, RMSNorm instead of LayerNorm, SwiGLU instead of GELU, GQA instead of MHA. LLaMA 3 is not weight-tied. The KV cache is why GQA exists.
GPT-2's learned wpe is gone. Position is a rotation of Q and K (RoPE, even/odd pairing, θ = 500,000 for LLaMA 3). LayerNorm becomes RMSNorm. The MLP becomes SwiGLU — three bias-free matrices. Attention becomes GQA. The language-model head is a separate matrix: LLaMA 3 is not weight-tied. Each layer keeps a KV cache so generate can prefill the prompt and then decode one token.
GQA is a filing cabinet with fewer drawers. Thirty-two clerks still ask thirty-two questions, but they share eight card boxes. The cache stores the boxes, not the questions.
RoPE, RMSNorm, SwiGLU and GQA each have a concept home. This stop is the assembled decoder. The modern block → · KV cache and GQA →
| Knob | LLaMA 3 8B |
|---|---|
| layers / Q heads / KV heads / head dim | 32 / 32 / 8 / 128 |
| hidden / vocab | 4096 / 128,256 |
| cache per token (bf16) | 128 KiB |
| weight tying | no |
Teaching batch: hidden 32, 4 query heads, 2 KV heads, length 8, vocab 50.
prefill · append K/V · GQA
GPT-2 block · four swaps · no weight tying
| Tensor | Shape |
|---|---|
| tokens | (2, 8) |
| q | (2, 4, 8, 8) |
| k, v (new) | (2, 2, 8, 8) |
| cache K after prefill | (2, 2, 8, 8) |
| logits | (2, 8, 50) |
RoPE runs on the new keys before they enter the cache. generate prefills the prompt, then feeds one token at a time with start_pos set to the current length.
"""LLaMA-style decoder: pre-norm, RMSNorm, SwiGLU, RoPE, GQA.
LLaMA 3 8B published: 32 layers, 32 query heads, 8 KV heads, head dim 128,
hidden 4096, vocab 128,256, RoPE θ = 500,000. Embeddings are NOT tied.
Teaching default is tiny. A KV cache plus greedy generate live here —
that is the reason GQA exists.
"""
from dataclasses import dataclass
import math
import torch
import torch.nn as nn
import torch.nn.functional as F
@dataclass
class LlamaConfig:
dmodel: int = 32
N: int = 2
H: int = 4
n_kv_heads: int = 2
vocabsize: int = 50
max_seq_len: int = 16
dff: int = 64
norm_eps: float = 1e-5
rope_theta: float = 500_000.0 # LLaMA 3, not the old 10_000
@property
def dk(self):
return self.dmodel // self.H
class RMSNorm(nn.Module):
"""Scale only — no mean subtraction, no bias."""
def __init__(self, dim, eps=1e-5):
super().__init__()
self.eps = eps
self.weight = nn.Parameter(torch.ones(dim))
def forward(self, x):
rms = torch.rsqrt(x.pow(2).mean(dim=-1, keepdim=True) + self.eps)
return x * rms * self.weight
class SwiGLU(nn.Module):
"""gate / up / down, no biases. silu(gate) * up, then down."""
def __init__(self, config):
super().__init__()
self.w_gate = nn.Linear(config.dmodel, config.dff, bias=False)
self.w_up = nn.Linear(config.dmodel, config.dff, bias=False)
self.w_down = nn.Linear(config.dff, config.dmodel, bias=False)
def forward(self, x):
return self.w_down(F.silu(self.w_gate(x)) * self.w_up(x))
class RotaryPositionalEmbedding(nn.Module):
"""Even/odd pairing (Meta-style), not the HF half-split."""
def __init__(self, head_dim, max_seq_len, theta):
super().__init__()
freqs = 1.0 / (theta ** (torch.arange(0, head_dim, 2).float() / head_dim))
pos = torch.arange(max_seq_len).float()
angles = torch.outer(pos, freqs)
self.register_buffer("cos_cached", torch.cos(angles))
self.register_buffer("sin_cached", torch.sin(angles))
def forward(self, x, start_pos=0):
# x: (B, H, S, dk) — rotate each token by its absolute position
B, H, S, dk = x.shape
even, odd = x[..., 0::2], x[..., 1::2]
cos = self.cos_cached[start_pos : start_pos + S].view(1, 1, S, -1)
sin = self.sin_cached[start_pos : start_pos + S].view(1, 1, S, -1)
rot = torch.stack([even * cos - odd * sin, even * sin + odd * cos], dim=-1)
return rot.view(B, H, S, dk)
class KVCache:
"""One layer: append keys and values along the token axis."""
def __init__(self):
self.k = None
self.v = None
def append(self, k_new, v_new):
if self.k is None:
self.k, self.v = k_new, v_new
else:
self.k = torch.cat([self.k, k_new], dim=2)
self.v = torch.cat([self.v, v_new], dim=2)
return self.k, self.v
def repeat_kv(x, n_rep):
"""(B, n_kv, S, d) → (B, H, S, d) by repeating each KV head."""
if n_rep == 1:
return x
B, n_kv, S, d = x.shape
return x.unsqueeze(2).expand(B, n_kv, n_rep, S, d).reshape(B, n_kv * n_rep, S, d)
class GQA(nn.Module):
def __init__(self, config):
super().__init__()
self.H = config.H
self.n_kv = config.n_kv_heads
self.n_rep = config.H // config.n_kv_heads
self.dk = config.dk
self.wq = nn.Linear(config.dmodel, config.H * config.dk, bias=False)
self.wk = nn.Linear(config.dmodel, config.n_kv_heads * config.dk, bias=False)
self.wv = nn.Linear(config.dmodel, config.n_kv_heads * config.dk, bias=False)
self.wo = nn.Linear(config.dmodel, config.dmodel, bias=False)
def forward(self, x, rope, start_pos=0, cache=None):
B, S, _ = x.shape
q = self.wq(x).view(B, S, self.H, self.dk).transpose(1, 2)
k = self.wk(x).view(B, S, self.n_kv, self.dk).transpose(1, 2)
v = self.wv(x).view(B, S, self.n_kv, self.dk).transpose(1, 2)
# rotate before caching so we never re-rotate the past
q, k = rope(q, start_pos), rope(k, start_pos)
if cache is not None:
k, v = cache.append(k, v)
t_all = k.size(2)
k_rep, v_rep = repeat_kv(k, self.n_rep), repeat_kv(v, self.n_rep)
scores = (q @ k_rep.transpose(-1, -2)) / math.sqrt(self.dk)
# causal: new queries sit at the end of the cached sequence
q_pos = torch.arange(start_pos, start_pos + S, device=x.device).view(S, 1)
k_pos = torch.arange(t_all, device=x.device).view(1, t_all)
scores = scores.masked_fill(k_pos > q_pos, float("-inf"))
out = scores.softmax(dim=-1) @ v_rep
return self.wo(out.transpose(1, 2).contiguous().view(B, S, self.H * self.dk))
class Block(nn.Module):
def __init__(self, config):
super().__init__()
self.attn = GQA(config)
self.ff = SwiGLU(config)
self.rms_1 = RMSNorm(config.dmodel, config.norm_eps)
self.rms_2 = RMSNorm(config.dmodel, config.norm_eps)
def forward(self, x, rope, start_pos=0, cache=None):
x = x + self.attn(self.rms_1(x), rope, start_pos, cache)
return x + self.ff(self.rms_2(x))
class Llama(nn.Module):
def __init__(self, config):
super().__init__()
self.config = config
self.wte = nn.Embedding(config.vocabsize, config.dmodel)
self.rope = RotaryPositionalEmbedding(config.dk, config.max_seq_len, config.rope_theta)
self.blocks = nn.ModuleList([Block(config) for _ in range(config.N)])
self.norm = RMSNorm(config.dmodel, config.norm_eps)
# LLaMA 3 is NOT weight-tied — a separate output matrix
self.lm_head = nn.Linear(config.dmodel, config.vocabsize, bias=False)
def forward(self, idx, start_pos=0, caches=None):
x = self.wte(idx)
for i, block in enumerate(self.blocks):
cache = None if caches is None else caches[i]
x = block(x, self.rope, start_pos, cache)
return self.lm_head(self.norm(x))
@torch.no_grad()
def generate(self, idx, max_new_tokens):
"""Prefill the prompt, then decode one token with a KV cache."""
caches = [KVCache() for _ in self.blocks]
logits = self.forward(idx, start_pos=0, caches=caches)
out = idx
for t in range(max_new_tokens):
nxt = logits[:, -1].argmax(dim=-1, keepdim=True)
out = torch.cat([out, nxt], dim=1)
logits = self.forward(nxt, start_pos=out.size(1) - 1, caches=caches)
return out
Test note. Forward (2, 8), assert logits and untied heads, run generate for 3 tokens, and check the cache grows to prompt + new tokens.
Tying lm_head to wte. That was true of GPT-2. LLaMA 3 8B is untied. Tying under-counts the published 8.03B.
RoPE θ = 10,000. LLaMA 2 used 10,000. LLaMA 3 uses 500,000.
Caching keys before RoPE. Rotate by the token's absolute position, then store. Otherwise you re-rotate the whole cache every step.
32 / 8 = 4.
No. The output matrix is separate.
Only the newest query is used. Past queries were consumed in earlier steps.