THE EVOLVING TRANSFORMER · STOP 04

LLaMA

The modern decoder: RoPE instead of wpe, RMSNorm instead of LayerNorm, SwiGLU instead of GELU, GQA instead of MHA. LLaMA 3 is not weight-tied. The KV cache is why GQA exists.

~14 min readGQA + KV cacheNot weight-tied

What you'll learn

  • Pre-norm + RMSNorm + SwiGLU + RoPE (θ = 500,000)
  • GQA: 32 query heads share 8 KV heads on LLaMA 3 8B
  • A KV cache, and generate that prefills then decodes
  • Why the head is not tied on LLaMA 3

Before you start

Four swaps, and a cache

GPT-2's learned wpe is gone. Position is a rotation of Q and K (RoPE, even/odd pairing, θ = 500,000 for LLaMA 3). LayerNorm becomes RMSNorm. The MLP becomes SwiGLU — three bias-free matrices. Attention becomes GQA. The language-model head is a separate matrix: LLaMA 3 is not weight-tied. Each layer keeps a KV cache so generate can prefill the prompt and then decode one token.

GQA is a filing cabinet with fewer drawers. Thirty-two clerks still ask thirty-two questions, but they share eight card boxes. The cache stores the boxes, not the questions.

RoPE, RMSNorm, SwiGLU and GQA each have a concept home. This stop is the assembled decoder. The modern block → · KV cache and GQA →

LLaMA 3 8B, then a tiny batch

KnobLLaMA 3 8B
layers / Q heads / KV heads / head dim32 / 32 / 8 / 128
hidden / vocab4096 / 128,256
cache per token (bf16)128 KiB
weight tyingno

Teaching batch: hidden 32, 4 query heads, 2 KV heads, length 8, vocab 50.

prefill · append K/V · GQA

MANIM Prefill cache; decode appends K/V; GQA shrinks the box.

GPT-2 block · four swaps · no weight tying

MANIM Four block swaps: RMSNorm, RoPE, SwiGLU, GQA.
TensorShape
tokens(2, 8)
q(2, 4, 8, 8)
k, v (new)(2, 2, 8, 8)
cache K after prefill(2, 2, 8, 8)
logits(2, 8, 50)

Rotate, then cache

RoPE runs on the new keys before they enter the cache. generate prefills the prompt, then feeds one token at a time with start_pos set to the current length.

llama.pytiny forward + cached generate
"""LLaMA-style decoder: pre-norm, RMSNorm, SwiGLU, RoPE, GQA.

LLaMA 3 8B published: 32 layers, 32 query heads, 8 KV heads, head dim 128,
hidden 4096, vocab 128,256, RoPE θ = 500,000. Embeddings are NOT tied.

Teaching default is tiny. A KV cache plus greedy generate live here —
that is the reason GQA exists.
"""

from dataclasses import dataclass
import math

import torch
import torch.nn as nn
import torch.nn.functional as F


@dataclass
class LlamaConfig:
    dmodel: int = 32
    N: int = 2
    H: int = 4
    n_kv_heads: int = 2
    vocabsize: int = 50
    max_seq_len: int = 16
    dff: int = 64
    norm_eps: float = 1e-5
    rope_theta: float = 500_000.0  # LLaMA 3, not the old 10_000

    @property
    def dk(self):
        return self.dmodel // self.H


class RMSNorm(nn.Module):
    """Scale only — no mean subtraction, no bias."""

    def __init__(self, dim, eps=1e-5):
        super().__init__()
        self.eps = eps
        self.weight = nn.Parameter(torch.ones(dim))

    def forward(self, x):
        rms = torch.rsqrt(x.pow(2).mean(dim=-1, keepdim=True) + self.eps)
        return x * rms * self.weight


class SwiGLU(nn.Module):
    """gate / up / down, no biases. silu(gate) * up, then down."""

    def __init__(self, config):
        super().__init__()
        self.w_gate = nn.Linear(config.dmodel, config.dff, bias=False)
        self.w_up = nn.Linear(config.dmodel, config.dff, bias=False)
        self.w_down = nn.Linear(config.dff, config.dmodel, bias=False)

    def forward(self, x):
        return self.w_down(F.silu(self.w_gate(x)) * self.w_up(x))


class RotaryPositionalEmbedding(nn.Module):
    """Even/odd pairing (Meta-style), not the HF half-split."""

    def __init__(self, head_dim, max_seq_len, theta):
        super().__init__()
        freqs = 1.0 / (theta ** (torch.arange(0, head_dim, 2).float() / head_dim))
        pos = torch.arange(max_seq_len).float()
        angles = torch.outer(pos, freqs)
        self.register_buffer("cos_cached", torch.cos(angles))
        self.register_buffer("sin_cached", torch.sin(angles))

    def forward(self, x, start_pos=0):
        # x: (B, H, S, dk) — rotate each token by its absolute position
        B, H, S, dk = x.shape
        even, odd = x[..., 0::2], x[..., 1::2]
        cos = self.cos_cached[start_pos : start_pos + S].view(1, 1, S, -1)
        sin = self.sin_cached[start_pos : start_pos + S].view(1, 1, S, -1)
        rot = torch.stack([even * cos - odd * sin, even * sin + odd * cos], dim=-1)
        return rot.view(B, H, S, dk)


class KVCache:
    """One layer: append keys and values along the token axis."""

    def __init__(self):
        self.k = None
        self.v = None

    def append(self, k_new, v_new):
        if self.k is None:
            self.k, self.v = k_new, v_new
        else:
            self.k = torch.cat([self.k, k_new], dim=2)
            self.v = torch.cat([self.v, v_new], dim=2)
        return self.k, self.v


def repeat_kv(x, n_rep):
    """(B, n_kv, S, d) → (B, H, S, d) by repeating each KV head."""
    if n_rep == 1:
        return x
    B, n_kv, S, d = x.shape
    return x.unsqueeze(2).expand(B, n_kv, n_rep, S, d).reshape(B, n_kv * n_rep, S, d)


class GQA(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.H = config.H
        self.n_kv = config.n_kv_heads
        self.n_rep = config.H // config.n_kv_heads
        self.dk = config.dk
        self.wq = nn.Linear(config.dmodel, config.H * config.dk, bias=False)
        self.wk = nn.Linear(config.dmodel, config.n_kv_heads * config.dk, bias=False)
        self.wv = nn.Linear(config.dmodel, config.n_kv_heads * config.dk, bias=False)
        self.wo = nn.Linear(config.dmodel, config.dmodel, bias=False)

    def forward(self, x, rope, start_pos=0, cache=None):
        B, S, _ = x.shape
        q = self.wq(x).view(B, S, self.H, self.dk).transpose(1, 2)
        k = self.wk(x).view(B, S, self.n_kv, self.dk).transpose(1, 2)
        v = self.wv(x).view(B, S, self.n_kv, self.dk).transpose(1, 2)
        # rotate before caching so we never re-rotate the past
        q, k = rope(q, start_pos), rope(k, start_pos)
        if cache is not None:
            k, v = cache.append(k, v)
        t_all = k.size(2)
        k_rep, v_rep = repeat_kv(k, self.n_rep), repeat_kv(v, self.n_rep)
        scores = (q @ k_rep.transpose(-1, -2)) / math.sqrt(self.dk)
        # causal: new queries sit at the end of the cached sequence
        q_pos = torch.arange(start_pos, start_pos + S, device=x.device).view(S, 1)
        k_pos = torch.arange(t_all, device=x.device).view(1, t_all)
        scores = scores.masked_fill(k_pos > q_pos, float("-inf"))
        out = scores.softmax(dim=-1) @ v_rep
        return self.wo(out.transpose(1, 2).contiguous().view(B, S, self.H * self.dk))


class Block(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.attn = GQA(config)
        self.ff = SwiGLU(config)
        self.rms_1 = RMSNorm(config.dmodel, config.norm_eps)
        self.rms_2 = RMSNorm(config.dmodel, config.norm_eps)

    def forward(self, x, rope, start_pos=0, cache=None):
        x = x + self.attn(self.rms_1(x), rope, start_pos, cache)
        return x + self.ff(self.rms_2(x))


class Llama(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config
        self.wte = nn.Embedding(config.vocabsize, config.dmodel)
        self.rope = RotaryPositionalEmbedding(config.dk, config.max_seq_len, config.rope_theta)
        self.blocks = nn.ModuleList([Block(config) for _ in range(config.N)])
        self.norm = RMSNorm(config.dmodel, config.norm_eps)
        # LLaMA 3 is NOT weight-tied — a separate output matrix
        self.lm_head = nn.Linear(config.dmodel, config.vocabsize, bias=False)

    def forward(self, idx, start_pos=0, caches=None):
        x = self.wte(idx)
        for i, block in enumerate(self.blocks):
            cache = None if caches is None else caches[i]
            x = block(x, self.rope, start_pos, cache)
        return self.lm_head(self.norm(x))

    @torch.no_grad()
    def generate(self, idx, max_new_tokens):
        """Prefill the prompt, then decode one token with a KV cache."""
        caches = [KVCache() for _ in self.blocks]
        logits = self.forward(idx, start_pos=0, caches=caches)
        out = idx
        for t in range(max_new_tokens):
            nxt = logits[:, -1].argmax(dim=-1, keepdim=True)
            out = torch.cat([out, nxt], dim=1)
            logits = self.forward(nxt, start_pos=out.size(1) - 1, caches=caches)
        return out

Test note. Forward (2, 8), assert logits and untied heads, run generate for 3 tokens, and check the cache grows to prompt + new tokens.

Mistakes this stop used to make

Tying lm_head to wte. That was true of GPT-2. LLaMA 3 8B is untied. Tying under-counts the published 8.03B.

RoPE θ = 10,000. LLaMA 2 used 10,000. LLaMA 3 uses 500,000.

Caching keys before RoPE. Rotate by the token's absolute position, then store. Otherwise you re-rotate the whole cache every step.

What to remember

  • LLaMA 3 8B: 32 / 32 / 8 / 128, hidden 4096, vocab 128,256, not weight-tied.
  • RoPE θ = 500,000. RMSNorm. SwiGLU. GQA.
  • The KV cache stores rotated K and V. Generate prefills, then decodes.

Check yourself

1. How many query heads share each KV head on LLaMA 3 8B?

32 / 8 = 4.

2. Is LLaMA 3 weight-tied?

No. The output matrix is separate.

3. Why cache K and V but not Q?

Only the newest query is used. Past queries were consumed in earlier steps.

Go to the source

Keep going