THE TRANSFORMER DEEP CUTS · CHAPTER 04
Attention does not know order. We have to put position in by hand. The modern way is to rotate the queries and keys.
Ash met oak. and Oak met ash. use the same three words. Attention on the raw embeddings would produce the same set of scores, just permuted. The meaning is not the same. Something has to mark who met whom.
Attention is permutation equivariant, not invariant: permute the input and the output permutes with it. It is still blind to which position was first. Position encodings break that symmetry.
Learned absolute embeddings (GPT-2, BERT) store one vector per slot, up to a max length. GPT-2 has 1,024 of them. Train at 1,024, ask for 2,048, and the extra slots have never been seen. The model does not extrapolate.
Sinusoidal encodings, from the 2017 paper, use a ladder of frequencies instead of a table:
pair \(i\) oscillates with frequency \(10000^{-2i/d}\). Low \(i\) is fast. High \(i\) is slow.
GPT-2 itself does not use this formula — it uses learned slots. If you still evaluate the sinusoid at GPT-2's width \(d = 768\), the first frequencies are:
| pair \(i\) | \(10000^{2i/768}\) | \(\omega_i\) |
|---|---|---|
| 0 | 1 | 1 |
| 1 | 1.024 | 0.976 |
| 2 | 1.049 | 0.953 |
| 10 | 1.271 | 0.787 |
| 50 | 3.32 | 0.301 |
| 100 | 11.0 | 0.091 |
Pair 50 still moves. It is not flat. Older tables that put \(\omega_1 = 0.0316\) and called pair 50 "completely flat" used the wrong exponent.
ω=1 → ladder → i=50 moves → not flat
ALiBi leaves \(Q\) and \(K\) alone and adds a bias to the scores: \(m \times (j - i)\), with a negative slope \(m\) per head. Nearby keys get a milder penalty. Far keys get a harsher one.
Four heads use slopes \(2^{-8/4}, 2^{-16/4}, \ldots\). Head 0 has slope \(0.25\). If a raw dot is 2.1 and the key sits 3 steps behind, the logit becomes \(2.1 + 0.25 \times (-3) = \) 1.35.
2.1 → −3 → 0.25 → 1.35
Because the bias is a function of the offset, ALiBi can step a little past the training length. It does not give you a free 8× context.
RoPE, from RoFormer (Su et al., Zhuiyi Technology, 2021 — not Microsoft), turns each even/odd pair of a query or key into a 2-D vector and rotates it by \(p \, \theta_i\).
the score between a query at \(m\) and a key at \(n\) depends only on the offset \(n-m\). Absolute position cancels.
That identity is the reason to use RoPE. Additive positions do not give you this. The rotation also keeps the vector's length, so it does not fight the \(\sqrt{d_k}\) scale.
Take the pair \((0.6,\, 0.4)\) and \(\theta = 1\) (what you get with \(d = 2\) and base 10,000). The same "oak" vector at five positions:
| position | rotated pair |
|---|---|
| 0 | ( 0.600, 0.400) |
| 1 | (-0.012, 0.721) |
| 2 | (-0.613, 0.379) |
| 3 | (-0.650, -0.311) |
| 4 | (-0.089, -0.716) |
Same token, five different queries. The norm stays \(0.721\) in every row.
LLaMA 1/2 use base 10,000. Llama 3 uses 500,000. A larger base makes the slow pairs even slower, which helps long context.
oak pair → rotate → only \(n-m\) → \(V\) alone
Interactive · A 2-D rotation
Move the query and the key. Does the score care about absolute position, or only the gap?
In a causal model, token 0 sees one key. Token 5 sees six. The number of visible keys is the position. Models trained with no extra position encoding still learn to use that cue (Haviv et al., 2022). Some stacks even interleave RoPE and NoPE layers.
sees 1 → sees 6 → count=position → causal cue
NoPE does not help a bidirectional encoder. Every token sees the same number of keys, so the mask leaks nothing.
RoPE does not "just work" past the training length. The fast pairs wrap too soon. Three common fixes:
Pretend the new positions are a stretch of the old ones. A 4,096 window is shown to the model as if it were 2,048. Simple, a bit blurry.
Raise the base so the slow frequencies stretch and the fast ones stay put. High-frequency pairs would otherwise break first.
Split the ladder into bands: keep the fastest, scale the slowest, interpolate the middle. Used to push LLaMA-style models toward 128K.
DeepSeek-V3's own long-context recipe (YaRN plus a decoupled RoPE key) lives in Decoding DeepSeek. The idea you need here: the frequencies are a knob, not a constant of nature.
Rotate after the Q/K maps and before the cache. Do not rotate \(V\). Do not overwrite a pair in place: the even component is needed to build the odd one.
"""Sinusoidal positions, ALiBi biases, and RoPE rotations.
RoPE is the scheme modern decoders use. The score between a query at
position m and a key at position n depends only on the offset n - m.
RoFormer (Su et al., Zhuiyi Technology, 2021) introduced it. It is not
a Microsoft paper.
"""
import math
import torch
def sinusoidal_pe(n_pos, d_model, base=10000.0):
"""Original Transformer additive encoding. Pair i uses frequency base^{-2i/d}."""
pos = torch.arange(n_pos).unsqueeze(1) # (T, 1)
# i = 0, 2, 4, ... so the exponent is 0, 2/d, 4/d, ...
freq = base ** (-torch.arange(0, d_model, 2) / d_model) # (d/2,)
angles = pos * freq # (T, d/2)
pe = torch.zeros(n_pos, d_model)
pe[:, 0::2] = torch.sin(angles)
pe[:, 1::2] = torch.cos(angles)
return pe
def alibi_bias(n_pos, n_heads):
"""ALiBi adds m * (j - i) to each score. Slopes shrink geometrically per head."""
# Press et al.: head h uses slope 1/2^{8h/H} (1-indexed in the paper).
slopes = torch.tensor([2 ** (-8 * (h + 1) / n_heads) for h in range(n_heads)])
dist = torch.arange(n_pos) - torch.arange(n_pos).unsqueeze(1) # (T, T), j - i
return slopes.view(n_heads, 1, 1) * dist # (H, T, T)
def rotate_pair(x, cos, sin):
"""Rotate even/odd pairs of a vector: [x0, x1, ...] -> R [x0, x1, ...]."""
x_even, x_odd = x[..., 0::2], x[..., 1::2]
# [x0, x1] rotated by theta is [x0 c - x1 s, x0 s + x1 c].
rot_even = x_even * cos - x_odd * sin
rot_odd = x_even * sin + x_odd * cos
out = torch.zeros_like(x)
out[..., 0::2] = rot_even
out[..., 1::2] = rot_odd
return out
def apply_rope(q, k, base=10000.0):
"""Rotate Q and K by their positions. Do not rotate V.
q, k: (..., T, d) with even d. Returns rotated q, k of the same shape.
"""
d = q.shape[-1]
t = q.shape[-2]
# Frequency i uses theta_i = base^{-2i/d}. Same ladder as the sinusoid.
theta = base ** (-torch.arange(0, d, 2, device=q.device, dtype=q.dtype) / d)
pos = torch.arange(t, device=q.device, dtype=q.dtype)
angles = pos.unsqueeze(1) * theta # (T, d/2)
cos, sin = torch.cos(angles), torch.sin(angles)
return rotate_pair(q, cos, sin), rotate_pair(k, cos, sin)
def rope_relative_identity(q, k, m, n, base=10000.0):
"""Check <R_m q, R_n k> == <q, R_{n-m} k> for 2-d vectors."""
d = q.numel()
theta = base ** (-torch.arange(0, d, 2, dtype=q.dtype) / d)
def rot(vec, pos):
angles = pos * theta
return rotate_pair(vec.view(1, d), torch.cos(angles), torch.sin(angles)).view(d)
left = rot(q, m) @ rot(k, n)
right = q @ rot(k, n - m)
return left, right
"RoFormer is a Microsoft paper." It is Su, Lu, Pan, Wen, and Liu at Zhuiyi Technology. arXiv 2104.09864.
Quoting GPT-2 as sinusoidal. GPT-2 uses learned slots. The sinusoid table above is the 2017 formula evaluated at width 768.
Rotating \(V\). Values are the notes. Only the question and the label need to know where they sit.
"Every modern LLM uses base 10,000." Llama 3 uses 500,000. Many long-context recipes change the base again.
To have a correct frequency table at a familiar width. GPT-2's actual positions are a learned table of 1,024 vectors.
It equals \(q \cdot (R_{n-m} k)\). Only the distance matters.
Every token sees the same number of keys, so the mask cannot encode position.
\(2.1 - 0.75 = 1.35\).