THE TRANSFORMER DEEP CUTS · CHAPTER 04

Position Encoding:
Absolute → RoPE → Long Context

Attention does not know order. We have to put position in by hand. The modern way is to rotate the queries and keys.

~22 min read2 interactivesTested PyTorch code

What you'll learn

  • Why a permutation of the tokens changes the meaning, but not the attention math
  • Learned absolute embeddings, sinusoids, and ALiBi
  • RoPE: rotate \(Q\) and \(K\), and the score depends on the offset
  • NoPE: a causal mask already leaks position
  • How NTK scaling and YaRN stretch RoPE to longer context

Before you start

The same bags, two stories

Ash met oak. and Oak met ash. use the same three words. Attention on the raw embeddings would produce the same set of scores, just permuted. The meaning is not the same. Something has to mark who met whom.

Attention is permutation equivariant, not invariant: permute the input and the output permutes with it. It is still blind to which position was first. Position encodings break that symmetry.

Add a vector, or add a wave

Learned absolute embeddings (GPT-2, BERT) store one vector per slot, up to a max length. GPT-2 has 1,024 of them. Train at 1,024, ask for 2,048, and the extra slots have never been seen. The model does not extrapolate.

Sinusoidal encodings, from the 2017 paper, use a ladder of frequencies instead of a table:

pair \(i\) oscillates with frequency \(10000^{-2i/d}\). Low \(i\) is fast. High \(i\) is slow.

GPT-2 itself does not use this formula — it uses learned slots. If you still evaluate the sinusoid at GPT-2's width \(d = 768\), the first frequencies are:

pair \(i\)\(10000^{2i/768}\)\(\omega_i\)
011
11.0240.976
21.0490.953
101.2710.787
503.320.301
10011.00.091

Pair 50 still moves. It is not flat. Older tables that put \(\omega_1 = 0.0316\) and called pair 50 "completely flat" used the wrong exponent.

ω=1  →  ladder  →  i=50 moves  →  not flat

MANIM Pair 50 still moves (ω≈0.301) — older tables used the wrong exponent.

A penalty that grows with distance

ALiBi leaves \(Q\) and \(K\) alone and adds a bias to the scores: \(m \times (j - i)\), with a negative slope \(m\) per head. Nearby keys get a milder penalty. Far keys get a harsher one.

Four heads use slopes \(2^{-8/4}, 2^{-16/4}, \ldots\). Head 0 has slope \(0.25\). If a raw dot is 2.1 and the key sits 3 steps behind, the logit becomes \(2.1 + 0.25 \times (-3) = \) 1.35.

2.1  →  −3  →  0.25  →  1.35

MANIM Leave Q/K alone; add a distance penalty to the score.

Because the bias is a function of the offset, ALiBi can step a little past the training length. It does not give you a free 8× context.

Rotate \(Q\) and \(K\). Leave \(V\) alone.

RoPE, from RoFormer (Su et al., Zhuiyi Technology, 2021 — not Microsoft), turns each even/odd pair of a query or key into a 2-D vector and rotates it by \(p \, \theta_i\).

the score between a query at \(m\) and a key at \(n\) depends only on the offset \(n-m\). Absolute position cancels.

That identity is the reason to use RoPE. Additive positions do not give you this. The rotation also keeps the vector's length, so it does not fight the \(\sqrt{d_k}\) scale.

Take the pair \((0.6,\, 0.4)\) and \(\theta = 1\) (what you get with \(d = 2\) and base 10,000). The same "oak" vector at five positions:

positionrotated pair
0( 0.600, 0.400)
1(-0.012, 0.721)
2(-0.613, 0.379)
3(-0.650, -0.311)
4(-0.089, -0.716)

Same token, five different queries. The norm stays \(0.721\) in every row.

LLaMA 1/2 use base 10,000. Llama 3 uses 500,000. A larger base makes the slow pairs even slower, which helps long context.

oak pair  →  rotate  →  only \(n-m\)  →  \(V\) alone

MANIM Rotate Q and K; the score depends only on the gap. Leave V alone.

Interactive · A 2-D rotation

Move the query and the key. Does the score care about absolute position, or only the gap?

2
5

The mask already tells you where you are

In a causal model, token 0 sees one key. Token 5 sees six. The number of visible keys is the position. Models trained with no extra position encoding still learn to use that cue (Haviv et al., 2022). Some stacks even interleave RoPE and NoPE layers.

sees 1  →  sees 6  →  count=position  →  causal cue

MANIM Visible-key count is the position cue in a causal model.

NoPE does not help a bidirectional encoder. Every token sees the same number of keys, so the mask leaks nothing.

NTK scaling and YaRN

RoPE does not "just work" past the training length. The fast pairs wrap too soon. Three common fixes:

Position interpolation

Pretend the new positions are a stretch of the old ones. A 4,096 window is shown to the model as if it were 2,048. Simple, a bit blurry.

NTK-aware

Raise the base so the slow frequencies stretch and the fast ones stay put. High-frequency pairs would otherwise break first.

YaRN

Split the ladder into bands: keep the fastest, scale the slowest, interpolate the middle. Used to push LLaMA-style models toward 128K.

DeepSeek-V3's own long-context recipe (YaRN plus a decoupled RoPE key) lives in Decoding DeepSeek. The idea you need here: the frequencies are a knob, not a constant of nature.

Sinusoid, ALiBi, RoPE

Rotate after the Q/K maps and before the cache. Do not rotate \(V\). Do not overwrite a pair in place: the even component is needed to build the odd one.

rope.pyrelative-position identity holds; sinusoid frequencies match d=768
"""Sinusoidal positions, ALiBi biases, and RoPE rotations.

RoPE is the scheme modern decoders use. The score between a query at
position m and a key at position n depends only on the offset n - m.
RoFormer (Su et al., Zhuiyi Technology, 2021) introduced it. It is not
a Microsoft paper.
"""
import math
import torch


def sinusoidal_pe(n_pos, d_model, base=10000.0):
    """Original Transformer additive encoding. Pair i uses frequency base^{-2i/d}."""
    pos = torch.arange(n_pos).unsqueeze(1)                     # (T, 1)
    # i = 0, 2, 4, ... so the exponent is 0, 2/d, 4/d, ...
    freq = base ** (-torch.arange(0, d_model, 2) / d_model)    # (d/2,)
    angles = pos * freq                                        # (T, d/2)
    pe = torch.zeros(n_pos, d_model)
    pe[:, 0::2] = torch.sin(angles)
    pe[:, 1::2] = torch.cos(angles)
    return pe


def alibi_bias(n_pos, n_heads):
    """ALiBi adds m * (j - i) to each score. Slopes shrink geometrically per head."""
    # Press et al.: head h uses slope 1/2^{8h/H} (1-indexed in the paper).
    slopes = torch.tensor([2 ** (-8 * (h + 1) / n_heads) for h in range(n_heads)])
    dist = torch.arange(n_pos) - torch.arange(n_pos).unsqueeze(1)  # (T, T), j - i
    return slopes.view(n_heads, 1, 1) * dist                      # (H, T, T)


def rotate_pair(x, cos, sin):
    """Rotate even/odd pairs of a vector: [x0, x1, ...] -> R [x0, x1, ...]."""
    x_even, x_odd = x[..., 0::2], x[..., 1::2]
    # [x0, x1] rotated by theta is [x0 c - x1 s, x0 s + x1 c].
    rot_even = x_even * cos - x_odd * sin
    rot_odd = x_even * sin + x_odd * cos
    out = torch.zeros_like(x)
    out[..., 0::2] = rot_even
    out[..., 1::2] = rot_odd
    return out


def apply_rope(q, k, base=10000.0):
    """Rotate Q and K by their positions. Do not rotate V.

    q, k: (..., T, d) with even d. Returns rotated q, k of the same shape.
    """
    d = q.shape[-1]
    t = q.shape[-2]
    # Frequency i uses theta_i = base^{-2i/d}. Same ladder as the sinusoid.
    theta = base ** (-torch.arange(0, d, 2, device=q.device, dtype=q.dtype) / d)
    pos = torch.arange(t, device=q.device, dtype=q.dtype)
    angles = pos.unsqueeze(1) * theta                           # (T, d/2)
    cos, sin = torch.cos(angles), torch.sin(angles)
    return rotate_pair(q, cos, sin), rotate_pair(k, cos, sin)


def rope_relative_identity(q, k, m, n, base=10000.0):
    """Check <R_m q, R_n k> == <q, R_{n-m} k> for 2-d vectors."""
    d = q.numel()
    theta = base ** (-torch.arange(0, d, 2, dtype=q.dtype) / d)

    def rot(vec, pos):
        angles = pos * theta
        return rotate_pair(vec.view(1, d), torch.cos(angles), torch.sin(angles)).view(d)

    left = rot(q, m) @ rot(k, n)
    right = q @ rot(k, n - m)
    return left, right

Mistakes learners make

"RoFormer is a Microsoft paper." It is Su, Lu, Pan, Wen, and Liu at Zhuiyi Technology. arXiv 2104.09864.

Quoting GPT-2 as sinusoidal. GPT-2 uses learned slots. The sinusoid table above is the 2017 formula evaluated at width 768.

Rotating \(V\). Values are the notes. Only the question and the label need to know where they sit.

"Every modern LLM uses base 10,000." Llama 3 uses 500,000. Many long-context recipes change the base again.

What to remember

  • Attention is order-blind. Absolute tables, sinusoids, ALiBi, and RoPE are four ways to mark position.
  • RoPE's score depends on the offset: \(\langle R_m q, R_n k \rangle = \langle q, R_{n-m} k \rangle\).
  • A causal mask already leaks position (NoPE). A bidirectional mask does not.
  • Long context is a frequency problem. Interpolation, NTK, and YaRN are three patches.

Check yourself

1. Why evaluate sinusoids at \(d=768\) if GPT-2 does not use them?

To have a correct frequency table at a familiar width. GPT-2's actual positions are a learned table of 1,024 vectors.

2. What does RoPE guarantee about \(q_m \cdot k_n\)?

It equals \(q \cdot (R_{n-m} k)\). Only the distance matters.

3. Why does NoPE fail in BERT?

Every token sees the same number of keys, so the mask cannot encode position.

4. ALiBi: raw dot 2.1, key 3 behind, slope 0.25. New logit?

\(2.1 - 0.75 = 1.35\).

Go to the source

Keep going