THE TRANSFORMER DEEP CUTS · CHAPTER 01

Self-Attention from
First Principles

Every modern language model is a stack of soft lookups. This chapter is that lookup: queries, keys, values, the scale, and the mask.

~20 min read2 interactives2 Manim clipsTested PyTorch code

What you'll learn

  • Attention as a soft lookup: a query, a row of keys, a row of values
  • Why the three maps \(W_Q\), \(W_K\), \(W_V\) exist
  • Why we divide by \(\sqrt{d_k}\) — variance, not a softness knob
  • Why the causal mask uses \(-\infty\) before softmax

Before you start

  • Dot products and softmax
  • Matrix shapes: \((T, d) \times (d, T) = (T, T)\)
  • A language model predicts the next token

The harbour chart table

A harbour master keeps a stack of slips. Each slip has a label ("reef", "pier", "channel") and a note ("stay clear", "lamp is lit"). A new question comes in: "where did the fog settle?"

The master does not pick one slip. They hold the question against every label, give each slip a weight, and read a mix of the notes. That mix is the answer they write down.

That is attention. The query is the question, the keys are the labels, and the values are the notes. The output is a weighted sum, not a hard pick.

The rest of the chapter is this lookup, written as matrices. One short sentence — Fog hid the pier. — is enough to see every shape.

Scores, weights, mix

Four tokens sit in a row. Each already has a vector. For a moment, pretend those vectors are the queries, keys, and values. Token \(i\) asks a question \(q_i\). Every token \(j\) offers a label \(k_j\) and a note \(v_j\).

weight \(j\) by how well key \(j\) matches query \(i\), then mix the values with those weights.

same token → q, k, v  ·  \(q_i \cdot k_j\)  →  softmax  →  \(\sum_j \alpha_{ij}\, v_j\)

MANIM Watch once: hid asks, every key answers, softmax makes weights, values flow into \(o_{\text{hid}}\) — pier carries most of the note. Use the beat buttons to jump.

In matrices, one multiply does every pair at once:

  1. Scores. \(QK^\top\) is a \(T \times T\) grid. Cell \((i, j)\) is \(q_i \cdot k_j\).
  2. Scale. Divide every cell by \(\sqrt{d_k}\).
  3. Weights. Softmax each row, so the weights of one query sum to 1.
  4. Mix. Multiply the weight grid by \(V\). Row \(i\) of the result is \(o_i\).
QT × d_k
Kᵀd_k × T
scoresT × T
× VT × d_k

Raw dots pick the wrong noun

If \(Q = K = V = X\), attention can only compare the embeddings you already have. That is often the wrong comparison.

Take the sentence The kettle sat beside the radio until it whistled. The word it should look at kettle, not radio. Give them three-dimensional embeddings:

TokenVectorRaw dot with it
it[1, 0, 0]—
kettle[0, 1, 0]0
radio[1, 0, 0]1

The raw dots pick radio. A learned key map \(W_K\) can rotate the space so "things that whistle" line up with the query:

\(W_K\)New keyDot with \(q_{\text{it}}\)
kettle[[0,1,0],[1,0,0],[0,0,1]][1, 0, 0]1
radio[0, 1, 0]0

Now kettle wins. The three maps \(W_Q\), \(W_K\), \(W_V\) exist so each token can ask a question, offer a label, and offer a note that are different views of the same embedding. They are ordinary linear layers, usually without a bias.

same vectors  ·  radio wins raw  →  \(W_K\) swaps axes  →  kettle wins

MANIM Watch once: raw dots pick radio; \(W_K\) rotates the space so kettle wins. Same embedding, better key.

Bare attention on \(X\) itself is a teaching device. Real self-attention always uses the three maps.

Variance grows with \(d_k\). That is the whole point.

Suppose every entry of \(q\) and \(k\) is independent, mean 0, variance 1. Then

each product \(q_i k_i\) has variance 1, and you add \(d_k\) of them. The raw score has standard deviation \(\sqrt{d_k}\).

Softmax cares about differences. If the scores wander over a wider and wider range, one of them dominates and the weights collapse to a one-hot spike. Dividing by \(\sqrt{d_k}\) puts the variance back to 1, for any head size.

A common demo holds the raw scores still and only turns the divisor. That demo teaches the wrong lesson: "bigger \(d_k\) makes softmax softer." In a real model the scores grow with \(d_k\). The scale is there to cancel that growth, not to act as a temperature knob you set by hand.

Var = d_k  →  spread  →  ÷√d_k  →  Var≈1

MANIM Scores grow with \(d_k\); the scale cancels that growth.

Interactive · What the scale actually does

As \(d_k\) grows, do raw dots stay put, or do they spread?

64

Hide the future with \(-\infty\), not with 0

A decoder writing token 3 must not read token 4. The usual fix is a lower-triangular mask: query \(i\) may see keys \(j \le i\).

Fill the illegal cells with \(-\infty\) before softmax. Then \(e^{-\infty} = 0\), and the remaining weights still sum to 1. Zeroing a weight after softmax leaves leftover mass. The row no longer sums to 1, and the leftover is a leak from the future.

query the must not see pier  ·  zero after softmax → leak  ·  \(-\infty\) before softmax → sum 1

MANIM Same sentence, query the. Zeroing pier after softmax leaks future mass; \(-\infty\) before softmax leaves a clean row.

Interactive · Four tokens, one lower triangle

Which keys may the query pier see?

allowed masked (\(-\infty\))

A full walk on Fog hid the pier.

Four tokens, \(d_k = 4\), so \(\sqrt{d_k} = 2\). Look at the last query, the one that must predict what comes after pier.

Foghidthepier
raw \(q \cdot k\)2125
÷ 21.00.51.02.5
softmax0.1410.0860.1410.632

If the values are the four standard basis vectors, the context vector is that weight row: mostly pier, a little of the others. The first query, Fog, sees only itself. Its row is \([1, 0, 0, 0]\) after the mask.

One head, in full

The function returns both the mix and the weight grid, so you can print the numbers above. The class wraps the three maps. Multi-head attention and \(W_O\) wait for the next chapter.

attention.pycausal pass matches a hand mask; −inf ≠ post-softmax zero
"""Scaled dot-product self-attention for one head.

This is the math of chapter 01 in a few lines. Multi-head attention,
the output projection W_O, and the rest of the block are chapter 02.
"""
import torch
import torch.nn.functional as F


def scaled_dot_product_attention(q, k, v, causal=True):
    """q, k, v: (batch, tokens, d_k). Returns (batch, tokens, d_k).

    Scores grow like sqrt(d_k) when the entries of q and k have variance 1.
    Dividing by sqrt(d_k) puts the scores back on a scale softmax can use.
    """
    d_k = q.shape[-1]
    # (B, T, d_k) @ (B, d_k, T) -> (B, T, T). One score per query-key pair.
    scores = q @ k.transpose(-2, -1) / d_k**0.5
    if causal:
        # Position j (a key) is in the future of position i (a query) when j > i.
        # Fill those cells with -inf so softmax sends their weight to 0.
        t = q.shape[-2]
        future = torch.triu(torch.ones(t, t, dtype=torch.bool, device=q.device), 1)
        scores = scores.masked_fill(future, float("-inf"))
    weights = F.softmax(scores, dim=-1)
    return weights @ v, weights


class SelfAttention(torch.nn.Module):
    """One head. Three learned maps turn the same tokens into Q, K, V."""

    def __init__(self, d_model, d_k):
        super().__init__()
        # No bias: the original paper and most modern decoders skip it here.
        self.W_q = torch.nn.Linear(d_model, d_k, bias=False)
        self.W_k = torch.nn.Linear(d_model, d_k, bias=False)
        self.W_v = torch.nn.Linear(d_model, d_k, bias=False)

    def forward(self, x, causal=True):
        # x: (batch, tokens, d_model)
        return scaled_dot_product_attention(
            self.W_q(x), self.W_k(x), self.W_v(x), causal=causal
        )

Mistakes learners make

"Bigger \(d_k\) makes softmax softer." Only if you hold the raw scores still. Real scores spread like \(\sqrt{d_k}\). The scale cancels that spread.

"A token always attends most to itself." That is true for \(q = k = x\) on a sphere. After learned maps, the diagonal is just another cell.

Masking with 0 after softmax. The row no longer sums to 1. Use \(-\infty\) before softmax.

"Attention has no parameters." The softmax mix does not. The three maps do. Every real layer uses them.

What to remember

  • Attention is a soft lookup: scores \(QK^\top\), scale, softmax, mix with \(V\).
  • \(W_Q\), \(W_K\), \(W_V\) exist because the raw embeddings compare the wrong things.
  • Divide by \(\sqrt{d_k}\) because \(\operatorname{Var}(q \cdot k) = d_k\). The scale is not a temperature toy.
  • Causal decoding hides the future with \(-\infty\) before softmax.

Check yourself

1. What does one row of the softmax grid represent?

The weights that one query puts on every key. They sum to 1.

2. Why not reuse the same vector as query and key?

A token may need to ask for something different from what it offers. The two maps learn those two jobs.

3. If \(d_k\) goes from 4 to 64 and you forget the scale, what happens to softmax?

Raw scores spread by \(4\times\). Softmax gets sharper, often collapsing to a spike.

4. Why \(-\infty\) and not 0?

Softmax of \(-\infty\) is 0 and the other weights renormalise. A 0 written after softmax does not renormalise.

Go to the source

Keep going