THE TRANSFORMER DEEP CUTS · CHAPTER 01
Every modern language model is a stack of soft lookups. This chapter is that lookup: queries, keys, values, the scale, and the mask.
A harbour master keeps a stack of slips. Each slip has a label ("reef", "pier", "channel") and a note ("stay clear", "lamp is lit"). A new question comes in: "where did the fog settle?"
The master does not pick one slip. They hold the question against every label, give each slip a weight, and read a mix of the notes. That mix is the answer they write down.
That is attention. The query is the question, the keys are the labels, and the values are the notes. The output is a weighted sum, not a hard pick.
The rest of the chapter is this lookup, written as matrices. One short sentence — Fog hid the pier. — is enough to see every shape.
Four tokens sit in a row. Each already has a vector. For a moment, pretend those vectors are the queries, keys, and values. Token \(i\) asks a question \(q_i\). Every token \(j\) offers a label \(k_j\) and a note \(v_j\).
weight \(j\) by how well key \(j\) matches query \(i\), then mix the values with those weights.
same token → q, k, v · \(q_i \cdot k_j\) → softmax → \(\sum_j \alpha_{ij}\, v_j\)
In matrices, one multiply does every pair at once:
If \(Q = K = V = X\), attention can only compare the embeddings you already have. That is often the wrong comparison.
Take the sentence The kettle sat beside the radio until it whistled. The word it should look at kettle, not radio. Give them three-dimensional embeddings:
| Token | Vector | Raw dot with it |
|---|---|---|
| it | [1, 0, 0] | — |
| kettle | [0, 1, 0] | 0 |
| radio | [1, 0, 0] | 1 |
The raw dots pick radio. A learned key map \(W_K\) can rotate the space so "things that whistle" line up with the query:
| \(W_K\) | New key | Dot with \(q_{\text{it}}\) | |
|---|---|---|---|
| kettle | [[0,1,0],[1,0,0],[0,0,1]] | [1, 0, 0] | 1 |
| radio | [0, 1, 0] | 0 |
Now kettle wins. The three maps \(W_Q\), \(W_K\), \(W_V\) exist so each token can ask a question, offer a label, and offer a note that are different views of the same embedding. They are ordinary linear layers, usually without a bias.
same vectors · radio wins raw → \(W_K\) swaps axes → kettle wins
Bare attention on \(X\) itself is a teaching device. Real self-attention always uses the three maps.
Suppose every entry of \(q\) and \(k\) is independent, mean 0, variance 1. Then
each product \(q_i k_i\) has variance 1, and you add \(d_k\) of them. The raw score has standard deviation \(\sqrt{d_k}\).
Softmax cares about differences. If the scores wander over a wider and wider range, one of them dominates and the weights collapse to a one-hot spike. Dividing by \(\sqrt{d_k}\) puts the variance back to 1, for any head size.
A common demo holds the raw scores still and only turns the divisor. That demo teaches the wrong lesson: "bigger \(d_k\) makes softmax softer." In a real model the scores grow with \(d_k\). The scale is there to cancel that growth, not to act as a temperature knob you set by hand.
Var = d_k → spread → ÷√d_k → Var≈1
Interactive · What the scale actually does
As \(d_k\) grows, do raw dots stay put, or do they spread?
A decoder writing token 3 must not read token 4. The usual fix is a lower-triangular mask: query \(i\) may see keys \(j \le i\).
Fill the illegal cells with \(-\infty\) before softmax. Then \(e^{-\infty} = 0\), and the remaining weights still sum to 1. Zeroing a weight after softmax leaves leftover mass. The row no longer sums to 1, and the leftover is a leak from the future.
query the must not see pier · zero after softmax → leak · \(-\infty\) before softmax → sum 1
Interactive · Four tokens, one lower triangle
Which keys may the query pier see?
Four tokens, \(d_k = 4\), so \(\sqrt{d_k} = 2\). Look at the last query, the one that must predict what comes after pier.
| Fog | hid | the | pier | |
|---|---|---|---|---|
| raw \(q \cdot k\) | 2 | 1 | 2 | 5 |
| ÷ 2 | 1.0 | 0.5 | 1.0 | 2.5 |
| softmax | 0.141 | 0.086 | 0.141 | 0.632 |
If the values are the four standard basis vectors, the context vector is that weight row: mostly pier, a little of the others. The first query, Fog, sees only itself. Its row is \([1, 0, 0, 0]\) after the mask.
The function returns both the mix and the weight grid, so you can print the numbers above. The class wraps the three maps. Multi-head attention and \(W_O\) wait for the next chapter.
"""Scaled dot-product self-attention for one head.
This is the math of chapter 01 in a few lines. Multi-head attention,
the output projection W_O, and the rest of the block are chapter 02.
"""
import torch
import torch.nn.functional as F
def scaled_dot_product_attention(q, k, v, causal=True):
"""q, k, v: (batch, tokens, d_k). Returns (batch, tokens, d_k).
Scores grow like sqrt(d_k) when the entries of q and k have variance 1.
Dividing by sqrt(d_k) puts the scores back on a scale softmax can use.
"""
d_k = q.shape[-1]
# (B, T, d_k) @ (B, d_k, T) -> (B, T, T). One score per query-key pair.
scores = q @ k.transpose(-2, -1) / d_k**0.5
if causal:
# Position j (a key) is in the future of position i (a query) when j > i.
# Fill those cells with -inf so softmax sends their weight to 0.
t = q.shape[-2]
future = torch.triu(torch.ones(t, t, dtype=torch.bool, device=q.device), 1)
scores = scores.masked_fill(future, float("-inf"))
weights = F.softmax(scores, dim=-1)
return weights @ v, weights
class SelfAttention(torch.nn.Module):
"""One head. Three learned maps turn the same tokens into Q, K, V."""
def __init__(self, d_model, d_k):
super().__init__()
# No bias: the original paper and most modern decoders skip it here.
self.W_q = torch.nn.Linear(d_model, d_k, bias=False)
self.W_k = torch.nn.Linear(d_model, d_k, bias=False)
self.W_v = torch.nn.Linear(d_model, d_k, bias=False)
def forward(self, x, causal=True):
# x: (batch, tokens, d_model)
return scaled_dot_product_attention(
self.W_q(x), self.W_k(x), self.W_v(x), causal=causal
)
"Bigger \(d_k\) makes softmax softer." Only if you hold the raw scores still. Real scores spread like \(\sqrt{d_k}\). The scale cancels that spread.
"A token always attends most to itself." That is true for \(q = k = x\) on a sphere. After learned maps, the diagonal is just another cell.
Masking with 0 after softmax. The row no longer sums to 1. Use \(-\infty\) before softmax.
"Attention has no parameters." The softmax mix does not. The three maps do. Every real layer uses them.
The weights that one query puts on every key. They sum to 1.
A token may need to ask for something different from what it offers. The two maps learn those two jobs.
Raw scores spread by \(4\times\). Softmax gets sharper, often collapsing to a spike.
Softmax of \(-\infty\) is 0 and the other weights renormalise. A 0 written after softmax does not renormalise.