THE TRANSFORMER DEEP CUTS · CHAPTER 08
The attention formula does not change. The mask and the loss do. That is the whole family tree.
Four junctions, one kind of car. The difference is which roads are open. Open every road and you have an encoder. Open only the roads behind you and you have a decoder. Open the prompt both ways and the answer one way and you have a prefix-LM. Open a second street for the source sentence and you have an encoder–decoder.
Chat models that hide the prompt from the loss but still use a causal attention grid are decoder-only. Loss masking and attention masking are different knobs. Do not mix them up.
Interactive · Which keys may a query see?
Pick a family. The highlighted cells are the allowed ones.
A causal mask and a next-token loss. GPT-2, Llama 3, Mistral, DeepSeek-V3. Training is parallel: every position is a label. Decoding is serial: one new query per step, with a KV cache.
The original CLIP paper uses this family for its text encoder. The tower is a GPT-2-style Transformer with a causal mask. The readout is the end-of-text token, which has attended to the whole caption. Calling that tower "an encoder stack only" is wrong: the mask is causal.
No causal mask. Every token sees every other token. Next-token prediction would then be trivial: the answer is sitting in the future, in plain sight. BERT is trained with masked language modelling instead.
Pick 15% of the tokens. Of those, 80% become [MASK], 10% become a random token, 10% stay as they were. The label is always the original word. The 10% "stay" stops the model from only looking at the mask slot; the 10% random stops it from assuming the visible token is always correct.
bi grid → 15% → 80/10/10 → originals
On a 128-token sequence that is 19.2 selected tokens: about 15.36 masks, 1.92 random, 1.92 left alone.
BERT-Base: 12 layers, width 768, 12 heads, 512 learned positions, ~110M. BERT-Large: 24 / 1024 / 16. Next-sentence prediction (NSP) was part of the original recipe and is now considered unhelpful. RoBERTa dropped it.
An encoder does not generate. Fine-tune a small head on [CLS] (or a better pool) for classification or span tagging. Sentence embeddings from a vanilla BERT pool are weak — that is why later bi-encoders exist, and why CLIP trains its own pair of towers.
The original Transformer and T5: a bidirectional encoder over the source, a causal decoder over the target, and a cross-attention that lets every target query read every source key. Three grids. At inference the encoder runs once; its K/V can be cached while the decoder writes.
enc bi → dec causal → cross → enc once
Translation, summarisation, and many older multimodal models use this layout. A modern decoder-only model can do the same jobs by packing "source + target" into one causal stream. The packed version is simpler to serve. The split version makes a cleaner cache for a long source.
A prefix-LM opens every road inside the prefix, and only the backward roads inside the suffix. Image tokens plus the prompt see each other. The answer is written causally, and it can see the whole prefix.
PaliGemma needs this mask. A plain causal triangle over [image | prompt | answer] would stop image tokens from looking right, and would stop the prompt from looking at later prompt tokens. That is the wrong picture.
Most instruction-tuned LLMs are still causal decoders. They zero the loss on the prompt so the model is not trained to reinvent the user's words. They do not unmask the prompt. GPT-3 did not flip the family either.
causal over all → prefix-LM → loss mask ≠ attn → two knobs
"""Attention masks for the four transformer families.
A 1 means "this query may look at this key". A 0 is masked out
(filled with -inf before softmax). The families differ by this grid
and by the training objective, not by a different attention formula.
"""
import torch
def causal_mask(n):
"""Decoder-only (GPT, LLaMA) and the original CLIP text encoder."""
return torch.tril(torch.ones(n, n, dtype=torch.bool))
def bidirectional_mask(n):
"""Encoder-only (BERT): every token may look at every other token."""
return torch.ones(n, n, dtype=torch.bool)
def encoder_decoder_masks(n_src, n_tgt):
"""T5 / original Transformer. Three grids, not one.
Encoder self-attention is bidirectional.
Decoder self-attention is causal.
Cross-attention: every target query may look at every source key.
"""
return {
"encoder_self": bidirectional_mask(n_src),
"decoder_self": causal_mask(n_tgt),
"cross": torch.ones(n_tgt, n_src, dtype=torch.bool),
}
def prefix_lm_mask(n_prefix, n_suffix):
"""PaliGemma-style prefix-LM.
The prefix (image tokens + the prompt) is bidirectional.
The suffix (the answer being written) is causal, and it can see
the whole prefix. Chat models that only mask the *loss* on the
prompt are not prefix-LM — they still use a causal attention grid.
"""
n = n_prefix + n_suffix
mask = torch.zeros(n, n, dtype=torch.bool)
# Prefix rows see the whole prefix.
mask[:n_prefix, :n_prefix] = True
# Suffix row i sees the prefix and suffix tokens up to i.
mask[n_prefix:, :n_prefix] = True
mask[n_prefix:, n_prefix:] = torch.tril(torch.ones(n_suffix, n_suffix, dtype=torch.bool))
return mask
"CLIP's text tower is a bidirectional encoder." The original paper uses a causal Transformer and reads the EOT token.
"Chat models are prefix-LM because they mask the prompt loss." Loss masking ≠ attention masking. Most chat models stay causal.
Training BERT with a next-token loss. The future is visible. The loss collapses.
Keeping NSP as a must-have. RoBERTa dropped it. Later encoder work did not bring it back.
The mask is bidirectional, so the next token is already visible to the query.
No. The original CLIP paper uses a causal mask and the EOT token as the readout.
No. Prefix rows do not see the suffix. The last answer token may see everyone.