THE TRANSFORMER DEEP CUTS · CHAPTER 08

Transformer Families

The attention formula does not change. The mask and the loss do. That is the whole family tree.

~18 min read1 interactiveTested PyTorch code

What you'll learn

  • Decoder-only, encoder-only, encoder–decoder, and prefix-LM, by mask
  • BERT's MLM recipe (15%, then 80/10/10) and why NSP was dropped
  • CLIP's text tower uses a causal mask in the original paper
  • PaliGemma needs a prefix-LM mask, not a plain causal one

Before you start

Same engine, four traffic rules

Four junctions, one kind of car. The difference is which roads are open. Open every road and you have an encoder. Open only the roads behind you and you have a decoder. Open the prompt both ways and the answer one way and you have a prefix-LM. Open a second street for the source sentence and you have an encoder–decoder.

Chat models that hide the prompt from the loss but still use a causal attention grid are decoder-only. Loss masking and attention masking are different knobs. Do not mix them up.

Look at the grid, not the name

Interactive · Which keys may a query see?

Pick a family. The highlighted cells are the allowed ones.

GPT, LLaMA, and the CLIP text tower

A causal mask and a next-token loss. GPT-2, Llama 3, Mistral, DeepSeek-V3. Training is parallel: every position is a label. Decoding is serial: one new query per step, with a KV cache.

The original CLIP paper uses this family for its text encoder. The tower is a GPT-2-style Transformer with a causal mask. The readout is the end-of-text token, which has attended to the whole caption. Calling that tower "an encoder stack only" is wrong: the mask is causal.

BERT, and why next-token is illegal

No causal mask. Every token sees every other token. Next-token prediction would then be trivial: the answer is sitting in the future, in plain sight. BERT is trained with masked language modelling instead.

Pick 15% of the tokens. Of those, 80% become [MASK], 10% become a random token, 10% stay as they were. The label is always the original word. The 10% "stay" stops the model from only looking at the mask slot; the 10% random stops it from assuming the visible token is always correct.

bi grid  →  15%  →  80/10/10  →  originals

MANIM Next-token is illegal on a bidirectional grid — MLM instead.

On a 128-token sequence that is 19.2 selected tokens: about 15.36 masks, 1.92 random, 1.92 left alone.

BERT-Base: 12 layers, width 768, 12 heads, 512 learned positions, ~110M. BERT-Large: 24 / 1024 / 16. Next-sentence prediction (NSP) was part of the original recipe and is now considered unhelpful. RoBERTa dropped it.

An encoder does not generate. Fine-tune a small head on [CLS] (or a better pool) for classification or span tagging. Sentence embeddings from a vanilla BERT pool are weak — that is why later bi-encoders exist, and why CLIP trains its own pair of towers.

Three attentions, not one

The original Transformer and T5: a bidirectional encoder over the source, a causal decoder over the target, and a cross-attention that lets every target query read every source key. Three grids. At inference the encoder runs once; its K/V can be cached while the decoder writes.

enc bi  →  dec causal  →  cross  →  enc once

MANIM Three attentions: encoder self, decoder self, cross.

Translation, summarisation, and many older multimodal models use this layout. A modern decoder-only model can do the same jobs by packing "source + target" into one causal stream. The packed version is simpler to serve. The split version makes a cleaner cache for a long source.

PaliGemma's mask, not a chat model's loss

A prefix-LM opens every road inside the prefix, and only the backward roads inside the suffix. Image tokens plus the prompt see each other. The answer is written causally, and it can see the whole prefix.

PaliGemma needs this mask. A plain causal triangle over [image | prompt | answer] would stop image tokens from looking right, and would stop the prompt from looking at later prompt tokens. That is the wrong picture.

Most instruction-tuned LLMs are still causal decoders. They zero the loss on the prompt so the model is not trained to reinvent the user's words. They do not unmask the prompt. GPT-3 did not flip the family either.

causal over all  →  prefix-LM  →  loss mask ≠ attn  →  two knobs

MANIM PaliGemma needs prefix roads open. Zeroing chat loss is a different knob.

Four functions, four grids

masks.pyprefix-LM: 3+2 tokens → bidirectional prefix, causal suffix
"""Attention masks for the four transformer families.

A 1 means "this query may look at this key". A 0 is masked out
(filled with -inf before softmax). The families differ by this grid
and by the training objective, not by a different attention formula.
"""
import torch


def causal_mask(n):
    """Decoder-only (GPT, LLaMA) and the original CLIP text encoder."""
    return torch.tril(torch.ones(n, n, dtype=torch.bool))


def bidirectional_mask(n):
    """Encoder-only (BERT): every token may look at every other token."""
    return torch.ones(n, n, dtype=torch.bool)


def encoder_decoder_masks(n_src, n_tgt):
    """T5 / original Transformer. Three grids, not one.

    Encoder self-attention is bidirectional.
    Decoder self-attention is causal.
    Cross-attention: every target query may look at every source key.
    """
    return {
        "encoder_self": bidirectional_mask(n_src),
        "decoder_self": causal_mask(n_tgt),
        "cross": torch.ones(n_tgt, n_src, dtype=torch.bool),
    }


def prefix_lm_mask(n_prefix, n_suffix):
    """PaliGemma-style prefix-LM.

    The prefix (image tokens + the prompt) is bidirectional.
    The suffix (the answer being written) is causal, and it can see
    the whole prefix. Chat models that only mask the *loss* on the
    prompt are not prefix-LM — they still use a causal attention grid.
    """
    n = n_prefix + n_suffix
    mask = torch.zeros(n, n, dtype=torch.bool)
    # Prefix rows see the whole prefix.
    mask[:n_prefix, :n_prefix] = True
    # Suffix row i sees the prefix and suffix tokens up to i.
    mask[n_prefix:, :n_prefix] = True
    mask[n_prefix:, n_prefix:] = torch.tril(torch.ones(n_suffix, n_suffix, dtype=torch.bool))
    return mask

Mistakes learners make

"CLIP's text tower is a bidirectional encoder." The original paper uses a causal Transformer and reads the EOT token.

"Chat models are prefix-LM because they mask the prompt loss." Loss masking ≠ attention masking. Most chat models stay causal.

Training BERT with a next-token loss. The future is visible. The loss collapses.

Keeping NSP as a must-have. RoBERTa dropped it. Later encoder work did not bring it back.

What to remember

  • Families differ by mask and by loss, not by a different attention formula.
  • Decoder: causal, next-token. Encoder: bidirectional, MLM. Enc–dec: three grids. Prefix-LM: bidirectional prefix, causal suffix.
  • CLIP text = causal. PaliGemma = prefix-LM. Chat GPT-style models = causal + loss mask on the prompt.

Check yourself

1. Why can't BERT use a next-token loss?

The mask is bidirectional, so the next token is already visible to the query.

2. Does CLIP's text tower use BERT's mask?

No. The original CLIP paper uses a causal mask and the EOT token as the readout.

3. Image tokens + prompt of length 3, answer of length 2. May the first image token see the last answer token?

No. Prefix rows do not see the suffix. The last answer token may see everyone.

Go to the source

Keep going