THE TRANSFORMER DEEP CUTS · CHAPTER 10
CLIP scores a pair. A VLM writes an answer. The picture becomes a prefix of tokens, the language model keeps doing language, and the mask has to change.
A red bicycle leans against a bakery window. You ask, "Where is the bicycle?" A contrastive model can pick the right caption from a list. It cannot type "against the bakery window." A VLM can, because the language model is still a decoder. The new job is to get the picture into that decoder as tokens.
PaliGemma's recipe is the simplest one that works: freeze or adapt a SigLIP ViT, project every patch into Gemma's width, and prepend those tokens as a prefix. No extra cross-attention tower. The code for this lives in Evolving Transformer 09. This page is the why.
SigLIP's width is 1,152. Gemma 2B's width is 2,048. The projector is one matrix, \(1152 \times 2048 = 2{,}359{,}296\) weights. The image tokens replace a run of <image> placeholders so the rest of the prompt stays a normal string.
256×1152 → 1152→2048 → prefix → decode
| Image | Grid | Tokens |
|---|---|---|
| 224 | 16 × 16 | 256 |
| 448 | 32 × 32 | 1,024 |
| 896 | 64 × 64 | 4,096 |
Keep every patch. A CLS would collapse the bicycle and the bakery into one vector. The decoder needs both. Counting 196 comes from pretending the patch is 16; PaliGemma's SigLIP-So400m uses 14.
Prefix-LM, as in chapter 08. The 256 image tokens plus "Where is the bicycle?" are bidirectional. "against the bakery window" is causal, and it can look at the whole prefix. The training loss sits on the suffix only. Teaching the model to reconstruct the prompt, or the patches, is a different objective.
Interactive · A tiny prefix-LM
Four image tokens, a 2-token prompt, a 2-token answer. Who can see whom?
Gemma uses RoPE, not a learned table. The 256 patch tokens take positions \(0 \ldots 255\). The prompt continues at 256. The answer continues after that. If you reset position ids at the text, the decoder cannot tell a patch on the left of the bicycle from a patch on the right. If you use a causal-only mask and skip RoPE on the image, both of PaliGemma's design choices are gone.
patches → continue → answer → prefill
At decode time the image prefix is prefilled once. Its keys and values sit in the cache. New answer tokens append as usual. Chapter 06 still applies.
A short MLP projector instead of one linear map. Still a prefix into a decoder. Same family, fancier adapter.
Frozen vision and language towers, with new cross-attention layers in the decoder. The picture is not prepended; it is read on the side.
A small transformer compresses the patches into a handful of query tokens, then those go into the LM. Fewer image tokens, extra parameters.
PaliGemma is the cleanest picture of "the image is a prefix." The others are worth knowing so you do not assume every VLM concatenates.
A causal mask over image + text. Image patches cannot look at each other to the right, and the prompt cannot either. PaliGemma is prefix-LM.
Learned positions over the packed sequence. Gemma uses RoPE. The positions must continue through the image, not restart.
196 tokens at 224. That is \(224/16\). The tower uses patch 14, so 256.
Loss on every token. The suffix is the answer. The prefix is context.
The decoder needs to point at parts of the picture — the bicycle, the window, the bakery. One vector throws that away.
Yes. The prefix is bidirectional.
At the first image token, then straight on through the prompt and the answer. No reset.