THE TRANSFORMER DEEP CUTS · CHAPTER 10

Vision-Language Models:
PaliGemma-style Fusion

CLIP scores a pair. A VLM writes an answer. The picture becomes a prefix of tokens, the language model keeps doing language, and the mask has to change.

~16 min read1 interactive

What you'll learn

  • How a linear projector turns vision width into language width
  • Why PaliGemma keeps all 256 patches instead of a CLS
  • The prefix-LM mask, and why RoPE must run through the image prefix
  • Where the loss is computed (the suffix only)

Before you start

  • CLIP — the vision tower this chapter reuses
  • Prefix-LM — the mask
  • RoPE — positions are not a table any more

The picture becomes a paragraph the model can attend to

A red bicycle leans against a bakery window. You ask, "Where is the bicycle?" A contrastive model can pick the right caption from a list. It cannot type "against the bakery window." A VLM can, because the language model is still a decoder. The new job is to get the picture into that decoder as tokens.

PaliGemma's recipe is the simplest one that works: freeze or adapt a SigLIP ViT, project every patch into Gemma's width, and prepend those tokens as a prefix. No extra cross-attention tower. The code for this lives in Evolving Transformer 09. This page is the why.

Patches, a linear map, a prefix

image 224 × 224
SigLIP ViT256 patches × 1,152
linear projector1,152 → 2,048
Gemma decoderprefix + prompt + answer

SigLIP's width is 1,152. Gemma 2B's width is 2,048. The projector is one matrix, \(1152 \times 2048 = 2{,}359{,}296\) weights. The image tokens replace a run of <image> placeholders so the rest of the prompt stays a normal string.

256×1152  →  1152→2048  →  prefix  →  decode

MANIM One matrix turns SigLIP patches into Gemma tokens — the picture becomes a prefix.

Patch 14, three image sizes

ImageGridTokens
22416 × 16256
44832 × 321,024
89664 × 644,096

Keep every patch. A CLS would collapse the bicycle and the bakery into one vector. The decoder needs both. Counting 196 comes from pretending the patch is 16; PaliGemma's SigLIP-So400m uses 14.

Image and prompt see each other. The answer does not leak.

Prefix-LM, as in chapter 08. The 256 image tokens plus "Where is the bicycle?" are bidirectional. "against the bakery window" is causal, and it can look at the whole prefix. The training loss sits on the suffix only. Teaching the model to reconstruct the prompt, or the patches, is a different objective.

Interactive · A tiny prefix-LM

Four image tokens, a 2-token prompt, a 2-token answer. Who can see whom?

allowed answer, causal blocked

The image tokens have positions too

Gemma uses RoPE, not a learned table. The 256 patch tokens take positions \(0 \ldots 255\). The prompt continues at 256. The answer continues after that. If you reset position ids at the text, the decoder cannot tell a patch on the left of the bicycle from a patch on the right. If you use a causal-only mask and skip RoPE on the image, both of PaliGemma's design choices are gone.

patches  →  continue  →  answer  →  prefill

MANIM Image tokens take positions 0…N−1; text continues the count — do not reset RoPE.

At decode time the image prefix is prefilled once. Its keys and values sit in the cache. New answer tokens append as usual. Chapter 06 still applies.

Not every VLM is a prefix

LLaVA

A short MLP projector instead of one linear map. Still a prefix into a decoder. Same family, fancier adapter.

Flamingo

Frozen vision and language towers, with new cross-attention layers in the decoder. The picture is not prepended; it is read on the side.

Q-Former / BLIP-2

A small transformer compresses the patches into a handful of query tokens, then those go into the LM. Fewer image tokens, extra parameters.

PaliGemma is the cleanest picture of "the image is a prefix." The others are worth knowing so you do not assume every VLM concatenates.

Mistakes learners make

A causal mask over image + text. Image patches cannot look at each other to the right, and the prompt cannot either. PaliGemma is prefix-LM.

Learned positions over the packed sequence. Gemma uses RoPE. The positions must continue through the image, not restart.

196 tokens at 224. That is \(224/16\). The tower uses patch 14, so 256.

Loss on every token. The suffix is the answer. The prefix is context.

What to remember

  • A VLM writes. CLIP scores. The jump is a decoder plus a way to feed it image tokens.
  • PaliGemma: SigLIP patches → linear 1,152→2,048 → prefix of a Gemma decoder.
  • 256 / 1,024 / 4,096 tokens at 224 / 448 / 896 with patch 14. Keep them all.
  • Prefix-LM mask. RoPE through the image. Loss on the suffix.

Check yourself

1. Why not pool the vision tower down to one CLS?

The decoder needs to point at parts of the picture — the bicycle, the window, the bakery. One vector throws that away.

2. May an image token attend to a later image token?

Yes. The prefix is bidirectional.

3. Where does RoPE start?

At the first image token, then straight on through the prompt and the answer. No reset.

Go to the source

Keep going