THE TRANSFORMER DEEP CUTS · CHAPTER 09
Two towers, one sphere. Matched image–caption pairs sit on the diagonal of a cosine grid. That is enough for zero-shot classification.
A photo of a copper kettle steaming on a blue tiled stove already comes with a sentence, somewhere on the web. CLIP treats that pair as a training case. The picture and the sentence should land near each other on a sphere. A bicycle caption should land far from the kettle picture.
The other pictures in the same batch are the negatives. Nobody mined "hard" ones. A larger batch is a larger set of things to push away. The paper used 32,768.
The image tower is a ResNet or a ViT. The ViT cuts the picture into patches, plus a CLS token, and reads the CLS. The text tower is a causal Transformer. It reads the end-of-text token, not "the last position" — padding would make those two different. Both towers project into the same width and L2-normalise.
The text mask is causal, as in chapter 08. The image tower is bidirectional over its patches.
Three of our own pairs, already unit length, temperature 1 so the logits are the cosines:
| kettle | bicycle | pine | |
|---|---|---|---|
| kettle | 0.80 | 0.60 | 0.00 |
| bicycle | 0.60 | 0.80 | 1.00 |
| pine | 0.96 | 1.00 | 0.80 |
Row softmax plus column softmax, each against labels \(0,1,2\). Average the two cross-entropies: 1.057. The bicycle row is in trouble — it likes "pine" more than "bicycle". Training would push that off-diagonal down.
The paper multiplies the cosines by a learned logit_scale, stored as \(\log(1/0.07) \approx 2.659\), so the scale starts at 14.3 and is clamped later. Without that scale the softmax is too soft.
\(I\) and \(T\) are L2-normalised batches. \(y\) is \(0,1,\ldots,N-1\). Same loss both ways so an image finds its caption and a caption finds its image.
3×3 grid → row CE → col CE → loss 1.057
Interactive · Warm the temperature
At \(\tau = 1\) the grid is mild. At the paper's 14.3, what happens to the diagonal?
A ViT does not see pixels one by one. It sees non-overlapping squares.
224 → ÷14 → 16²=256 → ±CLS
| Image | Patch | Grid | Patches | Used by |
|---|---|---|---|---|
| 224 | 32 | 7 × 7 | 49 | CLIP ViT-B/32 |
| 224 | 14 | 16 × 16 | 256 | CLIP ViT-L/14, SigLIP / PaliGemma |
| 448 | 14 | 32 × 32 | 1,024 | PaliGemma at 448 |
CLIP's ViT also prepends a CLS token, so the sequence is 50 or 257. PaliGemma's SigLIP tower keeps all 256 patches and drops the CLS. That difference is the next chapter.
To label a new picture as kettle / bicycle / pine, embed the three captions once, cache them, embed the picture, and take the argmax cosine. No new linear head. Prompt templates ("a photo of a …") and an ensemble of them help; the paper used 80.
captions → image → argmax → not a captioner
CLIP is not a captioner. It scores pairs. Generating "the kettle is on a blue stove" is a different job, for a decoder that can write. That is chapter 10.
The towers reuse the block you already have. The new math is the grid.
"""Symmetric contrastive loss used by CLIP.
One image encoder, one text encoder, one shared sphere. A batch of N
pairs makes an N by N grid of cosine similarities. The loss pulls the
diagonal up (matched pairs) and the off-diagonal down (the other items
in the same batch — in-batch negatives, not specially mined hard ones).
"""
import torch
import torch.nn.functional as F
def clip_logits(image_feat, text_feat, logit_scale):
"""image_feat, text_feat: (N, d). logit_scale is exp(learned temperature).
Features must already be L2-normalised. The paper initialises the
temperature at 0.07, i.e. logit_scale = 1/0.07 ≈ 14.3, and stores
it in log space so it cannot go negative.
"""
image_feat = F.normalize(image_feat, dim=-1)
text_feat = F.normalize(text_feat, dim=-1)
# (N, d) @ (d, N) -> (N, N). Row i is image i against every caption.
return logit_scale * image_feat @ text_feat.T
def clip_loss(image_feat, text_feat, logit_scale):
"""Symmetric cross-entropy: image-to-text plus text-to-image, averaged."""
logits = clip_logits(image_feat, text_feat, logit_scale)
labels = torch.arange(logits.size(0), device=logits.device)
loss_i = F.cross_entropy(logits, labels) # rows: each image finds its caption
loss_t = F.cross_entropy(logits.T, labels) # columns: each caption finds its image
return 0.5 * (loss_i + loss_t), logits
def num_patches(image_size, patch_size):
"""A 224 image with patch 14 becomes a 16 by 16 grid: 256 patches."""
side = image_size // patch_size
return side * side
Reading the last text position instead of EOT. Padding makes those two different. Gather at the EOT id.
"In-batch negatives are hard negatives." They are whatever else landed in the batch. Hard negatives are mined. CLIP does not mine.
Forgetting the temperature. The learned scale is part of the loss, not a demo slider you can drop.
Counting 224/16 = 196 for PaliGemma. PaliGemma's SigLIP uses patch 14, so 256 patches at 224.
So each image finds its caption and each caption finds its image. One direction alone lets the other tower collapse.
256 and 49.
A cosine. Not a generated sentence.