THE TRANSFORMER DEEP CUTS · CHAPTER 09

CLIP: Learning Images
and Text Together

Two towers, one sphere. Matched image–caption pairs sit on the diagonal of a cosine grid. That is enough for zero-shot classification.

~16 min read1 interactiveTested PyTorch code

What you'll learn

  • A dual encoder and a symmetric contrastive loss
  • In-batch negatives, and the learned temperature \(1/0.07 \approx 14.3\)
  • Patch size 14 on a 224 image gives 256 patches
  • Zero-shot classification by scoring captions, not by training a new head

Before you start

  • Families — CLIP's text tower is causal
  • Cosine similarity and cross-entropy

A caption is a label you did not have to write

A photo of a copper kettle steaming on a blue tiled stove already comes with a sentence, somewhere on the web. CLIP treats that pair as a training case. The picture and the sentence should land near each other on a sphere. A bicycle caption should land far from the kettle picture.

The other pictures in the same batch are the negatives. Nobody mined "hard" ones. A larger batch is a larger set of things to push away. The paper used 32,768.

An image encoder and a text encoder

The image tower is a ResNet or a ViT. The ViT cuts the picture into patches, plus a CLS token, and reads the CLS. The text tower is a causal Transformer. It reads the end-of-text token, not "the last position" — padding would make those two different. Both towers project into the same width and L2-normalise.

The text mask is causal, as in chapter 08. The image tower is bidirectional over its patches.

Rows and columns, then average

Three of our own pairs, already unit length, temperature 1 so the logits are the cosines:

Rows are images: copper kettle, red bicycle, lone pine. Columns are those three captions.
kettlebicyclepine
kettle0.800.600.00
bicycle0.600.801.00
pine0.961.000.80

Row softmax plus column softmax, each against labels \(0,1,2\). Average the two cross-entropies: 1.057. The bicycle row is in trouble — it likes "pine" more than "bicycle". Training would push that off-diagonal down.

The paper multiplies the cosines by a learned logit_scale, stored as \(\log(1/0.07) \approx 2.659\), so the scale starts at 14.3 and is clamped later. Without that scale the softmax is too soft.

\(I\) and \(T\) are L2-normalised batches. \(y\) is \(0,1,\ldots,N-1\). Same loss both ways so an image finds its caption and a caption finds its image.

3×3 grid  →  row CE  →  col CE  →  loss 1.057

MANIM Pull the diagonal up; push in-batch negatives down. Symmetric CE both ways.

Interactive · Warm the temperature

At \(\tau = 1\) the grid is mild. At the paper's 14.3, what happens to the diagonal?

1.0

224 ÷ 14 = 16, and \(16^2 = 256\)

A ViT does not see pixels one by one. It sees non-overlapping squares.

224  →  ÷14  →  16²=256  →  ±CLS

MANIM 224 ÷ 14 = 16, and 16² = 256 patches.
ImagePatchGridPatchesUsed by
224327 × 749CLIP ViT-B/32
2241416 × 16256CLIP ViT-L/14, SigLIP / PaliGemma
4481432 × 321,024PaliGemma at 448

CLIP's ViT also prepends a CLS token, so the sequence is 50 or 257. PaliGemma's SigLIP tower keeps all 256 patches and drops the CLS. That difference is the next chapter.

Classify by writing a sentence

To label a new picture as kettle / bicycle / pine, embed the three captions once, cache them, embed the picture, and take the argmax cosine. No new linear head. Prompt templates ("a photo of a …") and an ensemble of them help; the paper used 80.

captions  →  image  →  argmax  →  not a captioner

MANIM Classify by writing a sentence — CLIP scores pairs, it does not generate.

CLIP is not a captioner. It scores pairs. Generating "the kettle is on a blue stove" is a different job, for a decoder that can write. That is chapter 10.

The loss, not the towers

The towers reuse the block you already have. The new math is the grid.

clip_loss.pysymmetric CE; 224/14 = 256 patches
"""Symmetric contrastive loss used by CLIP.

One image encoder, one text encoder, one shared sphere. A batch of N
pairs makes an N by N grid of cosine similarities. The loss pulls the
diagonal up (matched pairs) and the off-diagonal down (the other items
in the same batch — in-batch negatives, not specially mined hard ones).
"""
import torch
import torch.nn.functional as F


def clip_logits(image_feat, text_feat, logit_scale):
    """image_feat, text_feat: (N, d). logit_scale is exp(learned temperature).

    Features must already be L2-normalised. The paper initialises the
    temperature at 0.07, i.e. logit_scale = 1/0.07 ≈ 14.3, and stores
    it in log space so it cannot go negative.
    """
    image_feat = F.normalize(image_feat, dim=-1)
    text_feat = F.normalize(text_feat, dim=-1)
    # (N, d) @ (d, N) -> (N, N). Row i is image i against every caption.
    return logit_scale * image_feat @ text_feat.T


def clip_loss(image_feat, text_feat, logit_scale):
    """Symmetric cross-entropy: image-to-text plus text-to-image, averaged."""
    logits = clip_logits(image_feat, text_feat, logit_scale)
    labels = torch.arange(logits.size(0), device=logits.device)
    loss_i = F.cross_entropy(logits, labels)       # rows: each image finds its caption
    loss_t = F.cross_entropy(logits.T, labels)     # columns: each caption finds its image
    return 0.5 * (loss_i + loss_t), logits


def num_patches(image_size, patch_size):
    """A 224 image with patch 14 becomes a 16 by 16 grid: 256 patches."""
    side = image_size // patch_size
    return side * side

Mistakes learners make

Reading the last text position instead of EOT. Padding makes those two different. Gather at the EOT id.

"In-batch negatives are hard negatives." They are whatever else landed in the batch. Hard negatives are mined. CLIP does not mine.

Forgetting the temperature. The learned scale is part of the loss, not a demo slider you can drop.

Counting 224/16 = 196 for PaliGemma. PaliGemma's SigLIP uses patch 14, so 256 patches at 224.

What to remember

  • Two towers, L2-normalised, one \(N \times N\) cosine grid, symmetric cross-entropy.
  • Temperature starts at 14.3. Batch size is the negative set (32,768 in the paper).
  • Patch 14 on 224 is 256 patches. CLIP ViT adds a CLS; PaliGemma's SigLIP does not.
  • Zero-shot = score captions. Generation needs a decoder.

Check yourself

1. Why both a row loss and a column loss?

So each image finds its caption and each caption finds its image. One direction alone lets the other tower collapse.

2. How many patches does a 224 image contribute at patch 14? At patch 32?

256 and 49.

3. What does CLIP output for a new picture and the word "kettle"?

A cosine. Not a generated sentence.

Go to the source

Keep going