THE EVOLVING TRANSFORMER · STOP 08

CLIP

Two tiny encoders, one shared space. A red-tile image and a short caption learn to find each other by contrastive loss.

~12 min readContrastiveTwo towers

What you'll learn

  • A ViT image tower (patches + CLS) and a causal text tower
  • L2-normalised embeddings and a learned temperature
  • Symmetric InfoNCE: the diagonal of the B×B matrix is the match
  • Read the EOT token, not blindly the last position

Before you start

From generating tokens to matching pairs

Every earlier stop was a language model. CLIP is two encoders. A 16×16 RGB tile is cut into 4×4 patches (16 tokens) plus a CLS. A short caption runs through a causal transformer. Both project into a 16-d space, L2-normalise, and a learned logit_scale (init log(1/0.07)) scales the B×B cosine matrix. The loss is symmetric cross-entropy on that matrix. The text tower gathers the last non-pad token — the EOT — because a padded last index is the pad, not the sentence.

Lay photographs on one table and index cards on another. Training shoves matching pairs together and pushes the rest apart. At test time you only write new cards (“a red tile”, “a blue tile”) and see which photo sits closest.

The concept chapter owns the loss and zero-shot story. This stop is a pair of tiny towers you can forward. CLIP concept →

A batch of two tiles

TensorShapeNote
images(2, 3, 16, 16)RGB tiles
patches(2, 16, 32)16 = (16/4)²
image + CLS(2, 17, 32)
caption ids(2, 8)
tower outputs(2, 16)embed_dim
logits(2, 2)image × text

Own tiny encoder pair

clip.pyforward + (2, 2) logits + EOT gather
"""Tiny CLIP: two towers, one shared embedding space, symmetric InfoNCE.

Own example pair — a 16×16 RGB tile and a short caption. The text
tower reads the EOT token (last non-pad id), not blindly x[:, -1].
"""

from dataclasses import dataclass
import math

import torch
import torch.nn as nn
import torch.nn.functional as F


@dataclass
class CLIPConfig:
    dmodel: int = 32
    H: int = 4
    N: int = 2
    dff: int = 64
    embed_dim: int = 16
    dropout: float = 0.0
    # image
    image_size: int = 16
    patch_size: int = 4
    C: int = 3
    # text
    vocab_size: int = 40
    max_text_len: int = 8
    pad_id: int = 0

    @property
    def dk(self):
        return self.dmodel // self.H

    @property
    def num_patches(self):
        return (self.image_size // self.patch_size) ** 2


class TransformerBlock(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.ln1 = nn.LayerNorm(config.dmodel)
        self.ln2 = nn.LayerNorm(config.dmodel)
        self.wq = nn.Linear(config.dmodel, config.dmodel)
        self.wk = nn.Linear(config.dmodel, config.dmodel)
        self.wv = nn.Linear(config.dmodel, config.dmodel)
        self.wo = nn.Linear(config.dmodel, config.dmodel)
        self.ff = nn.Sequential(
            nn.Linear(config.dmodel, config.dff),
            nn.GELU(),
            nn.Linear(config.dff, config.dmodel),
        )
        self.H = config.H
        self.dk = config.dk

    def forward(self, x, mask=None):
        B, S, _ = x.shape
        h = self.ln1(x)
        q = self.wq(h).view(B, S, self.H, self.dk).transpose(1, 2)
        k = self.wk(h).view(B, S, self.H, self.dk).transpose(1, 2)
        v = self.wv(h).view(B, S, self.H, self.dk).transpose(1, 2)
        scores = (q @ k.transpose(-1, -2)) / math.sqrt(self.dk)
        if mask is not None:
            scores = scores.masked_fill(mask[:, :, :S, :S] == 0, float("-inf"))
        x = x + self.wo((scores.softmax(-1) @ v).transpose(1, 2).contiguous().view(B, S, -1))
        return x + self.ff(self.ln2(x))


class TextEncoder(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.wte = nn.Embedding(config.vocab_size, config.dmodel)
        self.wpe = nn.Embedding(config.max_text_len, config.dmodel)
        self.blocks = nn.ModuleList([TransformerBlock(config) for _ in range(config.N)])
        self.ln = nn.LayerNorm(config.dmodel)
        self.proj = nn.Linear(config.dmodel, config.embed_dim, bias=False)
        self.register_buffer("mask", torch.tril(torch.ones(1, 1, config.max_text_len, config.max_text_len)))

    def forward(self, tokens):
        B, S = tokens.shape
        pos = torch.arange(S, device=tokens.device)
        x = self.wte(tokens) + self.wpe(pos)
        for block in self.blocks:
            x = block(x, self.mask)
        x = self.ln(x)
        # EOT = last non-pad token in each row (pad_id is 0)
        lengths = (tokens != 0).sum(dim=1).clamp(min=1) - 1
        eot = x[torch.arange(B, device=x.device), lengths]
        return self.proj(eot)


class PatchEmbedding(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.proj = nn.Conv2d(config.C, config.dmodel, config.patch_size, config.patch_size, bias=False)

    def forward(self, x):
        return self.proj(x).flatten(2).transpose(1, 2)


class ImageEncoder(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.patch = PatchEmbedding(config)
        self.cls = nn.Parameter(torch.randn(1, 1, config.dmodel) * 0.02)
        self.pos = nn.Parameter(torch.randn(1, config.num_patches + 1, config.dmodel) * 0.02)
        self.blocks = nn.ModuleList([TransformerBlock(config) for _ in range(config.N)])
        self.ln = nn.LayerNorm(config.dmodel)
        self.proj = nn.Linear(config.dmodel, config.embed_dim, bias=False)

    def forward(self, images):
        x = self.patch(images)
        cls = self.cls.expand(x.size(0), -1, -1)
        x = torch.cat([cls, x], dim=1) + self.pos
        for block in self.blocks:
            x = block(x, mask=None)  # bidirectional
        return self.proj(self.ln(x)[:, 0])


class CLIP(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.image = ImageEncoder(config)
        self.text = TextEncoder(config)
        self.logit_scale = nn.Parameter(torch.tensor(math.log(1 / 0.07)))

    def encode_image(self, images):
        return F.normalize(self.image(images), dim=-1)

    def encode_text(self, tokens):
        return F.normalize(self.text(tokens), dim=-1)

    def forward(self, images, tokens):
        img = self.encode_image(images)
        txt = self.encode_text(tokens)
        logits = self.logit_scale.exp() * img @ txt.T  # (B, B)
        labels = torch.arange(logits.size(0), device=logits.device)
        loss = (F.cross_entropy(logits, labels) + F.cross_entropy(logits.T, labels)) / 2
        return logits, loss

Test note. Forward two random tiles and two padded captions. Assert every row of the shape table. Loss is finite. EOT readout uses the last non-pad index.

two towers · logits (2,2) · match pairs

MANIM Two towers on one sphere; logits (2,2) image × text.

Mistakes this stop used to make

Reading x[:, -1] on padded text. The last index is often pad. Gather the last non-pad id.

Forgetting L2 norm. Without it the temperature cannot turn a dot product into cosine.

Comparing CLIP to LLaMA's RMSNorm. These towers still use LayerNorm. RMSNorm arrives in the decoder of the next stop.

What to remember

  • Two towers, one embed_dim, L2 norm, learned temperature.
  • InfoNCE on a B×B matrix. Labels are 0…B−1.
  • Image = CLS. Text = EOT (last non-pad), not the padded tail.

Check yourself

1. How many patches does a 16×16 image with 4×4 patches make?

(16/4)² = 16.

2. What shape are the similarity logits for batch 2?

(2, 2).

3. Why not take the last text position?

Padding. The EOT is the last real token.

Go to the source

Keep going