THE EVOLVING TRANSFORMER · STOP 08
Two tiny encoders, one shared space. A red-tile image and a short caption learn to find each other by contrastive loss.
Every earlier stop was a language model. CLIP is two encoders. A 16×16 RGB tile is cut into 4×4 patches (16 tokens) plus a CLS. A short caption runs through a causal transformer. Both project into a 16-d space, L2-normalise, and a learned logit_scale (init log(1/0.07)) scales the B×B cosine matrix. The loss is symmetric cross-entropy on that matrix. The text tower gathers the last non-pad token — the EOT — because a padded last index is the pad, not the sentence.
Lay photographs on one table and index cards on another. Training shoves matching pairs together and pushes the rest apart. At test time you only write new cards (“a red tile”, “a blue tile”) and see which photo sits closest.
The concept chapter owns the loss and zero-shot story. This stop is a pair of tiny towers you can forward. CLIP concept →
| Tensor | Shape | Note |
|---|---|---|
| images | (2, 3, 16, 16) | RGB tiles |
| patches | (2, 16, 32) | 16 = (16/4)² |
| image + CLS | (2, 17, 32) | |
| caption ids | (2, 8) | |
| tower outputs | (2, 16) | embed_dim |
| logits | (2, 2) | image × text |
"""Tiny CLIP: two towers, one shared embedding space, symmetric InfoNCE.
Own example pair — a 16×16 RGB tile and a short caption. The text
tower reads the EOT token (last non-pad id), not blindly x[:, -1].
"""
from dataclasses import dataclass
import math
import torch
import torch.nn as nn
import torch.nn.functional as F
@dataclass
class CLIPConfig:
dmodel: int = 32
H: int = 4
N: int = 2
dff: int = 64
embed_dim: int = 16
dropout: float = 0.0
# image
image_size: int = 16
patch_size: int = 4
C: int = 3
# text
vocab_size: int = 40
max_text_len: int = 8
pad_id: int = 0
@property
def dk(self):
return self.dmodel // self.H
@property
def num_patches(self):
return (self.image_size // self.patch_size) ** 2
class TransformerBlock(nn.Module):
def __init__(self, config):
super().__init__()
self.ln1 = nn.LayerNorm(config.dmodel)
self.ln2 = nn.LayerNorm(config.dmodel)
self.wq = nn.Linear(config.dmodel, config.dmodel)
self.wk = nn.Linear(config.dmodel, config.dmodel)
self.wv = nn.Linear(config.dmodel, config.dmodel)
self.wo = nn.Linear(config.dmodel, config.dmodel)
self.ff = nn.Sequential(
nn.Linear(config.dmodel, config.dff),
nn.GELU(),
nn.Linear(config.dff, config.dmodel),
)
self.H = config.H
self.dk = config.dk
def forward(self, x, mask=None):
B, S, _ = x.shape
h = self.ln1(x)
q = self.wq(h).view(B, S, self.H, self.dk).transpose(1, 2)
k = self.wk(h).view(B, S, self.H, self.dk).transpose(1, 2)
v = self.wv(h).view(B, S, self.H, self.dk).transpose(1, 2)
scores = (q @ k.transpose(-1, -2)) / math.sqrt(self.dk)
if mask is not None:
scores = scores.masked_fill(mask[:, :, :S, :S] == 0, float("-inf"))
x = x + self.wo((scores.softmax(-1) @ v).transpose(1, 2).contiguous().view(B, S, -1))
return x + self.ff(self.ln2(x))
class TextEncoder(nn.Module):
def __init__(self, config):
super().__init__()
self.wte = nn.Embedding(config.vocab_size, config.dmodel)
self.wpe = nn.Embedding(config.max_text_len, config.dmodel)
self.blocks = nn.ModuleList([TransformerBlock(config) for _ in range(config.N)])
self.ln = nn.LayerNorm(config.dmodel)
self.proj = nn.Linear(config.dmodel, config.embed_dim, bias=False)
self.register_buffer("mask", torch.tril(torch.ones(1, 1, config.max_text_len, config.max_text_len)))
def forward(self, tokens):
B, S = tokens.shape
pos = torch.arange(S, device=tokens.device)
x = self.wte(tokens) + self.wpe(pos)
for block in self.blocks:
x = block(x, self.mask)
x = self.ln(x)
# EOT = last non-pad token in each row (pad_id is 0)
lengths = (tokens != 0).sum(dim=1).clamp(min=1) - 1
eot = x[torch.arange(B, device=x.device), lengths]
return self.proj(eot)
class PatchEmbedding(nn.Module):
def __init__(self, config):
super().__init__()
self.proj = nn.Conv2d(config.C, config.dmodel, config.patch_size, config.patch_size, bias=False)
def forward(self, x):
return self.proj(x).flatten(2).transpose(1, 2)
class ImageEncoder(nn.Module):
def __init__(self, config):
super().__init__()
self.patch = PatchEmbedding(config)
self.cls = nn.Parameter(torch.randn(1, 1, config.dmodel) * 0.02)
self.pos = nn.Parameter(torch.randn(1, config.num_patches + 1, config.dmodel) * 0.02)
self.blocks = nn.ModuleList([TransformerBlock(config) for _ in range(config.N)])
self.ln = nn.LayerNorm(config.dmodel)
self.proj = nn.Linear(config.dmodel, config.embed_dim, bias=False)
def forward(self, images):
x = self.patch(images)
cls = self.cls.expand(x.size(0), -1, -1)
x = torch.cat([cls, x], dim=1) + self.pos
for block in self.blocks:
x = block(x, mask=None) # bidirectional
return self.proj(self.ln(x)[:, 0])
class CLIP(nn.Module):
def __init__(self, config):
super().__init__()
self.image = ImageEncoder(config)
self.text = TextEncoder(config)
self.logit_scale = nn.Parameter(torch.tensor(math.log(1 / 0.07)))
def encode_image(self, images):
return F.normalize(self.image(images), dim=-1)
def encode_text(self, tokens):
return F.normalize(self.text(tokens), dim=-1)
def forward(self, images, tokens):
img = self.encode_image(images)
txt = self.encode_text(tokens)
logits = self.logit_scale.exp() * img @ txt.T # (B, B)
labels = torch.arange(logits.size(0), device=logits.device)
loss = (F.cross_entropy(logits, labels) + F.cross_entropy(logits.T, labels)) / 2
return logits, loss
Test note. Forward two random tiles and two padded captions. Assert every row of the shape table. Loss is finite. EOT readout uses the last non-pad index.
two towers · logits (2,2) · match pairs
Reading x[:, -1] on padded text. The last index is often pad. Gather the last non-pad id.
Forgetting L2 norm. Without it the temperature cannot turn a dot product into cosine.
Comparing CLIP to LLaMA's RMSNorm. These towers still use LayerNorm. RMSNorm arrives in the decoder of the next stop.
(16/4)² = 16.
(2, 2).
Padding. The EOT is the last real token.