FINDING NEEDLES · CHAPTER 17
A leaf photo has no words to match. Put photos and questions in one space, and a sentence can find a picture. Split a scanned page into patches, and a question can find the corner that answers it.
A botanical society keeps an archive of leaf photos. The files are named IMG_0412.jpg, IMG_0413.jpg, and so on. Nobody wrote captions. A visitor types “a red lobed leaf”.
Every method so far in this series compares words, or vectors made from words. None of these files has a word. Keyword search returns nothing, and a text encoder has nothing to encode.
The archive also holds scanned field-notebook sheets: handwriting, pencil sketches, a lot of empty paper. A scanner’s text layer misses the handwriting and cannot read a sketch at all.
Train an image encoder and a text encoder so that a photo and a sentence describing it land near each other. Then the photo is a point, the question is a point, and search is the nearest-neighbour lookup you already know. For a page, keep one point per small square of the page, so a question can match the one corner that matters.
A shared space is built by contrastive training on matched pairs: a photo and its caption are pulled together, and every other caption in the batch is pushed away. CLIP was trained this way on 400 million image–caption pairs from the web. The training itself is the subject of The Evolving Transformer’s CLIP stop; this chapter only uses the result.
photo · shared · towers · cross
For search, the result is all you need: one function turns a photo into a vector, another turns text into a vector, and the two vectors can be compared directly.
Here are four photos in a toy space with four axes. Real axes have no names, and there are hundreds of them. These are made by hand so you can read them.
| Photo | Shows | (lobed, needle, toothed edge, autumn red) |
|---|---|---|
| IMG_0412.jpg | oak leaf, summer | (0.9, 0, 0.1, 0.1) |
| IMG_0413.jpg | pine needles | (0, 1, 0, 0) |
| IMG_0420.jpg | maple leaf, autumn | (0.8, 0, 0.2, 0.9) |
| IMG_0431.jpg | elm leaf | (0.1, 0, 0.9, 0.2) |
The “Shows” column is for you. The search never sees it. Only the vectors exist.
The text encoder turns “a red lobed leaf” into (0.7, 0, 0, 0.7): half lobed, half red. Score every photo by cosine, exactly as in chapter 2.
encode the words with the text tower, the photo with the image tower, then take the cosine. Only the encoders are new.
| Rank | Photo | Cosine |
|---|---|---|
| 1 | IMG_0420.jpg (maple, autumn) | 0.985 |
| 2 | IMG_0412.jpg (oak, summer) | 0.776 |
| 3 | IMG_0431.jpg (elm) | 0.229 |
| 4 | IMG_0413.jpg (pine) | 0.000 |
The red maple wins at 0.985. The oak is lobed but green, so it comes second at 0.776. Pine shares no axis with the query and scores 0.000.
Everything from earlier chapters carries over. Encode the photos once, offline. Store the vectors in a vector index. At query time, encode the text and look up the nearest photos. To search by photo instead (“more leaves like this one”), run the image encoder on the query photo and use the same index.
The visitor now asks for “samara seed sketch”: a drawing of the winged seed a maple drops. Two scanned sheets compete.
Sheet 12 is the one the visitor wants. Cut each sheet into a 2 × 3 grid of patches, and give each patch a vector on three axes: the word “samara”, a seed drawing, and anything else.
The single-vector way averages the six patches into one page vector, and averages the query’s two token vectors into one query vector. Then it takes one cosine.
Sheet 12’s average is (1/6, 1/6, 4/6). It points mostly at blank paper. Its cosine with the query is 0.333. Sheet 31’s average is (0.3, 0.3, 0.8), which scores 0.469. The averaged vectors rank the wrong sheet first.
Late interaction keeps every patch vector. At query time, each query token looks at every patch, keeps its best match, and the page score is the sum of those best matches. This is the scoring rule ColBERT introduced for text passages. ColPali applies it to page images.
for each query token \(q_i\), find the page patch \(p_j\) it matches best; add up those best scores. A blank patch never wins a max, so it costs nothing.
MaxSim ranks sheet 12 first, 2.00 to 1.20. The pooled cosine ranked it second, 0.333 to 0.469. Same patches, same query; only the scoring rule changed.
Interactive · How much blank paper can a page hold?
Add blank patches to sheet 12. Which score notices, and when does the pooled cosine flip to the wrong sheet?
With no blank patches, the pooled cosine for sheet 12 is 1.000. With two it is 0.577, still ahead of 0.469. With three it drops to 0.426 and loses. MaxSim stays at 2.00 the whole way. A real page is mostly margin and white space, so this is the common case, not the edge case.
Standard library only. The photo vectors, patch vectors, and query vectors are made by hand; a real system gets them from trained encoders.
"""Multimodal search on a toy leaf-photo archive.
Part 1: photos and a text query share one 4-D space; rank by cosine.
Part 2: scanned notebook sheets as patch vectors; MaxSim versus one
averaged vector per sheet.
All vectors are made by hand. A real system gets them from trained
image and text encoders with hundreds of dimensions.
"""
import math
# Part 1. Axes: lobed, needle, toothed edge, autumn red.
PHOTOS = {
"IMG_0412.jpg": [0.9, 0.0, 0.1, 0.1], # oak leaf, summer
"IMG_0413.jpg": [0.0, 1.0, 0.0, 0.0], # pine needles
"IMG_0420.jpg": [0.8, 0.0, 0.2, 0.9], # maple leaf, autumn
"IMG_0431.jpg": [0.1, 0.0, 0.9, 0.2], # elm leaf
}
# What the text encoder returns for "a red lobed leaf".
QUERY_TEXT = [0.7, 0.0, 0.0, 0.7]
def dot(a, b):
return sum(x * y for x, y in zip(a, b))
def cosine(a, b):
return dot(a, b) / (math.sqrt(dot(a, a)) * math.sqrt(dot(b, b)))
def search_photos(query=QUERY_TEXT):
"""Text in, photos out: the same cosine as text-to-text search."""
scores = {name: cosine(query, vec) for name, vec in PHOTOS.items()}
return sorted(scores.items(), key=lambda item: -item[1])
# Part 2. Patch axes: the word "samara", a seed drawing, anything else.
LABEL, SKETCH, BLANK = [1, 0, 0], [0, 1, 0], [0, 0, 1]
def sheet_12(blanks=4):
"""A handwritten label, a seed sketch, and empty paper."""
return [LABEL, SKETCH] + [BLANK] * blanks
# A typed page of mixed notes: every patch is a little on topic.
SHEET_31 = [[0.6, 0, 0.8], [0, 0.6, 0.8]] * 3
# Query "samara seed sketch": one vector per query token.
QUERY_TOKENS = [[1, 0, 0], [0, 1, 0]]
def maxsim(query_tokens, patches):
"""Each query token takes its best patch; add those best scores."""
return sum(max(dot(q, p) for p in patches) for q in query_tokens)
def mean(vectors):
return [sum(col) / len(vectors) for col in zip(*vectors)]
def pooled(query_tokens, patches):
"""The single-vector way: average each side, then one cosine."""
return cosine(mean(query_tokens), mean(patches))
def numbers_per_page(patches=1024, dim=128):
"""Storage for one page under late interaction."""
return patches * dim
if __name__ == "__main__":
for name, score in search_photos():
print(name, round(score, 3))
for label, sheet in [("sheet 12", sheet_12()), ("sheet 31", SHEET_31)]:
print(label, maxsim(QUERY_TOKENS, sheet), round(pooled(QUERY_TOKENS, sheet), 3))
print(numbers_per_page()) # 131072
ColPali cuts each page image into a 32 × 32 grid, 1024 patches, and keeps a 128-number vector for each. That is 1024 × 128 = 131,072 numbers per page. A single-vector page index stores one vector of a few hundred numbers.
Small index, fast nearest-neighbour search. Blurs small details into the page average. Fine for photos where the whole picture is the subject.
Hundreds of times the storage. Scoring touches every patch of every candidate. Finds the label in the corner, the number in the chart, the sketch in the margin.
The usual compromise is the funnel from chapter 9: a cheap search proposes a few hundred candidate pages, and MaxSim reorders only those. The patch vectors can also be stored in fewer bits, since each one only needs to win or lose a max.
Mixing encoders from two models. A text vector from one model and an image vector from another live in different spaces. Their cosine is a number, but it means nothing. Both towers must come from the same jointly trained pair.
Indexing only the text layer of a scan. Handwriting, sketches, and charts are invisible to it. Sheet 12’s label and sketch would both be lost, and the page would look empty.
Averaging patches to save space. It brings back the blur MaxSim removed. On sheet 12, the pooled cosine falls from 1.000 to 0.333 as blank paper is added, and the page drops below a vaguer one.
Expecting a photo encoder to read fine print. A model trained on photos and short captions is good at “red lobed leaf”, and weak at the small text on a specimen label. Page retrieval models are trained on document images for that reason.
Its vector is (0, 1, 0, 0). The query is (0.7, 0, 0, 0.7). They share no non-zero axis, so the dot product is 0.
Only the query encoder. Run the image encoder on the query photo and look it up in the same photo index.
A blank patch scores 0 against both query tokens. Each token already has a patch scoring 1, so the blank never wins the max.
Two or fewer. At two it is 0.577, above sheet 31’s 0.469. At three it is 0.426, below.
Each page carries 131,072 numbers. Scoring them all is slow. Use a cheap first stage to pick candidates, then MaxSim on those.