FINDING NEEDLES · CHAPTER 17

Multimodal search

A leaf photo has no words to match. Put photos and questions in one space, and a sentence can find a picture. Split a scanned page into patches, and a question can find the corner that answers it.

~13 min read4 photos, 2 sheetsTested Python

What you'll learn

  • Why text-to-image search is ordinary vector search once both sides share one space
  • How a text query ranks photos by cosine, with numbers you can check
  • Why one averaged vector per scanned page drowns small details in blank paper
  • How MaxSim scores a page patch by patch, and what that costs in storage

Before you start

A photo has no words to match

A botanical society keeps an archive of leaf photos. The files are named IMG_0412.jpg, IMG_0413.jpg, and so on. Nobody wrote captions. A visitor types “a red lobed leaf”.

Every method so far in this series compares words, or vectors made from words. None of these files has a word. Keyword search returns nothing, and a text encoder has nothing to encode.

The archive also holds scanned field-notebook sheets: handwriting, pencil sketches, a lot of empty paper. A scanner’s text layer misses the handwriting and cannot read a sketch at all.

Train an image encoder and a text encoder so that a photo and a sentence describing it land near each other. Then the photo is a point, the question is a point, and search is the nearest-neighbour lookup you already know. For a page, keep one point per small square of the page, so a question can match the one corner that matters.

Two encoders, one set of axes

A shared space is built by contrastive training on matched pairs: a photo and its caption are pulled together, and every other caption in the batch is pushed away. CLIP was trained this way on 400 million image–caption pairs from the web. The training itself is the subject of The Evolving Transformer’s CLIP stop; this chapter only uses the result.

photo · shared · towers · cross

MANIM Multimodal search: shared spaces so text finds images.

For search, the result is all you need: one function turns a photo into a vector, another turns text into a vector, and the two vectors can be compared directly.

Here are four photos in a toy space with four axes. Real axes have no names, and there are hundreds of them. These are made by hand so you can read them.

PhotoShows(lobed, needle, toothed edge, autumn red)
IMG_0412.jpgoak leaf, summer(0.9, 0, 0.1, 0.1)
IMG_0413.jpgpine needles(0, 1, 0, 0)
IMG_0420.jpgmaple leaf, autumn(0.8, 0, 0.2, 0.9)
IMG_0431.jpgelm leaf(0.1, 0, 0.9, 0.2)

The “Shows” column is for you. The search never sees it. Only the vectors exist.

The same cosine, across modalities

The text encoder turns “a red lobed leaf” into (0.7, 0, 0, 0.7): half lobed, half red. Score every photo by cosine, exactly as in chapter 2.

encode the words with the text tower, the photo with the image tower, then take the cosine. Only the encoders are new.

RankPhotoCosine
1IMG_0420.jpg (maple, autumn)0.985
2IMG_0412.jpg (oak, summer)0.776
3IMG_0431.jpg (elm)0.229
4IMG_0413.jpg (pine)0.000

The red maple wins at 0.985. The oak is lobed but green, so it comes second at 0.776. Pine shares no axis with the query and scores 0.000.

Everything from earlier chapters carries over. Encode the photos once, offline. Store the vectors in a vector index. At query time, encode the text and look up the nearest photos. To search by photo instead (“more leaves like this one”), run the image encoder on the query photo and use the same index.

One vector per page blurs the corner that matters

The visitor now asks for “samara seed sketch”: a drawing of the winged seed a maple drops. Two scanned sheets compete.

Sheet 12 is the one the visitor wants. Cut each sheet into a 2 × 3 grid of patches, and give each patch a vector on three axes: the word “samara”, a seed drawing, and anything else.

sheet 12sheet 31 label(1,0,0) sketch(0,1,0) blankblankblankblank (.6,0,.8)(0,.6,.8)(.6,0,.8) (0,.6,.8)(.6,0,.8)(0,.6,.8) blank = (0,0,1)every patch a little on topic
Two sheets cut into six patches. Sheet 12 has two strong patches and four blank ones.

The single-vector way averages the six patches into one page vector, and averages the query’s two token vectors into one query vector. Then it takes one cosine.

Sheet 12’s average is (1/6, 1/6, 4/6). It points mostly at blank paper. Its cosine with the query is 0.333. Sheet 31’s average is (0.3, 0.3, 0.8), which scores 0.469. The averaged vectors rank the wrong sheet first.

Let each query token find its own patch

Late interaction keeps every patch vector. At query time, each query token looks at every patch, keeps its best match, and the page score is the sum of those best matches. This is the scoring rule ColBERT introduced for text passages. ColPali applies it to page images.

for each query token \(q_i\), find the page patch \(p_j\) it matches best; add up those best scores. A blank patch never wins a max, so it costs nothing.

  1. Sheet 12, token “samara” (1, 0, 0). Best patch is the label: dot 1. The sketch and blanks score 0.
  2. Sheet 12, token “seed sketch” (0, 1, 0). Best patch is the sketch: dot 1. Total 1 + 1 = 2.00.
  3. Sheet 31, both tokens. The best any patch offers either token is 0.6. Total 0.6 + 0.6 = 1.20.

MaxSim ranks sheet 12 first, 2.00 to 1.20. The pooled cosine ranked it second, 0.333 to 0.469. Same patches, same query; only the scoring rule changed.

Interactive · How much blank paper can a page hold?

Add blank patches to sheet 12. Which score notices, and when does the pooled cosine flip to the wrong sheet?

4

With no blank patches, the pooled cosine for sheet 12 is 1.000. With two it is 0.577, still ahead of 0.469. With three it drops to 0.426 and loses. MaxSim stays at 2.00 the whole way. A real page is mostly margin and white space, so this is the common case, not the edge case.

Both searches in one file

Standard library only. The photo vectors, patch vectors, and query vectors are made by hand; a real system gets them from trained encoders.

multimodal.pymaple 0.985 · MaxSim 2.00 vs 1.20
"""Multimodal search on a toy leaf-photo archive.

Part 1: photos and a text query share one 4-D space; rank by cosine.
Part 2: scanned notebook sheets as patch vectors; MaxSim versus one
averaged vector per sheet.

All vectors are made by hand. A real system gets them from trained
image and text encoders with hundreds of dimensions.
"""
import math

# Part 1. Axes: lobed, needle, toothed edge, autumn red.
PHOTOS = {
    "IMG_0412.jpg": [0.9, 0.0, 0.1, 0.1],  # oak leaf, summer
    "IMG_0413.jpg": [0.0, 1.0, 0.0, 0.0],  # pine needles
    "IMG_0420.jpg": [0.8, 0.0, 0.2, 0.9],  # maple leaf, autumn
    "IMG_0431.jpg": [0.1, 0.0, 0.9, 0.2],  # elm leaf
}

# What the text encoder returns for "a red lobed leaf".
QUERY_TEXT = [0.7, 0.0, 0.0, 0.7]


def dot(a, b):
    return sum(x * y for x, y in zip(a, b))


def cosine(a, b):
    return dot(a, b) / (math.sqrt(dot(a, a)) * math.sqrt(dot(b, b)))


def search_photos(query=QUERY_TEXT):
    """Text in, photos out: the same cosine as text-to-text search."""
    scores = {name: cosine(query, vec) for name, vec in PHOTOS.items()}
    return sorted(scores.items(), key=lambda item: -item[1])


# Part 2. Patch axes: the word "samara", a seed drawing, anything else.
LABEL, SKETCH, BLANK = [1, 0, 0], [0, 1, 0], [0, 0, 1]


def sheet_12(blanks=4):
    """A handwritten label, a seed sketch, and empty paper."""
    return [LABEL, SKETCH] + [BLANK] * blanks


# A typed page of mixed notes: every patch is a little on topic.
SHEET_31 = [[0.6, 0, 0.8], [0, 0.6, 0.8]] * 3

# Query "samara seed sketch": one vector per query token.
QUERY_TOKENS = [[1, 0, 0], [0, 1, 0]]


def maxsim(query_tokens, patches):
    """Each query token takes its best patch; add those best scores."""
    return sum(max(dot(q, p) for p in patches) for q in query_tokens)


def mean(vectors):
    return [sum(col) / len(vectors) for col in zip(*vectors)]


def pooled(query_tokens, patches):
    """The single-vector way: average each side, then one cosine."""
    return cosine(mean(query_tokens), mean(patches))


def numbers_per_page(patches=1024, dim=128):
    """Storage for one page under late interaction."""
    return patches * dim


if __name__ == "__main__":
    for name, score in search_photos():
        print(name, round(score, 3))
    for label, sheet in [("sheet 12", sheet_12()), ("sheet 31", SHEET_31)]:
        print(label, maxsim(QUERY_TOKENS, sheet), round(pooled(QUERY_TOKENS, sheet), 3))
    print(numbers_per_page())  # 131072

A thousand vectors per page

ColPali cuts each page image into a 32 × 32 grid, 1024 patches, and keeps a 128-number vector for each. That is 1024 × 128 = 131,072 numbers per page. A single-vector page index stores one vector of a few hundred numbers.

One vector per page

Small index, fast nearest-neighbour search. Blurs small details into the page average. Fine for photos where the whole picture is the subject.

One vector per patch

Hundreds of times the storage. Scoring touches every patch of every candidate. Finds the label in the corner, the number in the chart, the sketch in the margin.

The usual compromise is the funnel from chapter 9: a cheap search proposes a few hundred candidate pages, and MaxSim reorders only those. The patch vectors can also be stored in fewer bits, since each one only needs to win or lose a max.

Mistakes learners make

Mixing encoders from two models. A text vector from one model and an image vector from another live in different spaces. Their cosine is a number, but it means nothing. Both towers must come from the same jointly trained pair.

Indexing only the text layer of a scan. Handwriting, sketches, and charts are invisible to it. Sheet 12’s label and sketch would both be lost, and the page would look empty.

Averaging patches to save space. It brings back the blur MaxSim removed. On sheet 12, the pooled cosine falls from 1.000 to 0.333 as blank paper is added, and the page drops below a vaguer one.

Expecting a photo encoder to read fine print. A model trained on photos and short captions is good at “red lobed leaf”, and weak at the small text on a specimen label. Page retrieval models are trained on document images for that reason.

What to remember

  • A shared space makes text-to-photo search ordinary vector search. Only the encoders change.
  • “A red lobed leaf” ranks the maple 0.985, the oak 0.776, the elm 0.229, the pine 0.000.
  • One averaged vector per sheet ranks sheet 31 over sheet 12, 0.469 to 0.333, because sheet 12 is mostly blank.
  • MaxSim sums each query token’s best patch: sheet 12 scores 2.00, sheet 31 scores 1.20.
  • The price is storage: 1024 × 128 = 131,072 numbers per page.

Check yourself

1. Why does the pine photo score exactly 0?

Its vector is (0, 1, 0, 0). The query is (0.7, 0, 0, 0.7). They share no non-zero axis, so the dot product is 0.

2. What do you change to search by photo instead of by text?

Only the query encoder. Run the image encoder on the query photo and look it up in the same photo index.

3. Why doesn’t adding blank patches change sheet 12’s MaxSim?

A blank patch scores 0 against both query tokens. Each token already has a patch scoring 1, so the blank never wins the max.

4. With how many blank patches does the pooled cosine still prefer sheet 12?

Two or fewer. At two it is 0.577, above sheet 31’s 0.469. At three it is 0.426, below.

5. Why not use MaxSim on every page in the archive for every query?

Each page carries 131,072 numbers. Scoring them all is slow. Use a cheap first stage to pick candidates, then MaxSim on those.

Go to the source

Keep going