FINDING NEEDLES · CHAPTER 06

Sentence embeddings

A shopper asks “Will this tent keep the rain out?”. The answer you want says “waterproof flysheet”. You need one vector for each sentence, and a way to train those vectors so the right pair ends up close.

~15 min read3 question–answer pairsTested Python

What you'll learn

  • Why averaging word vectors misses order and negation
  • What a bi-encoder is, and why it can pre-compute every answer
  • How a batch of pairs supplies its own negatives
  • What the temperature in the contrastive loss does

Before you start

  • Word embeddings — words as vectors, compared by cosine
  • Self-attention — how an encoder reads a word in context
  • Softmax, and the log of a probability

Questions and answers rarely share words

A camping shop has hundreds of product answers. Shoppers type questions. “Will this tent keep the rain out?” and “Waterproof flysheet with taped seams.” share no word at all, so TF-IDF scores the pair 0.

Chapter 5 gave each word a vector. Now you want one vector for the whole sentence, so a question and its answer can be compared with a single cosine.

A sentence embedding is a fixed-length vector for a whole piece of text. It is only useful if it was trained for the job: pairs that belong together must land close, and pairs that do not must land apart. The training signal is the pairs themselves.

The cheap answer, and where it breaks

The easy sentence vector is the average of its word vectors. Drop the stop words, add up the rest, divide by how many there are. It costs nothing and it works for rough topic matching. It also throws away two things that matter.

paraphrase · encode · towers · ANN

MANIM Sentence embeddings bridge paraphrases BM25 misses.
Sentence ASentence BCosine of the averages
the stove fits inside the potthe pot fits inside the stove1.000
the tent is waterproofthe tent is not waterproof0.981

The first pair uses the same words in a different order. The average is a sum, and a sum does not care about order, so both come out as (0.425, 0.425). The cosine is exactly 1.000.

The second pair differs by one word. With toy vectors tent = (1, 0), waterproof = (0, 1), and not = (0.5, 0), the averages are (0.5, 0.5) and (0.5, 0.333). One word in three is diluted, and the cosine is still 0.981. A shopper who wants a waterproof tent would be shown one that is not.

Read the whole sentence, then pool

The fix is a transformer encoder. It reads every word with attention to every other word, so “not” changes the vector of “waterproof” before anything is pooled. Then you average the output token vectors into one sentence vector. This is mean pooling, but over words that already know their context.

A bi-encoder runs that same encoder, with the same weights, on each text separately. A question goes in and a vector comes out. An answer goes in and a vector comes out. The score is the cosine of the two.

Question“Will this tent keep…”
Encodershared weights
Vector qmean pool
Answer“Waterproof flysheet…”
Encodersame weights
Vector astored once

Bi-encoder

Each text is encoded alone. Answer vectors are computed once and stored. A new question costs one encoder pass plus cheap dot products.

Cross-encoder

Question and answer go into the encoder together, so every word can attend across the pair. More accurate, but one full pass for every pair you score.

The gap is large. To find the most similar pair among 10,000 sentences, a cross-encoder must score 10,000 × 9,999 / 2 = 49,995,000 pairs. A bi-encoder encodes 10,000 sentences and does the rest with vector math. The Sentence-BERT paper reports that this search drops from about 65 hours with BERT as a cross-encoder to about 5 seconds.

Chapter 9 brings the cross-encoder back, as a second pass over a short list.

Every other answer in the batch is a negative

An encoder fresh from pre-training does not put questions near their answers. You fine-tune it on pairs. Take a batch of three (question, answer) pairs. Encode all six texts. Build the 3 × 3 table of cosines. The diagonal holds the true pairs. Every off-diagonal cell is a wrong answer that came free with the batch: an in-batch negative.

The vectors below are hand-made 3-D unit vectors, standing in for a half-trained encoder.

Question ↓ · Answer →a1 flysheeta2 shella3 stakes
q1 tent in rain0.80.00.6
q2 jacket in rain0.80.60.0
q3 tent in wind0.60.00.8

Rows 1 and 3 are fine: the true answer has the highest cosine. Row 2 is wrong. The jacket question sits closer to the tent’s flysheet (0.8) than to the jacket’s own shell (0.6). Both talk about rain.

The loss treats each row as a three-way choice. Divide the cosines by a temperature τ, take a softmax, and charge −log of the probability on the true answer. This is the InfoNCE loss.

how surprised the model is that question i goes with answer i, given the other answers in the batch. Average over the rows.

  1. With τ = 0.1, row 2’s cosines 0.8, 0.6, 0.0 become logits 8, 6, 0.
  2. The softmax gives the flysheet 0.881, the shell 0.119, and the stakes almost nothing. The loss for row 2 is −log(0.119) = 2.127.
  3. Rows 1 and 3 put 0.881 on the right answer, so each costs 0.127. The batch loss is (0.127 + 2.127 + 0.127) / 3 = 0.794.

The gradient on q2 pulls it toward the shell and pushes it away from each wrong answer in proportion to that answer’s probability. The flysheet, at 0.881, gets almost all the push. The stakes, already far away, get almost none. The loss spends its effort on the confusing negative.

Real encoders use 384 to 1,024 dimensions and batches of hundreds of pairs, so each question sees hundreds of negatives per step at no extra encoding cost.

How hard to punish a near miss

Cosines live between −1 and 1. Without the temperature, a softmax over 0.8, 0.6, and 0.0 is nearly flat and every row looks half wrong. Dividing by a small τ stretches the gaps. Small τ means a confident model, a tiny loss on the right rows, and a big loss on the wrong one.

τRow 1 and 3 lossRow 2 lossBatch loss
1.00.8191.0190.886
0.10.1272.1270.794
0.050.0184.0181.351

Interactive · Temperature

Slide τ. Watch the probability each question puts on its true answer. The pink bar is the jacket question the encoder gets wrong.

At τ = 1.0 the right rows and the wrong row cost almost the same, so the training signal is weak. At τ = 0.05 the wrong row costs 4.018 and dominates. Common values sit between 0.01 and 0.1.

The batch you can re-run

sentence_vectors.pyInfoNCE at τ = 0.1 → 0.794
"""Sentence vectors for camping-gear questions.

Part 1: averaging word vectors cannot see order, and barely sees "not".
Part 2: the contrastive loss with in-batch negatives that trains a
bi-encoder, on three question-answer pairs and hand-made 3-D vectors.
"""
import math

# --- Part 1: average the word vectors --------------------------------

# Toy 2-D word vectors. Stop words ("the", "is") are dropped first.
WORD_VECS = {
    "stove": [1.0, 0.0], "pot": [0.0, 1.0], "fits": [0.5, 0.5], "inside": [0.2, 0.2],
    "tent": [1.0, 0.0], "waterproof": [0.0, 1.0], "not": [0.5, 0.0],
}
STOP = {"the", "is"}


def mean_pool(sentence):
    """Average the vectors of the words that are left. Word order is lost here."""
    vecs = [WORD_VECS[w] for w in sentence.lower().split() if w not in STOP]
    return [sum(col) / len(vecs) for col in zip(*vecs)]


def cosine(left, right):
    dot = sum(x * y for x, y in zip(left, right))
    size = math.sqrt(sum(x * x for x in left)) * math.sqrt(sum(y * y for y in right))
    return dot / size


# --- Part 2: one training batch for a bi-encoder ---------------------

QUESTIONS = [
    "Will this tent keep the rain out?",
    "Is this jacket good in the rain?",
    "How do I pitch the tent in wind?",
]
ANSWERS = [
    "Waterproof flysheet with taped seams.",
    "Breathable shell rated for heavy showers.",
    "Stake the windward side first.",
]

# What a half-trained encoder might return. Every vector has length 1,
# so the dot product is the cosine. Real encoders use 384 to 1,024 numbers.
Q_VECS = [[0.8, 0.0, 0.6], [0.8, 0.6, 0.0], [0.6, 0.0, 0.8]]
A_VECS = [[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0]]


def similarity_matrix(qs=None, answers=None):
    """S[i][j] = cosine of question i and answer j. The diagonal is the true pairs."""
    qs = Q_VECS if qs is None else qs
    answers = A_VECS if answers is None else answers
    return [[cosine(q, a) for a in answers] for q in qs]


def row_losses(sim, tau):
    """InfoNCE per question: -log softmax(row / tau) at the true answer.

    Every other answer in the batch is a negative. No extra data needed.
    """
    losses = []
    for i, row in enumerate(sim):
        logits = [s / tau for s in row]
        top = max(logits)
        log_total = top + math.log(sum(math.exp(x - top) for x in logits))
        losses.append(log_total - logits[i])
    return losses


def info_nce(sim, tau=0.1):
    """The batch loss is the mean over the rows."""
    losses = row_losses(sim, tau)
    return sum(losses) / len(losses)


def pairs_to_score(n):
    """A cross-encoder must run once for every pair of n sentences."""
    return n * (n - 1) // 2

Mistakes learners make

Shipping averaged word vectors for questions. “Fits inside the pot” and “the pot fits inside” come out identical, and a “not” barely moves the vector. Fine for topic tags. Wrong for answers.

Using a raw pre-trained encoder. A model trained only to fill in masked words was never asked to put a question near its answer. Mean-pooled vectors from it can rank worse than averaged word vectors. Use a model fine-tuned on pairs.

False negatives in the batch. If two shoppers ask the same thing, each one’s answer is a “negative” for the other. The loss pushes apart two texts that belong together. Remove duplicates before you batch.

Mixing vectors from two models. Each encoder learns its own space. A question encoded by one model and answers encoded by another give cosines that mean nothing. Re-encode everything when you change models.

What to remember

  • Averaging word vectors loses order (cosine 1.000 for swapped words) and dilutes “not” (cosine 0.981).
  • A bi-encoder runs one shared encoder on each text alone, then pools. Answers are encoded once. 10,000 sentences need 49,995,000 cross-encoder passes but only 10,000 bi-encoder passes.
  • In-batch training uses every other answer in the batch as a negative. Our batch loss at τ = 0.1 is 0.794, and the wrong row costs 2.127 of it.
  • The temperature sets how sharply a near miss is punished.

Check yourself

1. Why do “the stove fits inside the pot” and “the pot fits inside the stove” get the same averaged vector?

They contain the same words. An average is a sum divided by a count, and a sum ignores order.

2. A batch has 3 pairs. How many negatives does each question see?

Two: the other two answers in the batch. With 256 pairs, each question would see 255.

3. Which wrong answer gets most of the push in row 2, and why?

The flysheet. The push from each wrong answer is weighted by its softmax probability, and the flysheet holds 0.881 of it.

4. Why can a bi-encoder search thousands of answers fast, and a cross-encoder cannot?

The bi-encoder’s answer vectors do not depend on the question, so they are computed once in advance. A cross-encoder must read each question–answer pair together, so nothing can be pre-computed.

Go to the source

Keep going