Finding Needles

From the inverted index to agents that search for themselves.

01

How search works

Indexes, postings lists, and Boolean AND.

02

TF-IDF

A count in the note, times how rare the word is.

03

BM25

TF-IDF with a cap on repeats and a fair length penalty.

04

A small search engine

Index, BM25, and a top-k heap in one file.

05

Word embeddings

Words as points, learned from the company they keep.

06

Sentence embeddings

One vector per sentence, trained with contrastive pairs.

07

Dense retrieval

Two encoders, one dot product, search by meaning.

08

Vector databases

Approximate nearest neighbours: IVF, HNSW, and filters.

09

Reranking

Retrieve wide with a bi-encoder, reorder with a cross-encoder.

10

Hybrid search

Fuse BM25 and dense ranks with reciprocal rank fusion.

11

Query understanding

Fix, expand, and route the query before you search.

12

RAG fundamentals

Retrieve, pack the context, then generate with citations.

13

Chunking

How you split the text decides what you can find.

14

Beyond the basic pipeline

Grade the retrieval, retry, and know when not to retrieve.

15

Evaluating search and RAG

Recall@k, MRR, nDCG, and grading the answer.

16

Graphs and structured retrieval

Entities and edges for questions that span documents.

17

Multimodal search

One space for images and text, and searching page images.

18

Production search architecture

Stages, budgets, caches, and freshness at scale.

19

Agentic search

A model that plans, searches, reads, and searches again.

20

Chroma Context-1

Case study: a search agent trained to retrieve.