FINDING NEEDLES · CHAPTER 19

Agentic search

Some questions cannot be searched in one go, because the second query depends on what the first one found. So you let the model search, read, and search again.

~15 min read8 hut notesTested Python

What you'll learn

  • Why one search with the whole question fails on a multi-part question
  • The agent loop: plan, search, read, decide
  • How to split a question into sub-questions that depend on each other
  • Why tracking what is already known saves searches
  • The stop conditions every agent needs, including a budget

Before you start

Ringing the hut office

You want to stay at a mountain hut near Lake Grau with your dog in July. You ask: “Can I bring my dog to the hut closest to Lake Grau, and what does a night cost in July?”

A good booking clerk does not answer that in one look. First they find which hut is closest. Only then can they open that hut’s house rules and that hut’s price list. The second and third look-ups use a name the first one found.

A RAG pipeline searches once and then writes. An agent puts the search inside a loop. After each search it reads what came back, writes down what it now knows, and decides: search again, or stop. The model becomes the one typing into the search box.

The booking knowledge base

Eight short notes. The search is a toy: a note’s score is how many of the query’s words it contains, ties go to the lower id, and it returns the top 2.

hut · plan · stop · cost

MANIM Agentic search: plan, call, refine; stop on enough evidence.
IdNote
1Lake Grau trail guide. The Edelweiss hut is the closest hut to Lake Grau, a 20 minute walk from the shore.
2Lake Grau swimming. The lake stays cold until late June.
3Edelweiss hut house rules. Dogs are welcome on a lead for 10 euro per night extra.
4Edelweiss hut prices. Summer season, July to September: 48 euro per night with half board.
5Falkenstein hut prices. In July a bed costs 39 euro per night.
6Falkenstein hut house rules. Pets are not allowed inside.
7Booking policy. Cancel up to 7 days before arrival for a full refund.
8Edelweiss hut winter season. December to April: 52 euro per night.

The answer needs notes 1, 3 and 4. Search once with the whole question. Note 1 scores 7, note 5 scores 5, note 4 scores 4, and note 3 scores 3. The top 2 are notes 1 and 5.

That is worse than a miss. Note 5 is the Falkenstein hut’s July price, 39 euro. A writer handed notes 1 and 5 would say “Edelweiss, 39 euro a night”, a confident wrong answer stitched from two huts. The dog rule never arrives, because the question says “dog” and the note says “dogs”, a different word to a keyword search, and the one query had to spend its words on everything at once.

A better ranker would not fix this. Notes 3 and 4 do not mention Lake Grau. They only become relevant once you know the name Edelweiss, and the question does not contain it.

Plan, search, read, decide

Plansplit the question into sub-questions
Searcha tool call: one query, a few results
Readpull facts out of what came back · write them down
Decideall answered? out of budget? otherwise loop to Search

In a real system a language model does every box. It writes the plan as text, emits a tool call such as search("Edelweiss hut dogs"), the program runs the search and pastes the results back into the model’s context, and the model reads them and chooses the next call. That pattern of thinking, acting, and observing is called ReAct.

This chapter replaces the model with fixed rules, so you can follow every step by hand and the check script can test it. The loop is the part to learn. Swapping the rules for a model changes how good the choices are, not the shape of the loop.

Sub-questions with holes in them

SlotNeedsQuery template
hut—closest hut to Lake Grau
dogshut{hut} hut dogs
july_pricehut{hut} hut price per night July

Each sub-question fills one slot. Two of them have a hole, {hut}, that only the first answer can fill. That dependency is what makes the question multi-hop. The agent may only run a sub-question whose needs are already known.

The agent keeps two lists as it goes. Known facts: the slots filled so far. Seen notes: every note id it has already read. Search skips seen notes, so each call brings something new. And when the agent reads a note, it checks it against every open slot, not only the one it searched for.

Reading is careful too. A price only counts if the note names the hut the agent is asking about. Without that rule, note 5 would fill the July price with Falkenstein’s 39 euro, the same mistake as the one-shot search.

Three slots, two searches

  1. Search “closest hut to Lake Grau”. Note 1 scores 5, note 2 scores 2. Read both. Note 1 says “The Edelweiss hut is the closest hut”, so hut = Edelweiss. Note 2 is about swimming and fills nothing.
  2. Fill the hole and search “Edelweiss hut dogs”. Note 1 is already seen, so it is skipped. Note 3 scores 3. Notes 4 and 8 tie at 2, and the lower id, 4, wins the second place. Read both.
  3. Read note 3: dogs welcome on a lead, 10 euro extra. Read note 4: it names Edelweiss and July, so july_price = 48 euro. That slot was not the one being searched for, but the agent checks every open slot.
  4. Decide. Every slot is filled. Stop with the reason “all sub-questions answered”.

The agent used 2 searches and read 4 notes. The plan had three sub-questions, but the third never needed its own search. Writing down what you already know is what saved it.

The answer it can hand to a writer: Edelweiss hut; dogs yes, on a lead, 10 euro a night extra; 48 euro per night in July. Each fact comes with the note it was read from, which is what the writer cites.

When to give up is part of the design

A loop with no exit will keep searching. Every search costs time, tokens, and money, and a model that is unsure tends to keep rephrasing. So the loop checks its exits before every search.

Stop reasonWhat happenedWhat you return
all sub-questions answeredevery slot filledthe facts and their notes
search budget spenthit the cap on searcheswhat is known, and what is not
would only repeat a querythe next query was already askedwhat is known, and what is not
no new notes to readsearch returned nothing unseenwhat is known, and what is not

Two runs show the other exits. With a budget of one search, the agent learns the hut and stops: “search budget spent”. Now delete note 4, the Edelweiss price list, and run again with the normal budget. The agent learns the hut, then the dog rule. Its third search, “Edelweiss hut price per night July”, returns notes 5 and 6. Note 5 scores 4 but is about Falkenstein, so the reader rejects it. The next step would be the same query again, so it stops after 3 searches: “would only repeat a query”.

That last answer is the honest one: the hut is Edelweiss, dogs are allowed, the July price is not in the knowledge base. Saying “unknown” is a valid result. Inventing 39 euro is not.

What a real agent does instead of stopping

A model-driven agent would rewrite the failing query before giving up: “Edelweiss summer rates”, “Edelweiss half board”. It might switch tools, from keyword search to dense search, or to an exact-match grep for “Edelweiss”. That is query rewriting from chapter 14, now done inside the loop. The budget still applies; it only buys more tries before the exit.

The agent you can re-run

hut_agent.py2 searches · 3 slots filled
"""A scripted search agent over a mountain-hut booking knowledge base.

No language model. The "policy" is a few fixed rules, so every run is
the same and you can follow it by hand. A real agent asks an LLM to
write the plan, the queries and the stop decision; the loop is the same.
"""
import re

KB = {
    1: "Lake Grau trail guide. The Edelweiss hut is the closest hut to Lake Grau, a 20 minute walk from the shore.",
    2: "Lake Grau swimming. The lake stays cold until late June.",
    3: "Edelweiss hut house rules. Dogs are welcome on a lead for 10 euro per night extra.",
    4: "Edelweiss hut prices. Summer season, July to September: 48 euro per night with half board.",
    5: "Falkenstein hut prices. In July a bed costs 39 euro per night.",
    6: "Falkenstein hut house rules. Pets are not allowed inside.",
    7: "Booking policy. Cancel up to 7 days before arrival for a full refund.",
    8: "Edelweiss hut winter season. December to April: 52 euro per night.",
}

QUESTION = "Can I bring my dog to the hut closest to Lake Grau, and what does a night cost in July?"


def words(text):
    return set(re.findall(r"[a-z0-9]+", text.lower()))


def search(query, kb=KB, exclude=(), k=2):
    """Toy keyword search: score = how many query words the note contains.

    Ties go to the lower id. Notes in `exclude` are skipped, so the agent
    is never shown the same note twice.
    """
    q = words(query)
    scored = [(len(q & words(text)), doc) for doc, text in kb.items() if doc not in exclude]
    scored = [(s, doc) for s, doc in scored if s > 0]
    scored.sort(key=lambda row: (-row[0], row[1]))
    return [doc for _, doc in scored[:k]]


# The plan: three sub-questions. {hut} is filled in once it is known.
PLAN = [
    {"slot": "hut", "needs": [], "query": "closest hut to Lake Grau"},
    {"slot": "dogs", "needs": ["hut"], "query": "{hut} hut dogs"},
    {"slot": "july_price", "needs": ["hut"], "query": "{hut} hut price per night July"},
]


def read(slot, text, known):
    """Pull one fact out of a note, or None. One rule per slot."""
    low = text.lower()
    if slot == "hut":
        m = re.search(r"the (\w+) hut is the closest hut", low)
        return m.group(1).capitalize() if m else None
    hut = known.get("hut", "").lower()
    if not hut or hut not in low:
        return None                       # a note about some other hut
    if slot == "dogs" and "dogs" in low:
        return "yes, on a lead, 10 euro extra" if "welcome" in low else "no"
    if slot == "july_price" and "july" in low:
        m = re.search(r"(\d+) euro per night", low)
        return int(m.group(1)) if m else None
    return None


def run(kb=KB, plan=PLAN, max_searches=4):
    """Plan, search, read, decide. Return the facts and the trace."""
    known, seen, asked, trace = {}, set(), set(), []
    while True:
        open_steps = [s for s in plan if s["slot"] not in known]
        if not open_steps:
            return known, trace, "all sub-questions answered"
        if len(trace) == max_searches:
            return known, trace, "search budget spent"
        ready = [s for s in open_steps if all(n in known for n in s["needs"])]
        if not ready:
            return known, trace, "stuck: a needed fact is missing"
        step = ready[0]
        query = step["query"].format(**known)
        if query in asked:
            return known, trace, "would only repeat a query"
        asked.add(query)
        docs = search(query, kb, exclude=seen)
        if not docs:
            return known, trace, "no new notes to read"
        learned = []
        for doc in docs:                  # read every note returned ...
            seen.add(doc)
            for s in open_steps:          # ... and fill ANY open slot it answers
                if s["slot"] not in known:
                    fact = read(s["slot"], kb[doc], known)
                    if fact is not None:
                        known[s["slot"]] = fact
                        learned.append(s["slot"])
        trace.append((query, docs, learned))


def one_shot(question=QUESTION, kb=KB, k=2):
    """The non-agent baseline: one search with the whole question."""
    return search(question, kb, k=k)

Mistakes learners make

Running dependent sub-questions in parallel. “{hut} hut dogs” cannot run before the hut is known. Parallel calls are fine only for sub-questions with no holes.

Reading only for the slot you searched for. Note 4 arrived during the dog search and answered the price. An agent that ignored it would have spent a third search.

Accepting a fact about the wrong entity. Note 5 is a real July price, for another hut. Every extracted fact must be tied to the entity it is about.

No budget. Without a cap and a repeat check, an agent that cannot find the price keeps rephrasing until someone pays the bill.

Filling a gap with a guess. When the loop stops early, return what is unknown as unknown. The writer downstream can say so.

What to remember

  • One search with the whole question returned notes 1 and 5, mixing the Edelweiss name with the Falkenstein price of 39 euro.
  • The loop is plan, search, read, decide. A model normally does each step; here fixed rules do.
  • Decompose into sub-questions. A sub-question with a hole waits for the answer that fills it.
  • Keep known facts and seen notes. The agent answered three slots with 2 searches and 4 notes read.
  • Stop on: all answered, budget spent, a repeated query, or nothing new. Return unknowns as unknown.

Check yourself

1. Why can’t a better ranker fix the one-shot search?

Notes 3 and 4 never mention Lake Grau. They only match once you know the name Edelweiss, and that name is not in the question.

2. Why did the July price not need its own search?

Note 4 came back in the dog search. The agent checks every note against every open slot, and note 4 names Edelweiss and July.

3. Why is note 1 not returned by the second search, even though it scores 2?

It is in the seen list. Search skips notes the agent already read, so every call brings something new.

4. Note 4 is deleted. What should the agent answer about the price?

That it is not in the knowledge base. It stops after 3 searches and returns the hut and the dog rule as known, the price as unknown.

Go to the source

Keep going