FINDING NEEDLES · CHAPTER 19
Some questions cannot be searched in one go, because the second query depends on what the first one found. So you let the model search, read, and search again.
You want to stay at a mountain hut near Lake Grau with your dog in July. You ask: “Can I bring my dog to the hut closest to Lake Grau, and what does a night cost in July?”
A good booking clerk does not answer that in one look. First they find which hut is closest. Only then can they open that hut’s house rules and that hut’s price list. The second and third look-ups use a name the first one found.
A RAG pipeline searches once and then writes. An agent puts the search inside a loop. After each search it reads what came back, writes down what it now knows, and decides: search again, or stop. The model becomes the one typing into the search box.
Eight short notes. The search is a toy: a note’s score is how many of the query’s words it contains, ties go to the lower id, and it returns the top 2.
hut · plan · stop · cost
| Id | Note |
|---|---|
| 1 | Lake Grau trail guide. The Edelweiss hut is the closest hut to Lake Grau, a 20 minute walk from the shore. |
| 2 | Lake Grau swimming. The lake stays cold until late June. |
| 3 | Edelweiss hut house rules. Dogs are welcome on a lead for 10 euro per night extra. |
| 4 | Edelweiss hut prices. Summer season, July to September: 48 euro per night with half board. |
| 5 | Falkenstein hut prices. In July a bed costs 39 euro per night. |
| 6 | Falkenstein hut house rules. Pets are not allowed inside. |
| 7 | Booking policy. Cancel up to 7 days before arrival for a full refund. |
| 8 | Edelweiss hut winter season. December to April: 52 euro per night. |
The answer needs notes 1, 3 and 4. Search once with the whole question. Note 1 scores 7, note 5 scores 5, note 4 scores 4, and note 3 scores 3. The top 2 are notes 1 and 5.
That is worse than a miss. Note 5 is the Falkenstein hut’s July price, 39 euro. A writer handed notes 1 and 5 would say “Edelweiss, 39 euro a night”, a confident wrong answer stitched from two huts. The dog rule never arrives, because the question says “dog” and the note says “dogs”, a different word to a keyword search, and the one query had to spend its words on everything at once.
A better ranker would not fix this. Notes 3 and 4 do not mention Lake Grau. They only become relevant once you know the name Edelweiss, and the question does not contain it.
In a real system a language model does every box. It writes the plan as text, emits a tool call such as search("Edelweiss hut dogs"), the program runs the search and pastes the results back into the model’s context, and the model reads them and chooses the next call. That pattern of thinking, acting, and observing is called ReAct.
This chapter replaces the model with fixed rules, so you can follow every step by hand and the check script can test it. The loop is the part to learn. Swapping the rules for a model changes how good the choices are, not the shape of the loop.
| Slot | Needs | Query template |
|---|---|---|
| hut | — | closest hut to Lake Grau |
| dogs | hut | {hut} hut dogs |
| july_price | hut | {hut} hut price per night July |
Each sub-question fills one slot. Two of them have a hole, {hut}, that only the first answer can fill. That dependency is what makes the question multi-hop. The agent may only run a sub-question whose needs are already known.
The agent keeps two lists as it goes. Known facts: the slots filled so far. Seen notes: every note id it has already read. Search skips seen notes, so each call brings something new. And when the agent reads a note, it checks it against every open slot, not only the one it searched for.
Reading is careful too. A price only counts if the note names the hut the agent is asking about. Without that rule, note 5 would fill the July price with Falkenstein’s 39 euro, the same mistake as the one-shot search.
The agent used 2 searches and read 4 notes. The plan had three sub-questions, but the third never needed its own search. Writing down what you already know is what saved it.
The answer it can hand to a writer: Edelweiss hut; dogs yes, on a lead, 10 euro a night extra; 48 euro per night in July. Each fact comes with the note it was read from, which is what the writer cites.
A loop with no exit will keep searching. Every search costs time, tokens, and money, and a model that is unsure tends to keep rephrasing. So the loop checks its exits before every search.
| Stop reason | What happened | What you return |
|---|---|---|
| all sub-questions answered | every slot filled | the facts and their notes |
| search budget spent | hit the cap on searches | what is known, and what is not |
| would only repeat a query | the next query was already asked | what is known, and what is not |
| no new notes to read | search returned nothing unseen | what is known, and what is not |
Two runs show the other exits. With a budget of one search, the agent learns the hut and stops: “search budget spent”. Now delete note 4, the Edelweiss price list, and run again with the normal budget. The agent learns the hut, then the dog rule. Its third search, “Edelweiss hut price per night July”, returns notes 5 and 6. Note 5 scores 4 but is about Falkenstein, so the reader rejects it. The next step would be the same query again, so it stops after 3 searches: “would only repeat a query”.
That last answer is the honest one: the hut is Edelweiss, dogs are allowed, the July price is not in the knowledge base. Saying “unknown” is a valid result. Inventing 39 euro is not.
A model-driven agent would rewrite the failing query before giving up: “Edelweiss summer rates”, “Edelweiss half board”. It might switch tools, from keyword search to dense search, or to an exact-match grep for “Edelweiss”. That is query rewriting from chapter 14, now done inside the loop. The budget still applies; it only buys more tries before the exit.
"""A scripted search agent over a mountain-hut booking knowledge base.
No language model. The "policy" is a few fixed rules, so every run is
the same and you can follow it by hand. A real agent asks an LLM to
write the plan, the queries and the stop decision; the loop is the same.
"""
import re
KB = {
1: "Lake Grau trail guide. The Edelweiss hut is the closest hut to Lake Grau, a 20 minute walk from the shore.",
2: "Lake Grau swimming. The lake stays cold until late June.",
3: "Edelweiss hut house rules. Dogs are welcome on a lead for 10 euro per night extra.",
4: "Edelweiss hut prices. Summer season, July to September: 48 euro per night with half board.",
5: "Falkenstein hut prices. In July a bed costs 39 euro per night.",
6: "Falkenstein hut house rules. Pets are not allowed inside.",
7: "Booking policy. Cancel up to 7 days before arrival for a full refund.",
8: "Edelweiss hut winter season. December to April: 52 euro per night.",
}
QUESTION = "Can I bring my dog to the hut closest to Lake Grau, and what does a night cost in July?"
def words(text):
return set(re.findall(r"[a-z0-9]+", text.lower()))
def search(query, kb=KB, exclude=(), k=2):
"""Toy keyword search: score = how many query words the note contains.
Ties go to the lower id. Notes in `exclude` are skipped, so the agent
is never shown the same note twice.
"""
q = words(query)
scored = [(len(q & words(text)), doc) for doc, text in kb.items() if doc not in exclude]
scored = [(s, doc) for s, doc in scored if s > 0]
scored.sort(key=lambda row: (-row[0], row[1]))
return [doc for _, doc in scored[:k]]
# The plan: three sub-questions. {hut} is filled in once it is known.
PLAN = [
{"slot": "hut", "needs": [], "query": "closest hut to Lake Grau"},
{"slot": "dogs", "needs": ["hut"], "query": "{hut} hut dogs"},
{"slot": "july_price", "needs": ["hut"], "query": "{hut} hut price per night July"},
]
def read(slot, text, known):
"""Pull one fact out of a note, or None. One rule per slot."""
low = text.lower()
if slot == "hut":
m = re.search(r"the (\w+) hut is the closest hut", low)
return m.group(1).capitalize() if m else None
hut = known.get("hut", "").lower()
if not hut or hut not in low:
return None # a note about some other hut
if slot == "dogs" and "dogs" in low:
return "yes, on a lead, 10 euro extra" if "welcome" in low else "no"
if slot == "july_price" and "july" in low:
m = re.search(r"(\d+) euro per night", low)
return int(m.group(1)) if m else None
return None
def run(kb=KB, plan=PLAN, max_searches=4):
"""Plan, search, read, decide. Return the facts and the trace."""
known, seen, asked, trace = {}, set(), set(), []
while True:
open_steps = [s for s in plan if s["slot"] not in known]
if not open_steps:
return known, trace, "all sub-questions answered"
if len(trace) == max_searches:
return known, trace, "search budget spent"
ready = [s for s in open_steps if all(n in known for n in s["needs"])]
if not ready:
return known, trace, "stuck: a needed fact is missing"
step = ready[0]
query = step["query"].format(**known)
if query in asked:
return known, trace, "would only repeat a query"
asked.add(query)
docs = search(query, kb, exclude=seen)
if not docs:
return known, trace, "no new notes to read"
learned = []
for doc in docs: # read every note returned ...
seen.add(doc)
for s in open_steps: # ... and fill ANY open slot it answers
if s["slot"] not in known:
fact = read(s["slot"], kb[doc], known)
if fact is not None:
known[s["slot"]] = fact
learned.append(s["slot"])
trace.append((query, docs, learned))
def one_shot(question=QUESTION, kb=KB, k=2):
"""The non-agent baseline: one search with the whole question."""
return search(question, kb, k=k)
Running dependent sub-questions in parallel. “{hut} hut dogs” cannot run before the hut is known. Parallel calls are fine only for sub-questions with no holes.
Reading only for the slot you searched for. Note 4 arrived during the dog search and answered the price. An agent that ignored it would have spent a third search.
Accepting a fact about the wrong entity. Note 5 is a real July price, for another hut. Every extracted fact must be tied to the entity it is about.
No budget. Without a cap and a repeat check, an agent that cannot find the price keeps rephrasing until someone pays the bill.
Filling a gap with a guess. When the loop stops early, return what is unknown as unknown. The writer downstream can say so.
Notes 3 and 4 never mention Lake Grau. They only match once you know the name Edelweiss, and that name is not in the question.
Note 4 came back in the dog search. The agent checks every note against every open slot, and note 4 names Edelweiss and July.
It is in the seen list. Search skips notes the agent already read, so every call brings something new.
That it is not in the knowledge base. It stops after 3 searches and returns the hut and the dog rule as known, the price as unknown.