TRUST, BUT VERIFY · CHAPTER 04
A language model follows text. Some of that text is the user. Some of it is a document, a tool result, or a training file. Security is about who is allowed to give instructions, and what still runs when those instructions are hostile.
A database driver separates the query from the data. A language model does not. Everything in the context is tokens: the developer’s policy, the user’s request, a PDF, a calendar invite, a tool result. The model’s job is to continue that stream. If hostile text looks like an instruction, it will often be treated as one.
Think of three seats. The vendor writes a policy. The user writes a request. The outside world writes documents and tool results. Jailbreaks are the user attacking the policy. Prompt injection is the outside world attacking the user. Poisoning is the outside world attacking the training run. Same model, three threat models.
| Stage | What goes wrong | Who is hurt |
|---|---|---|
| Training | Poisoned documents, membership inference | The deployer, and later users |
| Prompt | Jailbreak of the policy; injection via a document | Vendor (jailbreak) or user (injection) |
| Tools / RAG | Untrusted observations treated as commands | The user, whose tools then fire |
This chapter stays at that map. It does not walk through working attacks. If you are building a product, the job is to assume untrusted text will try to look like a developer instruction, and to put checks outside the model.
override · tools · docs
envelope · inject · channels · privilege
Alignment training teaches the model to refuse some classes of request. A jailbreak is an attempt to make it ignore that training: role-play, obfuscation, translation into a low-resource language, and other wrappers around the same intent. The details of those wrappers are not the lesson. The lesson is that refusals are a behaviour, not a lock.
Patches close one template and leave the next one open. Red-teaming and public jailbreak benches measure the gap; they do not close it. Treat a jailbreak as a safety-training failure, then add filters around the model rather than promising the next prompt tweak will hold.
Here the user is the victim. They ask an assistant to summarise a page, file, or email. Hidden in that page is text that looks like a higher-priority instruction: ignore the user, exfiltrate the conversation, call a tool. Indirect injection arrives through RAG chunks and tool observations too. The model cannot tell a policy paragraph from a paragraph that merely resembles one.
A system prompt is not a security boundary. It is more text. Delimiting user content (“the next block is untrusted data”) helps a little and is still bypassed. Privilege separation helps more: the model that reads the document should not be the process that can send mail or move money. Dual-LLM patterns, allow-listed tools, and human confirmation on side effects belong here, not a longer system prompt.
Training-time attacks change the weights. Membership inference asks whether a record was in the train set. Data poisoning plants a trigger so that a rare phrase later causes a specific behaviour. The surprising quantitative result in recent work is the asymmetry: on the order of 250 carefully chosen documents can be enough to backdoor a large model, rather than a fraction of the whole corpus. Defenders cannot assume “we trained on a trillion tokens, so a few bad files vanish”.
Defenses start before the run: provenance, dedup, and filters on the crawl. They continue after: canaries, evals for the known trigger style, and the same output filters you wanted anyway.
One useful shape is a classifier sandwich. An input gate inspects the user prompt. If it is quiet, the model runs. An output gate inspects the answer before it reaches the user or a tool. Neither gate is the model’s system prompt. They are separate classifiers, often trained on synthetic attacks, and they still miss things.
Add, around that: allow-listed tools, no side effects without confirmation, retrieved text treated as data, and logging. Defense is a stack. The model is one layer in it.
The file below is the shape of the check, not a lock. The labeler always returns “ok” on ordinary text. Production gates are trained models, and they fail. Do not ship this as a filter.
"""A defensive input/output gate. No attack strings live here. The sandwich is the idea: one classifier inspects the user input, the model runs only if that gate is quiet, and a second classifier inspects the outgoing answer. System prompts are not this gate. """ # Labels a real gate would emit. The page only needs the control flow. ALLOW = "allow" BLOCK = "block" def input_gate(text, blocked_labels=None): """Return BLOCK if a coarse labeler flags the prompt as disallowed. The labeler is a stand-in. Production gates are trained classifiers, not keyword lists, and they still fail. Treat this as the shape of the check, not a security boundary. """ blocked_labels = set(blocked_labels or []) label = _coarse_label(text) return BLOCK if label in blocked_labels else ALLOW def output_gate(text, blocked_labels=None): """Same shape as the input gate, applied to the model's answer.""" return input_gate(text, blocked_labels) def sandwich(prompt, generate, blocked_in=None, blocked_out=None): """Run the model only if both gates allow the turn.""" if input_gate(prompt, blocked_in) is BLOCK: return BLOCK, None answer = generate(prompt) if output_gate(answer, blocked_out) is BLOCK: return BLOCK, None return ALLOW, answer def _coarse_label(text): """Placeholder labeler used by the check. Always 'ok' on this page.""" return "ok" if text else "empty"
Equating jailbreaks with injection. Jailbreaks: user versus policy. Injection: untrusted document versus user. The fix for one (better refusal training) does not fix the other (privilege separation).
“The system prompt will hold.” It is more tokens. Attackers write tokens too. Put enforcement outside the generator.
Keyword filters as the plan. Obfuscation exists. Classifiers help and they are fallible. Combine them with tool allow-lists and human confirmation.
Assuming scale washes out poisoning. A few hundred crafted documents can be enough. Curate the crawl.
This is prompt injection. The user is the victim. The PDF is untrusted text that looks like an instruction.
The model does not have a trusted computing base. Policy text and document text are the same kind of token. Enforcement has to live outside the generator.
Separate input and output classifiers, so a request or an answer can be blocked even if the generator would have complied.
No. Recent measurements put a successful backdoor on the order of 250 documents, not a fraction of the crawl.