Retrieval Should Be a Tool, Not a Pipeline Stage

The default RAG architecture has one shape. Embed the question, retrieve the nearest chunks, paste them above the prompt, generate. It runs in that order for every question, and the order was chosen while you were building the system — which is to say, before you had seen any of the questions it would be asked.

That is a strange amount of confidence to bake into a control flow. Some questions need three documents. Some need a number computed from a database. Some need nothing at all.

What retrieving on every turn actually costs

I built an assistant over Indian mutual-fund portfolios. Real questions it gets, and what a fixed pipeline does with them:

• “What is my XIRR?” — retrieval runs and returns five chunks of fund factsheets. The answer is a computation over the user's own transactions. Not one of the five chunks is relevant.
• “Compare the expense ratios of these two funds.” — the answer is two rows in a table. Retrieval returns prose about expense ratios in general.
• “Thanks, that helps.” — retrieval runs.

The wasted latency and tokens are the obvious cost and the smaller one. The real damage is that irrelevant context is not neutral. Five chunks that half-match the question are actively worse than no chunks, because the model has been handed material that looks like evidence and has to decide, unaided, to ignore it. Give a model a factsheet paragraph about XIRR methodology and ask it for the user's XIRR, and you have built a machine for producing a plausible number from the wrong source.

A fixed pipeline has no way to express no documents needed. It also has no way to express that was not it, look again. One shot, no feedback, whatever the vector index returned is what the model gets.

Retrieval as one tool among several

The alternative is to stop treating retrieval as a stage and start treating it as a capability the model can invoke. The assistant has six tools; searching documents is one of them, sitting alongside tools that compute returns, fetch holdings, and compare funds.

const searchDocuments = {
  name: "search_documents",
  description:
    "Search fund factsheets, scheme documents and filings. " +
    "Use for policy, methodology and qualitative questions. " +
    "Do NOT use for figures about the user's own portfolio.",
  schema: z.object({
    query: z.string().min(3).describe("A focused natural-language query"),
    limit: z.number().int().min(1).max(10).default(5),
  }),
};

Two things are worth noticing. The description spends most of its words on when not to call this, which matters more than the capability itself — a tool that is always plausible gets called always. And the schema is narrow: a focused query and a bounded limit, not a free-form options bag.

What has not changed is the retrieval itself. Inside the tool it is still the pipeline from the hybrid retrieval post — dense and lexical search fused by reciprocal rank, reranked down to about five chunks. In our case both halves run as one SQL round trip. None of that ranking work is wasted by this change. The only thing that moved is the decision about whether and when to run it.

Validation errors are a message, not an exception

Once the model is choosing arguments, it will get them wrong. The instinct is to throw, catch at the loop boundary, and return a generic failure. That wastes the most useful property of an agent loop: there is another turn coming, and the model can read.

const parsed = tool.schema.safeParse(rawArgs);

if (!parsed.success) {
  // Not a throw. This becomes the tool result the model sees next turn.
  return {
    ok: false,
    error: parsed.error.issues
      .map((i) => `${i.path.join(".")}: ${i.message}`)
      .join("; "),
  };
}

Hand the model back limit: must be <= 10 and it calls again with a valid limit. No retry logic, no repair prompt, no parsing of the model's apology. Returning typed errors as tool results is the cheapest reliability win available in an agent loop, and it costs one safeParse.

What the model does with the choice

Three behaviours appear that a fixed pipeline cannot produce.

It skips. Portfolio arithmetic goes straight to the computation tools. No retrieval, no irrelevant context, and the whole class of wrong-source answers described above simply stops occurring.

It retrieves after computing. Asked why a fund underperformed, it fetches the return first, then searches documents with a query informed by what it found. A fixed pipeline retrieves before it knows anything, using only the user's words. This one retrieves knowing the answer is -4.2%, which is a materially better query.

It looks twice. If the first search comes back thin, the model reformulates and searches again. This is the behaviour I underestimated most. Users ask questions in their own vocabulary, and the second attempt — written by a model that has now seen what the corpus does and does not contain — is often the one that works.

Where retrieval is the wrong tool entirely

Giving the model a choice of tools only helps if the other tools are trustworthy. In this system every figure a user sees — XIRR, asset allocation, fund comparison — is computed deterministically in TypeScript and returned as a tool result. The model never does arithmetic. It decides which computation to ask for, and it phrases the answer, but the number is not its output.

That is worth stating alongside the retrieval argument because they are the same argument. Retrieval is how you get prose you did not write. Deterministic tools are how you get numbers the model cannot invent. The failure mode of a fixed pipeline is that it treats retrieval as the only way to ground an answer, so every question gets routed through the one mechanism, including the questions where grounding means running a calculation.

The guards that make the loop safe

Handing control flow to a model requires bounding what it can do with it. Four guards, none optional.

Ownership scoping lives in the tool layer. The tools take a session-bound user id from the runtime, never from the model's arguments, and every query is scoped by it in SQL. A prompt instructing the model to only access its own user's data is not a security control; this is the whole argument in prompt injection is an authorisation problem, and making retrieval model-invoked raises the stakes rather than changing the answer.

An iteration cap. Models can spiral — call a tool, read a disappointing result, call it again with a near-identical query, forever. Cap the loop, and when the cap is hit, answer with what has been gathered and say so, rather than failing.

Typed errors returned to the model, as above.

Provider-exact message assembly. Tool-call and tool-result blocks have to be reconstructed into the conversation in the precise shape the provider expects, including on the turns where the model called two tools at once. This is tedious rather than difficult, and it is where a hand-rolled loop actually costs you time. It is also why the abstraction layer that promises to hide provider differences tends to leak exactly here.

Tool selection is now a thing you can get wrong

Moving the decision into the model creates a new failure mode, so it needs its own number. The evaluation post argues for a golden set scored in CI; this architecture adds a category to it. Ours has four: numeric accuracy, retrieval quality, refusal compliance, and tool selection — did the model reach for the right capability at all.

Keeping that separate from retrieval quality matters, because the fixes are unrelated. A tool-selection failure is a description problem: the wording of search_documents made it sound applicable to portfolio figures. A retrieval failure is a chunking or ranking problem. Averaging them into one score tells you the assistant got worse and nothing about where to look.

When the fixed pipeline is the right answer

This is a trade, and there are systems where it is a bad one.

• Single-corpus question answering. A documentation chatbot where every question is about the docs should retrieve every time. The decision has one correct answer, so paying a model to make it buys nothing and adds a way to be wrong.
• Latency budgets under a second. Every decision is a round trip. A fixed pipeline has one model call; this can have three or four. Search-as-you-type cannot afford it.
• Smaller models. Tool selection is a capability, and below a certain level models call everything or nothing. A weak chooser is worse than no chooser.
• No evaluation harness. If you cannot measure tool selection, you cannot detect it degrading when you reword a description. Do not move control flow into the model until you can score it.

The short version

A fixed retrieval stage encodes a build-time guess about every future question, and the guess is wrong for any turn that needs a computation, a second attempt, or nothing. Expose retrieval as a tool with a description that says when not to use it, keep the hybrid pipeline unchanged inside it, and return validation failures to the model as results it can act on. Scope ownership in the tool layer rather than the prompt, cap the iterations, and score tool selection as its own eval category. Then, if your product is a docs chatbot with a one-second budget, ignore all of this and retrieve every time.