Skip to content
SJ
All writing

·11 min read

Building an Agent Loop That Knows When to Stop

Starting an agent loop is the easy part. The hard part is making it stop before it has burned forty dollars retrying a step that will never work.

Agents / LLM / AI / Architecture / Backend


The minimal agent loop fits in thirty lines. Read the user's message, call the model, check whether the model wants to use a tool, execute the tool, feed the result back, repeat until the model says it is done. Every framework ships this loop. The tutorials stop there, and the interesting problems all start there.

Because the default failure is not a wrong answer. It is an expensive silence. The model calls a tool that errors, interprets the error as something to fix, retries, gets the same error, retries again, and continues until it hits the context limit or the provider's rate ceiling. You get a four-thousand-token apology and a bill. The agent did not crash. It did what the loop told it to: keep going until the model emits a final response without a tool call.

This post is about the parts of an agent loop that tutorials leave out: how it decides to stop, how it avoids repeating work, and where the budget lives.

How the loop works, and what each step costs

Every agent loop is a sequence of model calls where each call can produce tool invocations. The model sees the entire conversation so far, decides whether to call a tool or respond, and the orchestrator acts on that decision. The loop terminates when the model responds without requesting a tool call.

That description hides three cost facts:

  • Each iteration sends the full history. The input token count grows with every step. Step one sends the system prompt and the user message. Step five sends all of that plus four rounds of tool calls and their results. By step ten the context is large and every step is more expensive than the last.
  • Tool results can be arbitrarily large. A tool that returns a full file, a database dump, or an API response with pagination metadata can double the context in one step.
  • The model does not know what it costs. It has no token counter, no spend tracker, no incentive to be brief. If the system prompt says “be thorough,” it will be thorough until something external stops it.

This means the orchestrator, not the model, is responsible for termination. Relying on the model to “decide it is done” is correct for the happy path and catastrophically wrong for every other path.

Exit conditions that actually fire

The model's own stop (a response with no tool call) is one exit condition. You need at least three more, and all of them belong in the orchestrator, not in the system prompt.

  • A maximum step count. Hard, and checked before every model call. Ten to twenty steps covers virtually every legitimate task. If your agent needs fifty steps, that is an architecture problem rather than a limit problem.
  • A token budget. Track cumulative input and output tokens across the loop. When the budget is exhausted, stop the loop and return what you have. This is the ceiling that prevents a retry spiral from becoming a billing incident.
  • Repeated failure on the same tool. If the same tool has errored twice in a row with similar arguments, the third call will also fail. Detect the repetition and stop, or force the model to try a different approach by injecting a message that says the tool is unavailable.
  • A wall-clock timeout. Some tool calls are blocking (HTTP requests, shell commands, database queries). A tool that hangs for five minutes while the user waits is a worse experience than failing fast. Wrap every tool execution in a deadline.

When any of these fires, the agent should not vanish. It should return the best partial result it has, along with a note about what stopped it. A half-finished answer with “I reached the step limit before completing the final check” is useful. A silent timeout is not.

Token budgets as hard limits, not suggestions

A token budget is not a prompt instruction. “Keep your response under 2,000 tokens” is a suggestion the model follows imperfectly, and it says nothing about cumulative spend across the loop. The budget belongs in code.

let totalInputTokens = 0;
let totalOutputTokens = 0;
const MAX_INPUT = 200_000;
const MAX_OUTPUT = 30_000;

for (let step = 0; step < MAX_STEPS; step++) {
  const response = await model.call(messages);
  totalInputTokens += response.usage.input_tokens;
  totalOutputTokens += response.usage.output_tokens;

  if (
    totalInputTokens > MAX_INPUT ||
    totalOutputTokens > MAX_OUTPUT
  ) {
    // Graceful exit: summarise what we have
    break;
  }
  // ... handle tool calls
}

The input budget is the one that matters most, because input tokens grow with every iteration while output tokens stay roughly constant per step. A ten-step loop where each tool returns 4,000 tokens of context has consumed over 100,000 input tokens by the end, and most of them are re-reads of earlier results.

Truncate tool results before they enter the context. If a tool returns 20,000 characters, keep the first 2,000 and a note saying the rest was truncated. The model almost never needs the full output, and the savings compound across every subsequent step.

When retrying makes things worse

On a tool error, the model tries again, often with the same arguments or a trivial variation. This is the single most expensive failure mode in agent systems, and it looks like productive work from the outside.

Three patterns to watch for:

  • The identical retry. Same tool, same arguments, same error. The model has no new information and will produce the same result. Detect this by hashing the tool name and arguments; if the hash matches a recent call that failed, block it.
  • The cosmetic variation. The model changes whitespace, rephrases a query, or adds a comment to its arguments. The underlying issue (a missing file, a permission error, a nonexistent API endpoint) has not changed. Normalise the arguments before hashing.
  • The escalating context. Each retry adds the previous error to the conversation, so the model “learns” from failures. Except each learning attempt adds thousands of tokens and the model rarely changes strategy. It just apologises more thoroughly before doing the same thing.

The fix is a retry policy at the orchestrator level. Two attempts on the same tool, then either skip that tool (inject a system message marking it unavailable for this run) or terminate the loop with a partial result. The model will not impose this discipline on itself.

Observability inside the loop

An agent loop that runs ten steps and returns an answer is a black box unless you log the right things. The minimum set:

  • Per-step: step number, tool called, arguments (truncated), result status (success/error), latency, input tokens, output tokens.
  • Per-run: total steps, total tokens (input and output), total wall-clock time, exit reason (model stop, step limit, token budget, timeout, repeated failure).
  • Per-tool: aggregate success rate, average latency, average result size. This is what tells you which tool is responsible for most of your spend.

The exit reason is the most important field. If your dashboard shows that 40% of runs end on the step limit rather than on the model's own stop, your agent is not completing its tasks. If 15% end on repeated failure, a specific tool is broken in a way the model cannot work around.

I wrote about attributing every model call to an owner earlier. In an agent loop, the owner is the step number and the tool name. Without that, the provider invoice is one number and you are guessing which step to optimise.

Multi-step planning and the backtracking problem

Simple agent loops are linear: step one feeds step two feeds step three. The model picks the next action based on everything so far, and there is no branching. This works for tasks with an obvious sequence (look up a value, transform it, write the result) and fails for tasks that require exploration.

The failure looks like this: the model commits to an approach on step two, discovers on step six that it was wrong, and then tries to recover without undoing steps three through five. It cannot undo those steps, because the context is append-only. So it works around its own earlier results, producing something that is technically complete but structurally incoherent.

Two patterns that handle this in practice:

  • Plan-then-execute. Ask the model to produce a plan (as structured output) before it starts calling tools. Validate the plan, then execute it step by step. If a step fails, return to the planning phase with the failure as new context. The plan is short enough that replanning is cheap, and you avoid the six-step dead end.
  • Checkpoint and rollback. After each successful step, snapshot the conversation state. If the model signals that the current approach is not working (or if a tool fails repeatedly), roll back to the last checkpoint and inject a message saying the abandoned approach did not work. This is more complex to implement but handles exploratory tasks where the plan itself is uncertain.

Both patterns share a principle: the orchestrator manages the conversation history, not just the tool calls. The model sees a curated version of what happened rather than the raw append-only log of every failed attempt.

When you do not need an agent

An agent loop is the most expensive way to call a model. Each iteration is a full round trip, each round trip re-sends the history, and each tool call is a latency and error surface. Before building one, check whether the task requires it.

  • If the steps are known in advance, a pipeline (a fixed sequence of model calls with code between them) is cheaper, faster, and easier to debug. An agent adds value only when the model needs to choose what to do next.
  • If the task needs one tool call, skip the loop entirely. Use the model's tool-calling capability in a single request and parse the result. No loop, no budget, no retry logic.
  • If the task is retrieval plus generation, a RAG pipeline with a single model call at the end is an order of magnitude cheaper than an agent that searches, reads, searches again, reads again, and then writes.
  • If the “agent” always follows the same path, you have discovered the path. Hard-code it. The model is adding latency and cost without adding decision-making.

The agent loop is the right tool when the number of steps, the tools required, or the order of operations depends on what the model finds along the way. For everything else, it is overhead dressed as flexibility.

The short version

The model does not know what it costs and will not stop on its own when things go wrong. The orchestrator needs a step limit, a token budget, a wall-clock timeout, and repeated-failure detection, all in code, none in the prompt. Truncate tool results before they compound across iterations. Log the exit reason for every run so you can see whether your agent is completing tasks or hitting ceilings. For tasks where the plan might be wrong, add a planning phase or checkpoint-and-rollback so the model does not spend ten steps working around its own mistakes. And before building any of it, check whether a pipeline or a single model call would do.

Written by Saumya Jain

GenAI & Full Stack Engineer working on agentic RAG systems, NestJS microservices, and real-time systems. Currently open to remote work.

LinkedInGitHubResume