Skip to content
SJ
All writing

·11 min read

A Pipeline Where No Node Is Allowed to Raise

An exception at stage nine throws away the model calls you already bought. Make every node declare what it returns when it cannot do its job, and make that answer a value rather than a log line.

AI / LLM / Reliability / Architecture / Backend


The pipeline behind a personalised feed is fourteen nodes long. Ingest, deduplicate, cluster, assess, research, fact-check, summarise, rank, assemble. Most nodes call a model, every model call can fail, and the whole thing runs unattended.

The rule that made it dependable is a prohibition rather than a technique: no node may raise. Not “handle errors carefully”. A node either returns its output or returns a declared degradation, and the orchestrator has no catch block because there is nothing for it to catch.

What an exception costs in a pipeline like this

In a request handler, throwing is a perfectly good design. The response is a 500, the client retries, the work lost was one request's worth and it was cheap.

Nine nodes into a nightly run, the arithmetic is different. The exception discards several hundred model calls that have already been made and billed, along with an hour of wall clock, and it produces nothing for anybody. The cost of throwing is not the failure, it is the work you already bought. This is the same reason spend is worth attributing by stage: once you can see that one stage cost more than the other thirteen combined, throwing it away at stage nine stops looking like error handling and starts looking like the incident.

Catching the exception at the top is worse. Someone wraps the orchestrator in a try/catch, logs the error, and lets the run continue. Now the pipeline reports success every night while a stage contributes nothing, and the first person to notice is a user asking why the summaries have looked odd for three weeks.

Both options dodge a question that only the author of each node can answer: what should this stage produce when it cannot do its job? Asked at design time it usually has a good answer. Asked at 3am inside a catch block, it does not.

The fallback is part of the node's contract

So the node signature carries it. Every node returns its value plus whatever it had to give up to produce it:

type Degradation = {
  node: NodeName;
  reason: "provider_error" | "timeout" | "refusal"
        | "budget_exhausted" | "invalid_output";
  detail: string;
};

type NodeResult<T> = {
  value: T;                      // always present, possibly reduced
  degradations: Degradation[];   // empty on the happy path
};

// the orchestrator has no try/catch: there is nothing to catch
async function runPipeline(input: Input) {
  let state = seed(input);
  for (const node of NODES) {
    const result = await node.run(state);
    state = merge(state, result.value);
    state.degradations.push(...result.degradations);
    if (fatal(state.degradations)) return abort(state);
  }
  return state;
}

The interesting work is deciding, per node, what the reduced value is. Three categories cover almost everything.

Additive nodes skip. An enrichment that adds related links contributes nothing when it fails, and nothing is a complete and correct answer. The item continues without the links.

Substitutable nodes fall back to something deterministic. A summariser that fails uses the source's own opening sentences, flagged as unsummarised. It is worse, but it is a real summary of a real article, which is the property that matters.

Scoring nodes fall back to a neutral value. This one works cleanly because of how the ranking is built. Since the score is a sum of named terms, a missing judgement becomes a defined neutral contribution rather than a crash or a zero: the item ranks on the terms that are still available, and it neither benefits nor suffers from the assessment that did not happen. A list-ranking prompt has no equivalent of this, because there is no term to leave out.

One category does not degrade at all. If ingestion returns nothing, there is no run: no amount of graceful handling downstream invents articles. When a node that everything downstream depends on fails, it should stop the run cleanly and early, which is a different thing from an exception eight nodes later.

A degradation is a value that travels with the data

The degradation has to be attached to the output, not written to a log. Logs are for humans who are already looking. A field is available to everything downstream, and three consumers need it.

The interface can say so. An item that was never fact-checked can be presented as unverified rather than presented identically to one that passed. A system whose failures are invisible to its users is lying to them, and what separates that from graceful degradation is whether the reduced output is labelled.

Evaluation can separate two things that look alike. A quality score that drops because a provider was returning 529s all night is not a quality regression, and an eval harness that cannot tell the difference will send you looking for a prompt bug that does not exist. Excluding or bucketing degraded cases is only possible if degradation is recorded per item.

You can ask questions about it later. How many items shipped without fact-checking last week, and were they concentrated in one source? That is a query against a column, and it is unanswerable against a log line in a file.

Give the reasons a small closed vocabulary rather than free text. Five values that can be grouped and alerted on beat an English sentence that is unique per occurrence.

The deterministic floor

The strongest version of this is to give every model-dependent stage a path that does not involve a model at all. In a tool I built that writes shooting scripts for a trip, there are three interchangeable model providers behind a preference order, and behind all of them a built-in deterministic writer. With no credentials configured at all, the application still produces a script. It is duller and more mechanical. It is also complete, and nothing in the product is unreachable because an API key expired.

That floor is useful in three ways, and only one is reliability.

  • It is the last fallback when every provider in the preference order has failed, which does happen.
  • It makes the pipeline testable end to end with no provider, no spend and no flakiness. Most of a pipeline is ordinary software, and this is what lets you test it as such.
  • It tells you what the model is actually adding. Put both outputs side by side. If the deterministic floor is nearly as good for a given stage, you have found a stage that does not need a model, and you found it for the price of a comparison.

The caveat: a deterministic floor is real work, sometimes as much as the feature. It is worth it when availability matters more than peak quality, and it is not worth it when the model's output is the product rather than a component of it. Nobody needs a template-based fallback for a chat assistant.

Outages are not the only trigger

Once degradation is a first-class value, three other situations want the same mechanism rather than one of their own.

Budget exhaustion. When a run hits its cap, the optional stages should stop running and the run should complete with what it has. That is a planned degradation with a reason code, and routing it through the same path as a provider failure means the interface, the metrics and the eval harness all handle it without new code. A budget check that throws converts a cost control into an outage, which is the one thing it must not do.

Refusal. A model declining to answer is a legitimate outcome and belongs in the schema as a representable result. Treating it as an exception is a category error, and it produces the worst possible behaviour: a retry loop against a model that is refusing for a good reason.

Deadlines. Give every node a time budget and make the deadline produce the declared fallback rather than an abandoned promise. A node that hangs is indistinguishable from a node that is slow, and without a deadline the run does not finish, which is the failure mode with no error message at all.

Fallback rate is the health metric

A pipeline built this way always succeeds, which is why it needs a different alarm. Success is no longer evidence of health. Fallback and refusal rates are listed among the online signals worth collecting; in a system with declared degradations they are the primary one.

SELECT d.node,
       d.reason,
       count(*)                                        AS items,
       round(100.0 * count(*) / max(r.total), 1)       AS pct
FROM run_degradations d
JOIN run_totals r USING (run_id)
WHERE r.started_at >= now() - interval '7 days'
GROUP BY d.node, d.reason
ORDER BY pct DESC;

Two rules about reading it. Alert on the change, not the level. A node sitting at 2% degraded forever is a node with a known, acceptable failure rate. The same node at 30% is an incident, and the absolute number tells you nothing without yesterday's.

Cap the total. If nine of fourteen nodes degraded, the output is not a feed with some gaps in it, and shipping it because every individual node behaved correctly is the wrong call. Define a threshold beyond which the run aborts and yesterday's output stands. Graceful degradation without a ceiling is a machine for shipping progressively worse output while every run reports success, which is the failure this whole approach is supposed to prevent.

Where failing loudly is right

The prohibition is on nodes swallowing expected external failures. It is not a licence to catch everything, and in practice that distinction is what goes wrong.

  • Programming errors must throw. A provider timeout is an expected external failure. A null dereference, a schema mismatch in your own code, a violated invariant: those are bugs, and a bug that lands in a generic catch becomes a permanent, silent quality loss that nobody ever fixes. Catch the specific, named failures. Never wrap a node body in a bare catch-all.
  • Tests run in strict mode. Any degradation should fail the test suite. A fallback path that activates in CI is hiding the exact defect the suite exists to find, and the only way to surface it is a flag that turns every degradation into a failure.
  • Irreversible and financial actions do not degrade. “The verification step was unavailable, so we proceeded” means a control got skipped. If a step gates something that cannot be undone, its unavailability blocks the action.
  • Interactive requests with a person watching. One user, one request, retry available: an error message is clear and cheap, and a worse answer with no explanation is neither. This pattern is worth it when the work is unattended and sits on top of expensive partial state.
  • When the degraded output is indistinguishable to the consumer. If nothing downstream can tell that a stage was skipped, do not build the fallback. All it adds is an undetectable way to ship worse work.

The short version

An exception at stage nine throws away the model calls you already paid for, and a catch-all at the orchestrator converts that into a failure that nobody notices for three weeks. Make the fallback part of each node's contract, decided by the person who understands the stage: additive nodes skip, substitutable nodes fall back to something deterministic, scoring nodes fall back to a neutral term. Attach the degradation to the data rather than the log, with a small closed vocabulary of reasons, so the interface can label reduced output, the eval harness can separate an outage from a regression, and last week's question is a query. Give expensive stages a deterministic floor where availability matters, if only to learn what the model was adding. Route budget exhaustion, refusals and deadline misses through the same mechanism instead of inventing three. Then watch the fallback rate rather than the success rate, alert on its movement, and cap how much degradation a single run is allowed to accumulate before it ships nothing at all. Bugs still throw.

Written by Saumya Jain

GenAI & Full Stack Engineer working on agentic RAG systems, NestJS microservices, and real-time systems. Currently open to remote work.

LinkedInGitHubResume