·11 min read
Every Model Call Should Have an Owner
The invoice says one number. Your pipeline has fourteen stages, and one of them is most of the bill. Attribution is what turns model spend from a surprise into a design decision.
LLM / AI / Cost / Architecture / Backend / PostgreSQL
The provider dashboard gives you spend per API key per day, possibly split by model. Your feature has fourteen stages: ingest, deduplicate, cluster, assess, research, fact-check, summarise, and so on down the graph. Between those two facts there is no mapping at all.
So the first time anyone asks the useful question, which is not “how much are we spending” but “on what, and is that the right place for it”, the answer is that you do not know. You will then optimise the stage you can picture most easily, which is almost never the expensive one.
The fix is small and it has to be in place before you need it: no model call happens without declaring who it belongs to.
The bill has no structure, and averages hide the skew
Model spend in a multi-stage system is not evenly distributed. In a personalised news pipeline I built, one stage was roughly two thirds of the bill, and it was not the stage with the longest prompt. It was a short prompt that ran once per candidate, several hundred times a day, while the stage everyone assumed was expensive ran twelve times.
That is the general pattern. Cost per call is what you notice while writing the prompt; cost per stage is cost per call multiplied by how often that stage runs, and the multiplier spans three orders of magnitude across a single pipeline. Reasoning about the first number while ignoring the second is how teams spend a week shortening a prompt that accounts for 4% of spend.
Recording tokens per call is necessary and not sufficient. The trace table from the Postgres post already has the columns: model, input and output tokens, cost, feature. “Feature” is the level almost everyone stops at, and it answers the wrong question. What you need to be able to group by is narrower and more specific:
- Which stage. A named node in the graph, not a module or a service.
- Which run. So a single night's pipeline has a total, and two nights can be compared.
- Which subject. The reader, tenant or account the work was done for, because unit economics live here and nowhere else.
Attribution belongs at the call site, and should be impossible to omit
Nothing in the codebase is allowed to talk to the provider SDK directly. One wrapper, and its signature requires the attribution as ordinary arguments with no defaults.
// The stage list is a closed union. Adding a member is a diff
// someone reviews, which is the point: a new place that spends
// money should not be addable by accident.
export type Stage =
| "assess_candidate"
| "cluster_summary"
| "research_query"
| "fact_check"
| "reader_digest";
export async function callModel<T>(opts: {
stage: Stage; // required, no default
runId: string; // required
subject: string | null; // reader/tenant, or null for shared work
model: ModelId;
messages: Message[];
maxOutputTokens: number; // required: an unbounded call is a bug
schema?: ZodSchema<T>;
}): Promise<Completion<T>> {
const verdict = await checkBudget(opts); // see below
if (!verdict.ok) return refused(verdict); // a value, never a throw
const started = Date.now();
const res = await provider.create(toProviderShape(opts));
await recordTrace({
stage: opts.stage,
run_id: opts.runId,
subject: opts.subject,
model: opts.model,
input_tokens: res.usage.input,
cached_input_tokens: res.usage.cachedInput ?? 0,
output_tokens: res.usage.output,
cost_usd: priceFor(opts.model, res.usage), // versioned price table
latency_ms: Date.now() - started,
outcome: res.stopReason,
});
return parse(res, opts.schema);
}
Three details carry most of the value. stage is a closed union rather than a string, so a new spending site cannot be introduced without touching a type that someone reviews. maxOutputTokens is required, because an unbounded output is an unbounded bill and the default limits are generous. And priceFor reads a versioned price table that is stored with the trace, not a constant: provider prices change, and a cost history computed with today's prices for last quarter's calls is fiction.
Enforcement can be one line in CI. A grep that fails the build if the provider SDK is imported anywhere except the wrapper is crude, takes a minute to write, and is the only reason the discipline survives contact with a deadline.
What this buys immediately is a single query instead of an investigation:
SELECT stage,
count(*) AS calls,
round(sum(cost_usd), 2) AS spend,
round(sum(cost_usd) / count(*), 5) AS per_call
FROM llm_traces
WHERE created_at >= now() - interval '7 days'
GROUP BY stage
ORDER BY spend DESC;
The first time you run that, the ordering is usually a surprise. That surprise is the entire return on the work.
The lever is the candidate set, not the prompt
Once spend is attributed by stage, you can write down the arithmetic. For any stage:
stage cost = cost per call
x items the stage runs over
x subjects it runs per item
x runs per day
Prompt engineering attacks the first factor and can win maybe 20 to 40%. The other three factors are architectural, and they are where the multiples are:
Put the expensive stages behind the cheap ones. The rule I would keep in any pipeline: no model-dependent stage runs over the full candidate set. Deduplicate, drop the already-seen, apply hard exclusions, and rank with computed signals first. Assessment then runs over what survived, and the research stage, which is an order of magnitude dearer per call, runs only over the handful that will be shown.
Order the cheap filters cheapest-first. Take three-tier deduplication: a URL hash is free, a near-duplicate hash over the text costs microseconds, and an embedding comparison costs a model call. Running them in that order means the expensive tier only ever sees what the two free tiers could not resolve. Deduplication is usually justified as a quality feature; it is also one of the largest cost controls in the system.
Do per-item work once, not once per subject. In a personalised product this is the single biggest structural saving available. Whether an article is original reporting or a rewrite is a property of the article, so it is assessed once into a shared corpus and read by every reader whose feed considers it. Only the judgements that depend on the reader run per reader. This is also a property you find by looking at attributed traces: the same stage billed a hundred times against a hundred subjects with identical inputs is a cache key waiting to be written, and it is invisible in an invoice.
The per-item assessments from the ranking post have the property that makes this work: a judgement about one item does not change when another item arrives, so it can be computed once and reused. A list-ranking prompt has to be redone in full every run, for every reader, forever. That is a cost argument as much as an explainability one.
For a comparison at the other extreme, the retrieval system with no embedding pipeline is the same trade made deliberately in reverse: spend the model budget on expanding the query and selecting chunks, and spend nothing on maintaining an embedding index. Both decisions come from knowing which stage the money is in.
The number that matters is cost per reader per day
Total spend tells you whether you are alarmed this month. Unit cost tells you whether the product works. With a subject column it is a query rather than a modelling exercise:
WITH per_subject AS (
SELECT subject,
date_trunc('day', created_at) AS day,
sum(cost_usd) AS spend
FROM llm_traces
WHERE subject IS NOT NULL
AND created_at >= now() - interval '30 days'
GROUP BY 1, 2
)
SELECT round(avg(spend), 4) AS mean,
round(percentile_cont(0.5) WITHIN GROUP (ORDER BY spend), 4) AS p50,
round(percentile_cont(0.95) WITHIN GROUP (ORDER BY spend), 4) AS p95,
round(max(spend), 4) AS worst
FROM per_subject;
Look at the spread rather than the mean. Personalised systems have a heavy tail by construction: a reader following forty topics costs multiples of the median, and shared work amortises unevenly across them. The p95 reader is the one who decides whether a free tier is viable, whether a per-account cap is needed, and whether the long tail should be served by a cheaper model. None of those conversations can happen without the number, which is why the subject column is worth more than the stage column for anything commercial.
A budget is a check before the call, not a chart
A dashboard is observation. What stops an incident is a limit consulted on the way in, and the inputs are all available: you have the rendered prompt, so input tokens are countable before you send them, and you set maxOutputTokens, so the worst case is known.
type Verdict =
| { ok: true }
| { ok: false; cap: "run" | "stage_run" | "subject_daily" };
async function checkBudget(opts: CallOpts): Promise<Verdict> {
const est = estimateCost(opts); // countable, not guessed
const run = await spendForRun(opts.runId);
if (run + est > CAP_PER_RUN) return { ok: false, cap: "run" };
const stage = await spendForStageInRun(opts.runId, opts.stage);
if (stage + est > CAP_PER_STAGE_RUN) {
return { ok: false, cap: "stage_run" }; // the fan-out guard
}
if (opts.subject) {
const day = await spendForSubjectToday(opts.subject);
if (day + est > CAP_PER_SUBJECT_DAY) {
return { ok: false, cap: "subject_daily" };
}
}
return { ok: true };
}
Three caps cover the realistic failures. Per run catches the retry storm and the loop that will not terminate. Per subject per day catches the one account whose usage is pathological, before it is a month of margin. Per stage per run catches the fan-out bug, where a stage you expected to run twelve times runs four hundred times because a filter upstream stopped filtering. That last one is the cap that has saved me the most, because the bug is invisible in every other signal: the pipeline succeeds, the output looks plausible, and only the spend moved.
An iteration cap on an agent loop is not a substitute for a token cap. Ten iterations that each accumulate the full conversation cost far more than ten times the first call, because the context grows with the transcript.
The check returns a verdict rather than throwing. What should happen at a cap is a real design question with a real answer, and the answer is never an exception: skip the optional stage, serve the cheaper path, record which cap stopped you. A budget that raises has converted a cost control into an outage, which is the one thing it exists not to do. The caller decides; the check only reports.
The smaller levers, in the order they pay
These are real and they are second-order compared to the funnel. Worth doing, not worth doing first.
- Exact-hash completion caching, keyed on the rendered prompt rather than the question, with the tenant as a real column. The cache design and the leak it prevents is worth reading before building one.
- Provider-side prompt caching, which rewards putting the long stable material first and the variable material last. Prompt assembly order is a cost decision, which is not obvious from reading the prompt.
- Propagating cancellation. An abandoned stream that keeps generating is billed in full, and passing the abort signal upstream is one argument. On a product where people rephrase mid-answer this is a material share of spend.
- Per-stage model choice. Attribution is what makes this safe: move one stage to a smaller model, watch that stage's spend and the eval score for that stage together, and keep the change only if the second did not move. Without per-stage numbers this is a guess in both directions.
- Output ceilings. Output tokens are typically priced several times higher than input. A schema that asks for a 180-character line instead of a paragraph is a cost control disguised as a style choice.
When not to build this
The accounting is cheap but it is not free, and there are systems where it is the wrong thing to spend a week on.
- One stage, one model, low volume. If the feature is a single call per user action and the monthly bill is small, the invoice is the report. Add the wrapper anyway, because it costs nothing now and retrofitting attribution across forty call sites is a bad afternoon. Skip the ledger and the queries until there is a second stage.
- The model is not the dominant cost. Plenty of AI systems are dominated by storage, egress or the compute around an embedding index rather than by inference. Measure which before you build accounting for the wrong one.
- Latency is the binding constraint. If the thing limiting the product is that answers take nine seconds, cost attribution is not the instrument you need, although the same trace rows will answer both questions.
- Before there is a product. Unit economics for a prototype with eleven users is a spreadsheet exercise about an imagined future. The wrapper and the required
maxOutputTokensare the parts worth having on day one.
The short version
An invoice has no stages and your architecture is nothing but stages, so attribute every call at the point it is made: a named stage from a closed union, a run id, and the subject it was done for, through one wrapper that nothing is allowed to bypass. Store cost with the price table version, because provider prices move and uncorrected history is fiction. Then follow the arithmetic: stage cost is per-call cost multiplied by how many items, subjects and runs it touches, so the expensive stages belong behind the cheap filters, the cheap filters belong in cheapest-first order, and per-item work belongs in a shared corpus rather than repeated per reader. Track cost per subject per day and look at the p95 rather than the mean, because the tail is what decides pricing. Enforce caps per run, per subject and per stage before the call rather than watching a chart after it, and remember that a cap which throws has turned a cost control into an outage. Do the caching, the cancellation and the model tiering second, once you know which stage the money is in.