Testing Something That Answers Differently Every Time

Every other feature you ship has a safety net: change the code, run the tests, see what broke. An LLM feature has none of it by default. The same input gives a different answer each time, so there is nothing to assert equality against — and the usual response is to skip testing entirely and rely on someone trying it manually before release.

Which means changing the prompt, the model version, the retrieval settings or the temperature is a change with no signal. It might have improved things. It might have broken the case that matters most to your largest customer. Nobody knows, including you.

Why normal tests do not transfer

Snapshot tests are the instinct and they are actively harmful here. The output legitimately differs run to run, so the snapshot fails constantly, gets updated without being read, and after two weeks the team approves every diff reflexively. You have built a ritual that consumes attention and detects nothing.

The shift required is from correctness to aggregate quality. Not “did this produce the right string” but “across a hundred representative cases, did the proportion that met the bar go up or down.” That is a different kind of test, and it is the only kind that works.

Test the deterministic parts normally

Before reaching for evals, notice how much of an LLM feature is ordinary software that ordinary tests cover completely:

Retrieval. Given a query, is the correct chunk in the results? That is deterministic and assertable, and it is where most quality problems actually live.
Prompt assembly. Does the built prompt contain the right context, in the right order, within budget? A pure function returning a string.
Output parsing. Feed recorded model responses — including malformed ones you have seen — through your parser and assert the handling.
Tool execution. The tools themselves are normal code with normal tests. The model chooses them; it does not implement them.
Everything around it — auth, rate limits, persistence, cancellation.

Cover those with the tests you already know how to write, and the genuinely non-deterministic surface shrinks to one step: given this prompt and this context, was the generation good. That is the only part needing an eval.

Building an eval set that is worth running

An eval set is a collection of inputs with a definition of what an acceptable output looks like. The definition rarely needs to be a full expected answer, which is what makes this tractable.

Assertions ordered by how much they are worth relative to their cost:

Structural. Valid against the schema, enum in range, required fields present. Free, deterministic, catches real breakage.
Substring or reference. Does the answer cite the expected document? Contain the correct figure? For factual retrieval this is most of what you need.
Negative assertions. Often more valuable than positive ones. Does it avoid promising a refund? Avoid naming a competitor? Avoid claiming certainty on a question with no answer in the corpus?
Graded judgement for the genuinely open cases, which is where a judge model comes in.

Where the set comes from matters more than its size. Fifty cases drawn from real user queries — especially ones that went wrong — beat five hundred invented ones, because invented cases cluster around what you already imagined. Every production failure should become an eval case, exactly like a regression test.

And keep the hard cases in. There is a strong pull toward an eval set where everything passes, because it feels like success. A set at 100% has stopped measuring anything.

LLM-as-judge, and where it stops being trustworthy

For open-ended output, using a model to grade another model's output is the only scalable option. It works, with specific caveats that decide whether the numbers mean anything.

Ask for a specific judgement, not a rating. “Rate this answer 1–10” produces noise: mostly sevens and eights, unstable across runs. “Does this answer contain the refund window stated in the reference? Yes or no” is stable and actionable. Decompose quality into several binary questions rather than asking for one number.

Give it the reference material. A judge scoring factual accuracy without the source is just a second opinion with the same failure modes as the first.

Known biases, all measurable and all worth designing around:

Length. Judges favour longer answers. If a change made responses more verbose, the score rises without quality changing.
Position. In A/B comparisons the first option is favoured. Randomise order, or run both orders and keep only agreements.
Self-preference. Models rate their own family's output higher, so a judge from the same provider as the system under test flatters it.

The essential validation step, which is usually skipped: grade a sample by hand and check the judge agrees with you. If human and judge disagree on a quarter of cases, the eval is measuring the judge's taste rather than your system's quality, and every decision made from it is unfounded. Recheck that agreement whenever you change the judge prompt or model.

Running it in CI without it being unusable

Evals are slow and cost money per run, so treating them like unit tests does not work. What does:

A small fast subset on every pull request — perhaps twenty cases, structural assertions only, no judge. Catches the change that broke parsing or emptied the context.
The full set on merge or nightly, with the judge, reported as a trend rather than a gate.
Compare against the previous run, not an absolute bar. The useful signal is “this change moved the score from 78% to 71%”. An absolute threshold either blocks constantly or never.
Fix a seed and temperature where you can, so run-to-run variance does not swamp the effect you are trying to observe.
Report the failures, not the number. A dashboard saying 71% is not actionable. A list of which eight cases regressed is.

Version everything that affects output — prompt, model, retrieval settings — and record it with each run. Without that, a score movement cannot be attributed, and you will spend an afternoon investigating a prompt change that was actually a provider-side model update.

The signals that matter most are online

An eval set is a proxy for real usage and drifts from it as usage changes. The strongest quality signals come from production and cost nothing to collect:

Explicit thumbs up and down, sparse but unambiguous.
Regeneration and rephrasing rates — a user asking the same thing again is telling you the first answer failed.
Abandonment mid-stream, which distinguishes “wrong” from “too slow”.
Retrieval misses — how often nothing cleared the relevance threshold, which is a content gap rather than a model problem.
Fallback and refusal rates, which move sharply when something upstream breaks.

Feed the bad ones back into the eval set. That loop — production failure becomes eval case becomes regression protection — is the whole mechanism, and it is the same discipline as writing a test for every bug, applied to a system that cannot be asserted on directly.

The short version

Snapshot tests are worse than nothing here. Cover retrieval, prompt assembly, parsing and tools with ordinary tests until only generation is non-deterministic. Build an eval set from real queries, weighted toward failures, and prefer structural and negative assertions over graded opinion. Where you need a judge, ask binary questions, give it the reference, control for length and position bias, and validate it against your own grading. Compare runs rather than thresholds, and let production failures keep feeding the set.