·13 min read
You Cannot Prompt Away a Hallucination
My system prompt said every number must come from a tool. With that rule in place, an eval run still flagged 14 of 38 answers. A rule in a prompt is a request. A check in code is a control.
AI / LLM / Testing / Reliability / Architecture
In 2023 two lawyers in New York filed a brief that cited court decisions which did not exist. A chatbot had written the citations, with case names, courts and quotations, and the lawyers and their firm were ordered to pay a $5,000 penalty. In 2024 a Canadian tribunal held an airline responsible for a refund policy its website chatbot had made up for a grieving customer. The airline argued that it could not be liable for what the chatbot said. It lost.
Neither system was broken in the ordinary sense. Each produced fluent, well-formed, confident text, which is what a language model is built to produce. The text was false, and nothing between the model and the reader was in a position to notice.
That is how I think about hallucination now. A hallucination that reaches a user is a claim that no part of the system checked. You cannot instruct the model out of making such claims. You can decide, for each kind of claim your product prints, what it could be checked against, what does the checking, and what happens when the check fails.
Fluent is the only guarantee
A language model produces the next token from a probability distribution over what tends to follow the text so far. A true sentence and a false one come out of the same process, and the false one arrives with no marker on it. If the question asks for an expense ratio, the likeliest continuation is a plausible expense ratio, whether or not the model has ever seen the right one.
Two common fixes do less than people hope. Setting the temperature to zero makes the output more repeatable, which gives you the same wrong figure every time. Retrieval puts the right passage into the context, and the model can still misread it, merge two passages, or answer from memory when the passage does not cover the question.
So an application built on a model has one property it can rely on, which is that the output will sound right. Every other property (the figure is correct, the quotation exists, the policy is the one you publish) has to be supplied by something else. Both cases above ended the same way: the people who deployed or used the model owned what it said.
An instruction is a request
The first thing everyone does is write the rule into the prompt. I did too. In a portfolio assistant I built for mutual-fund investors, the system prompt has a section headed NUMBERS, and its first rule begins: "Never calculate, re-total, re-rank, or extrapolate a figure yourself. Every number you state must appear in a tool result."
That rule is worth having. It is also only text in the context window, with the same standing as everything else in there. The eval log shows what it was worth as a control. The harness records, for every run, the share of answers containing a figure that no tool returned during that turn. Across five runs with a live model on one day, that share was 23.3%, 13.5%, 7.9%, 5.3% and 36.8%. The target written in the evaluation doc is 2% or lower, and that is a target; no live run has met it.
The last two runs are the ones I keep coming back to. They were three minutes apart and logged against the same commit: 2 answers in 38, then 14 in 38. The log records a commit hash and not the working tree, so I cannot say what changed between them, and that is the lesson. A property that moves from 5% to 37% while I am editing wording is one I do not control through wording.
The prompt still matters, because it sets the base rate, and I would not delete the rule. But a prompt rule should be read the way you read a code comment that says "this function is never called with null". It states an intent, and something else has to enforce it.
Sort the output by claim
"Reduce hallucination" is too vague to build. What an answer contains is a set of claims of different kinds, and each kind has a different checker.
| Kind of claim | What it can be checked against | Checked by |
|---|---|---|
| A figure (an amount, a rate, a return) | The results of the tools called in this turn | Code |
| A citation or quotation | The ids and text of the documents retrieved | Code |
| A name from a closed set (a fund, an employer, a product code) | An allowlist | Code |
| A forecast, or a fact no source holds | Nothing | Nobody, so it is refused |
| An interpretation ("your portfolio is concentrated") | The figures it rests on | A person, or a model judge on a sample |
Writing that table for your own product is most of the design. The top three rows are mechanical, and they usually cover the claims that hurt most when wrong: the invented figure, the invented source, the invented name. The fourth row is a policy decision. The fifth is where a mechanical check runs out, and it is better to know that in advance than to discover it in a complaint.
Give the model less to invent
The cheapest check is the one you never need, because the model was not asked to produce the claim.
Compute what can be computed. In the same assistant, every figure a user sees is calculated in TypeScript and handed back as a tool result, so the model chooses which calculation to request and words the answer around it. I covered that design in the post on retrieval as a tool, so I will not repeat it. For hallucination, the consequence is that the correct figure is always sitting in the turn's tool results, which gives a check something to compare against.
Refuse what nothing grounds. Some questions have no source at all. "Will this fund beat that one next year?" cannot be answered by any tool, so any answer is invention. The prompt handles this with a specific instruction: say in one sentence that nothing grounds a forecast, then name the thing that can be shown instead, such as what the fund has returned since the user bought it. Refusals are their own category in the eval set, and a refusal passes only if it both declines and offers the alternative. A model told only to decline will stonewall, and users learn to rephrase until it stops.
Say when the data is absent. The rule I would copy into any assistant is the one for empty results: if the data is not there, say so as a fact about the portfolio ("you have no debt funds"), and do not fill the gap from memory. An empty tool result is where a model is most likely to fall back on what it remembers from training.
Constrain the form where you can. Enumerated fields and schemas remove whole classes of invented values, as described in the post on reliable JSON. The guarantee is narrow, though. A schema makes the output parse. It does not make the output true, and a well-formed wrong number is still a wrong number.
Check figures against tool results
With the figures computed by tools, the check is short. Extract every figure from the answer, extract every figure from the tool results of the same turn, and report any figure in the answer that has no match.
function unmatchedFigures(answer, toolResultsJson) {
const grounded = extractNumbers(toolResultsJson);
return extractNumbers(answer).filter(
(n) => !grounded.some(
(g) => Math.abs(g - n) <= Math.max(0.01, Math.abs(n) * 0.001),
),
);
}
The tolerance is 0.1% of the figure, with a floor of 0.01, which absorbs rounding and formatting without letting a different number through.
The harder decision is what counts as a figure. Run a naive number regex over a paragraph and it flags "3 funds", "2019" and "step 2". The extractor therefore keeps only tokens that carry financial meaning: anything with a currency symbol, a percent sign, a decimal point or comma grouping, and any bare integer of 1,000 or more. Bare integers below that are dropped, and so are bare four-digit years from 1990 to 2099.
One detail matters more than its size suggests. The eval harness's numeric grader and this runtime check import the same extractor from a shared library. If the offline grader and the online check disagreed about what a number is, the scorecard would describe a different system from the one users talk to.
A detector or a gate
What happens when the check finds something is a design decision, and in the portfolio assistant I have to be precise about it: the check does not block anything.
The answer streams to the browser token by token. The provenance check runs after the model has finished, on the complete text, so by then the user has already read the figure. When it finds an unmatched figure it writes a warning to the log with the request's trace id and increments a counter. Nothing in the interface marks the answer yet. That makes it a detector. It tells me the rate and gives me the exact turns to replay, and it protects no individual user from an individual wrong number.
I chose that knowingly, because a hard block on a streamed chat answer costs a lot. You either buffer the whole answer and lose the streaming, or you retract text someone is already reading. In a voice product the trade is worse, since a spoken sentence cannot be taken back, so there the check has to run on each sentence before it is spoken.
A gate fits when the output is an artefact that goes somewhere later. In a job-application pipeline I built, the output is a resume PDF, and nothing is submitted until a verification script passes. Its content pass requires each substantive line to match a sentence the generator is able to emit, and rejects any percentage that is not in the facts file. Two properties of that gate carry over to any other:
- It checks the thing that ships. It reads the rendered PDF, not the draft the PDF was built from, because the reader gets the PDF.
- It fails closed. A PDF it cannot read is a failure. A checker that returns empty text for an unreadable file will report zero unsupported lines, which looks the same as a pass.
So the question to ask of your own product is whether anyone is waiting on the output. If a person is reading it as it arrives, build a detector and make it loud. If the output is a document, an email or a database write, build a gate.
Put the rate on the scorecard
Once the check exists, the hallucination rate is a number, defined in a way that can be argued with: answers with at least one unmatched figure, divided by answers. In my harness it sits on the scorecard next to numeric accuracy, retrieval, refusal compliance and tool selection, appended to a history file on every run.
Three things about that number are easy to get wrong, and I got two of them wrong before I got them right.
A zero from a stub means nothing. The harness can run without an API key, using a stub in place of the model that routes by keyword and echoes the tool result back as its answer. Its hallucination rate is 0% on every run, because it has nothing to invent with. That figure appears on the project's scorecard, labelled as the keyless baseline, and it measures the tools, the retrieval and the guardrail code. It says nothing about a model. The live figures are the ones in the second section.
Recording a metric is different from gating on it. CI runs the keyless configuration, and the build fails on one condition only: numeric accuracy falling more than five points below the previous run on the same provider. The unmatched-figure rate is recorded and gates nothing. Given how far it moved between two live runs, gating on it today would produce a build that fails at random, which teaches a team to ignore the gate. It needs repeated runs and a stable baseline first.
A judge needs checking too. For the interpretive claims in the bottom row of the table, the plan is a faithfulness judge: a second model, from a different provider than the one being tested, asked whether each claim is supported by the tool results it is shown. It is written and wired in, and it has never run. Every row in the history file has that field empty. When it does run, it gets the treatment described in the post on evaluation: specific yes-or-no questions, the source material in front of it, and a hand-graded sample to confirm it agrees with a person.
What the check cannot see
The check matches values. It does not understand sentences, and that sets its limits.
- A real number in the wrong place passes. If the tools returned returns for two funds and the answer attaches each to the other, both figures match something in the tool results. The answer is wrong and the check reports nothing.
- Small numbers are not checked. "You have held this for 4 years" is a claim, and a bare 4 is below the extractor's threshold. That trade bought a check with few enough false alarms to be worth reading.
- Some flags are formatting. An answer that writes ₹2,08,323.40 as "₹2.08 lakh" has stated a true thing and still produces an unmatched 2.08. A year followed by a comma, as in "since 2019, the fund", is taken for a comma-grouped figure and flagged as well. So the rates above are a ceiling on invented figures, and each flagged turn still needs a person to look at it.
- Prose claims are out of scope. "Your portfolio is concentrated" contains no number to match.
There are also products where none of this is worth building:
- Invention is the job. Brainstorming, fiction, naming, a first draft of marketing copy. There is no source to check against, and a check would only get in the way.
- A person reviews every output before it has any effect. A code suggestion that has to compile and pass tests already has a verifier. So does a draft that a named person signs.
- There is no source of truth. A check needs something to compare with. If the facts are not written down anywhere, the first piece of work is writing them down.
The short version
A model's output is fluent whether or not it is true, and telling it not to invent things adds a sentence to the context without adding a control: with that rule in my prompt, a live eval run still flagged 14 answers in 38. List the kinds of claim your product prints and what each could be checked against. Compute the figures in code so the correct value is always in the turn, refuse the questions nothing grounds, and state missing data as missing. Then compare what the answer says with what the tools returned, using one extractor for both the eval grader and the runtime check. If someone is reading the output as it streams, the check is a detector, so log it, count it and replay the flagged turns. If the output is an artefact, make it a gate that reads the file that ships and fails when it cannot read it. Put the rate on the scorecard, distrust a zero that came from a stub, and remember the check sees values and not meaning. The model will always sound sure. Whether it is right is your code's job to find out.