A language model asked a question it cannot answer will not say so. It will produce the most plausible-sounding continuation available, in the same confident register it uses when it is right. That behaviour has a name — hallucination — and understanding why it happens is what separates teams who build reliable AI systems from teams who are perpetually surprised.
Why it happens
A language model predicts likely text. It is not looking anything up, and it has no internal flag for "I do not know this." When you ask about your refund policy, the model produces text shaped like a refund policy, because that is what the question calls for. Fluency and accuracy are separate properties, and only one of them is what the model optimizes for.
This is not a defect to be patched. It is what the technology is. The engineering task is not eliminating hallucination — it is building systems where hallucination is caught before it matters.
The five controls that actually work
1. Give it the source material
The single largest reduction comes from retrieval: find the relevant passages and put them in front of the model, then instruct it to answer only from that. A model with the policy in front of it does not need to invent one. This is what RAG is for, and it is the foundation everything else sits on.
2. Require citations
Make every claim link to the passage it came from. This does two things: it lets a reader verify in one click, and it makes fabrication visible, because an invented claim has nothing to cite. Systems that cite are meaningfully more trustworthy than systems that do not, even when the underlying model is identical.
3. Permit refusal, and reward it
Instruct the system explicitly that "the provided material does not answer this" is a correct and expected response. Then check that it actually does so, by asking questions you know are unanswerable. A system that never declines is a system that is guessing, and you should find that out during testing rather than in front of a customer.
4. Keep a real evaluation set
Fifty to two hundred questions with known-correct answers, run automatically on every change to the prompt, the model, or the documents. Without this you cannot tell whether a change helped, and you will not notice the day a model update degrades a task that used to work.
5. Put a human where it counts
Anything that reaches a customer, moves money, or informs a decision about a person gets reviewed. This is not a failure of automation; it is the design that lets you deploy at all.
Where hallucination is most dangerous
Not where people expect. The risk is highest when the output is mostly right — a summary with one invented figure, a citation to a real document with the wrong section, a correct-looking calculation with a transposed number. Obvious nonsense gets caught. Plausible errors get forwarded.
Design review steps for the plausible case: show the reviewer the source alongside the claim, rather than asking them to assess the output on its own.
The numbers question
One specific rule worth adopting: do not let a language model do arithmetic that matters. Have it extract the values and hand them to real code for the calculation. Models are good at reading a table and bad at reliably adding it up, and the failure is silent.
The same applies to anything with a deterministic right answer — eligibility rules, tax bands, date arithmetic. Use the model to interpret and conventional software to compute.
What good looks like
A system that cites its sources, says "I do not know" roughly as often as the material warrants, is measured against a fixed test set, and escalates anything consequential to a person. That combination makes hallucination a managed risk rather than a lurking one.
It is also what any vendor should be able to describe unprompted. If they cannot, ask how they measure accuracy — the answer will tell you a great deal. More questions worth asking are in our note on what separates a real build from a thin wrapper.
Want a second opinion on an AI system's accuracy claims? We will look at how it is measured.