Ask a general-purpose AI model what your refund policy is and it will tell you something. It will be fluent, plausible, formatted nicely, and completely invented, because it has never seen your refund policy. That failure mode has a name — hallucination — and retrieval-augmented generation, or RAG, is the standard engineering answer to it.
The idea, without the acronym
A language model can only reason about text placed in front of it. RAG is the practice of finding the right text first, then asking the model to answer using only that.
The sequence looks like this:
- Someone asks a question.
- The system searches your documents for the handful of passages most likely to contain the answer.
- Those passages are handed to the model along with the question and an instruction along the lines of "answer from this material; if it is not here, say so."
- The model writes the answer, and the system shows which source it came from.
It is closer to a very good research assistant than to a know-it-all. The model supplies the language; your documents supply the facts.
Why not just fine-tune the model?
This is the most common question, and the answer is usually cost and freshness.
- Fine-tuning adjusts the model's behavior by training it further on examples. It is good at teaching style, format, and task shape — "always respond as a structured claim summary." It is a poor and expensive way to teach facts, because every time a fact changes you have to train again.
- RAG changes what the model can see at the moment it answers. Update a document and the next answer reflects it immediately. No retraining, no waiting.
Most business problems are facts-that-change problems. Pricing, policies, procedures, inventory, case history. That is why RAG is the default architecture and fine-tuning is the exception.
Why RAG is the part worth paying for
Calling a model API is easy. Retrieval is where builds succeed or fail, because "search your documents" hides a lot of engineering:
- Chunking. Documents get split into passages. Split badly and you sever a sentence from the condition that qualifies it — which is how a system confidently quotes a rule without its exception.
- Retrieval quality. Pure keyword search misses paraphrases. Pure semantic search misses exact identifiers like part numbers. Serious systems use both.
- Permissions. If retrieval ignores access control, you have built a very efficient way for anyone to read HR files. Filtering must happen at retrieval, not in the prompt.
- Freshness. Something has to notice when a source document changes and re-index it, or your assistant will cite last year's policy forever.
How to tell whether it is working
Insist on these three properties before anyone calls a RAG system done:
- Citations. Every answer links to the passage it came from. A user who can check in one click will catch what evaluation misses.
- An honest "I do not know." A system that never declines is a system that is guessing. Refusal on thin evidence is a feature.
- A real evaluation set. Fifty to a hundred questions with known-good answers, run on every change. Without it you have vibes, not quality control.
What it needs from you
RAG makes your documentation load-bearing. If your policies are contradictory, out of date, or scattered across three drives and an inbox, retrieval will surface that inconsistency at speed and in front of staff.
That is not a reason to avoid it — it is usually the most valuable thing an early build reveals. But it does mean the first phase of many AI projects is unglamorous document and data work, and pretending otherwise is how projects slip.
Done properly, RAG turns a general model into something that knows your business, cites its sources, and admits its limits. That combination is what makes it safe to put in front of staff — and eventually customers. It is the same architecture behind most of the internal assistants we build as part of application development engagements.
Have a document set you would like to make answerable? That is the conversation to start with.