RAG explained: how AI answers from your documents instead of making things up

Ask a bare language model about your business and it will answer fluently and, quite possibly, wrongly — it has never seen your contracts, prices or policies. RAG (retrieval-augmented generation) is the standard fix, and the idea is simple enough to explain in a paragraph.

The idea in one paragraph

When you ask a question, the system doesn't send it straight to the model. First it retrieves: searches your documents — files, emails, records — for the passages most relevant to the question. Then it hands the model your question plus those passages and instructs it: answer from this material and cite it. The model becomes a reader of your business rather than a guesser about it. Ask "what's our cancellation policy for trade customers?" and the answer comes from your actual terms document, with a link to it.

Why the sources matter as much as the answer

A RAG answer arrives with receipts — the documents it drew from. That changes the trust equation: you don't have to believe the AI, you can check it in one click. For anything commercial — quotes, policies, client history — answers without sources shouldn't be trusted, and any system that can't show its sources isn't fit for business use. This is the backbone of what we call a Company Brain.

Where RAG goes wrong

• Retrieval misses. If the search step fetches the wrong passages, the model answers correctly from irrelevant material — confident and useless. Retrieval quality, not model quality, is where most bad answers come from.
• Stale documents. RAG answers from what's indexed. If the 2023 price list is in and the 2026 one isn't, you'll quote 2023 prices fluently. Freshness pipelines matter.
• Permissions ignored. Naive RAG searches everything for everyone — meaning anyone can ask their way into salaries or legal disputes. Permission-aware retrieval is non-negotiable and is real engineering.
• Contradictory sources. Businesses accumulate conflicting documents. Good systems surface the conflict; bad ones pick one silently.

What to ask any vendor

"How does retrieval respect permissions?" · "How fresh is the index?" · "Show me an answer with its sources." · "What happens when it doesn't know?" The last one is telling: the right behaviour is saying so — an honest "I can't find this" beats a fluent invention every time.

← All notes