AI
What is RAG, and why does it matter for business AI?
Retrieval-augmented generation, or RAG, is the technique that lets an AI model answer questions using your own documents and data rather than only what it learned during training. It is the difference between an assistant that sounds plausible and one that is actually useful for your business.
Contents
The short answer
RAG is a pattern with two steps. First, the system searches a collection of your own content — documents, product data, policies, past enquiries — and retrieves the passages most relevant to a question. Second, it gives those passages to a language model along with the question and asks it to answer using only that material.
The result is an answer grounded in your information, which can cite where it came from. That matters because a general-purpose model does not know your prices, your specifications, your terms or your customer history.
Why a plain chatbot is not enough
Public language models are trained on broad text up to a point in time. They know nothing about the price list you updated last week, the contract terms you offer, or the technical specification of a product you launched last month.
When asked about those things, a model tends to produce a fluent, confident, wrong answer. This is usually called hallucination, and in a business setting it is the main reason assistants lose the trust of the people who were meant to use them.
RAG addresses the problem by changing the task. Instead of asking the model to remember, you ask it to read — and reading a supplied passage is something these models do far more reliably than recalling facts.
How RAG works, step by step
- Prepare the content. Documents are cleaned and split into meaningful chunks — a section, a policy clause, a product entry — with metadata such as product, branch, date and access level.
- Index it. Each chunk is converted into an embedding (a numerical representation of its meaning) and stored in a searchable index, alongside a conventional keyword index.
- Retrieve. When a user asks a question, the system finds the chunks most relevant to it, filtering by permission so people only retrieve what they are allowed to see.
- Generate. The model receives the question and the retrieved chunks, with instructions to answer only from them and to say so when the answer is not present.
- Cite and log. The answer includes the sources used, and the interaction is logged for review and improvement.
Where RAG works well
RAG suits problems where the answer exists somewhere in your material but is hard for people to find quickly.
- Staff asking about policies, procedures and pricing without interrupting a colleague
- Customer questions about products, delivery, terms and documentation
- Sales teams finding the right specification or case reference during a call
- Support teams searching years of tickets and resolutions
- Reviewing long documents against a defined set of criteria
Where RAG goes wrong
RAG is not magic, and most failures are avoidable once you know where to look.
Poor retrieval is the biggest cause. If the right passage is never retrieved, the model cannot use it, and may fall back on inventing an answer. Chunking that splits a table from its heading, or a clause from the term it modifies, produces exactly this.
Stale or contradictory content is the second. If three versions of a price list exist, the system will happily quote whichever it finds. Content ownership and versioning matter as much as the technology.
Exact-match queries are a third. Part numbers, invoice references and names are poorly served by semantic search alone, which is why good systems combine it with keyword search.
Finally, permission mistakes. If access rules are not applied at retrieval time, a model can surface a document to someone who should not see it.
How to tell whether it is working
Build a set of real questions with known correct answers before launch, and run it against every change. Without that, nobody can say whether an update improved the system or quietly broke it.
Track the questions the system could not answer, and the ones users marked unhelpful. Those are your content and retrieval gaps, and they are usually cheaper to fix than to explain.
Does a small business need RAG?
Not always. If your knowledge fits on a few pages, a well-written FAQ may do more than any AI. RAG earns its cost when the volume of material is large, changes regularly, or is spread across systems, and when people spend real time searching it.
A sensible first step is to pick one narrow use — say, answering common product questions from your catalogue — and measure whether it saves time and answers correctly before expanding.
Frequently asked questions
Is RAG the same as fine-tuning a model?
No. Fine-tuning changes the model itself using training examples; RAG leaves the model unchanged and gives it relevant material at question time. For most business knowledge, RAG is cheaper, easier to update and easier to audit.
Does RAG send our data to an AI provider?
The retrieved passages are sent to the model provider with each question, so data handling terms matter. Business or enterprise API terms typically exclude training on your data, and sensitive content can be restricted or kept out of scope.
Can RAG work with data in a CRM or database?
Yes. Records can be retrieved through an API or database query and supplied to the model as context, with the same permission controls.
How long does it take to set up?
A focused assistant over a defined set of documents can take a few weeks. Larger systems with several sources, permissions and integrations take longer and are best delivered in phases.
Next step
If you are weighing up an assistant over your own documents, see our AI solutions or AI development service, or send us a short brief.