What is RAG? AI that answers from your own documents

RAG works in four steps. You cut your documents into small pieces, capture the meaning of each piece, find the pieces that match each question and let the language model answer with only those pieces, plus the source. The first two steps happen in advance, the last two with every question.

1. Cutting up

You don't send an eighty-page handbook in full with every question. That's slow and expensive, and the model loses the thread. So you cut the documents into pieces of a few paragraphs, with the title and the chapter attached. How you cut matters more than you'd think. Cut a table in half, and the price ends up in one piece and the product in the other.

2. Capturing the meaning

For each piece you create an embedding: a series of numbers that captures its meaning. These go into a vector database. Texts with the same meaning sit close together in it, even if they use different words. 'Can I send my order back?' then also finds the chapter 'Return conditions', even though the word return isn't in the question.

3. Searching

When someone asks a question, the system finds the pieces that resemble it most. Often you combine that with ordinary keyword search, for order numbers, product codes and names. Then a filter goes over the results. Is this user allowed to see this document? And is this the latest version?

4. Answering with a source

The language model gets the question, the pieces it found and a clear instruction: answer only on the basis of these pieces, name the source and say so honestly if the answer isn't in them. That last sentence is the most important part of the whole setup. An assistant that says 'I don't know, ask the team leader' is worth its weight in gold. An assistant that guesses costs you customers.

The model itself doesn't change. You don't train anything. If you change the returns policy, you replace the document, and from that moment on everyone gets the new answer. That's the big difference from fine-tuning, where you teach the model itself something new.

You need RAG when the knowledge is large, changes often, has to be restricted per user or is used by lots of people at once. If it's a handful of documents for one team, you don't need your own RAG system. Put them in a Claude Project or a custom GPT, and you're done today.

That second point often gets forgotten. The request is 'build a chatbot on our own knowledge', while it's about a price list, a style guide and ten sample quotes. You don't need to have anything built for that. In the APO method that's build layer 2: assistants and projects that your team creates and maintains itself. How to go about it is in build your own AI assistant.

An assistant is enough when

• it's a handful of documents you can update by hand
• one team uses the assistant, and everyone in that team is allowed to see everything
• an employee asks the question and checks the answer themselves
• you don't need to look back at who asked what

You need RAG when

• it's hundreds or thousands of documents, spread across SharePoint, Google Drive, a wiki or the document management system
• the documents change every week and you don't want someone to keep uploading them again
• not everyone is allowed to see everything, as with HR files, contracts and salary data
• customers ask questions themselves, or the answer has to land in another system, such as your helpdesk
• you want to measure how good the answers are and look back at what was asked

And sometimes RAG is the wrong answer

RAG is good at questions whose answer is in a text. 'What is our warranty period on pumps?' But 'where is order 4471?' isn't in a document. It's in your ERP. For that you need an integration, not a document search. An AI agent can do both: it looks up the policy and the order. Calculation questions, such as 'how much revenue did we make in March?', don't belong in RAG either. They belong in a report or an integration with your data.

Because then you can check whether it's right. An answer without a source has to be taken on trust. An answer with a source you check in ten seconds: click, read the passage, correct. That's the difference between a toy and something you can put in front of customers.

Three reasons:

• Control. The employee sees at a glance where the answer comes from. If the source is an old document, they see that straight away.
• Trust. People only really start using an AI assistant once they dare to rely on it. A source helps more with that than any explanation. After a few weeks they know: if it says so, it's written down somewhere.
• Fewer fabrications. A model that may only answer from the pieces it found and has to name the source makes things up less often. The risk isn't completely gone. That's why a human stays in the loop.

Citing sources also helps you improve. If the assistant gives a wrong answer with the right source, the problem is in the instruction or the model. If it picks the wrong source, the problem is in the search. If the source itself is wrong, the document needs updating. Without a source, you don't even know where to start.

So for everything that goes to a customer, we build it like this: the AI prepares the answer, an employee checks it and sends it. That principle also runs at one of our enterprise clients. There, 80 percent of the 6,000 customer questions a month now have a prepared answer before the ticket opens, compared with 30 percent at first.

RAG rarely goes wrong on the technology and almost always on the documents. Outdated versions, poor scans, permissions that don't match and knowledge that only lives in people's heads: those are the classics. A RAG system is exactly as good as the knowledge you put into it.

Outdated documents

The 2024 price list, the 2025 price list and 'price list FINAL v3 (2)'. A human knows which one is right. The AI doesn't. It finds three answers and picks one, or worse: mixes them. So clean up first. One version per document, one owner per document and old versions out of the source. Boring work, and it delivers more than the smartest model.

Poor scans and tricky layouts

A crooked scanned PDF from 2011, a table spread over three pages, a manual that's mostly pictures. These first have to be read properly (OCR) and cut up neatly. If that fails, the AI doesn't find the information, or only half of it. So always test with questions whose answer is in a table or a scan.

Permissions per user

If you throw all the documents onto one pile, anyone can ask anything. Including 'what does my colleague earn?', if the salary overview was in there. The AI has to follow the same permissions as the source: anyone who can't open a document in SharePoint doesn't get to see it through the assistant either. You arrange that during the build, not afterwards.

Knowledge that isn't written down

'Just ask Piet.' If the answer only lives in Piet's head, RAG can't find it. That's why a RAG project often starts with writing down what isn't written down anywhere yet. That's a win straight away, also when Piet wants to go on holiday.

Nobody owns it

After go-live, the policy changes, a product is added and a folder moves. If nobody is responsible for the documents, the assistant gets a little worse every month. Agree who maintains which source, and look every month at the questions it didn't know the answer to.

Wherever people spend all day looking things up in documents. Customer service checking the returns policy, the inside sales team searching product sheets, the new colleague who can't find the staff handbook.

• Customer service. For every customer question, the AI prepares an answer from the returns policy, the terms and the product information, with the source included. The employee checks and sends it. More on AI by department.
• Internal helpdesk. 'How many days of leave do I have left?' 'How do I claim travel expenses?' Questions HR and IT get again every week, answered from the staff handbook and the IT wiki.
• Inside sales and sales. Specifications, applications, earlier quotes. The AI looks it up in product sheets, so the inside sales team doesn't have to keep calling the technical colleague.
• Technical service. A technician dealing with a breakdown asks what error code E47 means on this type of machine. The answer comes from the manual, with the page number included.
• Advisory and legal. Searching contracts, advice and case files: which clients have a different notice period? Here, permissions per user matter even more.

Want an assistant like this on your website, so customers can ask questions themselves? Then first read when an AI chatbot works and when it doesn't.

RAG isn't a product but a technique: a way to let AI answer from your own knowledge. A chatbot, an assistant and an agent can all use that technique. The difference is in what they do next.

• A chatbot with RAG answers questions from your documents. It talks, but does nothing in your systems.
• An assistant with RAG helps an employee, for example with a draft answer and the source included.
• An agent with RAG looks up the policy, checks in the ERP where the order is, prepares a return and puts it to a human. Looking things up is just one of its tools.

In practice it often starts with RAG and grows into an agent, as soon as it turns out an answer alone isn't enough: the customer also wants something to happen. How that works is in what is an AI agent?

In the APO method, a custom RAG application is build layer 4: custom work that NinA builds, within the AI Partnership. We don't start with the technology, but with the questions. Which questions come in, how often, and how long does someone currently spend looking for the answer?

This is how we go about it:

1. 01Collect questions. A set of real questions from the past few months, with the right answer included. That becomes the test set.
2. 02Clean up the sources. Which documents lead, which versions can go, who owns them? Often the most work, and the most valuable.
3. 03Build with permissions. Connected to the source, such as SharePoint or Google Drive, with the same permissions per user. The data stays in the EU: on your own infrastructure or with a European provider.
4. 04Test against the test set. Only when the answers are good enough do the first users start working with it. We test every change again, including a new model.
5. 05Live, with a human in the loop. The AI prepares, an employee checks. We keep an eye on the questions it had no answer to.

We only build within the AI Partnership, from € 4,500 per month, because a RAG application needs maintenance: documents change, and so do models. The partnership starts with the € 6,000 AI Quick Scan, in which we calculate whether it pays off. Whatever goes live then runs under a maintenance contract from € 245 per month, including AI usage. All amounts excluding VAT. Code, data and documentation are yours.

And if a Claude Project turns out to be enough for you, we'll tell you that too.