RAG Explained: How AI Works With Your Own Documents

Key Takeaways

• With RAG, the AI first looks up the relevant passages in your documents and then writes its answer from exactly those passages.
• The model isn't trained on your data for this. When your documents change, the answers change with them.
• Quality depends on two things: tidy documents and a search that finds the right passages.
• A good system shows the source for every answer, says so when it finds nothing, and respects your access rights.
• For company knowledge, RAG is almost always a better choice than fine-tuning, which mainly adjusts a model's style and format.

RAG (retrieval-augmented generation) means that before an AI answers, it searches your own documents for the passages that fit the question and then writes its answer from those passages, with a reference to the source. The language model doesn't have to be retrained for this. How good the answers are therefore depends mostly on your documents and on the search. RAG is the usual technique behind AI search across company knowledge.

What does RAG stand for?

The three parts of the name describe the process. Retrieval: the system looks up the passages that match the question. Augmented: those passages are added to the question. Generation: the language model writes an answer from the question and the passages. The term comes from a research paper published in 2020. Today the idea sits behind most tools that answer questions about internal documents, from search across a shared drive to an assistant in customer service.

How does RAG work, step by step?

Part of the work happens once, in advance, and the rest happens with every question. Simplified, it looks like this:

1. Prepare the documents.The system collects the documents from the folders you've released and extracts their text. Scanned PDFs need text recognition (OCR) first. Outdated versions and duplicates should be cleared out beforehand.
2. Split them into passages.Long documents are cut into shorter passages, usually a few paragraphs each. That way the search can later find the one passage that answers the question, not just the whole document.
3. Capture the meaning.A so-called embedding turns each passage into a series of numbers that represents its meaning. These numbers are stored in a vector database. Many systems also build a classic keyword index alongside it.
4. Find the matching passages.When a question comes in, it's converted the same way. The system looks for passages with a similar meaning, even if they use different words, and uses the keyword index to find exact terms such as a contract number. The best hits are ranked by relevance.
5. Write the answer.The question, the passages found and a clear instruction go to the language model, for example: “Answer only from these sources, and say so if they don't answer the question.” The answer points to the documents it used.

An example: someone in a fiduciary firm wants to know which documents a new client with a sole proprietorship needs to hand in for their first year-end accounts. The system finds the internal checklist and a file note from the previous year, summarises both and links to the two documents. Anyone who wants to check is one click from the source. If the checklist was updated last month, the answer already reflects the change, without anyone retraining anything. Our fiduciary and accounting page shows how firms use this kind of search.

How is RAG different from an ordinary chatbot?

An ordinary chatbot answers from memory, meaning from what the model learned in training. It doesn't know your internal documents, its knowledge stops at a cut-off date, and when something is missing it often fills the gap with something plausible. With RAG, the model answers from your current documents and you can see where the answer came from. Think of it as an exam: without RAG it's closed book, with RAG it's open book. New knowledge arrives as soon as you file a document, without anyone having to change the model.

When should you use RAG, and when fine-tuning?

Use RAG for knowledge and fine-tuning, if at all, for style and format. Fine-tuning means training a model further on your own examples. That sharpens tone, layout or a narrowly defined task, but it's a poor way to teach facts that change. A fine-tuned model also can't show which document a statement came from. For the knowledge in your contracts, handbooks, minutes or quotes, RAG is almost always the better choice. And if your material is small, a handbook or a price list for example, it often fits straight into the request to the model, and you don't need RAG at all.

Is RAG safe for confidential documents?

It can be, once three things are settled: where the data is processed, who may see what, and what gets stored. The search index holds your passages along with their embeddings, so it needs the same protection as the documents themselves. With every question, the passages found are sent to the language model, which means the model provider's terms matter too, especially on where data is stored, how long it's kept and whether it's used for training. And the logs of questions and answers often contain information just as sensitive as the documents.

Personal data in your documents falls under the Swiss data protection act, just as it does anywhere else. Law firms, fiduciaries and other professions bound by confidentiality should also have the set-up checked against their professional rules. What to look for when choosing tools and a data location is covered in where your data goes. This section is not legal advice.

Where does RAG go wrong?

• The documents are out of date: three versions of the same template sit side by side, and the system quotes the old one.
• The text is missing: scanned documents without text recognition are invisible to the search.
• A passage loses its context: “the notice period is three months” doesn't help much if nothing says which contract the sentence belongs to.
• The search finds the wrong passages, and the model turns them into an answer that sounds certain and is wrong.
• Access rights get lost: suddenly people can find documents through the search that they couldn't open in the folder.
• The model mixes in general knowledge instead of sticking to the sources. There's more on this in why AI makes things up.

How can you tell a good RAG system?

• Every answer shows its sources, ideally with a link to the exact place in the document.
• When the system finds nothing relevant, it says so instead of guessing.
• Your existing access rights apply in the search as well.
• You know where the data is processed and have it in writing that it won't be used to train anyone else's models.
• New and changed documents are picked up automatically, and deleted ones disappear from the search.
• There's a list of typical questions from your team that you use to check the answers regularly.

Start with one manageable collection, such as your internal templates and guidelines, and gather 20 to 30 questions your team asks again and again. Check every answer against its source. You'll soon see whether problems come from the documents or from the search. Note which answers were wrong and why. Usually a document is missing or out of date, and sometimes the problem is how the documents were split. What to consider on data protection is covered in can confidential documents go into an AI?, and what this looks like as a finished solution is on the company knowledge page.