RAG

A language model only knows what was in its training data. A company's internal work instructions, price lists, contracts and product data are not part of it, and even public knowledge stops at the cutoff of that data. RAG closes the gap: before the model answers, a system pulls the relevant content and passes it to the model as context. The model then writes its answer from those passages and, if set up to do so, cites its source.

The term comes from a 2020 research paper by Patrick Lewis and colleagues at Facebook AI Research, University College London and New York University. In that paper, the retrieval component and the language model were trained together. Today RAG refers to any application that pairs a search with a language model, from an internal knowledge assistant to a customer service chat to research across a contract archive. The search can be full-text, vector-based or both combined.

RAG matters to executives because it decides whether an AI assistant works with the company's own knowledge or builds a plausible answer out of general knowledge. Quality depends more on the sources and the search than on the model. Rolling out RAG is therefore largely a question of knowledge management and access rights, and only after that a question of technology.

How RAG works at its core.

Four steps sit between the question and the answer. First, the documents in the knowledge base are prepared: a system splits them into short passages (chunking) and stores them so they can be searched. For vector search, a separate model converts each passage into embeddings, strings of numbers that represent what a text means. These usually live in a vector database.

When someone asks a question, the system searches for the best-matching passages in the second step. In the third step, those hits go into the model's input together with the question and the instructions. Finally, the model writes the answer and can state which document each claim came from.

The second step matters most. The model only works with what it is given. If the search misses the right passage, even the best model cannot use it. If the search finds an outdated version, the model will repeat the outdated rule fluently and convincingly. Passage size, the number of hits passed along and how they are ranked therefore often have more influence on the result than the choice of model.

Full-text search and vector search, alone or combined.

Full-text search finds passages that contain the words being searched for. It is strong on part numbers, product names, legal sections and technical terms. It fails when the question is worded differently from the document, for example when someone asks about “returns” and the work instruction only says “RMA.”

Vector search compares meaning instead of words, so it also finds passages that say the same thing in different terms. It is less reliable with exact identifiers, because two part numbers barely differ in meaning.

Many systems combine both methods and merge the result lists, an approach often called hybrid search. Some systems then re-rank the hits with a separate model. Which setup fits depends on the questions people ask. If they mostly search for numbers and names, good full-text search goes a long way. If they ask free-form questions about inconsistently written documents, vector search is needed. Combining the two covers both kinds of question and takes more effort to build and maintain.

The sources decide the quality.

A RAG system is only as reliable as the documents it draws on. Retrieved sources can be incomplete, outdated or misread. In many companies the same rules exist in several versions, with drafts next to current ones. And knowledge that lives only in people's heads has to be written down before an assistant can find it.

An inventory therefore comes before the technical build: which documents are current, which are outdated, and where are rules missing entirely? Current documents need a visible date and someone responsible for keeping them up to date. When a process changes, the change has to reach the knowledge base and the search index has to be rebuilt. Without that upkeep, the assistant will reliably quote the old rules.

Document format matters too. Scanned PDFs without text recognition, nested tables and slides with little text are hard to split into useful passages. Part of the work is bringing that content into a machine-readable form.

Access rights belong in the search.

An assistant may only search content that the person asking is allowed to see. That sounds obvious, but it is a common weak point: putting all documents into one shared index makes payroll lists, contracts or personnel files findable for everyone through the assistant. The OWASP project lists vector and embedding weaknesses, including missing access control and unintended data leakage, among its top ten risks for applications built on language models (2025 edition).

Rights therefore have to be enforced in the search, not just in the instructions to the model. An instruction like “Do not reveal confidential data” is not protection, because the model can ignore it or be talked out of it with clever questions. The only safe content is content the search never returns.

Documents can themselves contain instructions the model will follow, such as a planted email in a mailbox the system searches (indirect prompt injection). And personal data in the knowledge base falls under data protection law such as the GDPR: it needs a legal basis, and deletions have to reach the search index as well.

What a good implementation looks like.

A good RAG implementation is measured against real questions, not against a demo. To do that, you collect a list of typical everyday questions whose correct answer and source are known in advance. Every change can then be tested against that list: a new model, a different passage size, additional documents.

Good systems measure search and answer separately. First, was the right passage among the hits? Second, did the model render it correctly? Anyone who only scores the final answer cannot tell where an error came from.

Other signs of quality: every answer cites its source, and the source opens with one click. The system is allowed to say it found nothing instead of building a plausible answer out of a gap. Questions and answers are logged so that recurring gaps in the knowledge base become visible. And there is a set process for getting new or changed documents into the index.

These are the points to ask a vendor about.

What RAG costs and what drives it.

The cost of a RAG system falls into four buckets. The first is preparing the sources: reviewing documents, cleaning them up, converting them into readable formats.

The second is the search index. For vector search, every passage has to be converted into embeddings and stored, and that happens again whenever documents change. The third is model calls. Providers typically charge by the token, meaning the amount of text in input and output. Because RAG attaches the retrieved passages to every question, the input is much longer than the question itself. The number of hits passed along therefore drives ongoing cost directly.

The fourth bucket is upkeep: keeping documents current, rerunning tests, closing gaps. Which bucket dominates depends on the volume and condition of the documents, the number of questions, and the requirements for access rights and data protection. It makes sense to log every model call with its cost so it stays visible what the assistant actually costs to run.

Where the term stops working

Retrieved sources can be incomplete, outdated or misread, and the model can summarize a correct passage incorrectly. So RAG does not reliably prevent hallucinations; citing the source makes review possible, but it does not replace it. RAG is distinct from fine-tuning, which changes the model itself, from prompt engineering, which only shapes the instruction, and from MCP, which connects tools and data sources without searching anything on its own.