Retrieval-augmented generation (RAG) is a pattern in which a system first searches your own documents or data for passages relevant to a question, then gives those passages to a language model and asks it to answer from them. The model writes the answer; the retrieved sources supply the facts. A good RAG system shows which sources each answer came from, so a reader can check it.
The term comes from a 2020 research paper by Lewis and colleagues, which combined a language model (“parametric memory”) with a searchable index of Wikipedia (“non-parametric memory”) and showed more specific and factual output than the model alone (arXiv 2005.11401). The idea has since become the default way to put a large language model to work on company knowledge.
How RAG Works
A production RAG pipeline has two halves.
Indexing (offline):
- Collect the source material: policies, manuals, tickets, contracts, product data.
- Clean it and split it into chunks of a few hundred tokens, keeping headings, dates and document identifiers as metadata.
- Turn each chunk into embeddings and store them in a vector database, often alongside a keyword index.
Answering (online):
- Rewrite the user’s question into one or more search queries.
- Retrieve candidates with vector search and keyword search (BM25) together. This is called hybrid search.
- Rerank the candidates with a dedicated reranking model and keep the best few.
- Apply access rules, so a user only retrieves documents they are allowed to read.
- Put the passages into the prompt with instructions to answer only from them and cite them.
- Return the answer with links to the source passages, or say that the sources do not contain an answer.
Each step matters. Anthropic’s published tests on “contextual retrieval” found that adding context to each chunk before indexing, combining embedding search with keyword search, and reranking the results cut failed retrievals by 67% compared with embeddings alone on their benchmark (Anthropic). Most quality problems in RAG come from retrieval, not from the model.
When to Use RAG and When Not To
RAG fits when:
- The knowledge changes often and the model must reflect today’s version.
- Answers must be traceable to a source document.
- Different users may see different documents.
- The corpus is too large to fit in a model’s context window.
RAG is the wrong tool when:
- The knowledge base is small. The same Anthropic article notes that a knowledge base under about 200,000 tokens can simply be placed in the prompt, with prompt caching keeping cost down. This is sometimes called cache-augmented generation (CAG).
- The question is about numbers in a database. “What were sales in March?” needs a query against structured data (text-to-SQL), not a search over documents.
- The goal is a new style, format or behaviour. That is a job for prompting or fine-tuning, not retrieval.
Common Failure Modes
- Bad chunking. A chunk cut in the middle of a table or without its heading loses meaning, and retrieval misses it.
- Stale or duplicate sources. Three versions of the same policy produce contradictory answers.
- Retrieval that looks right but is not. Semantically similar text is not always the relevant text, for example the 2024 price list instead of the 2026 one.
- Ungrounded answers. The model fills gaps from its training data instead of saying “not found”. This is a common source of AI hallucination.
- Permission leaks. If access rules are not applied at retrieval time, the assistant can quote documents the user should not see.
- Indirect prompt injection. A retrieved document can contain instructions aimed at the model, so retrieved text must be treated as data, never as instructions.
Related Terms
RAG is one technique within context engineering, the wider practice of deciding what goes into the model’s context. In agent systems, retrieval often becomes a tool the agent calls when it needs it (“agentic RAG”), and a knowledge graph can add relationships that plain text search misses. Quality is measured with evaluations: retrieval recall, answer faithfulness to the sources, and answer relevance.
RAG in Practice
Retrieval quality is measured before answer quality. A small set of real questions, each paired with the passages that answer it, is built first, and chunking, the embedding model and hybrid search are tuned until the right passages come back; only then does work on the prompt begin. Typical builds use Python with LangChain or LangGraph, PostgreSQL with a vector extension or a dedicated vector store, and a model from OpenAI, Anthropic, Google, Azure or an open-weight model, depending on data residency needs.
Rules that hold up in production:
- Every answer links to the passages it used, so it can be checked.
- Retrieval applies the user’s own document permissions inside the query.
- Prices, totals and other figures come from a database query, not from the model’s reading of a document.
- When no passage supports an answer, the assistant says so instead of answering from general knowledge.
- Retrieval hit rate and answer quality are tracked in production with tools such as LangSmith.
See how this applies to RAG and CAG knowledge assistants and to a support bot over company knowledge.