Language models know a lot, but not your company’s documents, today’s prices or last week’s policy change. Retrieval-augmented generation (RAG) fixes this by fetching relevant information at question time and giving it to the model along with the question. The term comes from a 2020 paper by Patrick Lewis and colleagues; today RAG is the most common way to build assistants over private knowledge.

The two phases

A RAG system has an indexing phase (done in advance) and a query phase (done for each question).

Indexing

  1. Ingest documents: PDFs, web pages, wiki pages, tickets, databases.
  2. Clean and extract text, keeping useful structure such as headings and tables.
  3. Chunk text into passages — often a few hundred tokens each, with some overlap so ideas aren’t cut in half.
  4. Embed each chunk with an embedding model, producing a vector that represents its meaning.
  5. Store vectors and text, plus metadata (source, date, permissions), in a vector index or database.

Querying

  1. Embed the question with the same embedding model.
  2. Retrieve the most similar chunks — typically the top 5 to 20.
  3. Rerank (optional): a reranker model scores each candidate’s relevance more precisely and keeps the best.
  4. Build the prompt: instructions, the retrieved passages with their sources, and the question.
  5. Generate an answer that uses — and ideally cites — the passages.

Why RAG works well

  • Fresh knowledge without retraining: update documents and the answers update.
  • Citations: users can check the sources.
  • Access control: retrieve only documents the user is allowed to see.
  • Cost: far cheaper than fine-tuning for knowledge.

Improving retrieval quality

Most RAG failures are retrieval failures: the right passage was never found. Techniques that help:

  • Hybrid search: combine vector similarity with keyword search, which catches exact terms like product codes and names.
  • Better chunking: split on document structure (sections, paragraphs) rather than fixed lengths.
  • Metadata filters: restrict by date, product or department before similarity search.
  • Query rewriting: let a model rephrase vague or conversational questions into good search queries.
  • Reranking: apply a cross-encoder or reranking model to the top results.

Prompting the generator

Tell the model to answer only from the provided passages, to cite them, and to say when the information is not present. Research such as Lost in the Middle (2023) showed that models can use information at the start and end of long contexts better than information buried in the middle, so put the most relevant passages first and avoid stuffing in too many.

Evaluate both halves

Measure retrieval and generation separately:

  • Retrieval: for a set of test questions, is the correct passage in the top results?
  • Generation: is the answer correct, complete, faithful to the sources and properly cited?

Common mistakes to avoid

  • Blaming the model for retrieval failures. Check whether the right passage was retrieved before rewriting prompts or switching models.
  • One chunk size for everything. Contracts, FAQs and code need different chunking strategies.
  • Ignoring permissions. Without access filters, a RAG assistant can reveal documents to people who should not see them.
  • Stale indexes. Schedule re-indexing so updated and deleted documents are reflected in answers.

Signs your RAG system needs work

Answers cite irrelevant passages, miss information you know is in the documents, mix up versions, or say “not found” too often. Each points to a different fix: chunking, retrieval, metadata filtering or prompt instructions.

Key takeaways

  • RAG retrieves relevant passages at question time and gives them to the model as context.
  • Indexing involves cleaning, chunking, embedding and storing documents with metadata.
  • Hybrid search, smart chunking, filters and reranking improve retrieval the most.
  • Instruct the model to answer only from sources, cite them and admit when information is missing.
  • Evaluate retrieval and generation separately with a test set.

Frequently asked questions

Does RAG eliminate hallucinations?

It reduces them substantially but not completely. The model can still misread passages or fill gaps, so citations and evaluation remain important.

Do I need a vector database?

For small collections, a simple in-memory index or a vector extension in an existing database is enough. Dedicated vector databases help at large scale.

RAG or long context windows?

Long context lets you include more text directly, but retrieval is still cheaper, faster and more precise for large collections. Many systems combine both.