Explainer

What is RAG? Retrieval-augmented generation explained

RAG is the technique behind “chat with your documents”, AI search with citations and most company chatbots. Here’s how it works, without the jargon.

Key points

  • RAG = retrieve relevant information first, then let the AI generate an answer based on it.
  • It lets a chatbot answer from documents it was never trained on — your files, a company wiki, or today’s web.
  • It can show exactly which passage supports each answer, which makes checking easier.
  • It reduces but doesn’t eliminate errors: bad retrieval in means bad answers out.

The problem RAG solves

A large language model knows only what was in its training data, frozen at a cutoff date. It knows nothing about your company’s refund policy, last week’s meeting notes or a product launched yesterday. And when it doesn’t know, it may produce a plausible guess — a hallucination. Retraining a model every time information changes isn’t practical.

Retrieval-augmented generation offers a simpler route: at the moment you ask a question, fetch the relevant information and hand it to the model along with your question. The model then answers from that material, like an open-book exam. The term was popularised by a 2020 paper from Patrick Lewis and colleagues, which combined a retrieval system with a text generator and showed better results on knowledge-intensive tasks than the generator alone.

How RAG works, step by step

Preparation (done once, updated as documents change)

  1. Collect documents — PDFs, web pages, help articles, policies, notes.
  2. Split them into chunks of a few paragraphs each, so each piece covers one idea.
  3. Create embeddings. Each chunk is converted by an embedding model into a list of numbers that represents its meaning. Texts with similar meaning get similar numbers, even if they use different words.
  4. Store them in a search index, often called a vector database.

At question time

  1. Embed the question the same way.
  2. Retrieve the chunks whose meaning is closest to the question (often combined with ordinary keyword search).
  3. Augment the prompt: the system builds a prompt like “Answer the question using only these passages: [passages]. Question: [question]. Cite the passage you used.”
  4. Generate: the language model writes an answer grounded in those passages, ideally with citations.

Where you already use RAG

  • AI search with citations. When a chatbot searches the web and shows numbered sources, it’s using a web-scale version of the same idea: retrieve pages, then generate a cited answer. See our comparison of the best AI for research.
  • “Chat with your files”. Many features that let you upload documents and ask questions retrieve the relevant parts behind the scenes.
  • Customer-support bots that answer from a company’s help centre.
  • Internal knowledge assistants that search company documents, policies and tickets.

Benefits of RAG

  • Up to date. Update the documents, and answers update immediately — no retraining.
  • Traceable. Answers can point to the exact passage, so people can verify them.
  • Private knowledge. The model can use information that was never public.
  • Cheaper per question than sending entire document collections every time. Embedding is inexpensive; OpenAI’s pricing page, for example, lists its small embedding model at $0.02 per million input tokens (checked November 19, 2026).

Where RAG goes wrong

Retrieval misses the right passage

If the search step returns the wrong chunks — because the question uses different wording, the answer is spread across several sections, or the document is a scanned image — the model may answer from irrelevant text or fall back on general knowledge. Most RAG failures are retrieval failures.

Chunks lose context

A chunk that says “this does not apply to annual plans” is useless if the retriever doesn’t also return the paragraph explaining what “this” is.

The model misuses good passages

Even with the right text, a model can misread, over-generalise or merge details from different passages. Research such as the 2023 “Lost in the Middle” study found that models used information placed in the middle of long inputs less reliably than information at the beginning or end.

Poisoned or manipulated content

If retrieved content contains hidden instructions — for example, a web page saying “ignore previous instructions and…” — a model may follow them. This is called indirect prompt injection, and OWASP lists prompt injection as the top risk in its Top 10 for large language model applications. Anyone building RAG over untrusted content needs safeguards.

RAG vs long context vs fine-tuning

ApproachWhat it doesBest for
Paste into contextPut whole documents in the promptA few documents, one-off questions
RAGRetrieve relevant passages per questionLarge or changing collections; citations; cost control
Fine-tuningFurther train the model on examplesTeaching a style, format or specialised behaviour — not for frequently changing facts

With many current models offering context windows of up to a million tokens, pasting documents directly has become practical for more cases. RAG remains valuable when the collection is large, changes often, or you need each answer tied to a specific source.

Using RAG ideas as an everyday user

You can apply the core principle without building anything. When accuracy matters, give the chatbot the source material and ask it to stay within it:

“Using only the policy document below, answer my question. Quote the sentence that supports your answer. If the document doesn’t cover it, say so. DOCUMENT: [paste] QUESTION: [your question]”

That’s RAG done by hand — you are the retriever. Anthropic’s guidance on reducing hallucinations recommends exactly this kind of grounding and quoting.

A quick glossary

  • Chunk: a small piece of a document stored for retrieval.
  • Embedding: a list of numbers representing the meaning of a piece of text.
  • Vector database: a store that finds chunks with similar embeddings quickly.
  • Grounding: making the model base its answer on supplied sources rather than memory.
Editor’s note — add your own experience: If you’ve built or used a “chat with your documents” tool, describe one question it answered well and one it got wrong, and why you think retrieval failed.
Remove this box by adding your content here, or hide all notes with "showEditorNotes": false in site.config.json.

For small businesses

A help-desk assistant that answers from your own FAQ is one of the most practical AI projects for a small business — but test it with real customer questions, keep a human in the loop for anything involving money or complaints, and make it easy to reach a person. Our guide on using AI for a small business covers where to start.

Frequently asked questions

Is RAG the same as training a model on my data?

No. RAG doesn’t change the model. It finds relevant passages at question time and gives them to the model as context. Training (fine-tuning) changes the model’s parameters, which is slower, costlier and harder to update.

Do I need RAG if models have one-million-token context windows?

Sometimes not — for a few documents you can paste them in directly. RAG still helps with large or frequently changing collections, lower cost per question, and pointing answers at the exact source passage.

Does RAG eliminate hallucinations?

No. It reduces them for questions your documents can answer, but the model can still misread a passage, combine passages wrongly, or answer from general knowledge if retrieval misses the right text.

What is a vector database?

A database that stores embeddings — numerical representations of text meaning — so it can quickly find passages similar in meaning to a question.

Is “chat with PDF” a form of RAG?

Usually, yes. Many such features split your file into chunks, retrieve the relevant chunks for each question and pass them to the model. For short files, some apps simply include the whole file in the context.

Can I build a simple RAG system without coding?

Many chatbot apps let you upload files to a project or workspace and ask questions about them, which applies the same idea without any code.

Sources

  1. Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (opens in a new tab)
  2. Liu et al. (2023), Lost in the Middle: How Language Models Use Long Contexts (opens in a new tab)
  3. Anthropic — Reduce hallucinations (opens in a new tab)
  4. OWASP — LLM01:2025 Prompt Injection (opens in a new tab)
  5. OpenAI API pricing (embedding model prices) (opens in a new tab)

Facts, prices and features were checked against these sources on November 19, 2026. AI products change quickly; if you spot something out of date, please tell us.