What Is RAG (Retrieval-Augmented Generation)?

Ask a plain language model about your company's runbook, last quarter's numbers, or a library released last month, and it will either say it doesn't know or — worse — confidently make something up. That's not a bug; a model only knows what was in its training data, frozen at some past date. RAG (Retrieval-Augmented Generation) is how you get a model to answer from your information instead. And the idea is refreshingly ordinary.

The problem: a model with no access to your world

A large language model is brilliant at language and reasoning, but it has two hard limits: it doesn't know anything private (your docs, your codebase, your tickets), and its knowledge has a cutoff date. Retraining it on your data is slow and expensive, and it goes stale the moment anything changes. You need a way to give the model the right facts at question time.

What RAG actually is

RAG splits answering into two steps: retrieve, then generate.

1. Retrieve. When a question comes in, first search your own documents for the handful of passages most relevant to it.
2. Generate. Paste those passages into the prompt as context, and ask the model to answer using only that context.

The model isn't remembering your data — it's reading it, fresh, every time you ask. Update the document and the next answer reflects it. No retraining.

question ──▶ search your docs ──▶ top matching passages ──▶ LLM answers from them ──▶ grounded answer

How the retrieval works: embeddings

The search step isn't keyword matching — it's by meaning. Each passage is run through an embedding model that turns text into a vector (a list of numbers) such that passages with similar meaning land near each other. You store those vectors once. At question time you embed the question the same way and pull the passages whose vectors are closest. That's why RAG can find "the deploy broke the site" when your doc said "502 after release" — different words, same meaning.

RAG doesn't make the model smarter — it makes it informed. The intelligence was always there; retrieval just puts the right page in front of it before it answers.

Why it matters

Grounding answers in retrieved text does three things at once. It kills hallucination for anything covered by your docs, because the model is quoting, not guessing. It stays current — change the source, change the answer. And it lets you cite sources, so a reader can verify the claim. That combination is why RAG powers most "chat with your docs," internal-knowledge, and support-assistant tools you've seen.

The mental model to keep

Don't think of RAG as teaching the model your data. Think of it as an open-book exam: the model is the smart student, your documents are the book, and retrieval is the act of flipping to the right page before answering. The skill isn't the model — it's chunking your docs well, embedding them, and retrieving the right passage. Get that right and a modest model gives sharp, trustworthy answers about information it was never trained on.

Frequently Asked Questions

Does RAG retrain the model?

No. RAG retrieves relevant text at question time and puts it in the prompt; the model reads it fresh each run, so updating a document updates the answer with no retraining.

What's the difference between RAG and fine-tuning?

Fine-tuning changes the model's weights (slow, and static once done); RAG leaves the model untouched and feeds it your current documents at query time.

Does RAG stop hallucinations?

It sharply reduces them for anything your documents cover, because the model answers from retrieved text it can cite — but it can still err on topics outside that context.