What Is RAG? Retrieval-Augmented Generation, Explained

10 min readai, llm

Ask a large language model about your company’s refund policy and it will answer fluently, confidently, and quite possibly wrong. Not because the model is bad, but because your refund policy was never in its training data. A model’s knowledge is frozen at its training cutoff and averaged over the public internet. Your internal wiki, your contracts, your ticket history, yesterday’s incident report — none of it is in there, and no amount of clever prompting can conjure facts the model never saw.

Retrieval-augmented generation — RAG — is the standard fix. Instead of hoping the model knows the answer, you search your own documents at query time, take the most relevant passages, and paste them into the prompt so the model can answer from evidence instead of memory. The term comes from a 2020 paper by Lewis et al., but the idea is older and simpler than the acronym suggests: it is a search engine bolted onto a text generator.

The pipeline is easy to demo and genuinely hard to run well. This post walks through how it works end to end — embeddings, vector search, chunking, hybrid retrieval, reranking — and then spends real time on the parts the demos skip: how to evaluate it, how it fails, and when you should not build it at all.

The problem RAG solves

Two properties of LLMs create the gap. First, knowledge is frozen: whatever happened after the training cutoff does not exist for the model. Second, knowledge is generic: the model learned from public text, so it knows what refund policies typically look like, not what yours says. When you ask about specifics it was never trained on, it does what autocomplete does — produces the most plausible-sounding answer, which is exactly the failure you cannot tolerate in production.

RAG attacks both problems at once. Because retrieval happens at query time, the model sees whatever is in your index right now — so freshness becomes an indexing problem, not a retraining problem. And because the retrieved text is yours, the answer is grounded in your data instead of the internet’s average. The model supplies reading comprehension and fluent synthesis; your corpus supplies the facts.

The pipeline, end to end

Every RAG system has two phases. Ingestion runs offline: you collect documents, split them into chunks, run each chunk through an embedding model, and store the resulting vectors in an index — a dedicated vector database, or an extension like pgvector inside the Postgres you already run. Query time runs on every request: embed the user’s question with the same model, find the stored vectors nearest to it, pull the corresponding chunks, and assemble a prompt that contains both the question and the evidence.

System: Answer only from the context below. If the
context does not contain the answer, say so.

Context:
[1] "Refunds are issued to the original payment
    method within 5-7 business days of approval."
[2] "Requests made more than 90 days after purchase
    require manager review."

User: How long does a refund take?

That is the whole trick. The model never queries a database and never “looks anything up” itself — retrieval happens outside the model, and generation is an ordinary completion over a prompt that happens to contain your documents. Everything that makes RAG good or bad lives in the quality of what you put in that context block.

Embeddings: the geometry of meaning

The step that makes semantic search possible is the embedding model. It maps a piece of text to a list of numbers — a vector, typically hundreds to a few thousand dimensions — such that texts with similar meaning land near each other in that space. “How do I get my money back” and “refund eligibility rules” share almost no words, but a good embedding model places them close together, because it was trained on enormous numbers of examples of which texts belong together. Distance in this space (usually cosine similarity) becomes a stand-in for relevance: nearest neighbors to the query vector are the passages most likely to be about the same thing. At scale you rarely compare against every vector; approximate nearest-neighbor indexes such as HNSW trade a sliver of recall for orders-of-magnitude faster lookups.

Chunking: the unglamorous decision that matters most

You cannot embed a 200-page PDF as one vector — the embedding would average away everything specific. So you split documents into chunks, and how you split turns out to matter more than which vector database you pick.

Two refinements are worth knowing. Small-to-big retrieval embeds small chunks for precision but returns the surrounding section to the model for context. And contextual retrieval prepends a short generated summary of the document to each chunk before embedding, so the chunk carries its own context — Anthropic published a detailed writeup of the technique. Either beats tuning chunk size by trial and error.

Hybrid search and reranking

Embeddings are good at paraphrase and bad at exact strings. Ask for error code E4021, a part number, or a function name, and semantic search may return passages that are thematically related while missing the one chunk that literally contains the identifier. Classic keyword scoring — BM25, the workhorse of pre-LLM search — has the opposite profile: exact matches score highly, paraphrases score zero.

Production systems therefore run hybrid search: both a vector query and a keyword query, with results merged (reciprocal rank fusion is the common recipe). On top of that, a reranker — a smaller model that reads the query and each candidate chunk together and scores actual relevance — reorders the top candidates before they enter the prompt. Retrieve fifty candidates cheaply, rerank to the best five, and precision improves substantially for modest latency. If a RAG system is underperforming, hybrid retrieval plus a reranker is usually the highest-leverage upgrade available.

Why long context windows did not kill RAG

As of August 2026, flagship models from all three major vendors offer context windows around a million tokens. A million tokens is several novels — so why not skip the pipeline and paste the whole corpus into every prompt? Four reasons, none of which are going away:

There is also a quality argument: models are noticeably better at answering from a few relevant passages than at finding and combining scattered facts inside an enormous prompt. Long context and RAG are complements — big windows let you retrieve more generously, not skip retrieval.

Evaluation and the failure modes demos never mention

A RAG system fails at two independent layers, and you must measure them separately. Retrieval: did the right chunk come back at all? Build a gold set of real questions paired with the passages that answer them, and track hit rate — how often the correct passage appears in the top k. Generation: given the right chunks, did the model produce a faithful answer? That is a groundedness check, often scored by a second model comparing the answer against the retrieved evidence. If you only measure end-to-end answer quality, you cannot tell whether to fix your index or your prompt — and no prompt engineering will rescue a query whose answer was never retrieved.

The failure modes that surface in production, ranked by pain:

When not to use RAG

RAG is infrastructure, and infrastructure has carrying costs. Skip it when the corpus is small: if everything fits comfortably in the context window — a manual, a codebase’s docs, a contract — just put it in the prompt, cache it, and be done. A pipeline you did not build cannot go stale. Skip it when the problem is behavior rather than knowledge: if you want a particular tone, format, or domain dialect, that is fine-tuning territory, and stuffing style guides into context is the wrong tool. The full decision framework is in fine-tuning vs RAG vs prompting. And when users need a document rather than an answer, plain search with good ranking beats a paraphrased summary — not every query deserves a generation step.

The short version