Retrieval-augmented generation

A base model answers only from what it memorized. RAG fetches the right passages from an external corpus at query time and puts them in the prompt - so the agent answers from evidence, with citations, without retraining. This lesson is the full pipeline, end to end.

Two memories, one prompt

Every LLM ships with parametric memory: facts compressed into its weights during training. That memory is frozen at the knowledge cutoff, cannot cite a source, and confabulates when asked about something it only half-remembers. RAG adds a second, swappable store - non-parametric memory: an external corpus you index and query at inference time, injecting the retrieved passages into the prompt as context.^[1] The framing - parametric plus non-parametric memory - comes from the paper that coined RAG (Lewis et al., NeurIPS 2020), which paired a dense retriever over Wikipedia with a generator and proposed two variants: RAG-Sequence (one retrieved set for the whole answer) and RAG-Token (a different doc can inform each token).^[1]

Production RAG usually skips the joint fine-tuning: an off-the-shelf embedding model, a vector database, and a frozen instruction-tuned LLM, wired together. The architecture - retrieve, then generate - is what matters. Teams reach for it because it buys four things a bare model cannot:

The pipeline: an offline index and an online query

RAG is two paths that meet at a shared index. Offline, you build the index once (and re-build as the corpus changes). Online, every user query travels the second path and reads from that index. Seven stages total - load, chunk, embed, index offline, then embed, retrieve, rerank, generate online - and each one is an independent failure point.

Chunking - the highest-leverage stage

You chunk because embeddings represent a bounded span and lose fidelity over long inputs, and because you want to retrieve precise evidence, not whole documents. The tension: a chunk that is too small fragments a fact across chunks; too large dilutes the embedding with irrelevant text and wastes context budget. Strategies, roughly by sophistication:^[3]

Contextual Retrieval (Anthropic, 2024) is the standout upgrade: have an LLM write a 50 to 100 token blurb that situates each chunk in its parent document, and prepend it before embedding and BM25 indexing. On Anthropic's eval this cut the top-20 retrieval-failure rate by 35% (5.7% to 3.7%); adding contextual BM25 took it to 49% (to 2.9%); adding a reranker to 67% (to 1.9%). One-time cost is about $1.02 per million document tokens with prompt caching.^[2] Late chunking (Jina) solves the same problem cheaper, using only the embedding model.^[3]

Dense + sparse = hybrid search

Two retrievers with complementary failure modes. BM25 (sparse, lexical) scores by term frequency down-weighted by rarity - it nails exact matches: product codes, error strings, function names, rare proper nouns. It has zero notion of meaning ("car" is not "automobile"). Dense embeddings do the opposite - great at paraphrase, synonyms, cross-lingual - but can whiff on that one literal SKU. Most real queries are mixed-intent, so hybrid is the robust production default: run both, then fuse.^[4]

Fusion uses Reciprocal Rank Fusion (RRF), which looks only at each document's rank position in each list - never the raw scores:

score(d) = sum  1 / (k + rank(d))    # summed over every retriever's ranked list
                                     # rank(d) is 1-indexed;  k = 60 (default)

Rerank with a cross-encoder

First-stage retrieval uses a bi-encoder: query and document are embedded separately, so the model never sees them interact - fast and precomputable, but coarse. A cross-encoder reranker feeds the concatenated (query, document) pair through a transformer, letting attention model every query-token to doc-token interaction. That yields far sharper relevance - the single biggest precision win in most pipelines.^[4]

The cost is that a cross-encoder runs a full forward pass per candidate - nothing precomputes. Hence the standard two-stage pattern: retrieve wide and cheap (top-100 to top-1000 via BM25 + dense + RRF), then rerank down to the final top-5 to top-10 for the prompt.^[4] Anthropic found retrieving top-20 candidates beat top-10 and top-5 - recall matters at the retrieval stage; let the reranker do the pruning.^[2] Common rerankers: Cohere Rerank 3.5 and Voyage rerank-2.5 (a 32K context, about 8x Cohere's, with instruction-following) plus open-weight BGE rerankers.^[4]

Worked example: Q&A over a document set

Corpus: 40,000 internal engineering docs (~30M tokens) - far too big for any context window, so retrieval is mandatory. The index was built offline with recursive ~512-token chunks + contextual blurbs, stored in both an HNSW vector index and a BM25 index. Now a query arrives:

1. Dense retrieval. Embed the query; ANN returns the 100 semantically-nearest chunks (it understands "checkout service", "deploy").
2. Sparse retrieval. BM25 returns 100 chunks and reliably surfaces the literal ERR_2043 string, which a pure embedding search might smear into "generic error."
3. Fuse (RRF, k=60). One merged list of ~150 unique candidates; a chunk both retrievers liked floats to the top.
4. Rerank. A cross-encoder re-scores the top ~100 (query, chunk) pairs; the runbook chunk that literally explains ERR_2043 and references a June config change now scores highest.
5. Assemble. Top-5 reranked chunks + "answer only from the context and cite chunk IDs" + the question.
6. Generate. The LLM produces a grounded, cited answer.

Had the contextual-prepend step been skipped, the runbook chunk - whose raw text said "this timeout" without naming the service - might have ranked #60 and been dropped before reranking: a textbook retrieved but below the cutoff miss. Context recall = 1.0, faithfulness = 1.0 on this trace only because every stage did its job.

RAG vs long context

Frontier models now take up to ~1M tokens. So why retrieve at all? Because it depends on corpus size and workload - it is a trade-off, not a war.^[5]

Long context is not free of retrieval problems either: attention cost scales roughly quadratically, and models exhibit "lost in the middle" - recall degrades for facts buried mid-context. Stuffing more in also injects more noise. The two compose: use RAG to pre-filter a huge corpus down to a long-context-sized working set.^[5] This is a direct extension of context engineering - the art of what to put in the window.

Beyond the fixed pipeline

Agentic RAG

Classic RAG always retrieves once, with the raw query. Agentic RAG makes retrieval a decision: should I retrieve at all? Rewrite or decompose the query first? Which source? Are these chunks good enough, or do I search again (multi-hop)? Named patterns: Self-RAG (the model emits reflection tokens deciding when to retrieve and critiquing support), Corrective RAG (an evaluator grades docs and falls back to web search when they are weak), and HyDE (embed a hypothetical answer). Retrieval becomes one tool an agent calls deliberately - the bridge from RAG into the agent loop and agent memory.

GraphRAG

Baseline vector RAG retrieves the chunks most similar to the query - but some answers live in the aggregate, not any single chunk. Ask "what are the top 5 themes across all 40,000 docs?" and no chunk contains the answer. GraphRAG (Microsoft) fixes this: at index time an LLM extracts entities plus relationships into a knowledge graph, the Leiden algorithm clusters it into a community hierarchy, and an LLM pre-summarizes each community. Global search then map-reduces over those summaries.^[7] Reach for it for global / sensemaking questions - not simple fact lookup, where it is overkill.^[8]

Evaluate retrieval and generation separately

A wrong answer is either a retrieval miss (the evidence never arrived) or a generation failure (it arrived and the model still blew it) - and the fixes are completely different. This split is the single most useful diagnostic in RAG, and frameworks like RAGAS formalize it.^[9]

Read the pair: low context recall means fix retrieval (chunking, embeddings, top-k, hybrid, reranker). Good recall but low faithfulness means fix generation (prompt, model, or the context is too noisy/long).^[10] Build a golden eval set and treat these scores as a regression suite on every pipeline change.

Top failure modes

The well-known "seven failure points of RAG" plus operational issues - most are retrieval problems wearing a generation costume:

Check yourself