How RAG Actually Works: Embeddings, Indexes, and Ranking

The first mistake people make with RAG is treating it as a language-model feature. It is not. Retrieval-augmented generation is an information-retrieval system bolted to a language model, and most of its quality is decided before the model sees a single token.

The generator is the last mile. The hard part is deciding what evidence reaches it: how documents are split, how text becomes vectors, how approximate nearest-neighbor search avoids scanning the whole corpus, how keyword search rescues exact identifiers, how rerankers trade latency for precision, and how you know whether any of it worked. I wrote a broader map of RAG, CAG, KAG, and memory. This post zooms into the RAG path itself.

RAG is not “chat with your documents”; it is a ranking pipeline whose final answer can only be as good as the evidence it retrieved.

Here is the pipeline from first principles.

RAG starts before the query

The original RAG paper framed the model as two kinds of memory: parametric memory in the model weights, and non-parametric memory in an external document index.¹ That distinction is still the cleanest way to think about it. The model’s weights are hard to inspect and hard to update. The index is explicit, replaceable, filterable, and citable.

But that index is not made at query time. It is built earlier, in an ingestion path that usually matters more than the chatbot wrapper around it. Documents are parsed, normalized, split into chunks, enriched with metadata, embedded, and written to a store. Only after that does the query path begin.

Here is the full shape:

The thing to notice is the asymmetry. Document work is offline and can be expensive; query work is online and sits on the user’s latency budget. That is why production RAG systems precompute document vectors, keep metadata beside them, and reserve the expensive reading step for only a shortlist.

The first design choice is chunking. A chunk that is too small loses the context needed to answer. A chunk that is too large becomes a bag of unrelated facts, and its embedding becomes a blurry average of them. There is no universal chunk size because the right boundary depends on the corpus: legal clauses, code functions, support articles, and research papers have different natural units. The principle is simpler than the parameter: embed the smallest unit that can still answer a question.

Metadata is not decoration. Tenant id, source id, timestamp, document type, permissions, and section path are retrieval controls. If the system should never answer customer A from customer B’s document, that cannot be a prompt instruction. It has to be a filter before retrieval or during retrieval, because the best prompt in the world cannot unsee leaked context.

Embeddings are compressed meaning

An embedding model turns text into a vector, a list of floating-point numbers. OpenAI’s docs describe the core operational idea plainly: small distances between vectors suggest high relatedness, large distances suggest low relatedness.² Google makes the same point from the normalization side: for normalized embeddings, semantic similarity is mostly vector direction rather than magnitude.³

That is the useful half. The dangerous half is compression. A paragraph with hundreds of tokens becomes one fixed-length vector. That vector has to preserve enough about meaning to be searchable, but it cannot preserve everything. Names, numbers, punctuation, exact error strings, and table structure are easy to wash out.

Modern text retrieval usually uses a bi-encoder. One encoder maps the query to a vector. Another, often related or identical, maps every document chunk to a vector. Similarity is then a dot product, cosine similarity, or distance calculation between vectors. The win is that document vectors are precomputed.

The thing to notice is what the bi-encoder gives up. It makes retrieval cheap by forcing query and document to meet only through their vectors. A cross-encoder can read the query and candidate together, but it cannot be run against every chunk in a large corpus.

Sentence-BERT made this tradeoff vivid. The paper notes that comparing all pairs in a collection of 10,000 sentences with a BERT cross-encoder requires about 50 million inference computations, roughly 65 hours on a V100 GPU; SBERT reduces that to computing sentence embeddings plus cosine similarity, on the order of seconds for the same scale.⁴ Dense Passage Retrieval made the same bi-encoder idea central for open-domain QA and reported large gains over a strong Lucene-BM25 baseline on top-twenty passage retrieval accuracy.⁵

So embeddings buy scale by turning language into geometry. That is powerful. It is also why RAG systems miss things that look obvious to humans. Humans can see that ERR_CONN_RESET is the important token. A dense vector might decide the whole chunk is “about browser networking” and retrieve a nicer paragraph that never mentions the actual error.

Vector indexes avoid the full scan

If you have one thousand chunks, exact search is fine: compute the distance from the query vector to every document vector, sort, take the top k. If you have one million or one billion chunks, that full scan becomes the bottleneck. A vector database is mostly an answer to that problem.

To be precise, a vector database is not magic storage for AI. It is a system that stores vectors plus payload metadata, supports similarity search, and usually provides approximate nearest-neighbor indexes so retrieval does not compare against every vector. pgvector, for example, defaults to exact nearest-neighbor search for perfect recall, then offers approximate indexes such as HNSW and IVFFlat when speed matters more than exactness.⁶

The most common index to understand is HNSW, Hierarchical Navigable Small World. It builds a graph where nearby vectors are linked. Some nodes are promoted into sparse upper layers. Search starts high, greedily walks toward the query, drops to denser layers, and refines near the bottom.⁷

The thing to notice is that HNSW is a recall-speed trade. You do not ask it to prove the exact nearest neighbors. You ask it to find very good neighbors quickly. Parameters such as m, ef_construction, and ef_search control how much graph connectivity and candidate exploration you buy. Higher values usually improve recall and cost more memory, build time, or query time.

IVFFlat makes a different trade. It clusters vectors into lists, finds the cluster centers closest to the query, and scans only those lists. In pgvector, lists controls how many partitions exist and probes controls how many are searched at query time.⁶ Higher probes improves recall and costs latency. Search all lists and you are back near exact search.

Product quantization is the compression branch of the family. FAISS popularized the practical stack for billion-scale vector search, including compressed-domain search and GPU-optimized selection.⁸ The simple idea is to store an approximation of the vector so the index uses less memory and scans faster. The price is another layer of recall loss.

The production question is not “which vector database is best”. It is more concrete: how many vectors, what dimension, what latency budget, what recall target, what metadata filters, what update rate, and how painful is a false negative? A support bot can tolerate a slower search better than a wrong answer spoken with confidence.

BM25 is still load-bearing

Dense retrieval is good at semantic similarity. It is bad at exact matching. BM25 is almost the mirror image: it is a bag-of-words ranking function that scores documents by query terms, term frequency, inverse document frequency, and document length normalization.⁹ It does not understand paraphrase. It does understand that a rare literal token matters.

The standard BM25 shape is:

score(D, Q) = sum over query terms:
  idf(term) *
  term_frequency_saturation(term, D, k1) *
  document_length_normalization(D, b)

idf gives more weight to rare terms. k1 controls term-frequency saturation, so the tenth occurrence of a word is not ten times as important as the first. b controls how strongly document length normalizes the score. Lucene’s BM25Similarity defaults to k1 = 1.2 and b = 0.75, and its docs define k1 as saturation and b as length normalization.¹⁰

This is why BM25 keeps showing up in modern RAG systems even after dense retrieval got good. Ask for “the timeout for PAYMENT_WEBHOOK_SECRET” and dense retrieval may return a conceptually similar page about webhooks. BM25 will put the chunk containing PAYMENT_WEBHOOK_SECRET near the top because that exact rare token dominates.

To be clear, this does not mean keyword search is better. It means the two systems fail differently. Dense retrieval catches synonyms, paraphrases, and conceptual matches. Sparse retrieval catches names, ids, codes, and exact wording. A serious RAG system usually wants both.

Hybrid retrieval is two recall systems

Hybrid retrieval is the standard repair: run dense vector search and sparse keyword search, then merge the ranked lists. The important detail is that their scores are not naturally comparable. A cosine similarity from one system and a BM25 score from another do not live on the same scale.

That is why Reciprocal Rank Fusion is so useful. RRF ignores raw scores and combines ranks. For each document, it sums 1 / (k + rank) across retrievers. Cormack, Clarke, and Buettcher fixed k = 60 during pilot experiments and found that RRF consistently improved over individual systems and other fusion methods across several TREC settings.¹¹

Here is the ranking stack:

The thing to notice is the order. Dense and sparse retrieval optimize recall. Fusion builds a candidate pool. Reranking optimizes precision. Prompt assembly is downstream of all three.

This separation matters because each stage has a different job. If the right chunk is not in the initial pool, the reranker cannot recover it. If the pool is too large, the reranker becomes expensive. If the final context is too large, the language model can underuse the evidence even when it is present.

The right mental model is a funnel:

• Candidate generation: fetch enough plausible chunks that the answer is probably somewhere in the pool.
• Fusion: avoid trusting one retriever’s score scale too much.
• Reranking: spend expensive model attention only where it can change the final list.
• Context assembly: choose what the generator actually reads.

Most “RAG is hallucinating” bugs are not generation bugs. They are funnel bugs.

Reranking spends attention late

A bi-encoder compares separately embedded vectors. A cross-encoder reads the query and candidate together. That makes it more accurate for relevance judgments and much more expensive. Cohere’s reranking docs describe the production pattern directly: keep your lexical or semantic search system as first-stage retrieval, then rerank the results in a second stage.¹²

The algorithmic reason is simple. If you have 1,000,000 chunks, a cross-encoder over every query-chunk pair is too expensive. But if hybrid retrieval gives you the top 50, a cross-encoder can afford to read those pairs and return the best 5 or 10.

There is a middle ground called late interaction. ColBERT independently encodes query and document, but keeps token-level representations and performs a cheap interaction step later. The paper reports effectiveness competitive with BERT-based ranking while cutting query-time compute by orders of magnitude.¹³ The broader lesson is that ranking quality improves when the model can compare query tokens against document tokens rather than compressing the whole chunk into one vector.

BEIR is a useful reality check here. Its benchmark across heterogeneous retrieval tasks found BM25 to be a robust baseline, while reranking and late-interaction models achieved the best average zero-shot performance at higher computational cost.¹⁴ That is exactly the trade you see in production: cheap retrievers for breadth, expensive rerankers for judgment.

The practical failure mode is fetching too few candidates. If dense top 5 misses the literal identifier and BM25 is absent, reranking a beautiful but wrong shortlist gives you a more confidently wrong answer. Reranking is not retrieval. It is sorting the pool you already found.

The prompt is a lossy handoff

After retrieval and reranking, the system still has to build a prompt. This step looks boring and is not. The selected chunks have to fit into the context window with the system instruction, conversation history, tool schemas, and the user’s question. Citations have to survive. Duplicates have to be removed. Contradictions need policy. Permission boundaries still matter.

The simplest prompt assembly rule is: put each chunk in a labeled block with source metadata, then instruct the model to answer only from those blocks and cite the block ids. That does not guarantee faithfulness, but it gives the model an evidence shape it can use and gives the system something to audit.

Long context does not remove this step. “Lost in the Middle” showed that models often perform best when relevant information appears near the beginning or end of the context, and degrade when the relevant information is in the middle, even for long-context models.¹⁵ More context can help, but it can also bury the answer.

That is why context assembly should be treated as ranking, not concatenation. The first chunk often gets more attention. Adjacent chunks can help recover local context. Near-duplicates waste budget. A chunk with the exact answer but no source path is less useful than the same chunk with title, section, and timestamp.

The prompt is the handoff from retrieval to generation. If retrieval hands over a messy pile, the model has to become both reader and librarian under a token budget. That is not where you want the system doing its hardest work.

Evaluation has to split the pipeline

RAG evaluation fails when it grades only the final answer. A bad answer can come from bad retrieval, bad reranking, bad prompt assembly, or bad generation. You need to split those apart.

Here is the evaluation loop I would start with:

The thing to notice is that retrieval should be evaluated before generation. If the gold evidence is not in the top k, the generator never had a fair chance.

The core retrieval metrics are old but still useful:

• Recall at k: did any relevant chunk appear in the first k results?
• MRR: how early did the first relevant result appear?
• nDCG: did the ranking put more useful results above less useful ones?
• Latency and cost: how long did retrieval, fusion, and reranking take?

Then evaluate the generated answer separately:

• Faithfulness: is the answer supported by retrieved context?
• Answer correctness: does it actually answer the question?
• Citation accuracy: do cited chunks support the cited claims?
• Abstention: does the system say it does not know when retrieval fails?

BEIR exists because dense retrievers that look good in narrow settings do not always generalize out of distribution.¹⁴ MTEB makes the same point across embedding tasks: no one embedding method dominates every use case.¹⁶ RAGAS adds a practical evaluation frame for RAG systems, separating context relevance, faithfulness, and answer quality without always requiring human-written references.¹⁷

My bias is to keep a small hand-labeled eval set anyway. Synthetic and model-graded evals are useful for iteration, but a dozen painful real questions from production will teach you more than a leaderboard score. The eval set should include identifiers, stale docs, ambiguous questions, permission boundaries, and questions whose correct answer is “not enough information”.

Advanced RAG is mostly better retrieval

Once the basic funnel works, the advanced techniques are less mysterious. Most of them improve one of three things: the query, the candidate pool, or the decision to retrieve at all.

Query rewriting turns a conversational follow-up into a standalone search query. “What about pricing?” is a bad query without the previous turn. “What is the enterprise pricing policy for the backup product?” is searchable.

Multi-query retrieval asks several versions of the same question and unions the results. It buys recall at the cost of more retrieval calls and more fusion work.

HyDE generates a hypothetical answer document first, embeds that generated document, and uses the embedding to retrieve real documents. The generated document may contain false details, but the dense bottleneck is meant to land near relevant real documents in vector space.¹⁸

Self-RAG trains a model to decide when to retrieve, generate, and critique using reflection tokens rather than always retrieving a fixed number of passages.¹⁹ The important production lesson is not that every app should train Self-RAG. It is that retrieval itself can be conditional. Some questions need external evidence; some do not; some need the system to stop because the corpus does not contain the answer.

Filtered vector search moves metadata into the retrieval step. Qdrant’s docs make the problem concrete: filtering separately from vector search can break down because strict filters can disconnect the graph, while weak filters can leave too many vectors to rescore; Qdrant addresses this with filter-aware HNSW edges.²⁰ This is not an implementation footnote. In multi-tenant RAG, filtering is security.

The thread through all of these is the same: get the right evidence into a small, high-quality context window. The LLM is not the search engine. It is the reader after search is done.

Takeaways

If I had to compress the mechanism into a few transferable lessons:

1. RAG quality is decided before generation. The model can only answer from the evidence it sees, so chunking, filters, retrieval, fusion, and reranking are the load-bearing parts of the system.
2. Embeddings are useful because they are lossy. Compressing text into vectors makes search fast and semantic, but the same compression drops exact tokens, structure, and edge cases. Pair dense retrieval with sparse retrieval when correctness matters.
3. Approximate search is a trade, not a guarantee. HNSW, IVFFlat, and quantization buy latency by spending recall. Tune them against an eval set, not vibes.
4. Reranking cannot rescue a missing candidate. Fetch broadly first, fuse rankings without trusting raw score scales, then spend expensive attention on a shortlist.
5. Evaluate retrieval and generation separately. If the gold chunk never appears in top k, the answer failure belongs to retrieval. If it appears and the model ignores it, the failure belongs to prompt assembly or generation.

That is the useful demystification. RAG is not a sprinkle of embeddings on top of a chatbot. It is a search system with a language model at the end. Build it like one.