Embeddings, vector DBs, and RAG

A language model's weights are a closed book frozen at training time - retrieval hands it an open book, fetched by meaning, right when the question arrives.

An LLM only knows what it saw during training. Ask it about your company's internal wiki, a document written yesterday, or the exact number in a contract, and it will do the most dangerous thing it can: answer fluently and wrong, inventing a plausible detail because nothing in its weights says otherwise. Retrieval-Augmented Generation (RAG) fixes this by giving the model an external memory. Before answering, we embed the question into a vector, use it to retrieve the few most relevant document chunks by similarity, paste them into the prompt as context, and only then generate. The model stops guessing from memory and starts reading from sources it can cite.

From documents to vectors

You cannot search millions of words by re-reading them. Instead you turn text into geometry. Back in Lesson 12 we saw that an embedding model maps text to a point in a high-dimensional space where meaning becomes position - "invoice" and "billing" land close together even though they share no letters. The same model embeds both your documents and your question into that one shared space, so a question can find the passages that mean the same thing, not just the ones that spell it the same way.

Whole documents are too coarse to embed as one vector - a 40-page manual would collapse into a single blurry point. So we chunk: split each document into passages of a few hundred tokens (often with a little overlap so a sentence straddling a boundary isn't lost), and embed each chunk on its own. Each chunk becomes one point in the space, paired with its original text. That collection of (vector, text) pairs is your knowledge base.

The vector database: nearest neighbors, fast

Given the question's vector $q$, retrieval means finding the document vectors $d$ pointing in the most similar direction. That is exactly the cosine similarity from Lesson 9 - the dot product of the two vectors, normalised so only their angle matters, not their length:

Computing this against a handful of chunks is trivial. Against millions, comparing $q$ to every single vector (an exact linear scan) is too slow for a live request. A vector database - FAISS, Pinecone, pgvector - builds an index for approximate nearest-neighbor (ANN) search: it returns almost the true top-$k$ in sublinear time by cleverly avoiding most comparisons. You trade a sliver of exactness for orders-of-magnitude speed. That trade-off is the whole game at scale.

Retrieve - Augment - Generate

With the nearest chunks in hand, the rest is prompt assembly. Augment: stitch the retrieved passages into the prompt as a CONTEXT block, above the question, with an instruction to answer only from that context and cite it. Generate: the LLM does what it always does - predict the next tokens - but now the most relevant facts sit right there in its context window, so the highest-probability continuation is the grounded one. The citations let a human trace every claim back to a source. Trace one question through all four stages below.

# 1. Offline: chunk every document, embed it, store (vector, text) in the DB
index = VectorDB()
for chunk in split(docs, size=512, overlap=64):   # chunking, with overlap
    index.add(embed(chunk), chunk)                # same embedding model for docs + queries

# 2. Online: answer a question by retrieving, augmenting, generating
def rag(question, k=4):
    q      = embed(question)                     # question -> vector, same space as docs
    hits   = index.search(q, k)                 # ANN: cosine-nearest k chunks, sublinear
    ctx    = "\n".join(f"[{i+1}] {h.text}" for i, h in enumerate(hits))
    prompt = f"Use ONLY this context and cite [n].\n{ctx}\n\nQ: {question}"
    return llm(prompt)                          # grounded, citable answer

Check yourself