Feeding Knowledge to an LLM: RAG, CAG, KAG, and the Memory Layer

A trained language model knows exactly one thing: whatever was in its weights the day training stopped. That knowledge is frozen, it has a cutoff date, it cannot cite a source, and when you ask about something outside it the model does not go quiet - it guesses, fluently. For a chatbot that is annoying. For an agent acting on a real business’s data, a confident guess is a liability.

So the interesting engineering is almost never inside the model. It is in how you get the right external knowledge in front of the model at inference time, cheaply and verifiably. There are three named families of techniques for that, and they are easy to confuse because the acronyms rhyme: Retrieval, Cache, and Knowledge augmented generation. They are not competitors so much as three different answers to the question “where does the knowledge live and how does it reach the prompt”. And agents add a fourth axis that none of the three covers: memory that grows from interaction rather than sitting in a fixed corpus.

This post is a map. RAG retrieves knowledge per query, CAG preloads it once, KAG structures it as a graph, and the memory layer is the part that remembers you between conversations. The skill is matching the technique to the shape of your knowledge, not picking a favorite. Here is each one, what it costs, and when it earns its place.

RAG: retrieve, then generate

Retrieval-Augmented Generation, introduced by Lewis et al in 2020,¹ is the default answer and still the most common. The idea is to give the model a non-parametric memory: knowledge that lives outside the weights, in a store you control, and gets pulled in at query time.

The read path is mechanical. You chunk your documents into passages, run each through an embedding model to get a vector, and store those vectors in a vector database. At query time you embed the user’s question the same way, find the nearest chunks by vector similarity, paste the top few into the prompt as context, and ask the model to answer using only that context.

Here is the whole read path:

The thing to notice is that the model never sees your whole corpus, only the handful of chunks that survived the nearest-neighbor search. That is the entire value proposition and the entire failure surface at once.

The value: knowledge stays fresh because you update the store, not the model. Answers are scoped to a single tenant’s documents. The model can cite, because you know which chunks it was handed. And it scales to corpora far larger than any context window, because only top-k chunks ever reach the prompt.

The store underneath is a vector database doing approximate nearest-neighbor search, usually with an HNSW graph index so lookups stay sub-linear as the corpus grows. It can be a dedicated engine or a Postgres extension like pgvector; the index structure matters more than the brand.

Here is what “sub-linear” buys you. HNSW stacks the vectors into layers, sparse up top and complete at the base, and a lookup enters at the top, greedily walks to the nearest node, then drops a layer and repeats:

The thing to notice is that the upper layers let the search cross most of the space in a few long hops before it ever touches the dense base layer, which is why the cost grows with the logarithm of the corpus rather than its size. The recurring tax is that everything hinges on the retriever actually surfacing the right chunks, and a plain vector search frequently does not.

Why naive RAG disappoints, and what fixes it

Vector similarity is semantic: it is good at “these two passages mean roughly the same thing” and bad at “these two passages share this exact rare token”. Ask a dense retriever for an order id, a part number, a person’s surname, or an error code, and it will happily return passages that are about the topic while missing the one passage that contains the literal string. This is the single most common reason a RAG demo looks brilliant and the production system feels dumb.

The standard repair is hybrid retrieval: run a dense vector search and a sparse keyword search such as BM25 in parallel, then fuse the two ranked lists. The usual fusion is Reciprocal Rank Fusion, which scores each result by the sum of 1 / (k + rank) across both lists with a constant k of about 60,² so a document that ranks decently in either retriever floats up without either score having to be calibrated against the other. Dense recall catches paraphrase, sparse recall catches exact tokens, and the union is far more forgiving than either alone.

The second repair is reranking. Retrieval is tuned for recall, so you fetch more candidates than you need (say the top twenty to fifty) and then pass each one, paired with the query, through a cross-encoder reranker that reads both together and scores true relevance. You keep the best handful. It is slower per candidate than a vector lookup, which is exactly why it runs only on the shortlist.

Stacked together, the two repairs turn a single vector lookup into a small pipeline:

The thing to notice is that dense and sparse run in parallel and only converge at fusion, so paraphrase and exact-token recall both get a vote before the expensive cross-encoder ever runs on the survivors.

Everything else in production RAG is in service of the same goal of getting better chunks to the model: chunking strategy (too small loses context, too large dilutes the signal), query rewriting and expansion, and metadata filters that constrain the search to the right tenant or document set before similarity ever runs. When people say “RAG is hard”, they almost never mean the generation step. They mean retrieval quality.

CAG: cache the context instead of retrieving it

Retrieval buys scale, but it costs a hop on every query, a vector store to operate, and a permanent risk that the retriever misses. The paper that named the technique, “Don’t Do RAG: When Cache-Augmented Generation is All You Need”, makes a sharp observation:³ when your knowledge is small enough to fit inside the model’s context window, retrieval is solving a problem you do not have.

Cache-Augmented Generation preloads the entire relevant corpus into the context once, lets the model process it, and caches the resulting key-value attention state - the KV cache. Subsequent queries reuse that cached state instead of re-encoding the documents or running any retrieval. There is no embedding step, no vector database, no chunk-boundary problem, and no chance the retriever drops the one passage that mattered, because every passage is already in context.

Here is the contrast. The familiar RAG path runs retrieval on every query:

CAG collapses that into a one-time preload, after which every query answers straight from the cached state with no retrieval:

The thing to notice is that CAG moves the work from query time to load time, and pays for it in context budget rather than retrieval infrastructure.

To be clear, this is not an exotic research technique. The same mechanism shows up commercially as provider-side context caching or prompt caching: you mark a large stable prefix (a long system prompt, a policy document, a codebase) as cacheable, and the provider stores its KV state so repeated calls skip re-encoding it, cutting both latency and token cost on the cached portion. CAG is that idea taken to its logical end, where the cached prefix is your knowledge base.

The limits are exactly the limits of the context window. CAG only works while the corpus fits, the per-query cost scales with how much context you carry, and a frozen cache goes stale the moment the underlying documents change, so you re-warm it on update. It is the right tool for a bounded, slow-changing body of knowledge - a product manual, a policy set, an API reference - and the wrong tool for a corpus measured in gigabytes.

KAG: grounding in a knowledge graph

RAG and CAG both treat knowledge as a bag of text passages. That works until the question requires reasoning over relationships rather than recalling a passage: “which suppliers are two hops away from a sanctioned entity”, “what is the dosage interaction between these two drugs given this condition”. Vector similarity has no notion of a hop, and stuffing more passages into context does not create one.

Knowledge-Augmented Generation, from the KAG framework built on the OpenSPG engine,⁴ addresses this by grounding the model in a structured knowledge graph of entities and the typed relationships between them, then using logical-form-guided reasoning to traverse that graph. Instead of “find passages similar to the query”, the system can plan a multi-step retrieval over the graph and combine structured facts with text. KAG’s stated aim is professional domains where the traditional vector-similarity model is too noisy and where answers must be logically defensible, not merely plausible.

The earlier “two hops from a sanctioned entity” question is a good shape to picture. The graph stores the entities and the typed edges between them, and the answer is a literal traversal:

The thing to notice is that the answer is defined by the edge types and the hop count, not by passage similarity: vector search has no way to express “exactly two owns-then-supplies hops away”, but a graph traversal is exactly that query.

To be clear, KAG is not just “RAG with a graph bolted on.” It is worth separating two things that get blurred. Microsoft’s GraphRAG also builds a graph from your documents and summarizes communities within it to answer broad “what are the themes” questions.⁵ KAG pushes further into formal reasoning and logical planning over a curated domain graph. Both are reactions to the same gap in flat RAG; KAG leans toward structured, multi-hop, audit-friendly reasoning, while GraphRAG leans toward graph-assisted summarization.

The cost is the graph itself. Someone or something has to extract entities and relations, resolve duplicates, and maintain the schema as the domain evolves. That construction and upkeep is real work, which is why KAG earns its place in high-stakes, structured domains - medicine, law, finance, compliance - and is overkill for a FAQ bot.

The memory layer: what an agent keeps between turns

RAG, CAG, and KAG all answer “how does my fixed knowledge reach the model”. An agent has a second, different need: it has to remember what happened in this relationship over time. That is not a read-mostly corpus you author once; it is state that accumulates from every interaction. Conflating the two is a common design error, because the storage looks identical while the lifecycle is completely different.

It helps to split memory along two axes. The first is duration. Short-term memory is the context window itself plus any scratchpad the agent keeps within a session; it is fast and gone when the session ends. Long-term memory is an external store the agent writes to and reads back across sessions. The second axis is the kind of thing remembered, borrowing the cognitive-science split: semantic memory holds durable facts (“this user prefers metric units”, “their account tier is enterprise”), episodic memory holds specific past events (“last Tuesday they reported a failed payment”), and procedural memory holds learned how-to (“the steps that resolved this class of ticket”).

The thing to notice is that the long-term branches are usually implemented with the same vector store and retrieval machinery as RAG, which is why the two get conflated.

The difference is entirely on the write path. A RAG corpus is authored and ingested deliberately. A memory store has to decide, on the fly, what from a conversation is even worth keeping, extract it into a clean fact, deduplicate it against what is already known, reconcile contradictions (“they moved cities”), and sometimes let stale entries decay. Here is that write path, which is where the real work lives:

Open memory layers such as Mem0 and OpenMemory, and agent frameworks in the MemGPT and Letta lineage, are largely about getting that extraction-and-reconciliation pipeline right, because naive “embed every message” memory fills with noise and starts retrieving garbage within a day. Retrieval into the prompt is the easy half; deciding what deserves to be remembered is the hard half.

What each one trades away

None of these is strictly better. Each one buys a strength by paying a specific tax. They trade along freshness, scale, latency, operational complexity, and reasoning power. The honest comparison:

The pattern worth internalizing: RAG and CAG sit on a single axis, namely corpus size versus context window. KAG is on a different axis entirely, namely flat text versus structured relationships. And memory is orthogonal to all three, because it is about time rather than corpus shape. Real systems combine them. A support agent might use RAG over the help center, CAG for the always-loaded policy doc, and a memory layer for the customer’s history, all at once.

When to use which

A rough decision order, cheapest and simplest first:

The thing to notice is that the memory question is separate from the other three, not a leaf of the same tree: you answer it in addition to whichever knowledge approach you picked, not instead.

In plain terms. If your knowledge is small and slow-changing, reach for CAG before you stand up a vector database, because retrieval infrastructure you do not need is pure operational cost. If it is large or updates constantly, RAG is the workhorse, and the moment a plain demo disappoints, add hybrid retrieval and a reranker before you blame the model. If correct answers require traversing relationships rather than recalling passages, and the domain justifies the upkeep, build a graph and use KAG or GraphRAG. And if the thing is an agent that should feel like it knows the person across conversations, layer a memory store on top of whichever of those you chose, and spend your effort on the write path, because that is where memory systems actually fail.

Takeaways

A trained model still knows exactly one thing: whatever was frozen in its weights. Everything in this post is about the knowledge that lives outside them.

• The hard part of knowledge-augmented AI is not the model. It is matching the technique to the shape of your knowledge: its size, its rate of change, its structure, and whether it persists over time.
• RAG gives the model a non-parametric memory that stays fresh and scales, but it lives or dies on retrieval quality. Hybrid dense plus BM25 fused with RRF, then a cross-encoder reranker, is the difference between a demo and a product.
• CAG is the underrated default for bounded, stable corpora. If the knowledge fits the context window, preloading and caching the KV state skips the entire retrieval problem.
• KAG and GraphRAG exist because flat text retrieval cannot reason over relationships. Pay the graph-construction cost only when answers genuinely require multi-hop, auditable reasoning.
• Memory is not RAG. It shares the storage but inverts the difficulty: retrieval is easy, deciding what to remember and when to forget is hard.
• These are layers, not rivals. The interesting systems run several at once and keep each one pointed at the kind of knowledge it is actually good at.