Agent memory

An LLM forgets everything the instant a call returns - it is stateless between turns. Memory is the machinery that lets an agent carry what it learned across turns and across sessions: a taxonomy borrowed from human cognition, and a write, read, reflect loop that keeps it useful over time.

Why agents need memory at all

A base model is a pure function: tokens in, tokens out, no side effects, no recollection. Whatever an agent "knows" during a run lives in exactly one place - the context window it was handed this turn. That window is the agent's only working memory, and it has two hard limits: it is volatile (cleared between sessions, and compacted mid-session to make room) and capacity-limited (you cannot just keep appending). Worse, quality degrades before the window even fills - Chroma's 18-model study measured this "context rot" on trivial tasks, well short of the token ceiling.^[2]

So "remember more" cannot mean "stuff more into the prompt." Anthropic frames the real job as curating the smallest set of high-signal tokens that get the desired behavior, and fetching the rest just-in-time from outside the window.^[1] Memory is that "outside": a persistent store the agent writes to and reads from, so a small window can behave like a large, long-lived mind. This is the same instinct behind context engineering - decide what belongs in the window - pushed across time instead of just across one turn.

The taxonomy: human memory, mapped

The field did not invent a taxonomy - it borrowed cognitive science's. The dominant formalization is CoALA (Cognitive Architectures for Language Agents), and Letta, Mem0, and LangGraph all adopt it.^[3] The top split is working (short-term) vs long-term; long-term then divides into three kinds that map cleanly onto how humans remember.^[4]

A fourth layer, sensory memory, maps to the learned embedding representations of raw inputs - transient pre-processing rather than something the agent reads back. For agent-building purposes the working / episodic / semantic / procedural split is the one that earns its keep.^[5]

The loop: write, read, reflect

A taxonomy tells you what to store; the loop tells you how the store stays alive. Three operations run against long-term memory. Write and read are the obvious pair; reflect is what separates a real memory from a dumb log.

1. Write. Do not dump the whole transcript. An extraction step (usually a small LLM pass) pulls the salient facts from the new turn and persists them - Anthropic calls this structured note-taking: cheap durable memory the agent re-hydrates later.^[1]
2. Read. At each turn, retrieve just the few memories relevant to the current query - embed the query, run ANN vector search over the store, and inject the top matches. The store may hold thousands; only about 3 high-signal ones enter the window.
3. Reflect. Periodically, the agent reasons over its own memories to write new ones: consolidate duplicates, resolve contradictions, and synthesize higher-level insights ("this user is a power user") that no single raw event stated.

A concrete trace: read, then write-back

A returning user, session 5. The store already holds facts from earlier sessions.

1. Query. "Book me the usual table for Friday." Embed it to a vector.
2. Read. ANN search over this user's long-term memory returns the top matches: "prefers window seat at Luigi's, party of 2" (semantic, 0.83), "last 4 Friday bookings were 7:30pm" (episodic, 0.79), "vegetarian" (semantic, 0.61).
3. Assemble. Working context = system prompt + those ~3 retrieved memories (not the whole store) + the current turn.
4. Act. The model calls reserve(restaurant="Luigi's", time="19:30", seats=2, seat="window").
5. Write-back. An extract-then-update pass records the new episode and leaves unchanged facts alone.

ADD   episodic -> "Session 5: booked Luigi's Fri 7:30pm, window, party of 2"
NOOP  on "vegetarian" (unchanged - no duplicate)

Thousands of memories in the store, but only ~3 (about 150 tokens) ever entered the window. That is the whole game: small working set, big backing store, retrieval on demand - the memory version of the just-in-time principle.

How memory differs from RAG

Memory and RAG share the exact same retrieval substrate - embeddings + ANN search - which is why they are easy to conflate. But they point in opposite directions. Classic RAG is read-only retrieval over a corpus someone else curated; memory is read and write over a store the agent authors from its own experience, and keeps current.

In practice the line blurs - agentic RAG lets a model decide when and what to retrieve, and a memory system uses RAG-style retrieval on every read. The useful distinction is persistence + updating + reflection: RAG answers "what does the corpus say?"; memory answers "what have I learned, and is it still true?"

Frameworks: two ways to build it

Two designs dominate, and they sit at different altitudes. One gives the model low-level control of a memory hierarchy; the other runs an automatic pipeline around the model.

The canonical example: Generative Agents ("Smallville")

The clearest demonstration of the full loop is the Generative Agents simulation - 25 agents living in a sandbox town, "Smallville," each with a memory architecture rich enough to produce believable behavior (one throws a Valentine's party and others actually show up and remember it).^[4] It is a working reference design for all three operations.

1. Memory stream (write). Every observation is appended to a long-term memory stream - a timestamped, natural-language log of what the agent saw and did. Pure episodic write.
2. Scored retrieval (read). To act, the agent retrieves memories scored on three axes: recency (recent-weighted), importance (self-rated salience), and relevance (embedding similarity to the moment). Not just nearest-neighbor.
3. Reflection (reflect). Periodically the agent synthesizes its recent memories into higher-level reflections - abstract inferences that themselves become new memories and steer future planning.

That importance-weighted retrieval and the reflection tree are the parts a plain vector search does not give you - and they are exactly the "reflect" edge of the loop made concrete.^[4]

What breaks

Check yourself