Memory and Knowledge

Context engineering

Managing the context window

~10 minLesson 11 of 28Written by Claude Opus 4.8

Prompting optimized one static block of instructions. But an agent runs in a loop - accumulating tool outputs, spanning many turns, pulling in external data - so the real artifact you manage is the entire set of tokens the model sees on every step. Curating that set is context engineering, and it is the highest-leverage reliability lever you have.

From prompt engineering to context engineering

Anthropic draws the line cleanly: prompt engineering is "writing and organizing LLM instructions," while context engineering is "the set of strategies for curating and maintaining the optimal set of tokens during LLM inference."^[1] The question shifts from what words do I use? to what configuration of context is most likely to produce the behavior I want?

Andrej Karpathy's analogy makes it concrete: the LLM is like a CPU and its is its RAM - a small, fast, finite working store. Your job each step is deciding what to page into that limited RAM. Context is, in Anthropic's words, "a critical but finite resource," and the goal for any given step is "the smallest possible set of high-signal tokens that maximize the likelihood of the desired outcome."^[1]

The reframe in one line

Prompt engineering asks what do I say? and optimizes a mostly-static message. Context engineering asks what should be in the window right now? and optimizes a dynamic, curated state that you re-assemble every turn - system prompt, history, retrieved documents, tool results, and memory all competing for one budget.

The context window is a budget

Every token the model reads on a step is drawn from one finite pool. The prompt you own is only a sliver of it; in a long-running agent the window fills mostly with things you did not hand-write - turn-by-turn history and raw tool payloads. Here is a realistic snapshot of a coding agent thirty turns into a task, against a 200K-token window.

System + tools · 10K · 5% Message history · 48K · 24% Retrieved docs · 32K · 16% Tool results · 70K · 35% Long-term memory · 16K · 8% Headroom · 24K · 12%

Figure 1 · The context budget of one step. Raw tool results (35%) and history (24%) dominate - nearly 60% of the window is accumulated exhaust, not high-signal input. That is where the savings live.

The in-window set is the agent's : instantly available, but volatile and capacity-bounded.^[6] Everything else - long-term memory, the document corpus, the full tool catalog - lives outside and must be paged in deliberately.

Why more context is not better: context rot

The naive fix is "buy a bigger window." It does not work the way you would hope. Chroma benchmarked 18 frontier models (Claude 4, GPT-4.1, Gemini 2.5, Qwen3) and found that every one degrades as input length grows - even on trivial tasks, and even when the window is nowhere near full.^[2] This measured phenomenon is . A 200K window can start losing real accuracy around ~50K tokens of input; 1M-token models show clear effects well before they fill.

Two mechanisms drive it:

Quadratic attention

Self-attention scores every pair of tokens: n tokens gives n squared relationships. At 100K tokens that is ~10 billion pairwise interactions.^[1] Spread the same attention budget thinner and each token gets less of it - a built-in tension between context size and focus.

Training-distribution skew

Models see far more short sequences than long ones in training, so they have less experience with - and weaker specialized machinery for - long-range dependencies.^[2] Length itself becomes an out-of-distribution stressor.

Accuracy erodes long before the window is full, and a single semantically-similar-but-irrelevant distractor accelerates the fall.^[2]

Lost in the middle and poisoning distractors

A related, older finding - - gives a U-shaped curve: models attend best to the start and end of the window and can miss facts buried in the middle. Chroma also found that even one irrelevant-but-similar chunk lowers accuracy, and several compound it.^[2] The lesson: "more retrieved chunks" is not strictly better - retrieve less, but better.

The five moves that keep the working set small

If quality falls with length, the discipline is to keep the window small and high-signal every turn. Five strategies do the heavy lifting. Think of them as the verbs of context engineering.^[1]

Compaction. Near the window limit, summarize the history into a compact brief and continue from it. Bias the summary prompt toward recall - preserve architectural decisions, open bugs, and key facts while dropping redundant tool payloads.^[1] Watch out: compaction is lossy, so keep raw notes retrievable in case a dropped detail matters three steps later.
Just-in-time retrieval. Do not pre-load whole corpora "just in case." Store lightweight identifiers - file paths, IDs, URLs, queries - and fetch the content at runtime, exactly when it is needed.^[1] This is RAG, powered by embeddings, and it enables progressive disclosure: the agent pulls detail on demand.
Sub-agent isolation. Give a focused sub-task its own clean window. The sub-agent does the noisy exploration and returns a distilled 1,000 to 2,000-token summary to the orchestrator - never the full trace.^[1] The parent's context stays clean; the expensive middle steps are quarantined in a window you throw away.
Offload to memory. Have the agent write durable notes outside the window - a scratchpad file or a memory store - and re-hydrate them later. Cheap, persistent, low-overhead structured note-taking.^[1] What survives a context clear is only what you deliberately persisted.
Just-in-time tool loading. Every tool definition in the prompt costs tokens on every turn. Load only the tools the current step needs, discovering more on demand - for example via MCP - rather than front-loading a catalog of fifty schemas. Same progressive-disclosure principle as retrieval, applied to capabilities instead of content.

The through-line

All five do one thing: move low-signal tokens out of the window and keep only what this step needs. Retrieve less but better; summarize aggressively; isolate the mess; persist the essence.

Bloated vs curated: the same step, rewritten

Here is one agent step, assembled two ways. Same model, same task, same window size - but a very different set of tokens reaches the model.

Bloated context (~190K / 200K)

Every prior turn kept verbatim, raw tool payloads and all. All 50 tool schemas loaded up front, most irrelevant now. Whole documents pasted in "just in case." Nothing summarized, cleared, or offloaded. Slow, costly, and - per context rot - less accurate: the key fact is buried mid-window and drowned by distractors.

Curated context (~40K / 200K)

History compacted to a running brief: decisions plus open threads. Only the ~3 tools this step needs are loaded. JIT retrieval: top 3 chunks fetched now, by ID. Stale tool results cleared; durable facts live in memory files. High-signal, cheaper, faster - and more accurate because every token earns its place.

This is now a platform feature

Context management shipped as first-class API on Claude Sonnet 4.5 (Sept 2025): a memory tool - file-based, client-side CRUD in a developer-owned directory that persists across sessions^[5] - plus context editing, which auto-clears the oldest stale tool results once a token threshold is crossed (the clear_tool_uses_20250919 strategy), keeping the reasoning but dropping the now-useless raw payloads.^[4] On a 100-turn web-search eval, Anthropic reports +29% from context editing alone, +39% combined with the memory tool, and 84% fewer tokens. Sonnet 4.5 even tracks its own remaining budget.^[3]

Long context vs RAG is a false choice

The 2026 consensus is hybrid: use retrieval and external memory to keep the working set small by default, and reserve genuinely long context only for dependency-dense tasks that truly need it - precisely because of context rot. "Stuff everything in" loses to "retrieve less, but better."^[1]

Check yourself

Match each context-engineering move to what it does.

drop here

summarise old turns to reclaim tokens

drop here

store IDs, fetch content at runtime

drop here

clean window, return a distilled summary

drop here

durable notes stored outside the window

According to the context-rot research, when does model accuracy start to degrade as input grows?

Your agent's window is filling with dozens of raw tool-result payloads from earlier turns. What is the right move?

Name the five moves of context engineering that keep the working set small.

Each moves low-signal tokens out of the window - how? Try to state it, then check.

Lock it in

Context engineering is the successor to prompt engineering: curate the full set of tokens the model sees each step, not just the words you write.
The window is a finite budget - system, history, retrieved docs, tool results, and memory compete for it. In long runs, raw tool payloads and history dominate.
More context is not better. Context rot degrades every model before the window fills; lost-in-the-middle and distractors make buried facts worse.
Five moves keep the working set small: compact, retrieve just-in-time, isolate via sub-agents, offload to memory, load tools just-in-time.
Aim for the smallest set of high-signal tokens per step - the platform tooling that does this reports +39% task success and 84% fewer tokens.

Primary source

Anthropic, Effective context engineering for AI agents

The canonical essay: the prompt to context reframe, the finite-resource framing, right-altitude system prompts, and each of the five moves (compaction, JIT retrieval, sub-agents, note-taking) in one place.

Sources