Shipping to production

A working agent in a notebook is maybe a fifth of the job. Production is where the token bill, the p95 latency SLO, compounding per-step errors, and irreversible side-effects all bite at once. This lesson is the systems architecture, economics, reliability primitives, and the eval / observability / guardrail stack that turns a demo into a service you can actually trust.

Capability is not your blocker

The frontier lab surveys are blunt about where teams get stuck. In LangChain's State of Agent Engineering 2025 (1,340 respondents), 57% already run agents in production, but the top blockers are quality (~32%) and latency (~20%), not model IQ. Cost fell down the list year-over-year, because caching and routing worked. And there is a telling gap: 89% have observability, yet only ~52% run offline evals.^[1] Teams can see failures but cannot catch regressions before they ship.

The root problem: errors compound over long runs

A stateless request/response service fails one request at a time. An agent is a loop: reason, act, observe, reason again, each step mutating state the next step depends on. So per-step reliability does not add, it multiplies. If every step is independently reliable with probability p, an n-step task finishes end-to-end only p^n of the time.^[2] A very-good-looking 95% per step is a 36% task over 20 steps.

Two lessons fall straight out of this curve. First, measure the whole task, never the per-step average, the average hides the multiplication.^[2] Second, the practical fixes are architectural: shorten n (fewer, sharper steps), raise p at the riskiest steps (validation, a better tool interface), and add recovery so a single bad step does not restart the whole run.

Token economics: the cheapest token is the one you do not send

Production spend is dominated by three composable levers. Learn them in order, because they stack.

Lever 1: Prompt caching (kill the fixed prefix)

Every request re-sends a big stable prefix: system prompt, tool schemas, few-shot examples, policy docs. Prompt caching lets repeat requests re-read that prefix instead of paying to process it again. The Anthropic mechanics are exact and worth memorising:^[3]

1. A cache read is about 0.1x base input (a 90% discount); a cache write costs more, 1.25x for the 5-minute TTL, 2.0x for the 1-hour TTL. Caching pays off once a prefix is read more than a couple of times.
2. cache_control marks a breakpoint; the write is a hash of the entire prefix ending at that block, and reads walk backward up to 20 blocks to find a prior write.
3. Invalidation is hierarchical: tools, system, messages. Editing tool definitions busts everything below them, so keep tool schemas and the system prompt stable.
4. There is a per-model minimum cacheable prefix (for example ~1,024 tokens) below which caching is silently skipped. Verify with usage.cache_read_input_tokens: zero means it never happened.

# WRONG - hash changes every request -> always a miss
system = [{"text": f"Time: {now}. You are a support agent...",
           "cache_control": {"type": "ephemeral"}}]

# RIGHT - stable prefix cached; volatile bits placed AFTER the breakpoint
system = [{"text": STABLE_POLICY_AND_FEWSHOTS,          # ~8k stable tokens
           "cache_control": {"type": "ephemeral"}},   # breakpoint on LAST stable block
          {"text": f"Time: {now}."}]                    # volatile - not cached, fine

Lever 2: Model routing and cascades (stop paying frontier prices for easy work)

Roughly 40 to 60% of requests do not need a frontier model at all.^[4] Two shapes:

Reported outcome across the routing literature: cost cuts of 45 to 85% while keeping about 95% of quality.^[4] The same idea applies inside a run: dispatch cheap sub-tasks (classification, extraction, summarising a tool result) to a small model and reserve the frontier model for the reasoning that needs it.

Lever 3: Batching (for anything a human is not waiting on)

Async/batch APIs give a roughly 50% discount for latency-tolerant work: evals, backfills, bulk generation.^[5] Batch stacked on top of caching pushes effective savings past 90% on the right workloads.

Latency budgets and streaming

Latency is not one number: decompose it so you optimise the part users feel, time-to-first-token (TTFT, responsiveness), tokens/sec (throughput), and time-to-last-token (total wall clock). The single biggest perceived-latency win is streaming: whenever a human is waiting, return tokens as they generate so they start reading before the run finishes.^[7]

1. Smaller / faster tier. The same routing that cuts cost cuts latency: a Haiku-class model answers the easy half faster.
2. Cap max_tokens. Generation time scales with output length; bound it and stop runaway completions.^[7]
3. Parallelise and cache. Fire independent tool calls concurrently; a warm prompt cache also shortens TTFT on the stable prefix.

Reliability primitives: ship these from day one

Agents retry aggressively and hold state for a long time, so the classic distributed-systems toolkit is mandatory, applied to non-deterministic calls.^[8]

1. Bounded retries with exponential backoff plus jitter. Wrap every tool and model call; add timeouts and circuit breakers so one flaky dependency cannot take the run down.
2. Idempotency keys tied to the originating user request. Attach one to every state-changing action (DB writes, charges, emails, external API calls). Without it, an aggressive retry on a non-idempotent endpoint becomes a duplicate charge or a double-sent email.^[8]
3. Durable execution plus checkpointing. Persist progress so a crash resumes from the last checkpoint instead of restarting a long, expensive run. Anthropic's long-running harness makes git commits, a progress file, and a feature checklist the external memory that survives a lost context window.^[9]
4. Bound the loop and the budget. Set a max-iteration cap, a per-run token/cost budget, and loop detection (hash of recent tool plus args) so a stuck agent hits a wall instead of a runaway bill.^[2]

The production architecture

Now assemble it. This is where the eval / observability / guardrail work from the reliability module stops being three separate lessons and becomes one request path. Read it as a request flowing through the gateway into a central orchestrator, fanning down into capabilities, wrapped by guardrail rails and an observability plane that spans everything.

Human-in-the-loop on the irreversible, not on everything

For irreversible or high-stakes actions, refunds, deletes, external sends, pause and get a human decision. HITL in LangGraph is an interrupt() backed by a mandatory checkpointer, offering four decisions, approve / edit / reject / respond, and conditional interrupts that gate on the tool's arguments (auto-approve reads; interrupt only writes over a threshold or outside a workspace).^[11] Because it is checkpointed, a paused run resumes rather than restarts, the same durability primitive as above.

Deployment topology: do not kill agents mid-run

A normal rolling deploy assumes requests finish in milliseconds. Agents hold state for minutes to hours, so a rollout that drains the old version kills in-flight runs. The fix is a rainbow deployment: run old and new versions concurrently and shift traffic gradually, letting long runs finish on the version they started on.^[6] Pair it with a stateless orchestrator plus durable state store so any worker can resume any run from its last checkpoint.^[9]

The dashboard: what production actually watches

You cannot improve what you cannot see, and the goal is to minimise MTTD (mean time to detect) and MTTR (mean time to respond), which means designing observability in from the start, not bolting it on after an incident.^[13] Get raw traces flowing first; the tooling choice matters less than having the data.^[14] The canonical wire format is the OpenTelemetry GenAI convention: an invoke_agent span with child chat and execute_tool spans carrying gen_ai.usage.*_tokens and finish_reasons.^[15]

The cost / latency lever board

Reach for these in order: the top three do most of the work.

Check yourself