Reliability, Evaluation and Safety

Guardrails and failure modes

Loops, cost blowups, drift

~11 minLesson 22 of 28Written by Claude Opus 4.8

Tracing lets you see what your agent did. This lesson is about the two things you build once you can see: naming the handful of ways agents reliably break, and wrapping the loop in so the break is caught before it reaches a user, or a database.

Why agents fail in their own way

Ordinary software fails with a stack trace and a 500. Agents mostly do not. They are non-deterministic, run a multi-step loop, and accumulate state, so their worst failures are silent: goal drift, context loss, and quality decay emit no error code at all.^[4] Worse, the loop compounds mistakes: a wrong tool argument at step 2 quietly corrupts every step after it, because "the autonomous nature of agents means higher costs and the potential for compounding errors."^[1]

That combination, invisible and compounding, is why alerting on exceptions is necessary but nowhere near sufficient. You need two disciplines working together: a mental catalogue of the characteristic failure modes, and a layer of deterministic checks around the loop. Every item below pairs a risk with its mitigation.

The six failure modes you will actually hit

Synthesized from Anthropic's field reports and production taxonomies, almost every agent incident is one of these six. Learn to name them on sight: most of your traces will cluster into just a couple of these.^[4]

Failure mode	Risk	Fix
Infinite / looping	The agent calls the same tool with the same arguments over and over, or cycles between two dependent sub-goals, burning tokens and latency until a timeout kills the session.	A : a max-iteration cap ("a maximum number of iterations to maintain control"), plus loop detection on a hash of the recent tool plus args.^[1]
Error cascade	One bad tool argument silently poisons every later step. Anthropic's SWE-bench agent hit this with relative filepaths corrupting downstream edits.	Schema validation on every argument, plus a better : switching to absolute paths alone fixed it.^[1]
Cost / token blowup	Agents burn roughly 4x the tokens of chat, multi-agent systems roughly 15x; token budget alone explained roughly 80% of performance variance. One system spawned 50 subagents for a trivial query.	A hard token / cost budget per run, plus prompt-embedded scaling rules (simple lookup, 1 agent, 3 to 10 calls).^[2]
Context rot	Quality decays as the window fills: poisoning (an error is written in and reproduces), distraction (history overrides trained knowledge), confusion (irrelevant retrievals degrade output).^[4]	Retrieve and compact instead of dumping; a 200K window loses accuracy long before it fills.^[3]
Goal drift	No single step fails, yet small reasoning deviations accumulate until the output no longer serves the original intent. Invisible to error monitoring.^[4]	Re-anchor to the objective at intervals (restate the goal), and run evaluation alongside output evaluation.
Hallucinated tool args	The model invents arguments, picks the wrong tool, or seeks data where it does not live: "an agent searching the web for context that only exists in Slack is doomed from the start."	Strict tool schemas with validation and high-quality descriptions: one rewrite cut task time 40%.^[2]

Six failure modes, each paired with its mitigation. Learn to name them on sight; most traces cluster into just a couple.

The through-line

Notice how many fixes are not prompts: caps, budgets, schemas, absolute paths, re-anchoring. You cannot instruct a non-deterministic model into reliability. You constrain it from the outside.

An agent repeatedly calls the same search tool with identical arguments, burning tokens until it times out. Which guardrail directly stops this?

The guardrail sandwich

Guardrails are a pipeline, not a prompt. The core pattern, Chip Huyen's framing, is to put checks on both ends: input guards prevent injection and PII leaks before the model sees anything; output guards catch malformed, unsafe, or low-quality responses before a user does.^[7] The agent loop is the filling; the guards are the bread. A response is returned only if it passes both slices.

Tap a node to see what it does.

The guardrail sandwich. Input and output rails bracket the loop; an execution rail sits inside it, gating every tool call, the security-critical rail for agents.

NVIDIA's formalizes this into five rail types. Input and output are the bread; the middle three run inside the loop.^[5]^[6]

Rail	Runs on	Catches
Input	the user message, before the LLM	jailbreaks, prompt injection, PII, off-topic requests
Dialog	conversation flow	disallowed topics, out-of-scope turns
Retrieval	knowledge-base results	poisoned or irrelevant chunks before they enter context
Execution	every tool / action call	unsafe or out-of-policy actions, the agent-critical rail
Output	the model's response, before the user	toxicity, leaked secrets, hallucinations, malformed data

NeMo's five rail types. Input and output are the bread; dialog, retrieval, and especially execution run inside the loop.

A complementary tool, Guardrails AI, specializes the output rail: define a typed schema plus validators, and if the response fails, re-ask or repair rather than ship it, the same reflex as constrained decoding, applied as a safety net.^[7]

Reach for the deterministic guardrails first

Allow / deny lists are the cheapest, most reliable rail: constrain which tools, domains, file paths, SQL operations, or recipients an agent may touch. Unlike model-based moderation, they are deterministic, so they do not have a "95% ceiling." A vendor filter that catches "95% of attacks" is a security failure, not a win: it is the 5% that exfiltrates your data.^[9] Pair them with hard budgets, step limits, and timeouts: cost is a first-class SLO, because a runaway loop is also a runaway bill.^[2]

Human-in-the-loop: the last rail for irreversible actions

Some actions cannot be undone by an output filter: a refund, a delete, an email to a customer. For those, insert a (HITL) checkpoint: the agent proposes the action and pauses for a human decision. Anthropic's guidance is to "pause for human feedback at checkpoints or when encountering blockers," with sandboxed testing and meaningful oversight.^[1]

LangGraph is the canonical implementation: an interrupt() primitive pauses the graph and saves its state to a checkpointer (mandatory, "human-in-the-loop requires checkpointing to handle interrupts"). The human then picks one of four decisions, and Command(resume=...) continues from the saved checkpoint, not from scratch.^[8]

Decision	What it does
approve	Run the tool call exactly as proposed. The default for anything that already passed the rails.
edit	Fix the arguments first, then run. Powerful for catching a cascade; use sparingly, it forces re-evaluation.
reject	Skip the call and return feedback that steers the agent toward a better next step.
respond	Return a human message as a synthetic tool result, only valid for ask-user tools.

The four HITL decisions LangGraph exposes at an interrupt. Command(resume=...) continues from the saved checkpoint.

Gate on the arguments, not just the tool name

Interrupt selectively using a conditional when predicate that receives the ToolCallRequest: auto-approve reads, and interrupt only writes outside a workspace, or a refund whose amount exceeds a threshold. Put HITL on the irreversible, not on everything: a checkpoint on every call trains reviewers to rubber-stamp.^[8]

Here is the whole system in one trace, a refund agent, with each guardrail annotated where it fires. Watch the HITL edit intercept an error cascade that no schema check could catch.

invoke_agent  refund-agent  [4.2s, 11,540 tok]
| -- INPUT RAIL: prompt-injection scan on user msg -> PASS
+- chat          proposes lookup_order(id="OX-42")
+- execute_tool  lookup_order                        [OK, 0.3s]
| -- DENY-LIST: tool in {lookup_order, issue_refund} -> allowed
+- chat          proposes issue_refund(amount=250.00)
| -- EXECUTION RAIL + HITL: amount > $100 -> interrupt()
|      state checkpointed (thread=abc123); await human
|      human -> EDIT amount to 25.00        <- cascade caught
+- execute_tool  issue_refund(amount=25.00)          [OK]
+- chat          finish_reasons=[stop]
| -- OUTPUT RAIL: PII / tone / hallucination check -> PASS
end  done   total_cost=$0.04

One trace, four guardrails. The model misread the order total (250.00 vs 25.00); no schema would flag a well-formed number, but the execution rail plus HITL edit caught the cascade before $250 left the account.

In a well-designed agent, where should a human-in-the-loop interrupt fire?

Guardrails are not security

Guardrails harden an agent against accidents. They do not solve prompt injection. Any agent with private-data access, exposure to untrusted content, and a way to communicate out, Simon Willison's , can be driven to exfiltrate data with no code exploit, and no probabilistic filter reliably stops it. The next lesson is where we treat that as an architecture problem, not a rail.^[9]

Check yourself

Match each NeMo rail to what it catches.

drop here

jailbreaks and prompt injection before the LLM

drop here

unsafe or out-of-policy tool calls

drop here

toxicity and leaked secrets before the user

Name the five NeMo rail types and which two form the bread of the sandwich.

Which run inside the loop, and which is the agent-critical one? Try to state it, then check.

Lock it in

Six failure modes cover almost everything: infinite loops, error cascades, cost blowups, context rot, goal drift, and hallucinated tool args. Each pairs with a mitigation, most of them not a prompt.
The worst failures are silent. Goal drift and context loss emit no error code; only quality evals over traces catch them, so exception alerting is necessary but not sufficient.
Guardrail the sandwich: input rails before the model, output rails before the user, and, most important for agents, an execution rail gating every tool call.
Deterministic beats probabilistic. Allow / deny lists, budgets, step limits, and timeouts have no 95% ceiling; reach for them before model-based moderation.
Put humans on the irreversible. Interrupt only consequential actions, gate on arguments, and require a checkpointer so paused runs resume, not restart.

Primary source

Anthropic, Building Effective Agents

The one to read: stopping conditions (max iterations), compounding errors and the SWE-bench absolute-path fix, the ACI, and "sandboxing plus guardrails plus human oversight" all in a single canonical field report.

Sources