Evaluating agents

You cannot improve what you cannot measure, and agents are the hardest software to measure. They take many non-deterministic steps, reach the same goal by many valid paths, and invent solutions your grader never imagined. This lesson is a working eval toolkit: what to grade, how to grade it, the metrics that actually predict production, and the benchmarks worth knowing.

Why agent eval breaks classic testing

A traditional ML eval is a pure function: fixed input, one output, compare to a label. Agent eval violates every one of those assumptions. Four structural reasons make it genuinely hard, and you are grading a whole system (model plus scaffold plus tools plus prompts), never a model in isolation.

1. Multi-step, so error compounds. Each turn mutates state and mistakes propagate.^[1] If every step is 95% reliable, a 10-step task is only about 0.95^10, roughly 60% end-to-end. Per-step accuracy hides this, so always measure the whole task.
2. Non-deterministic, so one run is noise. Same prompt, different outcome (sampling, plus a simulated user or live environment that itself varies). A single pass or fail is a coin flip's worth of signal. Run each task k times and reason about a distribution.
3. Many valid paths, so there is no golden trajectory. "Cancel my order and refund the smaller item" has many correct tool sequences. Grading against one reference path produces false negatives. Mature benchmarks grade the final state instead.
4. Open-ended, so solutions are creative. Frontier models phrase correct answers in ways your string-match grader marks wrong. This pushes eval toward flexible grading, state checks and LLM judges, and away from exact match.

The central choice: outcome vs trajectory

This is the single most important distinction to internalize. Outcome eval checks the final state of the environment and is the default pass or fail signal. Trajectory eval inspects the full transcript, which tools in what order at what token cost, and is essential for process quality and debugging, but it over-penalizes valid alternate paths if you use it as the gate.

Task: "Remove the water bottle from order #W123 - keep the mug."

TRAJECTORY - the path the agent took
  get_user()
  get_order(#W123)
  cancel_order()               x  policy block
  <- backtrack, re-read policy
  modify_order(remove=bottle)
  reply: "Done - mug kept."
  6 steps, one wrong turn, one retry

OUTCOME - the final state of the world
  order #W123
    status = pending
    items  = [ mug ]
    water_bottle -> removed
  goal_state = MATCH

Golden-path grader -> FAIL   (extra cancel step != reference path)
Final-state grader -> PASS   (end state == goal, path-agnostic)

Offline vs online: two questions, both needed

Offline (automated) evals run on a curated task bank without real users: reproducible, fast, safe, and able to gate CI. Their weakness is that the tasks are synthetic proxies, and they need isolated, stateful environments that reset per run. Online evals observe real production traffic: sampled LLM-judge scoring, thumbs up or down, task-completion signals, A/B tests, guardrail triggers. Their weakness is that they are reactive, so a regression reaches users before you catch it.^[1]

Three grader families, cheapest first

Every grade is produced by one of three mechanisms. Layer them cheapest first: let an assert decide what an assert can, and reserve expensive graders for what is genuinely open-ended.^[1]

Assertion evals are unit tests for agents: cheap boolean checks on output or trajectory ("response contains no PII," "the SQL is read-only," "refund called at most once") and should be your first line of defense. Rubric evals hand an LLM judge an explicit written checklist ("did it cite a source? did it refuse the out-of-policy request?"); isolate one judge per dimension rather than asking a single call to score five things.^[1]

LLM-as-judge, built like an instrument

The LLM judge is the workhorse of scalable eval, but it is only trustworthy if you build it like a measurement instrument, not a vibe. Hamel Husain's guide distills the discipline:^[2]

1. Binary pass or fail, not a 1 to 5 Likert. Nobody consistently distinguishes a 3 from a 4; those numbers do not correlate with quality and are gamed by verbosity. Ask one question: "did the AI achieve the desired outcome, pass or fail?"
2. One principal domain expert. The authoritative human whose judgment defines "good." Do not proxy it with a PM, and do not play the expert yourself if you lack the domain.
3. Critiques are mandatory. Every verdict carries a written rationale, detailed enough to become a few-shot example in the judge prompt. Externalizing the reasoning is how the expert discovers their own implicit criteria.
4. Iterate to above 90% agreement. Expert labels roughly 30 examples, draft the judge using their critiques, measure agreement, refine. Alignment usually lands in about 3 iterations.
5. Score the judge with precision and recall, not raw agreement. On 90%-pass data, a judge that always says "pass" scores 90% agreement while catching zero failures. Track true-positive and true-negative rate separately.

Metrics beyond success rate

Success rate is the headline, but insufficient alone. Agents routinely trade latency and cost for accuracy, so you must see the whole frontier.^[1]

pass@k vs pass^k: the reliability wall

This is the distinction that defines 2025-era agent eval. Run each task n times. pass@k = "at least one of k tries succeeds," optimistic, and the right metric when a downstream verifier can pick the winner (for example, code that must pass tests). pass^k = "all k tries succeed," tau-bench's reliability metric, and the one that matters for unattended production where you do not get to retry.^[3][4]

Same trials, opposite story. An agent that succeeds 80% of the time per run climbs toward near-100% pass@k (any of k succeeds) yet collapses to roughly 2% pass^k (all 8 succeed). tau-bench's best agent had above 60% at pass^1 but below 25% at pass^8. pass@k asks "can it ever?"; pass^k asks "can I trust it every time?"

Benchmarks: capability ceilings, not your finish line

Public benchmarks are excellent sanity checks and capability ceilings, but they saturate, leak into training data, and get gamed (Goodhart's law: "when a measure becomes a target, it ceases to be a good measure"). Treat them as reference points and build a private eval set from your own failures as the thing you actually optimize. Here is the load-bearing map, one line each.

Check yourself