Observability and tracing

When an agent misbehaves it rarely throws a stack trace. It just quietly does the wrong thing across ten steps. You cannot debug, cost, or improve what you cannot see. This lesson makes an agent run legible: the trace, the span, the OpenTelemetry standard that names them, and how to read a waterfall to spot a bug in seconds.

Why agents need their own observability

Traditional monitoring (APM) assumes deterministic code paths and error codes: a request either returns 200 or throws. Agents break both assumptions. They are non-deterministic, so the same input can produce a different sequence of tool calls. They run a multi-step, stateful loop of reason, act, observe, reason again. And their worst failures are silent: goal drift, context loss, and quality degradation emit no error code at all.^[9]

So the unit of observability is not the log line, it is the trace. Hamel Husain's single most contrarian piece of advice: get data flowing first, "it doesn't matter whether you use an observability tool or just log to files," because you cannot do error analysis or write evals on runs you never recorded.^[1]

Trace and span: the two words that matter

A trace is one whole agent run. It is a tree of spans, each one a unit of work with a start time, an end time, and a bag of attributes. For agents, three span types cover almost everything: the agent turn, an LLM call, and a tool call. Every span records the same six load-bearing facts, and reading a trace is mostly a matter of scanning them.

Two attributes do outsized work. finish_reason tells you the model's intent: tool_calls means "I want to act" (an execute_tool span should follow); stop means "I'm done"; length means it was truncated mid-thought. And the per-span input token count is your context and cost meter: it grows every step as history accumulates, so a sudden jump flags a bloated payload.

The standard: OpenTelemetry GenAI conventions

You do not want a bespoke logging format per vendor. OpenTelemetry (OTel) formed a GenAI Special Interest Group in April 2024 and now defines semantic conventions for agent telemetry: a shared vocabulary of span names and attributes.^[2] This is the vendor-neutral wire format that Langfuse, Arize Phoenix, Datadog, Honeycomb, and New Relic all consume: instrument once, view anywhere.

The canonical agent trace is a three-level tree. Learn this shape; it is the mental model for every tool in this lesson.

The attribute gen_ai.operation.name is the discriminator that names each span. The rest are the facts above, with exact standardized keys:^[3]

A parallel spec, Arize's OpenInference, powers Phoenix and predates the OTel effort. The two are converging but not identical, worth knowing both names exist when you compare tools.^[8]

# structural attributes (safe) - the OTel GenAI convention
{
  "gen_ai.operation.name":        "chat",
  "gen_ai.request.model":         "claude-sonnet-4",
  "gen_ai.usage.input_tokens":     2400,
  "gen_ai.usage.output_tokens":    88,
  "gen_ai.response.finish_reasons": ["tool_calls"],
  "start_time": 2.1, "end_time": 3.0   # -> 0.9s latency for this call
}

What every agent span should capture

A span is only as useful as the attributes you attach to it. For agent work the high-value fields are consistent across the ecosystem, and capturing them is the difference between a trace you can debug and a pretty timeline you cannot.

1. Inputs and outputs. The full prompt sent to the model (system, history, and retrieved context) and the raw completion it returned. This is the single most important thing to record and the one most often truncated away.
2. Tool calls. The tool name, the exact arguments the model chose, and the raw result or error that came back. Most agent bugs are a wrong argument or a tool result the model then misreads.
3. Model metadata. Model id, temperature and sampling settings, and the stop reason, so a run is reproducible and you can tell a truncation from a real stop.
4. Tokens and cost. Prompt and completion token counts per call, which drive both the bill and the latency and let you find the step that is quietly blowing the budget.
5. Status and latency. Success or error, the error type on failure, and wall-clock duration, so you can rank spans by what is slow or breaking.

Reading a trace waterfall

Every backend renders a trace the same way: a waterfall, spans laid out on a horizontal time axis, indented to show nesting. Width is duration; the tree is the agent's control flow. Here is one real-shaped run of a research agent answering "Compare the battery life of the top 3 flagship phones." Read top to bottom, and watch the input token count climb.

Spotting the bug: a redundant loop

The trace makes the defect obvious. execute_tool #1 already ran web_search(q="phone battery 2026") and got results. Yet chat #2 read those results and re-proposed the identical search, so execute_tool #2 ran the same query again. That is a redundant loop: no step errored, but the model duplicated work it had already done.^[9]

This is the whole game. Anthropic found the same class of bug in their multi-agent system, subagents "duplicating work" and one query spawning "50 subagents," and the fix was the same: make the waste visible in traces, then constrain it.^[4] A note of caution for framework users: agent frameworks "create extra layers of abstraction that can obscure the underlying prompts and responses, making them harder to debug,"^[5] so keep the raw prompt and response visible in your trace (that is what content capture is for).

A reading checklist

Open any trace and run this scan. It turns a wall of spans into a diagnosis in under a minute.

1. Start at the root. Total latency, total tokens, total cost, final finish_reason. Is the end-state actually correct?
2. Scan the finish reasons. Every tool_calls should be followed by an execute_tool; a stray length means a truncated call that likely corrupted what came after.
3. Follow the token curve. Input tokens should grow gently. A sudden jump is a bloated tool payload dumped into context (context rot).^[6]
4. Hunt for repeats. Identical (tool, args) spans are a redundant loop; the same error string recurring is an error cascade where one bad step silently corrupts every later step.
5. Inspect the widest bars. The longest span owns your latency; the largest chat input owns your cost. Optimize those, ignore the rest.
6. Read the messages on the suspect span. Only now open the (redacted) content to confirm the root cause.

The tooling landscape

Hamel's advice stands: do not build custom logging infra, observability is not your product, so adopt a platform.^[1] Because they mostly speak OTel/OTLP now, the choice is about workflow fit, not lock-in. The five you will hear most, segmented by job-to-be-done:^[8]

Check yourself