Agent Observability: What to Trace When Nobody Is Watching

A failing web service throws a 500 and pages somebody. A failing agent returns a confident paragraph and nobody notices for a week. That asymmetry is the entire argument for agent observability, and it is why we instrument an agent before we let it run unattended.

The good news is that 2026 is the year this stopped requiring invention. There is now a standard vocabulary for what an agent run should emit, and the tools most teams already run emit it.

What is agent observability?

Agent observability is capturing every model call, tool call, and decision an agent makes as structured spans, so that a run can be replayed after the fact. Not "the agent ran and exited zero," but "here is the exact path it took, what it asked the model, what each tool returned, and where the tokens went."

The distinction that matters in practice: logs tell you the agent ran. Traces tell you what it did and why it took the path it took. When an agent produces a bad answer, the log will look identical to the day it produced a good one. The trace will not.

Why do agent failures stay invisible?

Because a wrong answer and a right answer have the same shape. Same status code, same content type, roughly the same latency, both fluent. Nothing in the response envelope signals that the output is garbage.

The failure modes that actually hurt in production are all quiet ones:

• A tool returned an empty result and the agent reasoned around the absence instead of stopping. No error was raised. The answer is now confidently built on nothing.
• A retry loop fired four times against a flaky endpoint. It succeeded, so nobody looked, and the run cost five times what it should have.
• A response hit the token limit and truncated mid-thought. The agent used the partial output as though it were complete.
• A conditional read false because an upstream field changed shape, so an entire step silently never ran.

None of those page anyone. Nobody files a ticket about an agent that is subtly wrong. They just quietly stop trusting it, and six weeks later somebody asks why the team went back to doing it by hand.

Which spans and attributes actually matter?

OpenTelemetry's GenAI semantic conventions define the vocabulary, which means you can instrument once and send the data anywhere that speaks OTLP.

The span types map cleanly onto how agents actually work:

• invoke_agent is the parent span for the whole run.
• chat wraps each individual model call.
• execute_tool wraps each tool invocation.

Nest them and a run reads as a tree instead of a wall of log lines. You can see that the agent called the search tool three times before the model call that produced the wrong answer, which is the thing you actually needed to know.

The attributes worth building alerts on are a short list: gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and gen_ai.response.finish_reasons. On the metrics side, gen_ai.client.operation.duration and gen_ai.client.token.usage cover latency and spend.

Finish reason is the underrated one. A truncated response is a silent quality failure that never raises an error anywhere in your stack, and it is visible in exactly one place: the finish reason on the span. If you instrument nothing else, instrument that.

Two practical notes. First, the conventions are in use today but still under active development, so pin what you build dashboards on and expect attribute names to move. Second, you likely already have data flowing: VS Code Copilot, OpenAI Codex, and Claude Code all emit OpenTelemetry data under these conventions, so your coding agents are readable in any OTLP backend before you write a line of custom instrumentation.

How do you tell a broken agent from an expensive one?

They look similar on a dashboard and they need different fixes. Four numbers separate them.

Cost per completed task, not cost per call. An agent that retries four times looks cheap per call and terrible per outcome. Per-call cost is the metric that makes a broken agent look efficient.

Tool error rate and empty-result rate, tracked separately. An empty result is not an error, which is precisely why it is the more dangerous of the two. Errors get handled. Empty results get reasoned around.

Step count distribution. Not the average, the distribution. A widening tail is the earliest signal of a loop, and it shows up well before the cost graph does.

Human intervention rate. How often did a person have to step in? This is the number an operator actually cares about, and it is the one most teams never measure because it does not live in the system.

An agent with a stable step count, a low empty-result rate, and a rising cost per task is expensive. An agent with a widening step tail and a climbing intervention rate is broken. Same spend curve, different problem.

What we instrument on our own agents

We run agents in production across three platforms: Smile PreVue, Howdy Dispatch, and RunLink. Behind them sits a set of agents doing unglamorous work on a schedule. A research agent that assembles editorial briefs. A publisher that writes and ships content. Weekly search index audits. Documentation-sync agents that keep project docs current against the actual code. Uptime monitors.

The pattern we keep across all of them, at concept level:

Every run emits a trace. Every quality gate emits an explicit pass or hold, never a silent skip. And a held output is louder than a shipped one.

That last rule came from a real failure, and it is the most useful thing we have learned about this. We built a quality gate that could stop bad content from publishing, and it worked. It caught things. But when it held something back, the run completed successfully and nothing announced it. From the outside, a day where the gate saved us looked exactly like a day where nothing was scheduled. We were flying blind on our own safety mechanism.

A silent hold is functionally identical to a silent failure. Both leave you with no signal and a false sense that the system is fine. Now a hold posts louder than a publish does, with the reason attached.

Traces are not evals. What is the difference?

These three get conflated constantly, and they answer genuinely different questions.

You need all three. Traces without evals means you can see exactly how the agent produced a wrong answer but nothing tells you it was wrong. Evals without traces means you know quality dropped but you cannot find where. Monitors without either means you know it is up, which is the least useful of the three facts. We have written separately about eval-driven agent development, so we will not re-explain evals here.

Where to start if you are running one agent today

Instrument the boundary first. One span per run, one span per tool call, plus token counts and finish reason on each model call. That alone catches most of what actually goes wrong, and it is an afternoon of work rather than a project.

Pick a backend that speaks OTLP. The point of a standard vocabulary is that you are not buying into a proprietary schema you will have to migrate off later.

Which brings up the cautionary note of the year. Amazon Bedrock Agents, launched in November 2023, became Bedrock Agents Classic and closed to new customers on July 30, 2026. Existing agents keep running and the model catalog is frozen at that date, with AWS pointing new production work at AgentCore. Nothing dramatic happened to anyone who was already running, but the lesson generalizes: the framework layer is churning, and it will keep churning. Your telemetry layer should not churn with it. Keep the observability vocabulary standard and vendor-neutral even when the framework underneath is not, and a framework migration becomes an inconvenience instead of a rewrite.

The honest summary is that agent observability is not sophisticated work. It is the boring instrumentation you would have put on any other production system, applied to a system that happens to fail more quietly than most. The sophistication is in deciding to do it before the agent is already running unattended.

If you are about to put an agent in front of real work and you want a second set of eyes on what to watch, that is a good conversation to have early. Read how we approach it in our agent orchestration methodology, or get in touch.