Agent traces have a runtime-decided shape, token-based cost, and cross-process tool hops that conventional APM was never built for — here is what OpenTelemetry's gen_ai.* conventions actually pin down, and where tracing an agent loop still breaks in production.
An agent that silently retries a failing tool call six times before giving up looks, from the outside, exactly like an agent that succeeded on the first try: the user gets an answer either way. The difference is 40x the token spend and a latency spike nobody can explain from the application logs, because the application logs only recorded the final response. This is the normal state of agent observability at most shops — not broken, just absent. Standard request logging was built for systems that make one call and return one answer. An agent loop makes an unpredictable number of calls, to an unpredictable number of tools, in an order the code itself doesn't decide in advance. If your tracing doesn't model that shape, you're debugging blind.
This post is about what it actually takes to trace an agent loop properly: why conventional APM instrumentation doesn't fit, what OpenTelemetry's GenAI semantic conventions standardize (and where they stop), and where the model still breaks down in practice.
A traditional web request gets one span: request in, response out, maybe a few child spans for a database query or a cache lookup. The span tree is shallow and its shape is known ahead of time — you wrote the code, you know what it calls.
An agent loop inverts both properties. The tree depth depends on how many reasoning-and-tool-call round trips the model decides to take, which is not fixed at deploy time — it's a runtime decision made by the model, conditioned on what the previous tool call returned. A request that resolves in one hop today might take five hops tomorrow because a search tool returned a worse result. And the cost model is different: a conventional span has a database query with a fixed cost curve; an LLM call has cost measured in tokens, which vary independently from latency in ways that matter for billing, not just for performance.
That means an agent trace has to capture things a normal APM span doesn't have a field for: which model served this specific step (agents increasingly route between models mid-conversation), how many input and output tokens it consumed, what tool it decided to call and why, and whether the result was good — a dimension no HTTP span has ever needed, because HTTP responses don't have a quality score.
Flattened into a waterfall, a single agent turn that plans, searches, and answers looks like this — a root span for the whole run, with LLM calls and tool calls as children, each with wildly different durations and no fixed count:
Two things about this tree matter more than they look like they should. First, there is no way to know its shape ahead of time — you can't hand-write "expect 5 children" into a dashboard the way you would for a fixed pipeline. Second, one of those children crosses a process boundary: the tool.call mcp:validate span represents a call to an external MCP server, which means the trace has to survive being serialized into a JSON-RPC message, sent over the wire, and reconstructed on the other side with the same trace ID — or you lose the thread entirely and get two disconnected traces instead of one.
OpenTelemetry has a gen_ai.* semantic convention namespace specifically for this shape of workload. It doesn't invent a new tracing model — spans, trace IDs, and context propagation are still plain OpenTelemetry — it standardizes the attribute names so that a trace produced by one team's agent framework means the same thing as a trace produced by another's. Without that, every vendor invents its own field names for "how many tokens did this cost," and no tool downstream can aggregate across them.
The attributes that matter most in practice:
gen_ai.system — which provider served the call (anthropic, openai, azure.ai.openai, and so on)gen_ai.request.model / gen_ai.response.model — the model you asked for versus the model that actually answered, which matter separately once you're using automatic fallback or routing between model tiersgen_ai.usage.input_tokens / gen_ai.usage.output_tokens — the fields that make cost dashboards possible at allgen_ai.operation.name — chat, text_completion, embeddings, distinguishing what kind of call this span representsgen_ai.tool.name / gen_ai.tool.call.id — which tool was invoked and the ID the model used to correlate its own call with the result it gets backDeliberately absent from that list: prompt and completion text. Earlier drafts of the convention put full message content directly on span attributes, and that turned out to be a bad idea for two reasons — spans get truncated or dropped by collectors when they exceed size limits, and prompt content routinely contains PII you don't want sitting in your tracing backend's default retention window. The convention has moved toward recording content as structured log records or span events correlated to the span by trace and span ID, which lets you route content into a system with different (usually shorter, access-controlled) retention than the trace metadata itself.
A minimal instrumented tool-call span looks like this using the OpenTelemetry Python SDK:
from opentelemetry import trace
tracer = trace.get_tracer("agent-loop")
with tracer.start_as_current_span("tool.call") as span:
span.set_attribute("gen_ai.tool.name", "web_search")
span.set_attribute("gen_ai.tool.call.id", tool_call.id)
result = run_tool(tool_call)
span.set_attribute("tool.result.status", "success" if result.ok else "error")
# content goes to a log record linked to this span, not a span attribute
logger.info("tool.result", extra={
"trace_id": span.get_span_context().trace_id,
"span_id": span.get_span_context().span_id,
"content": result.payload,
})
The pattern generalizes: every gen_ai.chat call gets the same treatment, with token usage and model identity as attributes and the actual prompt/completion routed to logs. That split is what lets a cost dashboard stay fast — it's aggregating small numeric attributes across millions of spans — while a separate, access-controlled system handles "show me exactly what the model saw on this specific run" for debugging.
The interesting failure mode shows up the moment a tool call leaves your process. If an agent calls a tool over MCP, the client-side span (tool.call mcp:validate in the diagram above) and whatever the MCP server does to handle that call are two separate processes with two separate tracing runtimes. Nothing in the MCP spec mandates a tracing mechanism, so by default those are two disconnected traces — the client sees "I called a tool, it took 30ms," and the server, if it's instrumented at all, has its own trace with no idea it was invoked as a step inside someone else's agent run.
The fix is the same one distributed tracing has used for HTTP for a decade: propagate the W3C Trace Context (traceparent) header alongside the request. When an MCP transport runs over HTTP, there's nothing stopping you from attaching that header to the request the same way you would for any REST call, and having the server's tracing SDK pick it up and continue the same trace ID instead of starting a new one. A tool call to a well-instrumented MCP server — Utilix's MCP server, for instance, exposing REST-backed developer utilities as callable tools — looks identical in the resulting trace whether it's validating a Luhn checksum or hitting a third-party API: the span records latency and success, not what the tool internally does. That symmetry is the point — the agent's tracing shouldn't need to know anything about the tool's implementation to stay coherent across the hop.
Where this gets harder is stdio-transport MCP servers, which are common for local tool integrations and don't have an HTTP header to carry context on. There, propagation has to happen inside the JSON-RPC envelope itself — stuffing trace context into a custom field in the request params — which works, but only if both sides agree to read it, and most MCP server implementations today don't.
Three things remain genuinely unsolved in most production setups.
Streaming spans. A chat span technically starts when the request goes out and should end when the response finishes, but a streamed response might take 8 seconds to fully arrive, and you often want progress visibility before the span closes — time-to-first-token separately from total completion time. Most SDKs handle this by recording gen_ai.server.time_to_first_token as an attribute set mid-stream and closing the span only once the last token arrives, but tooling for visualizing that in-flight state is inconsistent across backends.
Sub-agent fan-out. When an orchestrator spins up parallel sub-agents — a common pattern for research or code-review tasks that split work across workers — each sub-agent's trace needs to be a child of the orchestrator's span, not a sibling top-level trace. Getting that right means propagating trace context across whatever mechanism launches the sub-agent (a subprocess, a queue message, a separate service call), and it's easy to get wrong in exactly the way that makes a five-minute fan-out look, in your tracing backend, like five unrelated one-minute traces.
Attributing quality, not just cost. Token counts and latency are cheap to attach to a span in real time. Whether the answer was any good usually isn't known until later — an LLM-as-judge eval, a user thumbs-down, a downstream task failing three steps later. The common pattern is to write eval scores back onto the original span or trace ID asynchronously, after the fact, which means your tracing backend has to support attribute updates on spans that have already closed. Not every backend does this cleanly, and it's the single biggest gap between "we have traces" and "we can actually debug why this agent run produced a bad answer."
| Tool | Hosting | Built on OTel? |
|---|---|---|
| LangSmith | Managed SaaS | Own schema, not OTel-native |
| Langfuse | Open-source, self-hostable or managed | OTel-compatible ingestion |
| Arize Phoenix | Open-source, self-hostable or managed | Built on OpenInference, OTel-based |
| Datadog LLM Observability | Managed SaaS (part of Datadog APM) | OTel-compatible ingestion |
| Helicone | Open-source proxy, self-hostable or managed | Proxy-based, not span-model tracing |
None of these are interchangeable despite solving overlapping problems — a proxy-based tool like Helicone captures every request that flows through it without any code-level instrumentation, which is fast to adopt but blind to anything that doesn't go through the proxy (like an MCP tool call). A span-model tool built on OpenTelemetry gets you the cross-process propagation described above, but only if you actually instrument the boundary, which is manual work no proxy can do for you.
Instrumenting an agent loop isn't "add tracing to an LLM call" — it's accepting that the trace tree's shape is a runtime decision, that cost and quality are first-class span dimensions HTTP tracing never needed, and that the moment a tool call leaves your process (an MCP server being the common case), you need trace context propagated across that boundary by hand or you get two disconnected traces instead of one. OpenTelemetry's gen_ai.* conventions solve the naming problem so tools can interoperate; they don't solve fan-out, streaming, or after-the-fact quality attribution for you. If your agent's tracing setup can already answer "which specific step burned the tokens and was the result any good," you're ahead of most production deployments — that gap, not model quality, is usually why agent debugging still feels like guesswork.