Sign in Subscribe
OpenTelemetry

How to Instrument Your Distributed AI Agent Workflows With OpenTelemetry-Native Tracing (And Finally Debug Cross-Agent Failures)

Scott Miller

• 11 min read

I have enough context to write a thorough, expert-level post. Here it is: ---

Picture this: your multi-agent AI pipeline just silently returned a wrong answer to a paying customer. Agent A called Agent B, which called a retrieval tool, which called an LLM, which hallucinated, which caused Agent C to make a cascading bad decision. Your on-call engineer opens the logs and finds ten thousand unstructured lines of INFO: agent completed step. Nobody knows where the failure actually started. Sound familiar?

This is the defining observability crisis of 2026. As agentic AI systems have moved from demos into production, engineering teams have discovered that the debugging tooling they relied on for microservices simply does not map cleanly onto the non-linear, dynamic, tool-calling, multi-hop nature of agent workflows. The answer is not more logs. The answer is OpenTelemetry-native distributed tracing designed specifically for agent topologies.

In this guide, you will learn exactly how to instrument a multi-agent system end-to-end using OpenTelemetry (OTel), structure your spans to capture agent-specific semantics, propagate trace context across agent boundaries, and set up a backend that lets your team pinpoint cross-agent failures in seconds rather than hours.

Why Standard Logging Fails Distributed AI Agents

Traditional application logging was designed for request-response systems where a single thread of execution produces a linear sequence of events. Distributed AI agent workflows break every one of those assumptions:

  • Non-linear execution: Agents invoke sub-agents, tools, retrievers, and LLMs in dynamic, sometimes recursive patterns. There is no single log file that captures the full picture.
  • Asynchronous hand-offs: Agent A may enqueue a task for Agent B over a message queue. The correlation between the two is invisible in flat logs unless you explicitly wire it up.
  • LLM latency opacity: A 4-second response time could mean a slow model, a long prompt, a rate-limit retry, or a network issue. Logs say "LLM call finished." Traces show you exactly which segment took how long.
  • Tool call fan-out: A single agent step might call five tools in parallel. Without structured spans, you cannot tell which tool failed or which succeeded first.
  • Context loss at boundaries: When an agent passes work to another service via HTTP or a queue, the trace context is typically dropped unless you explicitly propagate it.

OpenTelemetry solves all of these problems. It gives you a single causally-linked trace tree that spans every agent, every tool call, every LLM invocation, and every hand-off, regardless of what language or runtime each component uses.

Understanding the OTel Primitives You Actually Need

Before writing a single line of instrumentation code, make sure your team has a shared mental model of the three OTel primitives that matter most for agent workflows:

Traces and Spans

A trace is the complete record of a single workflow execution, from the moment a user request arrives until the final response is returned. A span is one unit of work within that trace. Spans form a parent-child tree. In an agent context, the top-level span might be "orchestrator received user request," and its children might be "Agent A: plan step," "Agent B: retrieve context," and "LLM: generate response."

Attributes

Attributes are key-value pairs attached to a span. They are where you encode agent-specific metadata: the agent name, the model used, the prompt token count, the tool name, the retrieval score, the decision made. Attributes are what transform a generic span into a queryable, filterable record of agent behavior.

Context Propagation

This is the mechanism by which the trace ID and span ID travel across process boundaries. OTel uses the W3C TraceContext standard (traceparent header) for HTTP and provides propagators for message queues. Without explicit propagation, your trace tree breaks at every agent boundary.

Step 1: Set Up Your OTel SDK and Collector

Start with a centralized OTel Collector that all your agents export to. This decouples your instrumentation from your backend and lets you route telemetry to Jaeger, Tempo, Honeycomb, or any other backend without changing agent code.

Here is a minimal otel-collector-config.yaml:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 512
  resource:
    attributes:
      - key: deployment.environment
        value: production
        action: upsert

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true
  logging:
    verbosity: detailed

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resource, batch]
      exporters: [otlp/jaeger, logging]

Next, initialize the OTel SDK in each agent process. Here is a Python example using the opentelemetry-sdk package:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

def configure_tracer(agent_name: str) -> trace.Tracer:
    resource = Resource.create({
        "service.name": agent_name,
        "service.version": "1.0.0",
        "ai.agent.type": "orchestrator",  # or "tool_agent", "retrieval_agent", etc.
    })

    provider = TracerProvider(resource=resource)
    exporter = OTLPSpanExporter(endpoint="http://otel-collector:4317")
    provider.add_span_processor(BatchSpanProcessor(exporter))
    trace.set_tracer_provider(provider)

    return trace.get_tracer(agent_name)

Call configure_tracer("my-agent-name") once at startup in each agent service. The service.name resource attribute is critical: it is what your tracing backend uses to identify which agent produced each span.

Step 2: Instrument Agent Steps With Semantic Spans

The most common mistake teams make is creating spans that are too coarse. A single span called "agent_run" tells you nothing useful. You need spans that map to the semantic units of agent work: planning, tool selection, tool execution, LLM calls, and reflection/evaluation steps.

Here is a pattern for a ReAct-style agent loop:

import json
from opentelemetry import trace
from opentelemetry.trace import SpanKind, Status, StatusCode

tracer = configure_tracer("research-agent")

def run_agent_step(agent_state: dict, step_number: int) -> dict:
    with tracer.start_as_current_span(
        f"agent.step",
        kind=SpanKind.INTERNAL,
        attributes={
            "agent.name": "research-agent",
            "agent.step.number": step_number,
            "agent.step.type": "react_loop",
            "agent.input.length": len(json.dumps(agent_state)),
        }
    ) as step_span:

        # Planning sub-span
        with tracer.start_as_current_span("agent.plan") as plan_span:
            thought = call_llm_for_planning(agent_state)
            plan_span.set_attribute("agent.thought", thought[:500])  # truncate for safety
            plan_span.set_attribute("agent.action_decided", thought.get("action"))

        # Tool execution sub-span
        tool_name = thought.get("action")
        tool_input = thought.get("action_input")

        with tracer.start_as_current_span(
            "agent.tool.call",
            kind=SpanKind.CLIENT,
            attributes={
                "agent.tool.name": tool_name,
                "agent.tool.input": str(tool_input)[:500],
            }
        ) as tool_span:
            try:
                tool_result = execute_tool(tool_name, tool_input)
                tool_span.set_attribute("agent.tool.result_length", len(str(tool_result)))
                tool_span.set_status(Status(StatusCode.OK))
            except Exception as e:
                tool_span.record_exception(e)
                tool_span.set_status(Status(StatusCode.ERROR, str(e)))
                raise

        step_span.set_attribute("agent.step.completed", True)
        return {**agent_state, "last_observation": tool_result}

Notice the key practices here: spans are named with a consistent dot-notation hierarchy (agent.step, agent.plan, agent.tool.call), exceptions are recorded on the span rather than just logged, and the span status is explicitly set. These three habits alone will cut your mean-time-to-debug in half.

Step 3: Instrument Every LLM Call With Rich Attributes

LLM calls are the most expensive and most failure-prone operations in any agent workflow. They deserve their own dedicated span with a rich set of attributes. The OpenTelemetry community's GenAI semantic conventions (now stable as of early 2026) define a standard set of attribute names you should follow for interoperability:

def call_llm(prompt: str, model: str, temperature: float = 0.7) -> dict:
    with tracer.start_as_current_span(
        "gen_ai.completion",
        kind=SpanKind.CLIENT,
        attributes={
            # GenAI semantic conventions
            "gen_ai.system": "openai",          # or "anthropic", "google", etc.
            "gen_ai.request.model": model,
            "gen_ai.request.temperature": temperature,
            "gen_ai.request.max_tokens": 1024,
            # Prompt metadata (never log raw prompts in production without PII review)
            "gen_ai.prompt.token_count": count_tokens(prompt),
            "gen_ai.prompt.template_id": "research_planner_v3",
        }
    ) as llm_span:
        import time
        start = time.perf_counter()

        response = llm_client.complete(prompt=prompt, model=model, temperature=temperature)

        latency_ms = (time.perf_counter() - start) * 1000

        llm_span.set_attribute("gen_ai.response.model", response.model)
        llm_span.set_attribute("gen_ai.usage.prompt_tokens", response.usage.prompt_tokens)
        llm_span.set_attribute("gen_ai.usage.completion_tokens", response.usage.completion_tokens)
        llm_span.set_attribute("gen_ai.response.finish_reason", response.choices[0].finish_reason)
        llm_span.set_attribute("gen_ai.response.latency_ms", round(latency_ms, 2))

        # Flag potential issues
        if response.choices[0].finish_reason == "length":
            llm_span.set_attribute("gen_ai.response.truncated", True)
            llm_span.add_event("Response truncated at max_tokens limit")

        return response

The gen_ai.usage.prompt_tokens and gen_ai.usage.completion_tokens attributes are particularly valuable. When aggregated across a trace, they give you the true cost of a single agent workflow execution, which your finance team will thank you for.

Step 4: Propagate Context Across Agent Boundaries

This is where most teams fail. They instrument each agent perfectly in isolation but forget to carry the trace context across the network calls between agents. The result is a forest of disconnected traces instead of one unified tree.

HTTP-Based Agent-to-Agent Calls

If your agents communicate over HTTP (REST or gRPC), use OTel's built-in instrumentation libraries. For Python with httpx:

from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor

# Call this once at startup
HTTPXClientInstrumentor().instrument()

# Now every httpx request automatically injects the traceparent header
import httpx

async def call_downstream_agent(agent_url: str, payload: dict) -> dict:
    # The current span context is automatically propagated in the request headers
    async with httpx.AsyncClient() as client:
        response = await client.post(agent_url, json=payload)
        return response.json()

On the receiving agent's side, use the corresponding server instrumentation:

from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from fastapi import FastAPI

app = FastAPI()
FastAPIInstrumentor.instrument_app(app)

@app.post("/run")
async def run_agent(request: AgentRequest):
    # The incoming traceparent header is automatically extracted.
    # Any spans created here are children of the upstream agent's span.
    with tracer.start_as_current_span("agent.handle_request"):
        result = await process_request(request)
        return result

Queue-Based Agent Hand-offs

When agents communicate via a message queue (Kafka, RabbitMQ, Redis Streams), you must manually inject and extract context because there is no shared HTTP header mechanism. Here is the pattern:

from opentelemetry.propagate import inject, extract
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

# PRODUCER SIDE: Inject context into the message headers
def publish_agent_task(queue, task_payload: dict):
    carrier = {}
    inject(carrier)  # Populates carrier with traceparent and tracestate

    message = {
        "payload": task_payload,
        "otel_context": carrier,  # Embed trace context in the message envelope
    }

    queue.publish(message)

# CONSUMER SIDE: Extract context from the message and restore it
def consume_agent_task(message: dict):
    carrier = message.get("otel_context", {})
    ctx = extract(carrier)  # Reconstruct the OTel context

    with tracer.start_as_current_span(
        "agent.consume_task",
        context=ctx,  # This makes the new span a child of the upstream span
        kind=SpanKind.CONSUMER,
    ) as span:
        process_task(message["payload"])

This pattern works identically for Kafka message headers, SQS message attributes, and Redis stream fields. The key insight is that the trace context is just a string that can be serialized into any envelope format your queue supports.

Step 5: Add Span Events for Agent Decision Points

Span attributes capture the state at the start and end of a span. But agents make decisions during execution that you also need to capture. Use span events for these intermediate milestones:

with tracer.start_as_current_span("agent.reasoning_loop") as span:
    for iteration in range(max_iterations):
        observation = get_observation()

        # Record each reasoning iteration as a timestamped event
        span.add_event(
            "agent.iteration",
            attributes={
                "iteration.number": iteration,
                "iteration.observation_length": len(observation),
                "iteration.decided_to_continue": True,
            }
        )

        if should_stop(observation):
            span.add_event("agent.loop_terminated", attributes={
                "termination.reason": "goal_achieved",
                "termination.iteration": iteration,
            })
            break

Span events appear as timestamped markers on the span's timeline in tools like Jaeger and Grafana Tempo. They are invaluable for understanding the internal dynamics of a long-running agent loop without creating hundreds of child spans.

Step 6: Structure Your Span Naming Convention as a Team Contract

Inconsistent span naming is the silent killer of tracing programs. When one engineer names spans llm_call, another uses LLMCall, and a third uses call-to-openai, your queries and dashboards break and your team stops trusting the data. Establish a naming contract before you instrument anything:

  • Use dot-notation namespaces: agent.step, agent.tool.call, gen_ai.completion, retrieval.query
  • Prefer nouns over verbs for span names: agent.tool.execution rather than execute_tool
  • Include the agent role in the service name, not the span name: The span name describes the operation; the resource attribute describes who performed it
  • Never put variable data in span names: Use agent.tool.call with a agent.tool.name attribute, not call_web_search_tool. Variable span names destroy cardinality in your backend.
  • Document your conventions in a shared wiki page and enforce them in code review

Step 7: Build a Debugging Runbook Around Your Traces

Instrumentation without a workflow for using it is just expensive data collection. Here is a concrete debugging runbook your on-call engineers can follow when a cross-agent failure occurs:

Step 1: Find the Root Trace

Every agent workflow execution should be tagged with a workflow.run_id attribute on the root span. When an incident is reported, use this ID to pull the full trace in your backend. In Jaeger, this is a single search query. In Grafana Tempo, use TraceQL: { .workflow.run_id = "abc-123" }.

Step 2: Identify the First Error Span

Sort the trace timeline by span start time and filter for spans with status.code = ERROR. The earliest error span is your primary suspect. Check its recorded exception and attributes.

Step 3: Walk the Causal Chain

From the error span, walk up the parent chain to understand what triggered that operation. Walk down the child spans to see what downstream effects the error caused. This causal chain is the single most valuable thing distributed tracing gives you over logs.

Step 4: Check LLM Attributes for Semantic Failures

Not all agent failures are exceptions. If the trace shows no errors but the output was wrong, check the gen_ai.response.finish_reason attributes. A finish_reason: length means the model was cut off. Check gen_ai.response.latency_ms for timeout-adjacent behavior. Check your custom agent.thought attributes to see what the agent decided and why.

Step 5: Correlate With Metrics

OTel traces and metrics share the same resource attributes, so you can pivot from a specific failing trace to a dashboard showing whether this failure pattern is isolated or systemic. Look for spikes in gen_ai.usage.prompt_tokens that correlate with latency degradation.

Common Pitfalls and How to Avoid Them

Pitfall 1: Logging Raw Prompts as Span Attributes

It is tempting to capture the full prompt text on every LLM span. Resist this. Prompts often contain PII, customer data, or confidential context. Instead, log the prompt template ID and the token count. If you need full prompt capture for debugging, route it to a separate, access-controlled store and link it via a reference ID attribute on the span.

Pitfall 2: Creating Spans Inside Hot Loops

If your agent runs an embedding similarity search over 10,000 vectors in a tight loop, do not create a span per vector. Create one span for the entire retrieval operation with aggregate attributes. Excessive span creation adds latency and overwhelms your collector.

Pitfall 3: Forgetting to Shut Down the TracerProvider

The BatchSpanProcessor buffers spans in memory and flushes them on a timer. If your agent process exits before the flush interval, you lose the last batch of spans. Always call provider.shutdown() in a shutdown hook or atexit handler.

Pitfall 4: Using the Same Tracer Instance Across Threads Without Care

OTel's Python SDK uses context variables for span propagation, which are thread-local by default. If you spawn threads or use asyncio without properly propagating the context, child spans will be orphaned. Use context.copy() and pass the context explicitly when crossing thread or task boundaries.

Choosing a Backend: What Works Best for Agent Traces in 2026

Your choice of tracing backend matters more for agent workflows than it does for traditional microservices, because agent traces are deeper, wider, and carry more custom attributes. Here is a quick comparison of the leading options:

  • Grafana Tempo + Grafana UI: The best open-source choice for teams already running the Grafana stack. TraceQL is expressive enough to query across agent attributes. Scales horizontally with object storage backends.
  • Jaeger: Mature, battle-tested, and easy to self-host. The UI is less powerful than Grafana's but perfectly adequate for trace exploration. Best for teams that want simplicity over features.
  • Honeycomb: The strongest commercial option for high-cardinality attribute querying. If you are storing 50+ attributes per span (which agent traces often do), Honeycomb's columnar query engine shines. The cost scales with event volume, so model your span counts carefully.
  • Datadog APM: Best choice for teams already paying for Datadog. The AI Observability product added native GenAI span support in late 2025, making it a strong fit for agent workflows without custom configuration.

Conclusion: Traces Are the Source of Truth for Agent Systems

Distributed AI agent workflows are fundamentally different from the stateless microservices that most observability tooling was designed for. They are stateful, recursive, semantically rich, and failure modes are often logical rather than technical. A stack trace tells you a function threw an exception. A well-structured OTel trace tells you that Agent B hallucinated a tool name because Agent A passed it a truncated context, which happened because the retrieval agent returned too many tokens, which happened because someone changed the chunk size in a configuration file three days ago.

That level of causal clarity is not possible with logs. It requires distributed tracing done right: consistent span naming, rich semantic attributes, explicit context propagation at every agent boundary, and a team-wide commitment to treating traces as the authoritative source of truth for system behavior.

The good news is that the tooling has never been more mature. OpenTelemetry's GenAI semantic conventions are now stable, the collector ecosystem is robust, and every major backend has native support for high-cardinality attribute queries. There has never been a better time to invest in this layer of your stack.

Start small. Pick your most complex agent workflow, instrument it end-to-end using the patterns in this guide, and watch your next debugging session shrink from hours to minutes. That first win will sell the rest of your team on making OTel-native tracing a non-negotiable standard for every agent you ship.

Sign up for more like this.

Enter your email
Subscribe