AI Agents

How to Build an AI Agent Observability Pipeline with Distributed Trace Correlation in H2 2026: A Step-by-Step Guide for Enterprise Backend Teams

Scott Miller

08 Aug 2026 — 13 min read

Multi-agent systems have quietly become the backbone of enterprise automation in 2026. Orchestrators spawn sub-agents, sub-agents call tools, tools invoke external APIs, and somewhere in the middle a workflow fails silently, a token budget explodes, or a causality chain breaks in a way that your existing Grafana dashboard cannot explain. The root cause? Your observability pipeline was built for microservices, not for agents.

This guide is specifically for backend engineering teams who already have a working OpenTelemetry (OTel) collector infrastructure and now need to extend it to capture the full causal story of multi-agent workflows, including cross-agent span propagation, LLM call attribution, tool invocation latency, and workflow-level causality context, without forking your pipeline or maintaining a shadow observability stack.

By the end of this tutorial, you will have a working, production-grade AI agent observability pipeline that correlates distributed traces across agent boundaries, preserves cross-workflow causality, and feeds into your existing backends (Jaeger, Tempo, Honeycomb, or Datadog) with zero loss of trace context.

Why Standard Distributed Tracing Breaks for Multi-Agent Systems

Before writing a single line of instrumentation code, it is worth understanding why the problem is hard. In a classic microservice trace, the causal chain is linear and synchronous enough that W3C TraceContext headers propagate cleanly through HTTP or gRPC calls. Spans nest predictably. The parent-child relationship is unambiguous.

Multi-agent workflows violate almost every assumption that model makes:

Asynchronous fan-out: An orchestrator agent may spawn three sub-agents concurrently via a message queue. There is no HTTP call to carry a traceparent header.
Cross-process causality: A planning agent and an execution agent may run in separate containers, separate Kubernetes namespaces, or even separate cloud accounts. The trace context must survive serialization across all of these boundaries.
Non-deterministic re-entry: Agents can be retried, replanned, or looped. A span that appears to be a "child" of another span may actually represent the third iteration of the same logical step, which breaks naive parent-child span linking.
LLM call attribution: A single LLM completion can be triggered by multiple upstream agents contributing to the same prompt context window. Attributing cost, latency, and errors to the correct originating workflow requires more than a single parent span ID.
Tool call opacity: When an agent calls a tool (a code interpreter, a web browser, a database), the tool itself is often not OTel-instrumented. You lose the trace at the boundary.

The solution is not to replace OpenTelemetry. It is to extend it correctly using a combination of semantic conventions for GenAI (now stable in OTel 1.30+), span links for non-linear causality, baggage propagation for workflow-level context, and a thin agent-aware instrumentation layer that wraps your existing agent framework.

Architecture Overview: The Four-Layer Pipeline

Think of the observability pipeline as four cooperating layers stacked on top of your existing OTel infrastructure:

Agent Instrumentation Layer: SDK-level wrappers that emit spans, metrics, and logs conforming to OTel GenAI semantic conventions from inside each agent.
Context Propagation Layer: A propagation strategy that carries traceparent, tracestate, and custom workflow baggage across async message queues, event buses, and direct agent-to-agent calls.
Collector Enrichment Layer: An OTel Collector pipeline with custom processors that enrich agent spans with workflow metadata, resolve span links, and deduplicate retry spans.
Backend Correlation Layer: Trace backend configuration (Jaeger, Tempo, or Honeycomb) that groups spans by workflow ID, agent ID, and causality chain for human-readable trace visualization.

The diagram below represents the data flow in prose form: an orchestrator agent starts a root span, injects context into a message envelope, a sub-agent consumes the envelope, extracts context, creates a child span linked back to the orchestrator span, calls an LLM (emitting a GenAI span), calls a tool (wrapped by a tool instrumentation shim), and all spans flow through the OTel Collector into your existing backend.

Step 1: Establish Your OTel GenAI Semantic Convention Baseline

The OTel GenAI semantic conventions (stabilized through the gen_ai namespace in the 1.30 specification) define the standard attribute names for LLM and agent operations. Using these conventions ensures your agent spans are compatible with any OTel-aware backend and with community tooling like the OTel GenAI instrumentation libraries.

Key attributes you must emit on every LLM call span:

gen_ai.system: The model provider (e.g., openai, anthropic, google_vertex).
gen_ai.request.model: The specific model version (e.g., gpt-4.5-turbo, claude-4-opus).
gen_ai.usage.input_tokens and gen_ai.usage.output_tokens: Token counts for cost attribution.
gen_ai.operation.name: One of chat, text_completion, embeddings, or tool_call.
gen_ai.agent.id: A stable identifier for the agent instance emitting the span.
gen_ai.agent.name: A human-readable agent role name (e.g., planner, executor, critic).

Add these custom attributes for workflow-level causality, which go beyond the base spec:

workflow.id: A UUID generated at workflow initiation time, propagated through all agents in the workflow.
workflow.step: An integer or semantic label indicating which logical step of the workflow this span belongs to.
workflow.iteration: Incremented each time a step is retried or re-planned, preserving causality across loops.
agent.parent_id: The gen_ai.agent.id of the spawning agent, distinct from the OTel parent span ID.

Step 2: Instrument Your Agent Framework with a Thin Wrapper

Rather than instrumenting every agent class individually, build a reusable instrumentation wrapper that any agent in your system can inherit or compose. The following example uses Python and the opentelemetry-sdk package, but the pattern applies equally to TypeScript/Node.js, Go, or Java agent implementations.

Install Dependencies

pip install opentelemetry-sdk \
            opentelemetry-exporter-otlp-proto-grpc \
            opentelemetry-instrumentation-httpx \
            opentelemetry-semantic-conventions-ai

Create the Base Agent Tracer


# agent_tracer.py
import uuid
from contextlib import contextmanager
from opentelemetry import trace, baggage, context
from opentelemetry.trace import SpanKind, Link
from opentelemetry.baggage.propagation import W3CBaggagePropagator
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

TRACER = trace.get_tracer("enterprise.ai.agent", schema_url="https://opentelemetry.io/schemas/1.30.0")
_PROPAGATOR = TraceContextTextMapPropagator()
_BAGGAGE_PROPAGATOR = W3CBaggagePropagator()

class AgentTracer:
    def __init__(self, agent_id: str, agent_name: str, workflow_id: str = None):
        self.agent_id = agent_id
        self.agent_name = agent_name
        self.workflow_id = workflow_id or str(uuid.uuid4())

    @contextmanager
    def agent_span(self, operation: str, parent_context=None, links: list = None, iteration: int = 0):
        """
        Start a new agent operation span.
        parent_context: an OTel Context extracted from an incoming message envelope.
        links: a list of OTel Link objects for non-linear causality (e.g., linking to a
               previously completed planning span).
        """
        with TRACER.start_as_current_span(
            name=f"{self.agent_name}.{operation}",
            context=parent_context,
            kind=SpanKind.INTERNAL,
            links=links or [],
            attributes={
                "gen_ai.agent.id": self.agent_id,
                "gen_ai.agent.name": self.agent_name,
                "workflow.id": self.workflow_id,
                "workflow.iteration": iteration,
            }
        ) as span:
            # Inject workflow_id into baggage so it propagates automatically
            ctx = baggage.set_baggage("workflow.id", self.workflow_id)
            token = context.attach(ctx)
            try:
                yield span
            except Exception as exc:
                span.record_exception(exc)
                span.set_status(trace.StatusCode.ERROR, str(exc))
                raise
            finally:
                context.detach(token)

    def inject_context(self, carrier: dict) -> dict:
        """Inject current trace context AND baggage into a message envelope dict."""
        _PROPAGATOR.inject(carrier)
        _BAGGAGE_PROPAGATOR.inject(carrier)
        carrier["workflow_id"] = self.workflow_id
        return carrier

    @staticmethod
    def extract_context(carrier: dict):
        """Extract OTel context from an incoming message envelope."""
        ctx = _PROPAGATOR.extract(carrier)
        ctx = _BAGGAGE_PROPAGATOR.extract(carrier, context=ctx)
        return ctx

Instrument an LLM Call


# llm_call_wrapper.py
import time
from opentelemetry import trace

def traced_llm_call(tracer: AgentTracer, model: str, system: str, messages: list, llm_client):
    """
    Wraps an LLM completion call with a GenAI-spec span.
    """
    span_name = f"gen_ai.{system}.chat"
    with TRACER.start_as_current_span(
        name=span_name,
        kind=trace.SpanKind.CLIENT,
        attributes={
            "gen_ai.system": system,
            "gen_ai.request.model": model,
            "gen_ai.operation.name": "chat",
            "gen_ai.agent.id": tracer.agent_id,
            "workflow.id": tracer.workflow_id,
        }
    ) as span:
        start = time.monotonic()
        response = llm_client.chat(model=model, messages=messages)
        latency_ms = (time.monotonic() - start) * 1000

        span.set_attributes({
            "gen_ai.usage.input_tokens": response.usage.input_tokens,
            "gen_ai.usage.output_tokens": response.usage.output_tokens,
            "gen_ai.response.finish_reason": response.finish_reason,
            "llm.latency_ms": latency_ms,
        })
        return response

Step 3: Propagate Context Across Async Message Boundaries

This is the most critical and most commonly botched step. When your orchestrator agent publishes a task to a Kafka topic, an SQS queue, or an internal event bus, the OTel context does NOT propagate automatically. You must manually inject and extract it from the message envelope.

Publishing a Task (Orchestrator Side)


# orchestrator.py
from agent_tracer import AgentTracer

orchestrator = AgentTracer(agent_id="orch-001", agent_name="planner", workflow_id="wf-abc-123")

with orchestrator.agent_span("plan_workflow") as root_span:
    task_payload = {
        "task": "summarize_documents",
        "document_ids": ["doc-1", "doc-2"],
    }
    # Inject trace context into the message envelope BEFORE publishing
    envelope = orchestrator.inject_context(task_payload)
    kafka_producer.send("agent-tasks", value=envelope)
    root_span.add_event("task_published", {"task.id": envelope.get("task")})

Consuming a Task (Sub-Agent Side)


# sub_agent.py
from agent_tracer import AgentTracer
from opentelemetry.trace import Link, get_current_span

for message in kafka_consumer:
    envelope = message.value

    # Extract the parent context from the envelope
    parent_ctx = AgentTracer.extract_context(envelope)

    # Retrieve workflow_id from the envelope for continuity
    workflow_id = envelope.get("workflow_id")

    sub_agent = AgentTracer(
        agent_id="exec-001",
        agent_name="executor",
        workflow_id=workflow_id
    )

    # Use the extracted context as the parent. This creates a true child span
    # of the orchestrator's root span, even across process and queue boundaries.
    with sub_agent.agent_span("execute_task", parent_context=parent_ctx) as span:
        span.set_attribute("task.name", envelope.get("task"))
        result = execute_task(envelope)
        span.set_attribute("task.result.status", "success")

This pattern ensures that the sub-agent's span is a proper child of the orchestrator's planning span in the resulting trace waterfall, even though the two processes never shared a direct HTTP connection.

Step 4: Use Span Links for Non-Linear Causality

Span links are the underused superpower of OpenTelemetry for agent workflows. A span link allows you to declare that a span is causally related to another span without making it a direct parent-child. This is essential for three agent-specific scenarios:

Retry causality: When a sub-agent is retried, the retry span should link to the original failed span, not inherit from it as a child. This preserves the failure history without polluting the primary trace tree.
Aggregation causality: When a critic or aggregator agent synthesizes the outputs of three parallel executor agents, its span should link to all three executor spans, not just the last one.
Cross-workflow causality: When a new workflow is triggered by the completion of a prior workflow, the new root span should link to the terminal span of the triggering workflow.

Creating Span Links


from opentelemetry.trace import Link, SpanContext, TraceFlags

def build_span_link_from_envelope(envelope: dict) -> Link:
    """
    Reconstruct a SpanContext from a stored span reference
    (e.g., a completed executor span whose context was saved to a state store).
    """
    trace_id = int(envelope["trace_id"], 16)
    span_id = int(envelope["span_id"], 16)
    ctx = SpanContext(
        trace_id=trace_id,
        span_id=span_id,
        is_remote=True,
        trace_flags=TraceFlags(TraceFlags.SAMPLED),
    )
    return Link(context=ctx, attributes={"link.reason": envelope.get("link_reason", "causal")})

# Usage in a critic agent that aggregates three executor results
executor_links = [build_span_link_from_envelope(e) for e in completed_executor_envelopes]

with critic_agent.agent_span("aggregate_results", links=executor_links) as span:
    span.set_attribute("aggregation.source_count", len(executor_links))
    final_result = aggregate(results)

In Jaeger or Tempo, these links will render as dotted arrows between spans in the trace graph view, giving you a visual representation of the full causal DAG of your agent workflow.

Step 5: Configure the OTel Collector for Agent-Aware Enrichment

Your existing OTel Collector configuration handles microservice spans well. For agent spans, you need to add three additional pipeline components: a transform processor for attribute normalization, a groupbyattrs processor for workflow-level aggregation, and a filter processor to route agent spans to a dedicated backend or pipeline.

Collector Configuration (collector-config.yaml)


receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  # Enrich all spans with deployment metadata
  resource:
    attributes:
      - key: deployment.environment
        value: "production"
        action: upsert
      - key: team.name
        value: "backend-ai-platform"
        action: upsert

  # Normalize agent attribute names for consistency
  transform/agent_normalize:
    trace_statements:
      - context: span
        statements:
          # Ensure workflow.id is always present; fall back to trace ID if missing
          - set(attributes["workflow.id"], trace_id) where attributes["workflow.id"] == nil
          # Promote baggage-carried workflow.id to a first-class span attribute
          - set(attributes["workflow.id"], baggage["workflow.id"]) where baggage["workflow.id"] != nil

  # Deduplicate retry spans: set a canonical iteration label
  transform/retry_label:
    trace_statements:
      - context: span
        statements:
          - set(attributes["workflow.is_retry"], true) where attributes["workflow.iteration"] > 0

  # Group spans by workflow for aggregated metrics
  groupbyattrs/workflow:
    keys:
      - workflow.id
      - gen_ai.agent.name

  # Route agent spans to a dedicated pipeline
  filter/agent_spans:
    traces:
      span:
        - 'attributes["gen_ai.agent.id"] != nil'

  batch:
    timeout: 5s
    send_batch_size: 512

exporters:
  # Your existing Tempo/Jaeger/Honeycomb exporter
  otlp/tempo:
    endpoint: "http://tempo:4317"
    tls:
      insecure: true

  # Optional: dedicated agent trace backend
  otlp/agent_backend:
    endpoint: "https://your-agent-observability-backend:4317"
    headers:
      api-key: "${AGENT_OBS_API_KEY}"

  # Prometheus metrics from agent spans
  prometheusremotewrite:
    endpoint: "http://prometheus:9090/api/v1/write"

service:
  pipelines:
    # Main pipeline (all spans)
    traces/main:
      receivers: [otlp]
      processors: [resource, transform/agent_normalize, transform/retry_label, batch]
      exporters: [otlp/tempo]

    # Agent-specific pipeline (filtered)
    traces/agents:
      receivers: [otlp]
      processors: [resource, transform/agent_normalize, filter/agent_spans, groupbyattrs/workflow, batch]
      exporters: [otlp/agent_backend]

Step 6: Instrument Tool Calls with the Shim Pattern

Tools are the most common observability black hole in agent systems. When an agent calls a code interpreter, a web search API, or a SQL query tool, that call typically exits your instrumented code entirely. The tool shim pattern wraps every tool invocation in a CLIENT span that bridges the gap.


# tool_shim.py
from opentelemetry import trace
from functools import wraps

TRACER = trace.get_tracer("enterprise.ai.tools")

def traced_tool(tool_name: str, tool_type: str = "external_api"):
    """
    Decorator that wraps any agent tool function with an OTel CLIENT span.
    """
    def decorator(fn):
        @wraps(fn)
        def wrapper(*args, **kwargs):
            with TRACER.start_as_current_span(
                name=f"tool.{tool_name}",
                kind=trace.SpanKind.CLIENT,
                attributes={
                    "tool.name": tool_name,
                    "tool.type": tool_type,
                    "gen_ai.operation.name": "tool_call",
                }
            ) as span:
                try:
                    result = fn(*args, **kwargs)
                    span.set_attribute("tool.result.status", "success")
                    return result
                except Exception as exc:
                    span.record_exception(exc)
                    span.set_attribute("tool.result.status", "error")
                    span.set_attribute("tool.error.type", type(exc).__name__)
                    raise
        return wrapper
    return decorator

# Usage
@traced_tool("web_search", tool_type="external_api")
def web_search(query: str) -> dict:
    return search_api.search(query)

@traced_tool("sql_query", tool_type="database")
def run_sql(query: str) -> list:
    return db.execute(query)

Because the tool shim runs within the agent's active span context, the tool call span is automatically a child of the agent's current operation span. No additional context injection is needed for synchronous tool calls.

Step 7: Preserve Cross-Workflow Causality with a Workflow State Store

Enterprise agent systems rarely run a single workflow in isolation. Workflows trigger other workflows. A report generation workflow may be triggered by a data ingestion workflow that was itself triggered by a monitoring alert workflow. Preserving causality across this chain requires a lightweight workflow state store that records terminal span contexts and makes them available to downstream workflows as link sources.


# workflow_registry.py
import redis
import json
from opentelemetry import trace

redis_client = redis.Redis(host="redis", port=6379, decode_responses=True)

def save_workflow_terminal_span(workflow_id: str, span: trace.Span, ttl_seconds: int = 86400):
    """
    Save the terminal span context of a completed workflow to Redis.
    This allows downstream workflows to create span links back to this workflow.
    """
    ctx = span.get_span_context()
    record = {
        "trace_id": format(ctx.trace_id, "032x"),
        "span_id": format(ctx.span_id, "016x"),
        "workflow_id": workflow_id,
        "link_reason": "workflow_trigger",
    }
    redis_client.setex(f"workflow:terminal:{workflow_id}", ttl_seconds, json.dumps(record))

def get_upstream_workflow_link(upstream_workflow_id: str):
    """
    Retrieve a span link to the terminal span of an upstream workflow.
    Returns None if the upstream workflow context has expired or does not exist.
    """
    raw = redis_client.get(f"workflow:terminal:{upstream_workflow_id}")
    if not raw:
        return None
    return build_span_link_from_envelope(json.loads(raw))

# In a downstream workflow's initialization
upstream_link = get_upstream_workflow_link("wf-abc-123")
links = [upstream_link] if upstream_link else []

with downstream_agent.agent_span("start_report_workflow", links=links) as span:
    span.set_attribute("upstream.workflow_id", "wf-abc-123")
    # ... rest of workflow

Step 8: Build Workflow-Level Metrics from Span Data

Traces tell you the story of individual workflow runs. Metrics tell you the story of your system over time. Use the OTel Collector's span metrics connector to derive the following workflow-level metrics automatically from your agent spans, without writing separate metric instrumentation code:

workflow.duration_ms (histogram): End-to-end workflow duration, grouped by workflow.id pattern and gen_ai.agent.name.
agent.llm.tokens.total (counter): Total token consumption per agent per workflow, for cost attribution dashboards.
agent.tool.calls.total (counter): Tool invocation frequency, grouped by tool.name and tool.result.status.
agent.retry.rate (gauge): Ratio of spans with workflow.iteration > 0 to total spans, indicating planning instability.
workflow.causality_depth (histogram): The maximum span link depth in a workflow trace, indicating how complex your cross-workflow dependency graphs are becoming.

Add the span metrics connector to your Collector config to enable this automatically:


connectors:
  spanmetrics:
    namespace: ai_agent
    dimensions:
      - name: gen_ai.agent.name
      - name: workflow.id
      - name: tool.name
      - name: gen_ai.system
      - name: workflow.is_retry
    exemplars:
      enabled: true
    histogram:
      explicit:
        buckets: [10, 50, 100, 250, 500, 1000, 2500, 5000, 10000]

service:
  pipelines:
    traces/agents:
      receivers: [otlp]
      processors: [resource, transform/agent_normalize, filter/agent_spans, batch]
      exporters: [otlp/agent_backend, spanmetrics]

    metrics/agent_derived:
      receivers: [spanmetrics]
      processors: [batch]
      exporters: [prometheusremotewrite]

Step 9: Validate the Pipeline End-to-End

Before shipping to production, run this validation checklist against your staging environment:

Trace continuity test: Trigger a three-agent workflow (orchestrator, executor, critic). Verify in your trace backend that all three agents' spans share the same traceId and that the parent-child hierarchy is correct.
Baggage propagation test: Assert that workflow.id appears as a span attribute on every span in the workflow, including tool call spans and LLM call spans.
Span link test: Trigger a retry scenario. Verify that the retry span has a link to the original failed span with link.reason: causal.
Cross-workflow causality test: Trigger a downstream workflow from an upstream workflow. Verify that the downstream root span has a link to the upstream terminal span and that the link resolves correctly in the trace UI.
Tool opacity test: Invoke a tool call from within an agent span. Verify that the tool span appears as a child of the agent span in the trace waterfall.
Metric derivation test: Run 10 workflow executions and verify that ai_agent_duration_ms and ai_agent_llm_tokens_total metrics appear in Prometheus with correct label values.

Common Pitfalls and How to Avoid Them

Pitfall 1: Context Loss at Async Boundaries

The most common failure mode. If you forget to call inject_context(envelope) before publishing to a queue, the sub-agent will start a new root span with no parent, creating an orphaned trace. Enforce context injection at the message publisher level using a middleware wrapper, not at the call site.

Pitfall 2: Sampling Dropping Agent Spans

If your existing OTel Collector uses a probabilistic sampler set to 10%, you will lose 90% of your agent traces. Agent workflows are typically lower volume but higher value than microservice spans. Add a rule-based sampler that always samples spans where gen_ai.agent.id is present, and apply probabilistic sampling only to non-agent spans.

Pitfall 3: Span Explosion from Token-Level Logging

Some teams instrument every token or every prompt chunk as a separate span event. This creates millions of spans per workflow and overwhelms your trace backend. Use span events (not child spans) for intermediate LLM streaming chunks, and emit a single span per LLM call with aggregate token counts as attributes.

Pitfall 4: Workflow ID Drift

If different agents in the same workflow generate their own workflow.id independently, you lose the ability to correlate them. Always generate workflow.id at the orchestrator level and propagate it via baggage. Treat it as immutable for the lifetime of the workflow.

Pitfall 5: Ignoring the Collector Memory Limit

Agent workflows with deep span link graphs can produce very large trace payloads. Set the memory_limiter processor as the first processor in every pipeline to prevent OOM crashes in the Collector under load.

Conclusion: Observability Is a First-Class Agent Feature

In H2 2026, the teams winning with multi-agent systems are not the ones with the most sophisticated agents. They are the ones who can see what their agents are doing clearly enough to debug, optimize, and trust them in production. Observability is not an afterthought you bolt on after the agents are built. It is a first-class design constraint that shapes how agents communicate, how workflows are structured, and how context flows through your system.

The pipeline described in this guide gives your enterprise backend team a production-grade foundation: GenAI semantic conventions for standardized attribute naming, span links for non-linear causality, baggage propagation for workflow identity, a Collector enrichment layer that fits cleanly into your existing OTel infrastructure, and derived metrics for operational dashboards. None of it requires a new observability vendor or a shadow stack.

The agents will surprise you. Make sure your traces do not.