AI Observability

How Enterprise Backend Teams Can Build AI Agent Observability Pipelines That Correlate Distributed Trace Data With Model Inference Latency Spikes Across Multi-Provider Routing Layers in H2 2026

Scott Miller

20 Aug 2026 — 13 min read

By mid-2026, most enterprise backend teams have crossed the threshold from experimenting with AI agents to running them in production. And that shift has exposed a brutal truth: the observability stacks that served you perfectly well for microservices are almost completely blind to what makes AI agent pipelines fail.

A p99 latency spike on a traditional REST service points you toward a slow database query or a saturated thread pool. A p99 latency spike in an AI agent pipeline could be a cold model container on a secondary provider, a prompt that ballooned token count mid-flight, a routing layer that silently fell back to a slower model, or a chain of tool calls that cascaded into a 14-second response. Traditional traces show you the when. They rarely show you the why , at least not without significant instrumentation work.

This guide is a practical, opinionated tutorial for enterprise backend engineers who need to build observability pipelines that do three things simultaneously: capture distributed trace context across every hop of an AI agent workflow, correlate that trace data with model inference latency signals from multiple providers, and surface actionable diagnostics at the routing layer where provider selection decisions are made. Everything here is oriented toward H2 2026 tooling and architectural patterns.

Why Existing Observability Stacks Fall Short for Multi-Provider AI Agents

Before diving into implementation, it is worth being precise about the problem. Most enterprise teams arrive at AI agent observability with one of two broken assumptions:

Assumption 1: "We already have OpenTelemetry. We're covered." OpenTelemetry gives you excellent span propagation across HTTP and gRPC boundaries. But model inference calls carry semantics that generic HTTP spans cannot represent: token counts, sampled logprobs, finish reasons, KV-cache hit rates, and streaming chunk timing. Without semantic conventions specific to LLM calls, your traces are structurally correct but diagnostically hollow.
Assumption 2: "Each provider has a dashboard. We can just check those." Provider dashboards (OpenAI, Anthropic, Google Gemini, Mistral, and others) show you aggregate latency from their perspective. They cannot show you how that latency interacts with your routing logic, your retry budget, your agent's tool-call depth, or your downstream service SLAs. The correlation gap lives entirely in your infrastructure.

The real challenge in H2 2026 is the multi-provider routing layer. Virtually every mature enterprise AI deployment now routes inference requests across at least two or three providers, using frameworks like LiteLLM, PortKey, Martian, or custom gateway services. This routing layer is simultaneously the most powerful architectural tool you have and the biggest observability black hole. When latency spikes, you need to know whether the spike originated before the router (agent logic, prompt construction), inside the router (provider selection algorithm, load shedding), or after the router (provider-side cold starts, rate limiting, token generation speed).

The Architecture You Are Building

Here is the target architecture this tutorial will walk you through. Think of it as three interlocking planes:

The Trace Plane: OpenTelemetry-instrumented spans propagated from your agent orchestrator, through your routing gateway, to provider adapters, and back. Every span carries a shared trace_id and enriched AI-specific attributes.
The Metrics Plane: Time-series signals emitted at the routing layer: per-provider TTFT (time-to-first-token), inter-chunk latency, token throughput, error rates, and routing decision metadata (which provider was selected and why).
The Correlation Engine: A pipeline, typically built on an OpenTelemetry Collector with a custom processor or a stream processor like Apache Flink or Redpanda, that joins trace spans with metrics signals using the shared trace_id and a time-window join. This is where latency spikes get annotated with their root cause context.

The output feeds into your existing observability backend (Grafana + Tempo, Honeycomb, Datadog, or similar) and, critically, into a feedback loop that can influence routing decisions in near-real-time.

Step 1: Establish LLM-Aware Semantic Conventions in Your Spans

The OpenTelemetry GenAI semantic conventions (stabilized in the 1.x specification by early 2026) give you a standardized attribute namespace for LLM calls. Make these non-negotiable across every team touching AI infrastructure. Here is the minimum viable attribute set you should be emitting on every inference span:


gen_ai.system                  = "openai" | "anthropic" | "google" | "mistral" | ...
gen_ai.request.model           = "gpt-4.5" | "claude-4-opus" | "gemini-2.5-pro" | ...
gen_ai.request.max_tokens      = 4096
gen_ai.request.temperature     = 0.7
gen_ai.response.model          = "gpt-4.5-2026-06"   # actual model version served
gen_ai.usage.input_tokens      = 1842
gen_ai.usage.output_tokens     = 612
gen_ai.usage.total_tokens      = 2454

# Custom extensions your team should add:
ai.router.provider_selected    = "anthropic"
ai.router.provider_fallback    = false
ai.router.selection_strategy   = "latency_weighted"
ai.inference.ttft_ms           = 312
ai.inference.generation_ms     = 4210
ai.inference.chunk_count       = 47
ai.agent.tool_call_depth       = 3
ai.agent.step_index            = 2
ai.prompt.template_id          = "customer-support-v4"
ai.prompt.estimated_tokens     = 1790

The critical distinction here is between gen_ai.request.model (what you asked for) and gen_ai.response.model (what was actually served). In multi-provider environments, providers frequently serve requests from different underlying model versions or infrastructure tiers. That discrepancy is often the root cause of latency variance that looks completely random if you are only tracking the requested model name.

Instrument your router gateway to emit these attributes as early as possible in the span lifecycle. Do not wait for the response to close the span; use span events to record TTFT the moment the first streaming chunk arrives:


# Python example using OpenTelemetry SDK
from opentelemetry import trace
from opentelemetry.trace import SpanKind
import time

tracer = trace.get_tracer("ai.router", version="1.0.0")

def route_and_call(request, provider_client):
    with tracer.start_as_current_span(
        "gen_ai.inference",
        kind=SpanKind.CLIENT,
        attributes={
            "gen_ai.system": provider_client.system_name,
            "gen_ai.request.model": request.model,
            "ai.router.provider_selected": provider_client.provider_id,
            "ai.router.selection_strategy": router.current_strategy,
        }
    ) as span:
        request_start = time.monotonic_ns()
        first_chunk_received = False

        for chunk in provider_client.stream(request):
            if not first_chunk_received:
                ttft_ms = (time.monotonic_ns() - request_start) / 1_000_000
                span.add_event("gen_ai.first_token", attributes={
                    "ai.inference.ttft_ms": ttft_ms
                })
                first_chunk_received = True
            yield chunk

        span.set_attributes({
            "gen_ai.usage.input_tokens": response.usage.input_tokens,
            "gen_ai.usage.output_tokens": response.usage.output_tokens,
            "ai.inference.generation_ms": (time.monotonic_ns() - request_start) / 1_000_000,
        })

Step 2: Propagate Trace Context Through Your Routing Gateway

This step is where most teams quietly lose their correlation capability. Your routing gateway sits between the agent orchestrator and the provider APIs. If the gateway does not correctly propagate the W3C traceparent header (and your custom baggage), you end up with two disconnected trace trees: one for the agent logic, one for the provider call. They share a timestamp range but no structural relationship.

The fix depends on your gateway architecture:

For LiteLLM-Based Gateways

LiteLLM's proxy mode supports OpenTelemetry callbacks natively. Configure it to extract the incoming traceparent header and use it as the parent context for all outbound provider spans. Add this to your litellm_config.yaml:


general_settings:
  otel: true
  otel_exporter: otlp
  otel_endpoint: "http://otel-collector:4317"

litellm_settings:
  success_callback: ["otel"]
  failure_callback: ["otel"]
  # Propagate incoming trace context to provider calls
  forward_traceparent: true
  custom_attributes:
    ai.router.gateway: "litellm-proxy"
    deployment.environment: "production"

For Custom Gateway Services

If you have built a custom routing service (common in enterprises with strict security requirements), you need to explicitly extract and inject trace context at both the ingress and egress points:


from opentelemetry.propagate import extract, inject
from opentelemetry import context, trace

# At gateway ingress: extract context from incoming agent request
incoming_ctx = extract(request.headers)
token = context.attach(incoming_ctx)

try:
    # Build outbound headers for the provider API call
    outbound_headers = {}
    inject(outbound_headers)  # Injects traceparent + tracestate into outbound_headers

    response = provider_http_client.post(
        provider_endpoint,
        headers={**base_headers, **outbound_headers},
        json=payload
    )
finally:
    context.detach(token)

A subtlety worth calling out: if your gateway fans out a single agent request to multiple providers simultaneously (for A/B testing or ensemble routing), each fan-out call should be a child span of the same parent, not a separate root span. This lets you see, in a single trace waterfall view, that provider A responded in 800ms while provider B responded in 3.2 seconds, and your router correctly selected A's response.

Step 3: Build the Per-Provider Latency Metrics Layer

Distributed traces give you per-request detail. But to detect patterns (provider degradation trends, time-of-day latency curves, model version rollout impacts), you need a metrics layer running in parallel. The key is that every metric must be tagged with the same dimensional attributes as your spans, so you can pivot between aggregate trends and individual trace examples.

Emit the following metrics from your routing gateway using the OpenTelemetry Metrics API:


from opentelemetry import metrics

meter = metrics.get_meter("ai.router", version="1.0.0")

# Histograms (not gauges) for latency - you need percentile distributions
ttft_histogram = meter.create_histogram(
    name="ai.inference.ttft",
    description="Time to first token in milliseconds",
    unit="ms",
)

generation_histogram = meter.create_histogram(
    name="ai.inference.generation_duration",
    description="Total generation time in milliseconds",
    unit="ms",
)

token_throughput = meter.create_histogram(
    name="ai.inference.tokens_per_second",
    description="Output token throughput",
    unit="tokens/s",
)

routing_decisions = meter.create_counter(
    name="ai.router.decisions_total",
    description="Total routing decisions made",
)

fallback_counter = meter.create_counter(
    name="ai.router.fallbacks_total",
    description="Number of provider fallback events",
)

# Record with rich dimensional labels
def record_inference_metrics(result, provider, model, strategy, agent_id):
    labels = {
        "provider": provider,
        "model": result.response_model,
        "routing_strategy": strategy,
        "agent_id": agent_id,
        "fallback": str(result.was_fallback),
        "finish_reason": result.finish_reason,
    }
    ttft_histogram.record(result.ttft_ms, labels)
    generation_histogram.record(result.generation_ms, labels)
    token_throughput.record(result.tokens_per_second, labels)
    routing_decisions.add(1, labels)
    if result.was_fallback:
        fallback_counter.add(1, labels)

Configure your OpenTelemetry Collector to export these metrics to your time-series backend at a 15-second scrape interval for production. Use 1-second intervals only during active incident investigation; the cardinality cost at 1-second resolution across multiple providers and model versions adds up quickly.

Step 4: Build the Correlation Pipeline

This is the architectural centerpiece of the whole system. The correlation pipeline solves a specific problem: when your metrics dashboard shows a p95 TTFT spike on Anthropic Claude between 14:32 and 14:47 UTC, how do you automatically surface the specific traces that were affected, annotated with their full agent context?

The approach that works best at enterprise scale in 2026 uses the OpenTelemetry Collector's spanmetrics connector combined with a custom processor. Here is the logical flow:

Spans arrive at the Collector from your routing gateway.
The spanmetrics connector generates RED metrics (Rate, Error, Duration) from span data, keyed by your AI-specific attributes.
A custom latency_spike_detector processor compares incoming span duration against a rolling baseline per provider/model combination.
When a span exceeds the baseline by a configurable threshold (say, 2.5x the p75), the processor enriches the span with a ai.latency_anomaly = true attribute and a ai.latency_anomaly_severity score.
Anomalous spans are routed to a high-priority export pipeline that writes to both your trace backend and a dedicated anomaly event stream (Kafka or Redpanda topic).

Here is the relevant section of an otel-collector-config.yaml that wires this together:


receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 5s
    send_batch_size: 1000

  # Enriches spans with anomaly flags based on rolling baselines
  transform/ai_anomaly_detection:
    trace_statements:
      - context: span
        statements:
          # Flag spans where generation time exceeds provider p75 baseline
          - set(attributes["ai.latency_anomaly"],
              true) where attributes["ai.inference.generation_ms"] != nil
              and attributes["ai.inference.generation_ms"] >
              Double(attributes["ai.router.provider_p75_baseline_ms"]) * 2.5
          - set(attributes["ai.latency_anomaly_severity"],
              "critical") where attributes["ai.latency_anomaly"] == true
              and attributes["ai.inference.generation_ms"] >
              Double(attributes["ai.router.provider_p75_baseline_ms"]) * 5.0

  # Attach baseline values from an external lookup (populated by your metrics pipeline)
  attributes/inject_baselines:
    actions:
      - key: ai.router.provider_p75_baseline_ms
        from_context: provider_baseline_cache
        action: insert

connectors:
  spanmetrics:
    histogram:
      explicit:
        buckets: [50, 100, 200, 500, 1000, 2000, 5000, 10000, 30000]
    dimensions:
      - name: gen_ai.system
      - name: gen_ai.response.model
      - name: ai.router.selection_strategy
      - name: ai.router.provider_fallback
      - name: ai.agent.tool_call_depth
      - name: ai.latency_anomaly
    namespace: ai_router

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

  otlp/anomaly_stream:
    endpoint: anomaly-processor:4317
    sending_queue:
      enabled: true
      num_consumers: 10

  prometheusremotewrite:
    endpoint: "http://mimir:9009/api/v1/push"

service:
  pipelines:
    traces/standard:
      receivers: [otlp]
      processors: [attributes/inject_baselines, transform/ai_anomaly_detection, batch]
      exporters: [otlp/tempo, spanmetrics]

    traces/anomalies:
      receivers: [otlp]
      processors: [attributes/inject_baselines, transform/ai_anomaly_detection, batch]
      exporters: [otlp/anomaly_stream]

    metrics:
      receivers: [spanmetrics]
      processors: [batch]
      exporters: [prometheusremotewrite]

Step 5: Maintain Rolling Baselines Per Provider and Model Version

The anomaly detection in Step 4 is only as good as its baselines. Static thresholds (for example, "flag anything over 5 seconds") are too brittle for multi-provider environments where latency profiles differ dramatically by provider, model size, time of day, and input token count. You need rolling baselines that adapt.

The practical approach for most enterprise teams is a lightweight sidecar service that reads from your metrics backend and writes baseline values into a Redis cache that your Collector processor can query:


# baseline_updater.py - runs every 60 seconds
import redis
import requests
from datetime import datetime, timedelta

PROVIDERS = ["openai", "anthropic", "google", "mistral"]
MODELS = {
    "openai": ["gpt-4.5", "o3"],
    "anthropic": ["claude-4-opus", "claude-4-sonnet"],
    "google": ["gemini-2.5-pro", "gemini-2.5-flash"],
    "mistral": ["mistral-large-3", "mistral-medium-3"],
}

def fetch_p75_baseline(provider, model, window_minutes=60):
    """Query Mimir/Prometheus for the p75 TTFT over the last N minutes."""
    query = (
        f'histogram_quantile(0.75, '
        f'sum(rate(ai_router_ai_inference_ttft_bucket{{'
        f'provider="{provider}",model="{model}"'
        f'}}[{window_minutes}m])) by (le))'
    )
    response = requests.get(
        "http://mimir:9009/api/v1/query",
        params={"query": query}
    )
    result = response.json()
    if result["data"]["result"]:
        return float(result["data"]["result"][0]["value"][1])
    return None

def update_baselines():
    r = redis.Redis(host="redis", port=6379, decode_responses=True)
    for provider in PROVIDERS:
        for model in MODELS.get(provider, []):
            baseline = fetch_p75_baseline(provider, model)
            if baseline:
                key = f"baseline:{provider}:{model}:p75_ttft_ms"
                r.setex(key, 300, baseline)  # TTL of 5 minutes
                print(f"Updated {key} = {baseline:.1f}ms")

With adaptive baselines in place, your anomaly detector will correctly flag a 2-second TTFT on Gemini Flash (normally 180ms) while ignoring a 2-second TTFT on Claude Opus during a known high-load window where the baseline is already elevated.

Step 6: Build the Grafana Correlation Dashboard

All of this instrumentation work pays off at the dashboard layer. The key design principle is: every metric panel must be clickable to drill into the underlying traces. In Grafana with Tempo as your trace backend, this is achieved through exemplars.

Configure your Prometheus remote write to include exemplars, and make sure your spanmetrics connector is emitting exemplar trace_id values. Then build your dashboard with these core panels:

Provider TTFT Heatmap: A heatmap per provider showing the distribution of time-to-first-token over time. Latency spikes appear as color bands. Click any cell to see the exemplar traces for that time window.
Routing Decision Sankey: A flow diagram showing how requests were distributed across providers, with fallback paths highlighted. This surfaces routing strategy drift instantly.
Anomaly Event Timeline: A time-series panel showing ai.latency_anomaly = true spans over time, grouped by provider and severity. Correlate this visually with deployment events and provider status page incidents.
Tool Call Depth vs. Latency Scatter: A scatter plot with ai.agent.tool_call_depth on the X axis and total inference latency on the Y axis, colored by provider. This reveals whether your latency spikes are driven by agent complexity rather than provider issues.
Token Budget Burn Rate: A stacked area chart of gen_ai.usage.total_tokens per provider over time. Sudden spikes here often precede rate limiting events that show up as latency spikes 30 to 60 seconds later.

Step 7: Close the Loop With Routing Feedback

An observability pipeline that only alerts is only half a system. The full value comes from feeding latency anomaly signals back into your routing layer to influence provider selection in near-real-time. This is where the Kafka/Redpanda anomaly event stream from Step 4 becomes a control plane input.

Your routing gateway should subscribe to the anomaly stream and maintain a per-provider health score that decays toward neutral over time:


# provider_health_tracker.py
import asyncio
from collections import defaultdict
from aiokafka import AIOKafkaConsumer
import json
import math

class ProviderHealthTracker:
    def __init__(self):
        # Score from 0.0 (degraded) to 1.0 (healthy)
        self.scores = defaultdict(lambda: 1.0)
        self.decay_rate = 0.95   # Score recovers 5% per cycle
        self.penalty_map = {
            "warning": 0.15,
            "critical": 0.40,
        }

    async def consume_anomaly_events(self):
        consumer = AIOKafkaConsumer(
            "ai.latency.anomalies",
            bootstrap_servers="redpanda:9092",
            value_deserializer=lambda v: json.loads(v.decode())
        )
        await consumer.start()
        try:
            async for msg in consumer:
                event = msg.value
                provider = event.get("provider")
                severity = event.get("ai.latency_anomaly_severity", "warning")
                if provider:
                    penalty = self.penalty_map.get(severity, 0.15)
                    self.scores[provider] = max(0.0, self.scores[provider] - penalty)
        finally:
            await consumer.stop()

    async def decay_scores(self):
        """Gradually restore health scores every 30 seconds."""
        while True:
            await asyncio.sleep(30)
            for provider in list(self.scores.keys()):
                self.scores[provider] = min(1.0,
                    self.scores[provider] * (1 / self.decay_rate))

    def get_routing_weights(self):
        """Return normalized weights for latency-aware routing."""
        total = sum(self.scores.values()) or 1.0
        return {p: s / total for p, s in self.scores.items()}

Feed these weights into your router's provider selection logic. When Anthropic's health score drops to 0.4 due to a cluster of critical latency anomalies, the router automatically shifts a larger share of traffic to OpenAI and Google until the score recovers. The recovery is automatic; no on-call engineer needs to manually update routing rules at 2 AM.

Common Pitfalls and How to Avoid Them

After walking through the full pipeline, here are the failure modes that consistently trip up enterprise teams:

Cardinality explosion from model version attributes: Providers update model versions frequently. If you use gen_ai.response.model as a high-cardinality label in your metrics (not just your traces), you can easily generate tens of thousands of unique time series. Normalize model versions to major families in your metrics labels, and reserve the full version string for trace attributes only.
Clock skew between services: Distributed trace correlation relies on consistent timestamps. In multi-provider environments where some latency measurements come from provider response headers and others from your own clock, a 50ms clock skew can make a TTFT measurement look like it belongs to the wrong time window. Use NTP-synchronized clocks everywhere and treat provider-reported timestamps as advisory only.
Treating streaming and non-streaming calls identically: TTFT is only meaningful for streaming calls. For non-streaming (batch) inference calls, the meaningful latency metric is total response time. Mixing these in the same histogram without a streaming=true/false label produces a bimodal distribution that makes percentile calculations meaningless.
Ignoring prompt construction time: Teams frequently instrument the provider call but not the prompt assembly step. In agentic workflows with dynamic few-shot examples, retrieval-augmented context, and multi-turn history, prompt construction can take 200 to 800ms. If you omit this span, your trace waterfall will show a gap that makes the provider look slower than it actually is.
Sampling away your anomalies: Tail-based sampling is a great cost control tool, but if your sampling rules drop spans below a certain duration threshold, you may be discarding exactly the slow traces you need most. Configure your sampler to always retain spans with ai.latency_anomaly = true, regardless of other sampling rules.

Conclusion: Observability as a Routing Intelligence Layer

The pattern described in this guide represents a meaningful shift in how enterprise backend teams should think about AI observability. It is not a passive monitoring system. It is an active intelligence layer that makes your multi-provider routing smarter with every request that passes through it.

By the end of H2 2026, the teams that will have the most reliable, cost-efficient AI agent infrastructure are the ones who treated observability as a first-class engineering concern from the start: not bolted on after the first production incident, but designed into the routing gateway, the agent orchestrator, and the deployment pipeline from day one.

The tooling is mature enough to do this well right now. OpenTelemetry's GenAI semantic conventions are stable. The Collector's spanmetrics connector handles the trace-to-metrics bridge. Grafana's exemplar support closes the loop between dashboards and traces. The remaining work is the integration work, which is exactly what this guide has walked you through.

Start with Step 1 (semantic conventions) and Step 2 (context propagation). Get those right, and the rest of the pipeline becomes dramatically easier to build. The most expensive observability mistake you can make in a multi-provider AI environment is emitting data that looks complete but lacks the dimensional richness to answer the question that matters most: which provider, which model version, at which point in the agent workflow, caused this latency spike, and what should the router do differently next time?