When Distributed Tracing Lies: How One Enterprise Backend Team Rebuilt Their Agentic Observability Stack from Scratch

In early 2026, the platform engineering team at a mid-sized financial services firm called Meridian Capital Systems (name changed for confidentiality) was staring at a dashboard that told a perfectly coherent story. Their agentic backend, a multi-agent orchestration system powering automated compliance review and portfolio risk summarization, showed an average end-to-end latency of 1.4 seconds per request. Acceptable. Within SLA. Green across the board.

There was just one problem: their users were reporting waits of 8 to 12 seconds. Regularly. Reproducibly. Infuriatingly.

What followed was a six-week forensic investigation that fundamentally changed how their team thought about observability for agentic systems, and produced a cautionary tale that every backend team running multi-agent workloads in 2026 should hear.

The Setup: A Microservices Tracing Stack Pressed Into Agentic Service

Meridian's backend team had built a mature observability stack over several years. They ran OpenTelemetry collectors feeding into a self-hosted Grafana Tempo instance, with Prometheus for metrics and a Loki-backed log aggregation pipeline. For their traditional microservices, this stack was excellent. Spans were clean, parent-child relationships were accurate, and latency attribution was reliable.

When the team deployed their first agentic layer in late 2025, the natural instinct was to instrument it the same way. They wrapped each agent invocation in a span, propagated trace context through HTTP headers between agent-to-agent calls, and instrumented their tool execution layer using the same OpenTelemetry SDK they had always used. On paper, the architecture looked like this:

• Orchestrator Agent: Receives user request, decomposes into subtasks, delegates to specialist agents
• Compliance Agent: Queries a vector database, calls a regulatory lookup API, and invokes an LLM for synthesis
• Risk Agent: Pulls portfolio data, runs a quantitative scoring tool, and calls the same LLM endpoint
• Summarizer Agent: Waits for both upstream agents, merges outputs, produces a final structured response

The tool execution graph was a directed acyclic graph (DAG) with real fan-out and fan-in behavior. The tracing stack, designed for linear or loosely parallel microservice chains, had no idea what it was walking into.

The Root Cause: Three Ways Traditional Tracing Misleads in Agentic Graphs

After weeks of investigation, the team identified not one but three distinct failure modes where their microservices-oriented tracing stack produced actively misleading latency attribution data.

1. Span Parenting Collapsed Fan-Out Into a False Sequential Model

In a standard microservices trace, a parent span calls a child span, waits for it, and the wall-clock time of the parent naturally encompasses the child. Latency attribution is straightforward: if a child span took 900ms, that cost is visible in the waterfall.

In a multi-agent tool execution graph, fan-out is the norm. The Orchestrator Agent was spawning the Compliance Agent and the Risk Agent concurrently, using an async task queue. But because both agents received the same parent trace context (the Orchestrator's span ID), OpenTelemetry recorded them as sequential siblings under the same parent. The Tempo waterfall view displayed them stacked end-to-end, implying a combined latency of 4.2 seconds when the actual wall-clock cost was only 2.3 seconds (they ran in parallel).

More dangerously, when the team looked at which agent was the "bottleneck," the waterfall pointed to the Compliance Agent because it appeared last in the trace timeline. In reality, the Risk Agent was the slower of the two and was the true critical path. The team had spent two weeks optimizing the wrong agent.

2. LLM Token Generation Time Was Being Absorbed Into Tool Span Overhead

Each agent called a shared internal LLM gateway service. The gateway was instrumented with its own spans, covering the HTTP round-trip from the agent to the gateway and back. What the spans did not capture was the time-to-first-token (TTFT) versus total generation time breakdown inside the LLM provider's infrastructure.

The result was that a span labeled llm_gateway.invoke showing 3.1 seconds was attributed entirely to "network and gateway overhead" in the team's SLO dashboards. In reality, 2.6 seconds of that was pure LLM inference time, and 0.5 seconds was actual gateway and serialization overhead. When the team tried to optimize the gateway (the wrong target), they achieved negligible improvements and grew increasingly frustrated.

The fix required instrumenting TTFT as a dedicated metric, separating streaming chunk arrival events from span lifecycle events, and building a new derived metric that isolated inference latency from transport latency. None of this existed in their original stack.

3. Tool Retry Logic Created Phantom Spans That Inflated Aggregate Metrics

The Compliance Agent included a retry wrapper around its external regulatory API calls. When the API returned a 429 or a timeout, the agent silently retried up to three times before either succeeding or failing hard. Each retry attempt created a new child span under the same parent tool span.

In the Tempo UI, these retry spans were invisible at the individual trace level because they were collapsed under the parent. But in Prometheus, the span_duration_seconds histogram was counting every span, including retries. This meant that the aggregate P95 latency for the regulatory_api.call tool was being calculated across both successful first-try calls and multi-retry calls, without any label distinguishing them.

The aggregate P95 looked like 1.8 seconds. The actual P95 for successful first-attempt calls was 0.4 seconds. The team's SLO was set against the aggregate, which was being silently inflated by a retry rate they had never measured directly. They were meeting their SLO on paper while their real user experience was far worse.

The Rebuild: What an Agentic-Native Observability Stack Actually Needs

Once the team had diagnosed all three failure modes, they spent four weeks rebuilding their observability stack around a set of principles specifically designed for agentic, tool-using, multi-agent systems. Here is what they changed.

Principle 1: Model the Execution Graph, Not Just the Trace Tree

The team moved away from a purely hierarchical span model for inter-agent communication. Instead, they introduced a lightweight execution graph manifest: a structured JSON payload that each agent emitted at the start of its work, describing its known inputs, expected outputs, and the IDs of any agents it was spawning or waiting on. The Orchestrator collected these manifests and assembled a true DAG representation of each request's execution.

They built a custom Grafana panel (using the Node Graph visualization plugin) that rendered this DAG in real time, with edge weights representing actual wall-clock elapsed time on each dependency edge. For the first time, the team could see the true critical path of any given request, not just the longest span in a waterfall.

This single change immediately revealed that the Risk Agent was the true bottleneck in 73% of slow requests, not the Compliance Agent as the waterfall had implied.

Principle 2: Treat LLM Inference as a First-Class Telemetry Domain

The team adopted a set of LLM-specific telemetry conventions that had been gaining traction in the broader LLMOps community in 2026. Specifically, they instrumented their LLM gateway to emit the following as distinct, labeled metrics rather than burying them inside generic span durations:

• Time to First Token (TTFT): Measured from request dispatch to the arrival of the first streamed token
• Inter-Token Latency (ITL): Average time between consecutive tokens during generation
• Total Generation Time: Full wall-clock time from request to final token
• Prompt Token Count and Completion Token Count: Emitted as histogram observations, not just log fields
• Model Routing Label: Which underlying model version or provider endpoint served the request

These metrics were stored in a dedicated Prometheus job labeled llm_inference and visualized in a separate Grafana dashboard section. The immediate insight was that TTFT had a P99 of 4.1 seconds during peak hours, driven entirely by the shared LLM endpoint being throttled. This was the single largest contributor to user-perceived latency and had been completely invisible in the old stack.

Principle 3: Instrument Intent, Not Just Execution

One of the team's most important architectural decisions was to introduce what they called intent spans: spans that captured what an agent was trying to do, as distinct from what it actually executed. An intent span was opened the moment an agent decided to invoke a tool, and it carried attributes describing the agent's reasoning context: which task it was working on, what information it was trying to retrieve, and what decision it would make based on the tool's output.

This mattered enormously for debugging. When a tool call failed or was slow, the team could now see not just "the regulatory API took 3.2 seconds" but "the Compliance Agent was trying to verify a specific ISIN code, the regulatory API was slow, and as a result the agent fell back to a cached response from 6 hours ago." The causal chain from user experience degradation to agent behavior to tool failure was finally traceable end-to-end.

Principle 4: Separate Retry Attempts as First-Class Trace Citizens

The team refactored their retry wrapper to emit each attempt as a fully independent span with its own attributes, including an attempt_number label, an attempt_outcome label (success, timeout, rate-limited, error), and a retry_reason label. Retry spans were parented to a new tool invocation span that sat above them and represented the logical tool call, not any individual attempt.

This three-level hierarchy (intent span, tool invocation span, attempt spans) gave the team clean separation between logical operations and physical execution attempts. Their Prometheus histograms were now labeled by attempt_number, allowing them to compute first-attempt P95 latency separately from retry-attempt latency. The real first-attempt P95 for the regulatory API was 0.38 seconds. The retry rate was 22%, which was itself a critical signal they had never seen before.

The Results: Six Weeks Later

After deploying the rebuilt observability stack and acting on the insights it surfaced, Meridian's team reported the following outcomes:

• User-reported P95 latency dropped from 9.4 seconds to 2.1 seconds, achieved primarily by addressing LLM endpoint throttling (moving to a dedicated capacity tier) and optimizing the Risk Agent's quantitative scoring tool.
• The regulatory API retry rate dropped from 22% to 4% after the team implemented proper exponential backoff with jitter and negotiated a higher rate limit tier with the API provider.
• SLO accuracy improved dramatically: The team retired their old aggregate latency SLO and replaced it with three separate SLOs covering TTFT, tool execution critical path, and end-to-end agent response time. Each SLO now tracked something real and actionable.
• Mean time to diagnose (MTTD) for latency regressions dropped from 3.2 days to 4 hours, because the DAG visualization and intent spans made root cause identification immediate rather than requiring manual log correlation.

The Broader Lesson: Agentic Systems Are Not Microservices

The core mistake Meridian's team made was a conceptually understandable one. Microservices and multi-agent systems look superficially similar: both involve distributed components communicating over networks, both have latency budgets to manage, and both benefit from tracing. But the similarities are shallow.

Microservices execute deterministic code paths. Agents make decisions. Microservices call APIs. Agents invoke tools based on reasoning. Microservices have fixed dependency graphs. Agent execution graphs are dynamic, request-specific, and shaped by LLM outputs. Microservices retry on infrastructure failure. Agents may retry, reroute, or abandon tool calls based on semantic judgments about whether a result is good enough.

Every one of these differences has direct implications for how observability data should be collected, structured, and interpreted. Grafting a microservices tracing model onto an agentic system does not just produce incomplete data; as Meridian's experience showed, it produces misleading data that actively steers engineering teams toward the wrong optimizations.

What to Take Away If You Are Building Agentic Infrastructure in 2026

If your team is running multi-agent systems today, here are the concrete questions to ask about your current observability stack:

• Does your tracing model represent concurrency accurately? If your waterfall view always shows sequential spans, you are probably misattributing latency in any system with fan-out.
• Is LLM inference time isolated from transport and gateway time? If you cannot answer "what was the P95 TTFT for this agent over the last hour," you are flying blind on your most expensive and variable cost center.
• Are your retry attempts polluting your aggregate latency metrics? If your histogram does not have an attempt_number label, your P95 and P99 numbers are likely significantly higher than your actual first-attempt performance.
• Can you reconstruct the critical path of a slow request without reading raw logs? If the answer is no, your observability stack is a post-mortem tool, not an operational one.

Conclusion: Observability Debt Is Now an Agentic Problem

The observability tooling ecosystem in 2026 is catching up to the realities of agentic systems, but it is catching up unevenly. OpenTelemetry's semantic conventions for generative AI are maturing. LLMOps platforms are building richer agent tracing primitives. But the default path, reaching for the same distributed tracing stack that served you well in your microservices era, remains a trap for teams that have not explicitly thought through the differences.

Meridian's story is not unique. Across the industry, backend teams are discovering the same class of problem: dashboards that look healthy while users suffer, SLOs that are technically met while real performance is degraded, and optimization efforts aimed at the wrong bottlenecks because the tracing data told a plausible but incorrect story.

The fix is not a new vendor or a new tool. It is a new mental model: one that treats agent intent, tool execution graphs, LLM inference telemetry, and retry semantics as first-class observability concerns rather than afterthoughts bolted onto a microservices framework. Build that mental model first, and the tooling choices will follow naturally.

Your agentic system deserves an observability stack that understands what agents actually do. In 2026, building that stack is no longer optional. It is the difference between dashboards that tell you what you want to hear and dashboards that tell you what you need to know.