LLMOps

How to Build a Multi-Agent Pipeline Observability Dashboard That Surfaces Token Waste, Latency Outliers, and Runaway Agent Loops Before They Appear on Your Q3 2026 Cloud Invoice

Scott Miller

21 Jun 2026 — 10 min read

You deployed your multi-agent pipeline in January. By March, your cloud bill had quietly doubled. By June, it had tripled. Sound familiar? If you are running production AI systems in 2026, this is not a hypothetical horror story. It is a Tuesday.

The core problem is deceptively simple: multi-agent systems are observability black boxes by default. A single user request can fan out into a dozen sub-agent calls, each spawning tool invocations, retrieval steps, and LLM completions. Without a dedicated observability layer, you have no idea which agent is re-prompting unnecessarily, which tool call is adding 4 seconds of latency, or which orchestrator loop has silently started running 40 iterations instead of 4.

This guide walks you through building a practical, production-grade observability dashboard specifically designed to catch the three most expensive failure modes in multi-agent pipelines: token waste, latency outliers, and runaway agent loops. We will cover the instrumentation layer, the data model, the alerting logic, and the dashboard views, with real code you can adapt today.

Why Standard APM Tools Fall Short for Multi-Agent Systems

Traditional Application Performance Monitoring tools like Datadog, New Relic, and Dynatrace were built for request/response microservices. They think in terms of HTTP spans, database queries, and CPU utilization. Multi-agent pipelines break all of these assumptions in a few key ways:

Non-linear execution graphs: Agents branch, recurse, and call other agents conditionally. A flat trace waterfall does not capture this topology meaningfully.
Token economics as a first-class metric: Cost in LLM systems is denominated in tokens, not compute time. Standard APM tools have no concept of prompt tokens, completion tokens, or per-model pricing.
Semantic context loss: Knowing that a span took 2.3 seconds is useless without knowing which agent triggered it, which model was called, and what the input context window size was.
Loop detection requires state across spans: A runaway agent loop is not a slow span; it is a pattern of spans. You need cross-span logic that standard trace viewers do not provide out of the box.

This is why the LLMOps tooling landscape has matured significantly. Platforms like LangSmith, Arize Phoenix, Langfuse, and OpenLLMetry now provide agent-aware tracing. But even these tools require deliberate instrumentation and a purpose-built dashboard layer to surface the cost signals that actually matter before billing day arrives.

The Architecture of Your Observability Stack

Before writing a single line of instrumentation code, you need a clear picture of the stack. Here is the architecture we will build:

Layer 1: Instrumentation (OpenTelemetry + Semantic Conventions)

All traces, spans, and metrics flow from your agent code via OpenTelemetry (OTel). We will use the GenAI semantic conventions that were stabilized in the OTel spec in late 2025, which give us standardized attribute names for LLM calls, token counts, model names, and agent roles.

Layer 2: Collector and Storage

An OTel Collector receives spans and routes them to two destinations: a trace backend (Jaeger, Tempo, or a managed service like Honeycomb) for distributed tracing, and a time-series store (Prometheus + Thanos, or ClickHouse) for aggregated metrics and cost accounting.

Layer 3: The Dashboard (Grafana)

Grafana sits on top of both backends, giving us unified dashboards that combine trace-derived metrics with real-time cost calculations. We will build three purpose-built panels: the Token Waste Heatmap, the Latency Outlier Explorer, and the Agent Loop Watchdog.

Layer 4: Alerting

Grafana Alerting rules fire to Slack or PagerDuty when thresholds are breached, giving you a chance to intervene before a runaway loop runs up a $4,000 overnight bill.

Step 1: Instrument Your Agents with OpenTelemetry

The foundation of everything is clean, consistent instrumentation. Here is how to wrap your agent calls with OTel spans that carry all the attributes your dashboard will need.

First, install the dependencies:

pip install opentelemetry-sdk opentelemetry-exporter-otlp opentelemetry-instrumentation-httpx openllmetry-sdk

Next, set up your tracer and define a reusable decorator for agent steps:


import time
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.trace import SpanKind
from functools import wraps

# Initialize provider
provider = TracerProvider()
exporter = OTLPSpanExporter(endpoint="http://otel-collector:4317")
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("multi-agent-pipeline", "1.0.0")

def agent_span(agent_name: str, model: str):
    """Decorator that wraps an agent step in a rich OTel span."""
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            with tracer.start_as_current_span(
                f"agent.{agent_name}",
                kind=SpanKind.CLIENT,
            ) as span:
                span.set_attribute("gen_ai.system", "openai")
                span.set_attribute("gen_ai.request.model", model)
                span.set_attribute("agent.name", agent_name)
                span.set_attribute("agent.invocation_depth",
                    kwargs.get("depth", 0))

                start = time.monotonic()
                result = func(*args, **kwargs)
                duration_ms = (time.monotonic() - start) * 1000

                # Attach token counts from the LLM response
                if hasattr(result, "usage"):
                    span.set_attribute("gen_ai.usage.prompt_tokens",
                        result.usage.prompt_tokens)
                    span.set_attribute("gen_ai.usage.completion_tokens",
                        result.usage.completion_tokens)
                    span.set_attribute("gen_ai.usage.total_tokens",
                        result.usage.total_tokens)

                span.set_attribute("agent.duration_ms", duration_ms)
                return result
        return wrapper
    return decorator

The critical attributes here are agent.invocation_depth and the token usage fields. These are the raw signals that your dashboard will aggregate into cost and loop-detection metrics.

Step 2: Build a Token Cost Accounting Model

Token counts are meaningless without a pricing model attached. Create a small pricing config that maps model names to per-token costs. Keep this in a config file so you can update it without redeploying your instrumentation layer.


# pricing_config.yaml
models:
  gpt-4o:
    input_cost_per_1k:  0.0025
    output_cost_per_1k: 0.0100
  gpt-4o-mini:
    input_cost_per_1k:  0.00015
    output_cost_per_1k: 0.00060
  claude-3-7-sonnet:
    input_cost_per_1k:  0.003
    output_cost_per_1k: 0.015
  gemini-2-flash:
    input_cost_per_1k:  0.000075
    output_cost_per_1k: 0.000300

In your OTel Collector pipeline, add a transform processor that calculates the dollar cost of each span at collection time and attaches it as a new attribute:


# otel-collector-config.yaml (transform processor snippet)
processors:
  transform/add_cost:
    trace_statements:
      - context: span
        statements:
          - set(attributes["gen_ai.cost.usd"],
              (attributes["gen_ai.usage.prompt_tokens"] / 1000.0
                * resource.attributes["pricing.input_per_1k"])
              + (attributes["gen_ai.usage.completion_tokens"] / 1000.0
                * resource.attributes["pricing.output_per_1k"]))
            where attributes["gen_ai.usage.total_tokens"] != nil

Now every span that represents an LLM call carries a gen_ai.cost.usd attribute. This is the key that unlocks real-time cost dashboards rather than after-the-fact billing surprises.

Step 3: Detect Token Waste Patterns

Token waste is not simply "using a lot of tokens." It is using tokens unnecessarily. The three most common token waste patterns in multi-agent systems are:

Context stuffing: An agent passes the full conversation history to every sub-agent call, even when only the last two turns are relevant.
Redundant retrieval: A RAG agent retrieves the same documents multiple times within a single pipeline execution because there is no within-run cache.
Over-verbose system prompts: System prompts that were written during development and never trimmed, consuming 800+ tokens on every single call.

To surface these in your dashboard, emit a token efficiency ratio metric from your OTel Collector to Prometheus:


# In your agent code, after each LLM call:
from opentelemetry import metrics

meter = metrics.get_meter("agent-cost-meter")

token_efficiency_histogram = meter.create_histogram(
    name="agent.token_efficiency_ratio",
    description="Ratio of completion tokens to prompt tokens. "
                "Low values indicate context stuffing.",
    unit="ratio",
)

def record_token_efficiency(prompt_tokens: int,
                            completion_tokens: int,
                            agent_name: str,
                            model: str):
    if prompt_tokens > 0:
        ratio = completion_tokens / prompt_tokens
        token_efficiency_histogram.record(
            ratio,
            attributes={
                "agent.name": agent_name,
                "gen_ai.request.model": model,
            }
        )

A healthy token efficiency ratio for most task-oriented agents sits between 0.15 and 0.40. If you see an agent consistently below 0.05, it is almost certainly stuffing unnecessary context into its prompts. Flag it. Fix it. The savings are immediate.

Step 4: Identify Latency Outliers with P95/P99 Tracking

Average latency is a lie in multi-agent systems. Your mean agent response time might be a comfortable 800ms, while your P99 is sitting at 18 seconds because one agent occasionally receives a massive context window and stalls. That P99 is what your users actually experience at scale.

In Prometheus, define recording rules that pre-compute latency percentiles per agent and model:


# prometheus-rules.yaml
groups:
  - name: agent_latency
    interval: 30s
    rules:
      - record: agent:duration_ms:p95
        expr: |
          histogram_quantile(0.95,
            sum by (agent_name, model, le) (
              rate(agent_duration_ms_bucket[5m])
            )
          )

      - record: agent:duration_ms:p99
        expr: |
          histogram_quantile(0.99,
            sum by (agent_name, model, le) (
              rate(agent_duration_ms_bucket[5m])
            )
          )

      - record: agent:latency_outlier_ratio
        expr: |
          agent:duration_ms:p99 / agent:duration_ms:p95

The agent:latency_outlier_ratio metric is particularly powerful. A ratio above 3.0 means your P99 is more than three times your P95, which signals a heavy-tail latency problem that averages will never show you. In Grafana, set a threshold alert on this metric per agent name, and you will catch latency bombs before they cascade.

Step 5: Build the Runaway Loop Watchdog

This is the most critical component of the entire dashboard. A runaway agent loop is when an orchestrator agent keeps invoking sub-agents or tools in a cycle, either because the termination condition is never met or because the LLM keeps generating tool calls rather than a final answer. In production, these loops can run for minutes, consuming hundreds of dollars in tokens before any human notices.

The detection strategy has two parts: a depth counter (instrumented in Step 1) and a span count anomaly detector.

Part A: Depth-Based Loop Detection

In your orchestrator, pass and increment a depth counter on every recursive agent call:


MAX_AGENT_DEPTH = 10  # Tune per your pipeline's expected max depth
LOOP_ALERT_THRESHOLD = 7

class AgentOrchestrator:
    def run(self, task: str, depth: int = 0) -> str:
        if depth >= MAX_AGENT_DEPTH:
            # Emit a loop-breach span event before raising
            span = trace.get_current_span()
            span.add_event("agent.loop_limit_reached", {
                "agent.depth": depth,
                "agent.task_preview": task[:100],
            })
            raise RecursionLimitError(
                f"Agent depth limit of {MAX_AGENT_DEPTH} exceeded."
            )

        with tracer.start_as_current_span("orchestrator.step") as span:
            span.set_attribute("agent.invocation_depth", depth)

            # ... agent logic here ...

            if needs_sub_agent:
                return self.run(sub_task, depth=depth + 1)

Part B: Span Count Anomaly Detection via Prometheus

Even without a depth counter, you can detect loops by watching the rate of spans per trace ID. A normal pipeline execution for a given task type should generate a predictable number of spans. Define a baseline and alert on deviations:


# prometheus-alert-rules.yaml
groups:
  - name: agent_loop_watchdog
    rules:
      - alert: AgentLoopSuspected
        expr: |
          sum by (trace_id, agent_name) (
            increase(agent_span_total[2m])
          ) > 50
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Suspected agent loop in {{ $labels.agent_name }}"
          description: >
            Trace {{ $labels.trace_id }} has generated more than 50 spans
            in the last 2 minutes. Possible runaway loop. Current count:
            {{ $value }}.

Wire this alert to PagerDuty for critical pipelines and to Slack for non-critical ones. The 1-minute for clause prevents false positives from legitimately bursty but healthy pipelines.

Step 6: Assemble the Grafana Dashboard

Now that your metrics are flowing, build the three core dashboard panels in Grafana. Here is the panel configuration for each:

Panel 1: Token Cost by Agent (Last 24 Hours)

Use a Bar Chart panel with this PromQL query to show which agents are the biggest cost drivers:


sum by (agent_name, model) (
  increase(agent_token_cost_usd_total[24h])
)

Sort descending. Color-code by model so you can immediately see if an agent is using an expensive model when a cheaper one would suffice.

Panel 2: Token Efficiency Ratio Heatmap

Use a Heatmap panel against your agent.token_efficiency_ratio histogram. Set the color scale from red (ratio below 0.05, severe context stuffing) to green (ratio above 0.2, healthy). This panel makes prompt bloat immediately visible across your entire agent fleet.

Panel 3: Latency Outlier Scoreboard

Use a Table panel showing P50, P95, P99, and the outlier ratio side by side per agent. Add a Grafana threshold on the outlier ratio column: yellow above 2.0, red above 3.0. This is your first-glance latency health check.

Panel 4: Agent Loop Activity Feed

Use a Logs panel pointed at your trace backend, filtered for spans with the agent.loop_limit_reached event or where agent.invocation_depth exceeds your warning threshold (e.g., 7). This gives you a live feed of loop incidents with the task context attached, so you can immediately understand what the agent was trying to do when it went off the rails.

Step 7: Set Up Proactive Cost Budget Alerts

The final piece is the one that directly protects your Q3 invoice. Set a rolling 7-day cost budget per pipeline and alert when you are on track to exceed it:


# prometheus-alert-rules.yaml (cost budget section)
groups:
  - name: cost_budget_watchdog
    rules:
      - alert: PipelineCostBudgetWarning
        expr: |
          sum by (pipeline_name) (
            increase(agent_token_cost_usd_total[7d])
          ) > (
            # Budget thresholds per pipeline (USD)
            vector(500) * on() group_left
            label_replace(
              kube_configmap_annotations{
                configmap="pipeline-budgets"
              },
              "pipeline_name", "$1",
              "annotation_pipeline_name", "(.*)"
            )
          )
        labels:
          severity: warning
        annotations:
          summary: "Pipeline {{ $labels.pipeline_name }} nearing cost budget"
          description: >
            7-day rolling cost for {{ $labels.pipeline_name }} is
            ${{ $value | humanize }}. Review token usage immediately.

Store your budget thresholds in a Kubernetes ConfigMap or a simple YAML file so that product managers and engineering leads can adjust them without touching alert rule code.

Quick-Reference: The Five Metrics That Matter Most

If you take nothing else from this guide, instrument and monitor these five metrics on every multi-agent pipeline you run in production:

gen_ai.cost.usd per span: The dollar cost of each individual LLM call. Aggregated by agent, by model, and by pipeline.
agent.token_efficiency_ratio: Completion tokens divided by prompt tokens. Your early-warning signal for context stuffing.
agent:latency_outlier_ratio (P99/P95): The heavy-tail latency indicator. Anything above 3.0 needs investigation.
agent.invocation_depth: How deep the recursive agent call stack goes. Alert at 70% of your configured maximum.
agent_span_total rate per trace_id: Spans-per-minute for a given trace. Your primary runaway loop detector.

Common Pitfalls to Avoid

Building this dashboard is straightforward; keeping it accurate over time is where teams usually stumble. Watch out for these traps:

Stale pricing configs: Model pricing changes frequently. If your pricing YAML is six months old, your cost calculations are wrong. Automate pricing config updates from provider APIs or a community-maintained pricing registry.
Sampling too aggressively: It is tempting to sample traces at 10% to reduce storage costs. For multi-agent systems, sample at 100% for short-running pipelines and use tail-based sampling (keeping traces that contain errors or high costs) for long-running ones. You cannot detect a loop from a sampled trace.
Ignoring cached token discounts: Most major providers in 2026 offer significant discounts on cached prompt tokens. If your instrumentation does not distinguish between cached and uncached input tokens, your cost model will overestimate, and you will chase phantom waste.
Conflating agent retries with loops: A legitimate retry after a tool failure looks exactly like a loop in span-count metrics. Tag retry spans explicitly with agent.retry=true and exclude them from your loop detection queries.

Conclusion: Observability Is a Cost Control Strategy

The multi-agent systems that will survive Q3 2026 and beyond are not the ones with the most capable models. They are the ones whose teams actually know what is happening inside them. Token waste, latency outliers, and runaway loops are not exotic edge cases; they are the default behavior of uninstrumented agent pipelines running at scale.

The dashboard you have just built does three things that no cloud billing alert can do: it tells you which agent is wasting money, why it is wasting money, and when it started wasting money. That is the difference between a retrospective post-mortem and a proactive engineering culture.

Start with Step 1 today. Get the OTel spans flowing, attach token counts and costs, and build the loop watchdog first. You do not need a perfect dashboard on day one. You need enough signal to stop being surprised by your invoice. Everything else is iteration.

Your Q3 cloud budget will thank you.