Blind Spots in the Machine: How Enterprise Backend Teams Must Architect Multi-Agent Pipeline Observability When Token-Level Logging Disappears

There is a quiet but seismic shift underway in how foundation model providers expose usage data to their enterprise customers. Through the second half of 2026, major providers have been progressively deprecating granular, per-request token-level logging endpoints in favor of consolidated, aggregated billing dashboards. The stated reasons are reasonable enough: cost reduction on the provider side, simplified compliance surfaces, and cleaner UX for the majority of customers who only ever cared about their monthly bill.

For those customers, this is fine. For enterprise backend teams running production multi-agent pipelines, it is a quiet catastrophe.

When your observability strategy depends on pulling per-call token counts, latency breakdowns, and prompt/completion splits directly from the provider's logging API, and that API is replaced by a dashboard that tells you "you spent $4,312 this week," you have just lost the nervous system of your entire debugging and cost-attribution infrastructure. This post is a deep dive into what that actually means architecturally, and more importantly, how to rebuild that nervous system so it lives entirely within your own stack.

Why Granular Token Logging Mattered So Much in the First Place

Before we talk about solutions, it is worth being precise about what is being lost. In a multi-agent pipeline, a single user-facing action might trigger a cascade of model calls: a planning agent, several tool-calling subagents, a summarization agent, a validation agent, and a final synthesis step. Each of those calls has its own prompt token count, completion token count, model version, latency, and cost. Provider-side token logging gave backend teams three critical capabilities:

• Per-agent cost attribution: Understanding which agent in the pipeline is responsible for the majority of spend, enabling targeted optimization.
• Anomaly detection at the call level: Catching prompt injection attacks, runaway loops, or unexpectedly large context windows before they balloon into billing surprises.
• Latency decomposition: Separating model inference time from network round-trip time, which is essential when debugging SLA breaches in a pipeline with five or more sequential model calls.

Aggregated billing dashboards provide none of these. They are accounting tools, not engineering tools. Losing provider-side granular logging does not just make debugging harder; it removes the ground truth that many teams have been using to validate their own instrumentation. That second point is subtle but critical: if you have been using provider logs as a source of truth to cross-check your client-side measurements, you now need a new source of truth. You need to become your own provider, observability-wise.

The Core Architectural Principle: The Instrumented Proxy Layer

The foundational shift in thinking required here is this: your observability boundary must move from the provider's edge to your own infrastructure edge. Every model call your multi-agent pipeline makes must pass through an instrumented proxy layer that you own, operate, and trust. This is not a new idea in distributed systems, but it is newly non-negotiable for LLM workloads.

This proxy layer sits between your agent orchestration framework (whether that is a custom implementation, LangGraph, AutoGen, or a proprietary internal system) and the foundation model provider's API. It intercepts every request and response, captures the data you need, and forwards calls transparently. Critically, it must do this with negligible latency overhead: a well-implemented async instrumentation proxy should add no more than 2 to 5 milliseconds to each call, which is essentially invisible against typical model inference times of 500ms to several seconds.

What the Proxy Must Capture

The proxy is not just a logging shim. It needs to capture a specific set of signals that, in combination, reconstruct everything you previously got from provider-side logs:

• Request timestamp and trace ID: Wall-clock time at request dispatch, correlated with a distributed trace ID that spans the entire agent pipeline execution.
• Raw prompt payload hash: A SHA-256 hash of the serialized prompt (not the prompt itself, for data governance reasons) so you can detect identical or near-identical prompts being sent redundantly.
• Client-side token count (pre-call): Using a local tokenizer (tiktoken for OpenAI-compatible APIs, the provider's published tokenizer library for others) to count tokens before the call is dispatched. This is your replacement for the provider's "prompt tokens" field.
• Response timestamp and first-token latency: For streaming responses, capture the time to first token (TTFT) separately from time to completion. These are different signals and conflating them masks real performance problems.
• Completion token count (post-call): Count tokens in the completion using the same local tokenizer. For most modern tokenizers, this is accurate to within 1 to 2 tokens of the provider's own count.
• Model version and deployment ID: Providers increasingly route to different model versions under the same API endpoint name. Capture the exact model identifier from the response header or body.
• HTTP status code and error type: Distinguish between rate limit errors (429), context length errors (400 with specific body content), server errors (500/503), and timeout conditions. Each has a different remediation path.
• Agent identity and pipeline stage: Injected via request metadata or a thread-local context, this tells you which agent in your pipeline made this call and at what stage of execution.

Reconstructing Cost Attribution Without Provider Data

One of the most practically painful losses is cost attribution. When a provider's aggregated dashboard tells you your weekly spend, you cannot decompose that into "the summarization agent costs 3x what we expected" without per-call data. Here is how to reconstruct it entirely client-side.

Maintaining a Local Pricing Ledger

Your instrumented proxy layer should maintain a versioned, in-memory pricing configuration that maps model identifiers to per-token costs for both input and output tokens. This sounds simple, but there are important details:

• Version it explicitly: Providers change pricing. Your pricing ledger needs to be versioned with effective dates so that historical cost calculations remain accurate when you run retrospective analyses. Store it in your configuration management system, not hardcoded.
• Account for cached tokens: Several providers now offer reduced pricing for prompt tokens that hit a cache (prompt caching, KV cache reuse). Your proxy must detect cache-hit signals in the response and apply the correct discounted rate. Failing to do this will cause your client-side cost estimates to run 20 to 40 percent high for pipelines with repetitive system prompts.
• Handle batch vs. real-time pricing: Batch inference endpoints typically carry a 50 percent cost reduction. If your pipeline uses a mix of synchronous and batch calls, your cost ledger must track which pricing tier applies to each call.

With these in place, you can compute a estimated_cost_usd field for every single model call your system makes, in real time, without touching the provider's dashboard at all. The accuracy of these estimates against actual billing is typically within 2 to 5 percent, which is more than sufficient for engineering decision-making.

The Telemetry Pipeline: From Proxy to Storage

Capturing data at the proxy is only half the problem. You need to get that data into a queryable store efficiently, without creating a synchronous bottleneck in your critical path. The architecture here follows a well-established pattern from high-throughput distributed systems:

Async Emission via a Local Buffer

The proxy writes telemetry events to a local, in-process ring buffer (or a lightweight local queue like a Unix domain socket connected to a sidecar process). This write is non-blocking and happens after the response has been forwarded to the calling agent. The buffer drains asynchronously to a message broker: Kafka or Redpanda for high-volume deployments, or a managed service like AWS Kinesis or Google Pub/Sub for teams that prefer not to operate their own streaming infrastructure.

The key constraint here is that telemetry emission must never block or slow down the model call itself. If your observability infrastructure has an outage, your agents should continue operating normally, with telemetry events buffered locally until the pipeline recovers. This requires explicit backpressure handling and a bounded buffer with a defined overflow policy (drop oldest, not drop newest, since recent events are more valuable for real-time alerting).

The Dual-Write Pattern for Real-Time and Historical Analysis

From the message broker, implement a dual-write consumer pattern:

• Hot path: A stream processor (Apache Flink, Spark Structured Streaming, or a simpler solution like a Kafka Streams application) computes real-time aggregations: rolling 5-minute token spend per agent, error rates per model endpoint, p95 latency per pipeline stage. These aggregations feed into a time-series database (InfluxDB, Victoria Metrics, or Prometheus with remote write) and drive your real-time dashboards and alerting rules.
• Cold path: Raw telemetry events are written to columnar storage (Apache Parquet on S3/GCS, or a managed data warehouse like BigQuery or Snowflake) for ad-hoc analysis, cost reporting, and model performance retrospectives. Partition by date and agent identity for efficient querying.

This dual-write pattern gives you both the real-time visibility you need for operational monitoring and the historical depth you need for capacity planning and optimization work. Neither alone is sufficient.

Distributed Tracing Across Agent Boundaries

Token-level data is only one dimension of observability. The other critical dimension in a multi-agent system is understanding the causal chain: which agent called which other agent, in what order, with what inputs and outputs, and how long each step took. This is the domain of distributed tracing, and it deserves its own architectural attention.

Propagating Trace Context Through Agent Handoffs

Every agent-to-agent handoff in your pipeline must propagate a W3C Trace Context header (or an equivalent internal correlation ID if you are not using HTTP for inter-agent communication). This sounds obvious, but it is consistently the most common gap in multi-agent observability implementations. Teams instrument the model calls but forget to instrument the orchestration layer that dispatches agents and collects their results.

Your orchestration framework should inject a x-trace-id and x-span-id into every agent invocation context. Each agent appends its own span to the trace before making its model calls, so that the resulting trace tree shows you the complete execution graph: root orchestrator, child agents, model calls within each agent, tool calls within each model call, and the timing of every node.

OpenTelemetry (OTel) is the right standard to use here. The OTel semantic conventions for LLM calls (the gen_ai namespace, now well-established in the 2026 OTel specification) provide a standardized schema for recording model calls as spans with the correct attributes. Using OTel means your traces are compatible with any OTel-compatible backend: Jaeger, Tempo, Honeycomb, Datadog, Dynatrace, and others.

Span Attributes That Replace Provider Log Fields

When you record a model call as an OTel span, populate it with the following attributes to fully replace what you previously got from provider logs:

• gen_ai.system: The provider name (e.g., "openai", "anthropic", "google").
• gen_ai.request.model: The model name as sent in the request.
• gen_ai.response.model: The actual model version returned in the response (these can differ).
• gen_ai.usage.input_tokens: Client-side token count for the prompt.
• gen_ai.usage.output_tokens: Client-side token count for the completion.
• gen_ai.usage.cache_read_input_tokens: Tokens served from provider cache, if applicable.
• llm.agent.id: Your custom attribute identifying the agent making this call.
• llm.pipeline.stage: The named stage of your pipeline (planning, execution, validation, synthesis, etc.).
• llm.estimated_cost_usd: The computed cost for this call, using your local pricing ledger.

With these attributes on every span, your distributed trace is a complete, self-contained record of your pipeline's behavior that does not depend on any provider-side data at all.

Alerting Without Provider-Side Anomaly Detection

Provider dashboards sometimes included basic anomaly detection: alerts when your spend spiked, for example. With aggregated billing dashboards, even these coarse signals may be delayed by hours. You need to implement your own alerting layer, and it needs to be fast enough to catch runaway agent loops before they cause significant financial damage.

The Three Alerts Every Multi-Agent Pipeline Needs

Based on the failure modes most commonly seen in production multi-agent systems, these three alerting rules should be considered the minimum viable set:

1. Per-pipeline-run token budget alert: Every pipeline execution should have a maximum token budget. If a single run exceeds, say, 500,000 tokens, something has gone wrong (likely an agent loop or a context window that grew out of control). Alert immediately and optionally trigger a circuit breaker that halts the run. Implement this as a counter that the proxy increments per trace ID, with a threshold check on each increment.
2. Rolling spend rate alert: Compute a rolling 5-minute estimated spend rate across all model calls. If this rate exceeds 2x your baseline (calculated from the previous 7-day moving average for the same time window), fire an alert. This catches sudden traffic spikes, prompt injection attacks that trigger expensive completions, and infrastructure misconfigurations that cause duplicate requests.
3. Error rate spike alert: A sudden increase in 429 (rate limit) or 500 (server error) responses from the provider is an early warning signal for both provider-side incidents and your own traffic pattern problems. Alert when the error rate for any model endpoint exceeds 5 percent over a 2-minute window.

Data Governance and the Prompt Privacy Problem

There is one important constraint that shapes all of the above: in most enterprise environments, you cannot log raw prompt content. Prompts may contain PII, proprietary business data, or confidential customer information. This was actually one reason some teams relied on provider-side logging: the provider was already handling the data under a DPA, so the enterprise did not have to worry about their own logging infrastructure being a compliance liability.

Now that you are building your own instrumentation layer, you must design it with privacy in mind from the start. The recommended approach is a two-tier logging strategy:

• Structural metadata only (always logged): Token counts, latency, cost, model version, agent ID, trace ID, error codes. None of this contains sensitive content. Log it freely to your telemetry pipeline.
• Prompt content (logged only with explicit opt-in and redaction): If your use case genuinely requires prompt content for debugging (for example, in a development or staging environment), run all prompt content through a PII redaction pipeline before logging. Use a combination of regex patterns for structured PII (emails, phone numbers, SSNs) and a small, fast classification model for unstructured PII detection. Log the redacted version only, and only in environments where your data governance policy explicitly permits it.

This two-tier approach gives you the operational visibility you need without creating a compliance liability. It also means your telemetry infrastructure is safe to run in production from day one.

Putting It All Together: The Reference Architecture

To summarize the full stack described in this post, here is the reference architecture for a self-sufficient multi-agent observability system that is completely independent of provider-side logging:

1. Instrumented Proxy Layer: Intercepts all model API calls, counts tokens with a local tokenizer, computes estimated cost, captures latency signals, and emits OTel spans. Non-blocking, with async telemetry emission.
2. Local Buffer and Message Broker: Ring buffer in-process, draining to Kafka/Redpanda or a managed streaming service. Decouples the critical path from the observability pipeline.
3. Stream Processor (Hot Path): Computes real-time aggregations for dashboards and alerting. Feeds a time-series database and an alerting engine (Alertmanager, PagerDuty, etc.).
4. Columnar Storage (Cold Path): Raw telemetry events in Parquet on object storage or a data warehouse. Used for cost reporting, capacity planning, and retrospective analysis.
5. Distributed Trace Backend: OTel-compatible trace store (Tempo, Jaeger, Honeycomb, or similar) receiving spans from the proxy layer and the orchestration framework. Provides the causal execution graph for every pipeline run.
6. Versioned Pricing Ledger: Configuration-managed, versioned mapping of model IDs to token prices. Updated promptly when providers change pricing. Used by the proxy to compute estimated_cost_usd on every call.
7. PII Redaction Pipeline: Sits in front of any prompt content logging path. Always on, even in development environments, to build the habit.

The Uncomfortable Truth About Provider Dependency

The broader lesson here extends beyond the specific issue of token logging deprecation. For years, enterprise teams have been building observability strategies that rely on provider-side data as a source of truth. That was always a fragile dependency: providers can change their APIs, deprecate endpoints, alter data retention policies, or simply decide that granular logging is not a product feature they want to maintain.

The teams that will navigate the H2 2026 logging changes with the least disruption are the ones that already treated provider-side logs as a convenience rather than a dependency. They built client-side instrumentation first, used provider logs only as a cross-check, and maintained a proxy layer as a standard part of their LLM integration architecture. For everyone else, the deprecation is a forcing function, and it is worth treating it as an opportunity to build the right foundation rather than a patch to apply.

The good news is that the architecture described here is not significantly more complex than what you should have been building anyway. It follows established distributed systems patterns (proxy layers, async telemetry, dual-write pipelines, distributed tracing) applied to a new problem domain. The engineering investment is real but bounded, and the result is an observability stack that is more robust, more detailed, and more under your control than anything a provider dashboard was ever going to give you.

Conclusion

Foundation model providers removing granular token-level logging is not the end of LLM observability. It is the end of outsourced LLM observability, and that is probably a good thing. The teams that build their own instrumented proxy layers, maintain local pricing ledgers, implement distributed tracing across agent boundaries, and design privacy-safe telemetry pipelines will have deeper, more reliable visibility into their multi-agent systems than they ever had when they were reading it off a provider dashboard.

The shift requires real engineering work. But the alternative, flying blind in a production multi-agent pipeline where a single misconfigured agent can burn thousands of dollars in minutes, is not acceptable for any serious enterprise deployment. Build the stack. Own the data. The providers were never going to be your observability team anyway.