How One Enterprise Backend Team Discovered Their LLM Caching Layer Was Silently Serving Stale Tool-Call Results Across Concurrent Agentic Sessions

When you build a production-grade agentic AI system, you expect the hard problems to announce themselves loudly: a model hallucination that crashes a workflow, a timeout that surfaces in your APM dashboard, a malformed JSON response that throws a clear exception. What you do not expect is a bug that whispers. A bug that serves confidently wrong answers while every health check stays green.

This is a case study about exactly that kind of bug. It affected a mid-sized fintech company's internal AI platform team (names changed for confidentiality), and it lived undetected in production for nearly six weeks before a sharp-eyed QA engineer noticed something that didn't add up. The root cause was a caching layer that had been designed thoughtfully for a single-session world, then quietly promoted into a concurrent, multi-agent architecture without the right invalidation logic. The fix involved schema versioning at the tool-call result level, and it's now considered one of the most important architectural lessons the team has carried into 2026.

The System: A Multi-Agent Financial Research Platform

The team, which we'll call the Meridian platform team, had built an internal agentic AI system for financial research analysts. The platform allowed analysts to spin up autonomous research agents that could call a suite of internal tools: a live market data API, a proprietary earnings database, a regulatory filing search index, and a news sentiment aggregator.

Each agent session was orchestrated via a central LLM (a fine-tuned variant of a frontier model), which would decompose analyst queries into sub-tasks, issue tool calls, receive structured JSON results, reason over them, and produce research summaries. Sessions could run for several minutes, spawn parallel sub-agents for different data domains, and occasionally share context through a shared memory store.

To control costs and latency, the team had introduced an LLM output caching layer early in development. The logic was sensible at the time: if the LLM received an identical prompt (same system prompt, same user message, same tool schema), return the cached completion rather than making a new API call. This worked beautifully in staging, where sessions were sequential and tool schemas were stable.

Production, as it turned out, was a different story.

The Symptom: Confident Answers That Were Subtly Wrong

The first sign of trouble came from an analyst who noticed that two separate research reports, generated on the same morning for different clients, contained the same earnings-per-share figure for a company that had reported new results overnight. The figures weren't wildly wrong. They were just one quarter stale, and they were identical down to the decimal point.

At first, the team suspected a data pipeline issue. The earnings database was checked and confirmed to be current. The market data API was returning fresh values. The news sentiment aggregator was live. Every upstream data source was healthy. The agents, however, were not consuming those fresh values in certain scenarios.

Over the following week, the QA engineer, working with one of the senior backend engineers, began cataloguing the anomalies. The pattern that emerged was striking:

• Stale results appeared almost exclusively during high-concurrency windows, typically between 8:00 and 10:00 AM when analysts were starting their day.
• The stale data always matched a result that had been legitimately correct at some earlier point in the day or the previous session.
• Agents running in isolation, outside of peak concurrency, almost never exhibited the issue.
• The affected tool calls were always from the earnings database tool and the regulatory filing search tool, never from the live market data tool.

That last observation cracked the case open.

Digging In: How the Cache Key Was Constructed

The team audited their caching layer. The cache key for any LLM completion was a hash of three components:

1. The full system prompt.
2. The serialized conversation history (all user and assistant messages).
3. The tool definitions array passed to the model.

On the surface, this looked correct. If the tool definitions changed, the hash would change, and the cache would miss, forcing a fresh LLM call. The problem was not in the cache key for the LLM completion itself. The problem was one layer deeper: the tool-call result caching layer.

The team had a secondary cache, added later and somewhat informally, that cached the results returned by the tools themselves. The idea was reasonable: if an agent asked for Apple's Q4 earnings, and another agent in a different session asked for the same thing five minutes later, why make two round-trips to the earnings database? The tool result cache key was built from:

• The tool name (e.g., get_earnings_data).
• The arguments passed to the tool (e.g., {"ticker": "AAPL", "period": "Q4-2025"}).

There was no TTL short enough to matter. There was no session isolation. And critically, there was no versioning tied to the tool's output schema.

The Root Cause: Schema Drift Without Cache Invalidation

Here is where the subtle architectural failure becomes clear. Over the preceding two months, the team had quietly evolved the earnings database tool's response schema. What began as a simple flat JSON object had grown to include nested objects, new fields for non-GAAP metrics, and a revised structure for guidance data. The tool's name and its input arguments had not changed. Only the shape of its output had evolved.

Because the tool result cache key was built only from the tool name and input arguments, a cached result from an older schema version was considered a valid hit for a request expecting a newer schema version. The LLM, receiving this older-shaped JSON in its context, would reason over it faithfully and produce a response that was structurally coherent but factually stale or incomplete.

The concurrency angle explained the timing. During high-traffic morning windows, multiple agents would be invoking the same tools with the same arguments within seconds of each other. The first agent to execute a tool call would populate the cache. Every subsequent agent within the TTL window would receive that cached result, regardless of whether the underlying data had been updated or whether the result schema had drifted.

The live market data tool was immune because it had a very short TTL (15 seconds) set explicitly by the engineer who built it, who had correctly anticipated that freshness was critical. The earnings and regulatory tools had been given a TTL of 4 hours, based on the assumption that those data sources updated infrequently. That assumption was mostly correct, but it completely ignored the schema evolution dimension.

The Fix: Tool-Call Result Schema Versioning

The team's solution was elegant and has since become a standard pattern in their platform. It has three components:

1. Explicit Schema Versions on Every Tool Definition

Every tool in the platform now carries an explicit schema_version field in its definition object. This is a semantic version string (e.g., "2.3.0") that is incremented whenever the tool's output schema changes in any meaningful way, including additive changes. The version is maintained in a central tool registry, and a CI check enforces that any pull request modifying a tool's response model must also bump the schema version.

{
  "name": "get_earnings_data",
  "schema_version": "2.3.0",
  "description": "Retrieves earnings data for a given ticker and period.",
  "parameters": { ... }
}

2. Schema Version Included in the Cache Key

The tool result cache key now includes the tool's schema_version as a mandatory component. The key is constructed as:

cache_key = hash(tool_name + schema_version + sorted(tool_arguments))

This means that when the earnings tool's schema is bumped from 2.2.1 to 2.3.0, every previously cached result for that tool is effectively orphaned. The new requests generate new cache keys, hit misses, and fetch fresh results. The old entries expire naturally via TTL without any active purge needed.

3. A Cache Manifest for Observability

The team added a lightweight cache manifest: a structured log entry written on every cache hit that records the tool name, schema version, cache key, age of the cached entry, and the session ID of the agent consuming it. This manifest feeds into their observability stack and powers a dashboard that surfaces cache hit rates broken down by schema version. If an old schema version is still generating hits, it's a signal that something in the deployment pipeline failed to propagate the version bump correctly.

This observability layer alone has caught two subsequent schema drift incidents before they reached analysts.

Broader Lessons for Agentic AI Architecture in 2026

This incident is not unique to Meridian's team. As agentic AI systems mature and move from prototype to production, the caching patterns borrowed from traditional web and API development are being stress-tested in ways their original designers never anticipated. A few broader lessons stand out:

Tool-Call Results Are Not Static API Responses

In a traditional microservices architecture, a GET endpoint's response schema is governed by an API contract with explicit versioning in the URL or headers. In agentic systems, tool calls often feel more informal, more like internal function calls. This informality leads teams to underinvest in schema governance. Treating tool output schemas with the same rigor as public API contracts is no longer optional at enterprise scale.

Concurrency Multiplies the Blast Radius of Caching Bugs

A caching bug in a single-session system causes one wrong answer. The same bug in a concurrent multi-agent system causes the same wrong answer to propagate across every session that shares the cache within the TTL window. The blast radius scales with concurrency, and enterprise agentic platforms are inherently high-concurrency systems during peak usage.

Session Isolation and Cache Sharing Are in Tension

There is a genuine engineering tradeoff between cache efficiency (sharing results across sessions to reduce latency and cost) and session isolation (ensuring each session operates on data appropriate to its context and timing). The right answer is not always "never share cache across sessions." It is, rather, to be explicit about what the cache key encodes and what invariants it assumes. Schema versioning is one critical invariant. Data freshness requirements are another.

Silent Failures Are the Most Dangerous Failures in AI Systems

This bug produced no exceptions, no error logs, no failed health checks, and no anomalous latency. Every metric looked healthy. The only signal was a subtle factual discrepancy that a human analyst happened to notice. As AI systems are trusted with higher-stakes decisions, the industry needs much better tooling for detecting semantic correctness failures, not just operational failures. Cache manifests, result provenance tracking, and automated output consistency checks are becoming essential parts of the production AI stack.

What the Team Would Do Differently From Day One

When asked what they would change if they were rebuilding the platform from scratch, the Meridian team's senior engineer gave a clear answer: "We would treat the tool result cache as a first-class versioned data store from the beginning, not an afterthought optimization. Every cached artifact would carry its schema version, its data source version, and its session context requirements. We got lucky that the bug was caught by a human. We can't rely on that."

They have since published an internal architecture decision record (ADR) that mandates schema versioning for all tool-call result caches across the organization's AI platform teams. The ADR is now part of their onboarding checklist for any new agentic system that enters the design phase.

Conclusion

The Meridian incident is a reminder that as agentic AI systems grow in complexity and concurrency, the failure modes grow subtler. The caching layer that saves you money and latency in a simple pipeline becomes a liability if it is not designed with schema governance in mind. Tool-call result caching is a powerful optimization, but it demands the same discipline as any other versioned data contract in a production system.

Schema versioning is not a glamorous fix. It does not involve a new model, a new framework, or a novel algorithm. It is, at its core, just good software engineering applied to a new class of artifact. But in a world where AI agents are trusted to produce research, make recommendations, and drive decisions, "just good software engineering" is exactly what the moment calls for.

If your team is running agentic AI in production and your tool-call result cache keys do not include a schema version, this is worth checking today. The bug might already be whispering.