The 47-Day Silent Failure: How a Deprecated Tool Endpoint Hijacked an Enterprise Multi-Agent System's Production Decisions

It started with a routine compliance audit. Not a dramatic outage, not a cascade of 500 errors, and not an angry Slack message from a VP at 2 a.m. Just a quiet, methodical review of system logs that uncovered something that made the backend engineering team at a mid-sized fintech company go completely silent in their war room: for 47 consecutive days, their flagship multi-agent orchestration system had been routing live, production-level business decisions through a deprecated internal tool endpoint that was supposed to have been decommissioned in Q4 of 2025.

No alerts had fired. No dashboards had turned red. The system had continued to respond, to process, and to decide, all while silently calling a ghost.

This case study is a reconstruction of what happened, why it happened, and what the engineering and AI governance communities need to learn from it before the same thing happens to their own agentic pipelines.

Background: The Architecture That Made This Possible

The company (referred to here as "Meridian Financial" to preserve confidentiality) had built a multi-agent backend system over the course of 2024 and early 2025. The system was designed to automate a class of mid-tier credit decisioning tasks: flagging accounts for review, adjusting risk scores based on real-time signals, and triggering downstream workflows in their loan servicing platform.

The architecture followed a now-common pattern in enterprise agentic AI:

• An orchestrator agent (built on a major LLM provider's API) received structured task inputs from an upstream data pipeline.
• A set of specialist sub-agents handled discrete functions: data enrichment, regulatory lookup, scoring model calls, and audit logging.
• Each sub-agent was given a tool manifest, a JSON-defined list of callable endpoints representing internal microservices, external APIs, and data stores.
• A central router managed which tool manifest was loaded for each agent session based on environment configuration files.

The system had been running in production since mid-2025 and had processed hundreds of thousands of decisions. By most observable metrics, it was performing well. Latency was within SLA. Decision throughput was high. Downstream teams were happy.

The Deprecated Endpoint: A Ghost in the Manifest

In late November 2025, Meridian's platform engineering team decommissioned v1/risk-signal-enrichment, an internal microservice that had been replaced by a newer, more accurate version (v2/risk-signal-enrichment) with an updated data schema and improved model weights. The migration was handled through a standard deprecation process: the v1 service was flagged as deprecated, the new v2 endpoint was stood up, and internal documentation was updated.

What was not updated was the tool manifest loaded by the data enrichment sub-agent.

The v1 endpoint was not immediately shut down. It was left running in a "deprecated but live" state as a transitional measure, a common practice that, as this incident demonstrates, carries significant hidden risk in agentic architectures. The sub-agent continued calling v1 without complaint. The v1 service continued responding. And because the response schema between v1 and v2 was mostly compatible (differing only in three fields that the orchestrator had not been explicitly trained to validate), the orchestrator accepted the responses without error.

For 47 days, every risk-signal enrichment call made by the multi-agent system was pulling from a deprecated data model, feeding stale feature logic into credit decisions affecting real customers.

Why Nothing Caught It: The Observability Blind Spots

This is the part of the story that should concern every engineering leader building agentic systems today. The failure was not the result of negligence on any single person's part. It was the result of a set of observability assumptions that were designed for traditional microservice architectures and had not been adapted for multi-agent AI systems.

Blind Spot 1: Tool Calls Were Not Version-Audited

The system's logging infrastructure captured agent inputs and outputs at the task level, but it did not log which specific tool endpoint version was being called on each invocation. This is a critical gap. In a conventional microservice system, the calling service typically knows and declares the version it is targeting. In this agentic setup, the tool manifest was loaded at session initialization and assumed to be correct. Nobody was watching whether the manifest itself was current.

Blind Spot 2: Deprecation Signals Did Not Propagate to Agent Configs

The platform team had a deprecation registry, a service catalog that tracked the lifecycle state of every internal endpoint. However, there was no automated mechanism that checked whether any active agent tool manifest referenced an endpoint marked as deprecated. The registry and the agent configuration layer were completely decoupled. A human would have had to manually cross-reference them, and nobody had been assigned that responsibility.

Blind Spot 3: The LLM Did Not "Know" the Endpoint Was Deprecated

This is perhaps the most instructive blind spot of all. The orchestrator LLM was not given any metadata about endpoint lifecycle state. It had no concept of "deprecated" vs. "current." It simply called the tools listed in its manifest and trusted the responses. This is expected and, in isolation, correct behavior. But it underscores a fundamental architectural truth: LLMs in agentic systems are only as trustworthy as the tool environment they operate in. Garbage in, garbage out applies with particular force when the "garbage" looks structurally valid.

Blind Spot 4: Alerting Was Outcome-Focused, Not Process-Focused

Meridian's monitoring stack was tuned to detect outcome anomalies: decision volume spikes, downstream error rates, latency breaches. It was not tuned to detect process anomalies, such as an agent calling an endpoint that should no longer be in active use. This is a distinction that most teams building on top of agentic frameworks have not yet internalized. Outcome monitoring tells you when something has gone wrong at the surface. Process monitoring tells you how it went wrong, and often catches it before the outcome is affected.

What the Compliance Audit Actually Found

The audit was triggered as part of Meridian's standard quarterly AI governance review, a process the company had adopted in response to evolving financial services AI regulations that came into broader enforcement focus in early 2026. The auditors were not specifically looking for this problem. They were performing a routine data lineage trace: for a sample of credit decisions, they were mapping every data source and transformation that contributed to the final output.

When they traced the risk-signal enrichment calls for decisions made between late November 2025 and mid-January 2026, they found that the feature values being passed to the scoring model were inconsistent with what the v2 enrichment service would have produced. Specifically, three fields related to recent payment behavior weighting had been computed using the older v1 logic, which had a known calibration issue that v2 had been specifically designed to correct.

The practical impact varied by account segment. For the majority of decisions, the difference was within acceptable tolerance. But for a subset of approximately 2,300 accounts in a specific risk tier, the stale v1 logic had produced enrichment scores that were measurably more conservative than v2 would have produced. Some of those accounts had been flagged for additional manual review that, under v2 logic, they would not have been flagged for. Others had received adjusted risk scores that influenced downstream pricing in ways that may require remediation.

The compliance and legal teams were immediately looped in. The incident was classified as a material AI system integrity event under the company's internal governance framework.

The Fix: Immediate and Structural

The immediate response was straightforward: the tool manifest for the data enrichment sub-agent was updated to point to v2, the deprecated v1 endpoint was fully shut down, and the affected decision cohort was queued for re-evaluation using the correct enrichment logic.

The structural fixes took longer and were more significant:

1. Manifest-to-Registry Binding

The team built an automated reconciliation job that runs on every deployment and on a nightly schedule. It cross-references every tool endpoint declared in every active agent manifest against the service catalog. If any endpoint is flagged as deprecated or decommissioned, the job raises a blocking alert and, in the case of fully decommissioned services, automatically removes the tool from the manifest and notifies the owning team.

2. Tool Call Versioning in Logs

Every agent tool invocation now logs the full endpoint URI including version, the manifest version that was active for the session, and a hash of the tool definition at call time. This creates a complete, auditable record of not just what the agent decided but what tools it used to decide it.

3. Deprecation-Aware Tool Manifests

Tool manifests now include a lifecycle_state field for each tool entry, sourced directly from the service catalog at manifest generation time. The orchestrator's system prompt was updated to include an instruction set that causes it to flag and escalate tasks if any tool in its manifest is marked with a lifecycle state other than active. This is a lightweight but effective guardrail.

4. Process-Level Monitoring

A new observability layer was added specifically for agent behavior monitoring. It tracks which tools are being called, at what frequency, and whether the calling pattern matches expected baselines. Anomalous tool usage, including calls to endpoints that are not on the "expected active" list, now triggers a real-time alert to both the platform team and the AI governance function.

The Broader Lesson for Enterprise Agentic AI in 2026

By early 2026, multi-agent systems are no longer experimental. They are running payroll processes, making lending decisions, triaging support queues, and managing infrastructure. The tooling ecosystem around them has matured rapidly, with orchestration frameworks, agent memory systems, and LLM routing layers becoming staples of the enterprise backend stack.

But the operational discipline required to run these systems safely has not kept pace with the speed of adoption. Meridian's incident is not an edge case. It is a preview of a category of failure that will become increasingly common as agentic architectures grow in complexity and as the gap between "what the agent is calling" and "what the engineering team thinks it is calling" widens.

Here are the core principles this case study validates:

• Treat tool manifests as first-class infrastructure artifacts. They deserve the same versioning, testing, and lifecycle governance as your APIs and data schemas.
• Deprecation processes must include agent configuration audits. Any service deprecation checklist should include a mandatory step: search all active agent manifests for references to the endpoint being deprecated.
• Observability for agentic systems must be process-aware, not just outcome-aware. If you can only see what the agent decided and not how it decided it, you are flying blind.
• Compliance and AI governance teams need access to tool-call logs. Data lineage in an agentic world extends into the tool layer. Auditors need to be able to trace not just data sources but agent behaviors.
• "Deprecated but live" is a dangerous state in agentic environments. What is a convenient transitional measure in a human-driven microservice ecosystem becomes an invisible trap when autonomous agents are calling your services without human review of each call.

Conclusion: The Audit That Should Have Been the Monitoring

The most uncomfortable truth of Meridian's 47-day incident is this: the compliance audit worked exactly as intended. It caught the problem. But a compliance audit is a lagging indicator. It looks backward. By the time it found the deprecated endpoint, over two thousand customers had already been affected by decisions made with stale data.

The goal for any team running agentic systems in production should be to make the compliance audit boring. To make it a confirmation of what your monitoring already told you, rather than a discovery of what your monitoring missed. That requires investing in observability and governance infrastructure that is purpose-built for the agentic layer, not retrofitted from patterns designed for a world where humans were still the ones deciding which endpoints to call.

Meridian has since shared a sanitized version of their remediation framework with their industry peer group. Several other fintech firms have begun adopting similar manifest governance practices. The incident that nobody was supposed to know about has quietly become one of the most instructive case studies in enterprise agentic AI operations this year.

The 47 days are over. The question is: how many days are your agents currently running on a tool you forgot to update?