The Silent Data Bleed: How Kestrel Financial Rebuilt Its AI Anomaly Detection Pipeline After Multi-Tenant Inference Endpoints Exposed Customer Context Across Sessions

In early 2026, the engineering team at Kestrel Financial, a mid-size fintech serving roughly 340,000 retail and SMB customers across North America, made a discovery that stopped their roadmap cold. Their flagship AI-powered transaction anomaly detection system, which had been praised internally for its precision and speed, was silently bleeding customer financial context across concurrent inference sessions. The culprit was not a rogue developer, a misconfigured firewall, or a third-party breach. It was something far more insidious: the architectural assumption that a shared, multi-tenant large language model (LLM) inference endpoint would cleanly isolate session state between simultaneous requests.

It did not.

This is the story of how Kestrel discovered the problem, quantified its scope, and rebuilt their entire agent-driven pipeline from the ground up. It is also a cautionary tale for every fintech, insurtech, and financial services platform that has embraced agentic AI without fully stress-testing the boundaries between tenants at the inference layer.

Background: Kestrel's Agentic Anomaly Detection Architecture

By mid-2025, Kestrel had migrated their rule-based fraud detection system to a multi-agent architecture. The new system used a three-layer agent pipeline:

• Ingestion Agents: Streamed and normalized transaction events from payment rails, card networks, and ACH feeds in real time.
• Reasoning Agents: Consumed enriched transaction records, retrieved relevant customer behavioral profiles from a vector store, and submitted structured prompts to a hosted LLM inference endpoint to score anomaly likelihood and generate plain-language explanations for compliance teams.
• Action Agents: Routed flagged transactions to human reviewers, triggered automated holds, or escalated to a risk scoring model for further adjudication.

The LLM at the center of the reasoning layer was accessed via a shared inference endpoint provided by a major cloud AI platform. Kestrel had negotiated a high-throughput tier, allowing hundreds of concurrent requests. The endpoint was not a dedicated deployment; it was a pooled, multi-tenant service shared across the provider's customer base. At the time, this seemed like a reasonable cost and scalability trade-off. In practice, it became the source of a significant data integrity and privacy failure.

The Discovery: When Anomaly Scores Stopped Making Sense

The problem surfaced in February 2026 during a routine model performance audit. Kestrel's ML Ops lead, reviewing a batch of anomaly explanations generated over a 48-hour window, noticed something strange. Several explanation outputs referenced spending patterns, merchant categories, and even approximate balance thresholds that did not match the customer whose transaction was being analyzed.

One explanation, generated for a flagged transaction from a small-business owner in Phoenix, described the customer as "a frequent traveler with recurring international hotel charges and a pattern of high-value electronics purchases." The actual customer had never made an international transaction. The description matched, almost exactly, the behavioral profile of a different Kestrel customer whose transactions had been processed in a near-concurrent session milliseconds earlier.

The team initially suspected a bug in their vector store retrieval logic, assuming the wrong customer profile was being injected into the prompt. But after auditing the retrieval pipeline, all retrieved context was confirmed to be correct at the application layer. The contamination was happening after the prompt left Kestrel's infrastructure, inside the inference endpoint itself.

Root Cause Analysis: The KV Cache Problem in Shared Inference

After engaging their cloud AI provider and conducting their own analysis, Kestrel's engineering team identified the root cause: key-value (KV) cache residue in the shared inference pool.

Modern LLM inference systems use KV caching aggressively to reduce compute costs and latency. In a properly isolated, single-tenant deployment, this is safe. The cache is scoped to a single user or session. But in a high-throughput, multi-tenant pooled environment, the boundary between cache scopes can become porous under certain conditions, particularly when:

• Request batching groups prompts from different tenants into the same forward pass for efficiency.
• Prefix caching (a technique that reuses cached computation for shared prompt prefixes) is applied without strict tenant-level isolation keys.
• Session context windows are reused across requests when the pool recycles inference workers under load.

In Kestrel's case, the provider's prefix caching implementation was grouping requests that shared a common system prompt prefix, which Kestrel used across all their reasoning agent calls. Because the system prompt was identical across customers, the caching layer treated multiple customers' sessions as candidates for prefix reuse. Under high concurrency, residual token activations from one customer's context were bleeding into the next session's completion, particularly in the early tokens of the generated explanation where the model was "warming up" its contextual framing.

The provider acknowledged the behavior but categorized it as a known performance optimization trade-off rather than a security defect. This response, understandably, did not satisfy Kestrel's legal, compliance, or engineering teams.

Quantifying the Scope: A 90-Day Forensic Review

Kestrel immediately pulled 90 days of logged inference outputs and ran a systematic cross-contamination audit. They built a lightweight classifier trained to detect when an explanation output contained behavioral descriptors inconsistent with the target customer's verified transaction history. Key findings from the audit included:

• 2.3% of all explanation outputs contained at least one factual descriptor that could not be traced to the target customer's own data.
• Contamination was strongly correlated with peak load windows, with incidence rates rising to over 6% during periods of maximum concurrent throughput.
• In 0.4% of cases, the contamination was significant enough that the anomaly score itself was likely influenced, meaning a transaction may have been flagged or cleared based partly on another customer's behavioral context.
• No evidence was found of full account numbers, passwords, or explicit PII being surfaced in outputs. The contamination was primarily behavioral and contextual, not raw data exfiltration. However, from a regulatory standpoint, behavioral financial context is still considered sensitive personal financial information under applicable data protection frameworks.

Kestrel disclosed the findings to their legal and compliance teams, initiated a regulatory notification process with relevant financial oversight bodies, and began the architectural rebuild immediately.

The Rebuild: Four Architectural Pillars

Kestrel's engineering team, working alongside external AI security consultants, designed a new pipeline architecture around four core principles: hard tenant isolation, context minimization, deterministic auditability, and inference sovereignty.

Pillar 1: Dedicated, Single-Tenant Inference Deployment

The most impactful change was the most straightforward: Kestrel moved off the shared multi-tenant endpoint entirely. They provisioned a dedicated inference cluster using a self-hosted open-weight model fine-tuned on financial reasoning tasks, deployed on isolated GPU infrastructure within their own cloud VPC. No other organization's requests touch this cluster. KV cache state is scoped exclusively to Kestrel's own sessions, and the team controls the full inference stack, including batching policies, cache eviction rules, and concurrency limits.

The cost increase was significant, approximately 3.1x the prior per-token cost. However, when weighed against potential regulatory fines, reputational damage, and the cost of the forensic audit itself, the economics were not difficult to justify.

Pillar 2: Per-Customer Context Sandboxing at the Prompt Layer

Even with a dedicated endpoint, Kestrel implemented strict per-session context sandboxing at the application layer. Each reasoning agent now constructs a prompt that includes a cryptographically signed session token binding the inference request to a specific customer ID. A lightweight validation layer checks that the output references only entities and descriptors consistent with the input context before the explanation is passed downstream.

This acts as a defense-in-depth measure. Even if a caching anomaly were to occur at the infrastructure level, the validation layer would catch and quarantine the contaminated output before it influenced any downstream decision or audit log.

Pillar 3: Context Minimization and Structured Prompting

The original pipeline injected rich, free-text customer behavioral profiles into prompts, including narrative summaries generated by an earlier agent stage. This unstructured context was both verbose and difficult to validate. The new pipeline replaces free-text profiles with structured, schema-validated JSON context objects containing only the specific fields necessary for the anomaly reasoning task.

Each field in the context object is tagged with the customer ID it belongs to. The reasoning agent is prompted to reference only explicitly provided structured fields, and the output schema is constrained so that the model cannot generate free-form behavioral narratives. Instead, it produces a structured anomaly report with discrete scored dimensions, each traceable back to a specific input field.

This change reduced average prompt token length by 41% and made cross-contamination detection trivially easy: any output field referencing a value not present in the input schema is automatically flagged.

Pillar 4: Full Inference Audit Logging with Input-Output Hash Binding

Kestrel's old pipeline logged inference outputs but not the exact inputs sent to the endpoint. This made forensic analysis after the fact extremely difficult. The new architecture implements immutable, hash-bound audit logs that record the exact prompt payload, the model version, the inference configuration parameters, and the raw output for every single inference call. Inputs and outputs are cryptographically linked so that any tampering with either side of the log is detectable.

These logs are stored in an append-only, compliance-grade data store with a 7-year retention policy aligned with financial records requirements. Every anomaly decision made by the system is now fully reproducible and auditable from first principles.

Results: Six Months Post-Rebuild

By August 2026, Kestrel had been running the new architecture for approximately four months. The results were measurable across multiple dimensions:

• Cross-contamination rate: 0% detected across over 14 million inference calls in the post-rebuild audit window.
• False positive rate in anomaly detection dropped by 18%, largely attributed to the elimination of contaminated context influencing scoring decisions.
• Inference latency improved by 22%, a counterintuitive result explained by the elimination of shared-pool queuing delays and the tighter, more efficient structured prompts.
• Compliance team review time per flagged transaction fell by 31%, because structured, schema-validated explanations are faster to parse than free-text narratives.
• Kestrel received no regulatory enforcement action following their proactive disclosure, with regulators citing their rapid response and comprehensive remediation as mitigating factors.

Lessons for the Industry: What Every Fintech Needs to Know Right Now

Kestrel's experience is not unique. As agentic AI pipelines become standard infrastructure in financial services, the industry is collectively learning that the security model for LLM inference is fundamentally different from the security model for traditional APIs. A few hard-won lessons stand out:

Shared Inference Endpoints Are Not Equivalent to Shared Databases with Row-Level Security

Most engineering teams intuitively understand that a shared relational database requires careful row-level security to prevent one tenant from reading another's data. The equivalent mental model for LLM inference, specifically that KV caches, prefix caches, and batching policies can create analogous cross-tenant exposure, is not yet widely internalized. It needs to be.

Behavioral Context Is Regulated Data

There is a tendency to treat the "soft" outputs of AI systems, behavioral descriptions, spending pattern summaries, risk narratives, as less sensitive than raw PII. Regulators in 2026 do not share this view. Behavioral financial context derived from transaction history is protected personal financial information in most jurisdictions. Contamination of this context across customer boundaries is a reportable event, not just a model quality issue.

Audit Logging Must Cover the Full Inference Boundary

If you cannot reproduce exactly what was sent to an inference endpoint and exactly what came back, you cannot conduct a meaningful forensic investigation. This is non-negotiable for any AI system making or influencing financial decisions. Build immutable, hash-bound input-output logging from day one, not as an afterthought.

Dedicated Inference Is a Compliance Cost, Not Just a Performance Choice

For fintech teams evaluating the build-versus-buy calculus on inference infrastructure, the Kestrel case makes a compelling argument that dedicated inference should be treated as a compliance requirement for sensitive financial workloads, not merely a premium performance option. The shared-pool pricing advantage evaporates quickly when you factor in the audit, remediation, and regulatory notification costs of a contamination event.

Conclusion: The Invisible Boundary Problem in Agentic Finance

The Kestrel Financial case is a landmark example of a new class of AI infrastructure failure: not a model hallucination, not a prompt injection attack, not a data pipeline misconfiguration, but a silent boundary collapse at the inference layer that no one on the team thought to test for, because no one had a mental model for it yet.

As agentic AI systems take on more consequential roles in financial services, the industry needs to develop the same rigorous, adversarial thinking about inference infrastructure that it has long applied to databases, APIs, and network boundaries. The context window is not just a technical parameter. In a financial AI system, it is the perimeter of your customer's privacy.

Kestrel learned that lesson the hard way. The rest of the industry does not have to.