How One Enterprise Backend Team Discovered Their Agentic Workflow Versioning Strategy Was Incompatible With Rolling Deployments , and the Painful Refactor That Finally Made Zero-Downtime Agent Updates Possible

When the backend platform team at a mid-sized fintech company called Meridian Payments first deployed their agentic workflow system in late 2024, they celebrated. Their new AI-powered reconciliation agent could autonomously handle dispute classification, fraud signal correlation, and ledger anomaly triage, cutting manual review time by over 60%. It was, by every measure, a success story worth telling.

By early 2026, that same team was deep in a crisis they hadn't anticipated: their versioning strategy for agent workflows was silently incompatible with the rolling deployment model their Kubernetes infrastructure depended on. The result was a class of bugs so subtle and intermittent that it took three production incidents, two all-hands war rooms, and one very uncomfortable postmortem to finally diagnose the root cause.

This is the story of what went wrong, why it went wrong, and the architectural refactor that finally made zero-downtime agent updates a reality. If your team is running agentic systems in production, or planning to, this case study will save you months of pain.

The Setup: What Meridian's Agentic System Looked Like

Meridian's reconciliation agent was built on a multi-step agentic workflow framework. Each "agent run" was a stateful execution graph: a series of LLM-driven reasoning steps, tool calls, memory reads, and conditional branching decisions. The workflow was defined as a versioned YAML schema stored in a central config repository, and each version was tagged with a semantic version string like workflow-schema: v2.4.1.

At any given time, a workflow execution could span multiple minutes, sometimes longer if it was waiting on external API responses or human-in-the-loop approval gates. The agent's state was serialized to a shared Redis cluster between steps, allowing the system to be horizontally scaled across multiple pods.

The deployment model was standard for their infrastructure team: a Kubernetes rolling update, where new pods running the updated agent code were gradually spun up while old pods were gracefully terminated. At peak load, they ran 12 pods. A typical rolling update would cycle through all 12 pods over roughly 8 to 10 minutes.

On paper, everything looked fine. In practice, a time bomb was ticking.

The First Incident: "Ghost Steps" in Production

The first warning sign appeared during a routine deployment in January 2026. A small but statistically significant number of agent runs started producing what the team internally called "ghost steps": workflow executions that appeared to complete successfully in the logs but produced outputs inconsistent with the expected schema. Downstream consumers of the agent's output, including a fraud scoring microservice and a human review queue, started receiving malformed payloads.

The team's first instinct was to blame a bug in the new code. They rolled back, ran their full test suite, found nothing, and re-deployed. The ghost steps disappeared. They filed a ticket, labeled it "flaky," and moved on.

The second incident, three weeks later, was worse. This time, several long-running agent executions simply stalled indefinitely. The runs were alive in Redis, their state checkpoints intact, but no pod was picking them up for continuation. A manual restart of the affected runs cleared the queue, but the team had now lost confidence in the system's reliability.

Diagnosing the Real Problem: A Version Mismatch Hiding in Plain Sight

The breakthrough came during a late-night debugging session led by Meridian's senior platform engineer, Tariq. While tracing a stalled run end-to-end through their distributed tracing system, he noticed something strange in the span metadata: the run had been initiated by a pod running agent code version 2.5.0, but the step that stalled had been picked up for continuation by a pod running version 2.6.0.

During a rolling deployment, both old and new pods coexist in the cluster for 8 to 10 minutes. Any long-running agent execution that was started on an old pod and then paused at a checkpoint could be resumed by a new pod, one running entirely different code with a different understanding of the workflow state schema.

Here is where the versioning strategy failed catastrophically. The team had versioned their workflow definition schemas carefully, but they had made a critical assumption: that the in-memory and serialized state structures used during a live execution would always be backward compatible. They were not.

Specifically, version 2.6.0 had introduced a refactored step-context object. A field called tool_call_trace had been restructured from a flat list to a nested dictionary to support parallel tool execution. When a v2.6.0 pod picked up a checkpoint written by a v2.5.0 pod, it attempted to deserialize the old flat-list format into the new nested-dictionary structure. The deserialization did not throw an error. It silently produced a malformed context object, which then propagated through the rest of the execution, producing garbage outputs or causing the run to stall at the next conditional branch that expected the new structure.

The bug was invisible in unit tests because unit tests don't simulate mid-execution pod handoffs. It was invisible in integration tests because those tests ran against a single version of the code at a time. It only manifested in production, during rolling deployments, when two versions of the agent were simultaneously alive in the cluster.

Why This Problem Is Uniquely Hard for Agentic Systems

Traditional stateless microservices handle rolling deployments gracefully because each request is self-contained. A request that starts on a v1 pod ends on a v1 pod. There is no shared mutable state that crosses pod boundaries mid-request.

Agentic workflows break this assumption in several ways:

• Long execution spans: An agent run can last minutes, hours, or even days. This is far longer than any rolling deployment window, meaning a single run will almost certainly encounter multiple pod generations.
• Serialized intermediate state: Agents checkpoint their reasoning state to external stores between steps. This state is version-sensitive: it encodes assumptions about the shape of data that the next step will consume.
• Tool call and memory schemas evolve: As agent capabilities are updated, the schemas for tool call results, memory entries, and scratchpad objects change. These changes are often not treated with the same rigor as API contracts.
• Conditional branching depends on state shape: Unlike a simple data pipeline, an agent's next action depends on reasoning over its accumulated context. A subtly malformed context object does not cause an immediate crash; it causes a wrong decision, which may only surface much later in the run.

In short, agentic systems are stateful, long-lived, and schema-sensitive in ways that most deployment tooling was not designed to handle.

The Painful Refactor: Four Pillars of Version-Safe Agent Deployment

Meridian's team spent six weeks redesigning their deployment and versioning architecture. The refactor touched their agent runtime, their state serialization layer, their deployment pipeline, and their Kubernetes configuration. Here is what they built.

Pillar 1: Execution-Scoped Version Pinning

The team introduced a concept they called execution-scoped version pinning. When an agent run is initiated, the exact version of the agent code that started it is written into the run's metadata in Redis. Every subsequent step of that run, regardless of which pod picks it up, must be executed by a pod running that exact version.

To implement this, they modified their Kubernetes deployment to run two sets of pods simultaneously during any rollout: a "current" deployment and a "previous" deployment. The previous deployment is not terminated until all in-flight runs pinned to it have completed or timed out. Only then does the previous deployment scale down to zero.

This is conceptually similar to blue-green deployment, but applied at the run level rather than the traffic level. New runs are always routed to the current version. Old runs are always routed to the version that created them.

Pillar 2: Explicit State Schema Versioning with Migration Contracts

The team introduced a formal state schema versioning system, modeled loosely on database migration patterns. Every serialized checkpoint now includes a state_schema_version field. When a new version of the agent code introduces changes to the checkpoint state structure, the developers are required to write an explicit migration contract: a function that takes a checkpoint in the old schema version and produces a valid checkpoint in the new schema version.

These migration contracts are tested in a dedicated test suite that simulates mid-execution pod handoffs by serializing state with v(N) code and deserializing it with v(N+1) code. This test suite is now a required gate in their CI pipeline. No deployment proceeds if a migration contract is missing or if the migration test suite fails.

Pillar 3: Graceful Run Draining Before Pod Termination

The team configured their Kubernetes pods with an extended terminationGracePeriodSeconds and implemented a custom pre-stop lifecycle hook. When a pod receives a SIGTERM signal, it stops accepting new run assignments immediately but continues to execute steps for any runs it currently holds. It also broadcasts its "draining" status to the run coordinator, which stops assigning new steps to it.

For runs that are paused at a checkpoint and not actively executing on the draining pod, the coordinator reassigns them to a pod running the same pinned version. Only if no such pod exists (a rare edge case during major version upgrades) does it fall back to the migration contract path.

Pillar 4: A Version-Aware Run Coordinator

The final piece was a redesigned run coordinator service. Previously, the coordinator was a simple work queue: it pulled pending steps from Redis and dispatched them to any available pod. After the refactor, the coordinator is version-aware. It maintains a registry of which pods are running which agent version, and it routes step execution requests to pods that match the version pinned to each run.

The coordinator also exposes a deployment readiness API that the CI/CD pipeline queries before initiating a rollout. If the coordinator reports that more than a configurable threshold of runs are in-flight on the current version, the pipeline waits before proceeding. This prevents deployments from beginning during traffic spikes, when the number of long-running runs is highest.

The Results: What Zero-Downtime Agent Updates Actually Looks Like

After deploying the refactored architecture in March 2026, Meridian ran their first rolling update under the new system. The results were striking compared to the previous experience:

• Zero ghost steps were produced during or after the deployment window.
• Zero stalled runs were observed. All in-flight runs on the previous version completed normally, served by the previous deployment pods that remained live until the queue drained.
• The deployment window extended from the original 8 to 10 minutes to approximately 22 minutes, due to the run draining period. The team considered this an acceptable tradeoff for correctness.
• The migration contract test suite caught two schema incompatibilities during the refactor period itself, before they ever reached production. Both would have caused the exact class of silent corruption bugs they had experienced previously.

The Broader Lesson: Agentic Systems Demand Deployment-Aware Architecture

Meridian's experience is not unique. As agentic AI systems move from prototypes into production enterprise infrastructure in 2026, teams across industries are discovering that the deployment assumptions baked into their existing infrastructure were designed for a stateless world. Agentic workflows are not stateless. They are long-lived, stateful, schema-sensitive processes that can span hours and cross dozens of pod boundaries during their lifetime.

The engineering community has well-developed patterns for handling stateful systems in rolling deployments: database migration strategies, backward-compatible API versioning, consumer group management in event streaming. What has been missing, until teams like Meridian's are forced to build it, is the equivalent discipline applied to agent execution state.

The key principles that emerge from this case study are worth stating plainly:

• Treat agent checkpoint state as a versioned API contract, not an implementation detail.
• Never assume that a new pod can safely resume a run started by an old pod without an explicit compatibility guarantee.
• Build deployment pipelines that are aware of in-flight agent runs, not just HTTP traffic or queue depth.
• Test cross-version state handoffs explicitly, in CI, before every deployment.

Conclusion: The Deployment Problem Is the Next Frontier for Agentic AI

The AI community has spent enormous energy on making agents smarter, faster, and more capable. The operational question of how to deploy and update those agents safely in production has received far less attention. Meridian's painful refactor is a preview of the challenges that will face every enterprise team that runs agentic workloads at scale.

The good news is that the patterns exist. They are borrowed, adapted, and extended from distributed systems engineering, database versioning, and deployment automation. They are not exotic or inaccessible. They simply require that teams stop treating agent deployment as equivalent to deploying a stateless REST service, and start treating it as the stateful, long-lived, version-sensitive operation that it actually is.

The teams that internalize this lesson early will build agentic systems that are not just capable, but operationally trustworthy. In enterprise software, trustworthiness is the feature that matters most.