How to Build a Multi-Agent Pipeline Graceful Degradation Layer That Automatically Reroutes Agent Workloads to Fallback Foundation Models During Provider Outages
It's 2:47 AM on a Tuesday. Your enterprise batch processing window is in full swing, churning through thousands of high-priority document summarizations, contract extractions, and compliance checks. Then your primary foundation model provider goes dark. No warning. No ETA. Just a cascade of 503 Service Unavailable errors flooding your logs while your pipeline grinds to a halt and your SLA clock keeps ticking.
This scenario is no longer a hypothetical. In 2026, with agentic AI pipelines embedded deep into enterprise workflows at companies ranging from financial institutions to healthcare networks, a single provider outage can trigger a business-critical incident within minutes. The uncomfortable truth is that most teams build for the happy path and bolt on resilience as an afterthought.
This tutorial walks you through building a Graceful Degradation Layer (GDL) for your multi-agent pipeline: a purpose-built subsystem that detects primary model provider failures, intelligently reroutes agent workloads to ranked fallback foundation models, preserves task context, and recovers without human intervention. By the end, you will have a production-ready architecture you can adapt to any agentic framework, whether you are using LangGraph, AutoGen, CrewAI, or a custom orchestration stack.
Why "Retry Logic" Is Not Enough
The first instinct when a model API call fails is to add exponential backoff and retry. That works fine for transient blips, but it is dangerously insufficient for true provider outages, which can last anywhere from 15 minutes to several hours. During a peak enterprise processing window, a retry loop that keeps hammering a dead endpoint does three harmful things:
- Burns your agent's token budget on failed calls that count against rate limits once the provider recovers.
- Blocks your pipeline's thread pool, preventing other agents from completing work that could proceed on a fallback model.
- Destroys task context if your agent framework times out and discards in-progress state before recovery.
A Graceful Degradation Layer operates at a higher level of abstraction. It sits between your agent orchestrator and the raw model API clients, acts as an intelligent circuit breaker, and makes routing decisions based on real-time provider health, task criticality, and model capability profiles. Think of it as an air traffic control tower for your agent workloads.
Core Architecture: The Four Pillars of a GDL
Before writing a single line of code, you need to understand the four components that make a GDL work in production:
1. The Provider Health Monitor
A lightweight, asynchronous process that continuously probes each registered foundation model provider using minimal "canary" requests. It maintains a rolling health state for each provider: HEALTHY, DEGRADED, or UNAVAILABLE. The monitor feeds into a shared state store (Redis works well here) so every agent worker in your pool can read provider status without making redundant health checks themselves.
2. The Circuit Breaker Registry
Borrowed from microservices resilience patterns, a circuit breaker per provider tracks consecutive failure counts and response latencies. When thresholds are breached, the breaker trips to OPEN state, immediately blocking new requests to that provider without waiting for another timeout. After a configurable cool-down window, it enters HALF-OPEN state and allows a single probe request to test recovery.
3. The Capability-Aware Fallback Router
This is the brain of the GDL. Not all foundation models are interchangeable. A task requiring 128K-token context cannot be silently rerouted to a model with a 32K window. The router maintains a capability profile for each registered model and matches incoming agent tasks to the best available fallback based on required capabilities, not just availability.
4. The Task Context Preservation Buffer
When a reroute happens mid-task, you need to reconstruct the agent's working context on the fallback model. The buffer serializes in-flight task state (conversation history, tool call results, intermediate reasoning steps) to a durable store before the switch, then replays the minimum necessary context on the new provider to resume work without starting from scratch.
Step 1: Define Your Provider Registry and Capability Profiles
Start by codifying what each foundation model in your roster can and cannot do. Here is a Python dataclass structure that captures the properties you will need for routing decisions:
from dataclasses import dataclass, field
from enum import Enum
from typing import List, Optional
class ProviderStatus(Enum):
HEALTHY = "healthy"
DEGRADED = "degraded"
UNAVAILABLE = "unavailable"
@dataclass
class ModelCapabilityProfile:
provider_id: str # e.g., "openai-gpt4o", "anthropic-claude4", "google-gemini2"
display_name: str
max_context_tokens: int
supports_function_calling: bool
supports_vision: bool
supports_streaming: bool
average_latency_ms: int # baseline p50 latency under normal load
cost_per_1k_tokens: float # used for cost-aware routing decisions
priority_rank: int # 1 = primary, 2 = first fallback, 3 = second fallback, etc.
capability_tags: List[str] = field(default_factory=list)
# e.g., ["code-generation", "long-context", "multilingual", "reasoning"]
# Example registry
PROVIDER_REGISTRY = [
ModelCapabilityProfile(
provider_id="openai-gpt4o",
display_name="GPT-4o (OpenAI)",
max_context_tokens=128000,
supports_function_calling=True,
supports_vision=True,
supports_streaming=True,
average_latency_ms=420,
cost_per_1k_tokens=0.005,
priority_rank=1,
capability_tags=["reasoning", "code-generation", "long-context", "multilingual"]
),
ModelCapabilityProfile(
provider_id="anthropic-claude4",
display_name="Claude 4 (Anthropic)",
max_context_tokens=200000,
supports_function_calling=True,
supports_vision=True,
supports_streaming=True,
average_latency_ms=510,
cost_per_1k_tokens=0.006,
priority_rank=2,
capability_tags=["reasoning", "long-context", "document-analysis", "multilingual"]
),
ModelCapabilityProfile(
provider_id="google-gemini2",
display_name="Gemini 2.0 Pro (Google)",
max_context_tokens=1000000,
supports_function_calling=True,
supports_vision=True,
supports_streaming=True,
average_latency_ms=380,
cost_per_1k_tokens=0.004,
priority_rank=3,
capability_tags=["reasoning", "long-context", "code-generation", "multimodal"]
),
]
The priority_rank field establishes your default fallback order, but the router will override this order when a lower-priority model is a better capability match for a specific task type.
Step 2: Build the Async Provider Health Monitor
The health monitor runs as a background asyncio task, completely decoupled from your agent workers. It writes status updates to Redis so all workers share a single source of truth without polling overhead:
import asyncio
import time
import redis.asyncio as aioredis
import httpx
import json
CANARY_PROMPTS = {
"openai-gpt4o": {
"url": "https://api.openai.com/v1/chat/completions",
"payload": {"model": "gpt-4o", "messages": [{"role": "user", "content": "ping"}], "max_tokens": 1}
},
"anthropic-claude4": {
"url": "https://api.anthropic.com/v1/messages",
"payload": {"model": "claude-4-sonnet", "messages": [{"role": "user", "content": "ping"}], "max_tokens": 1}
},
"google-gemini2": {
"url": "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-pro:generateContent",
"payload": {"contents": [{"parts": [{"text": "ping"}]}]}
}
}
HEALTH_CHECK_INTERVAL_SECONDS = 15
CANARY_TIMEOUT_SECONDS = 5
DEGRADED_LATENCY_THRESHOLD_MS = 2000
class ProviderHealthMonitor:
def __init__(self, redis_url: str, api_keys: dict):
self.redis_url = redis_url
self.api_keys = api_keys # {"openai-gpt4o": "sk-...", ...}
self._running = False
async def start(self):
self._running = True
self.redis = await aioredis.from_url(self.redis_url)
asyncio.create_task(self._monitor_loop())
async def _monitor_loop(self):
while self._running:
tasks = [self._check_provider(pid) for pid in CANARY_PROMPTS]
await asyncio.gather(*tasks, return_exceptions=True)
await asyncio.sleep(HEALTH_CHECK_INTERVAL_SECONDS)
async def _check_provider(self, provider_id: str):
config = CANARY_PROMPTS[provider_id]
headers = self._build_headers(provider_id)
start = time.monotonic()
status = ProviderStatus.HEALTHY
try:
async with httpx.AsyncClient(timeout=CANARY_TIMEOUT_SECONDS) as client:
resp = await client.post(config["url"], json=config["payload"], headers=headers)
elapsed_ms = (time.monotonic() - start) * 1000
if resp.status_code >= 500:
status = ProviderStatus.UNAVAILABLE
elif resp.status_code == 429 or elapsed_ms > DEGRADED_LATENCY_THRESHOLD_MS:
status = ProviderStatus.DEGRADED
else:
status = ProviderStatus.HEALTHY
except (httpx.TimeoutException, httpx.ConnectError):
status = ProviderStatus.UNAVAILABLE
await self.redis.set(
f"gdl:provider_status:{provider_id}",
json.dumps({"status": status.value, "checked_at": time.time()}),
ex=60 # expire after 60s so stale data never silently persists
)
def _build_headers(self, provider_id: str) -> dict:
key = self.api_keys.get(provider_id, "")
if "openai" in provider_id:
return {"Authorization": f"Bearer {key}", "Content-Type": "application/json"}
elif "anthropic" in provider_id:
return {"x-api-key": key, "anthropic-version": "2024-06-01", "Content-Type": "application/json"}
return {"Authorization": f"Bearer {key}", "Content-Type": "application/json"}
Notice the ex=60 TTL on every Redis write. This is a critical safety mechanism: if the health monitor itself crashes, provider status keys expire within 60 seconds and your router will treat unknown providers as unavailable rather than assuming they are healthy based on stale data.
Step 3: Implement the Circuit Breaker Registry
The circuit breaker operates at the call level, catching failures that slip through before the health monitor's next polling cycle. This is especially important in the first 15 seconds of a provider outage, when the monitor has not yet detected the problem:
import threading
from collections import deque
class CircuitBreaker:
def __init__(self, provider_id: str, failure_threshold: int = 5,
recovery_timeout_seconds: int = 60, half_open_max_calls: int = 1):
self.provider_id = provider_id
self.failure_threshold = failure_threshold
self.recovery_timeout = recovery_timeout_seconds
self.half_open_max_calls = half_open_max_calls
self._state = "CLOSED" # CLOSED = normal, OPEN = blocking, HALF-OPEN = testing
self._failure_count = 0
self._last_failure_time: Optional[float] = None
self._half_open_calls = 0
self._lock = threading.Lock()
@property
def is_available(self) -> bool:
with self._lock:
if self._state == "CLOSED":
return True
if self._state == "OPEN":
if time.time() - self._last_failure_time >= self.recovery_timeout:
self._state = "HALF-OPEN"
self._half_open_calls = 0
return True
return False
if self._state == "HALF-OPEN":
return self._half_open_calls < self.half_open_max_calls
return False
def record_success(self):
with self._lock:
self._failure_count = 0
self._state = "CLOSED"
def record_failure(self):
with self._lock:
self._failure_count += 1
self._last_failure_time = time.time()
if self._state == "HALF-OPEN" or self._failure_count >= self.failure_threshold:
self._state = "OPEN"
class CircuitBreakerRegistry:
def __init__(self):
self._breakers: dict[str, CircuitBreaker] = {}
def get(self, provider_id: str) -> CircuitBreaker:
if provider_id not in self._breakers:
self._breakers[provider_id] = CircuitBreaker(provider_id)
return self._breakers[provider_id]
def available_providers(self) -> List[str]:
return [pid for pid, cb in self._breakers.items() if cb.is_available]
Step 4: Build the Capability-Aware Fallback Router
This is where the intelligence lives. The router takes a task's requirements and returns the best available provider, combining health monitor data, circuit breaker state, and capability matching:
@dataclass
class TaskRequirements:
required_context_tokens: int
requires_function_calling: bool = False
requires_vision: bool = False
requires_streaming: bool = False
required_capability_tags: List[str] = field(default_factory=list)
task_criticality: str = "normal" # "normal", "high", "critical"
class CapabilityAwareFallbackRouter:
def __init__(self, registry: List[ModelCapabilityProfile],
circuit_breakers: CircuitBreakerRegistry,
redis_client):
self.registry = sorted(registry, key=lambda m: m.priority_rank)
self.circuit_breakers = circuit_breakers
self.redis = redis_client
async def select_provider(self, task: TaskRequirements) -> Optional[ModelCapabilityProfile]:
candidates = []
for profile in self.registry:
# Step 1: Check circuit breaker
cb = self.circuit_breakers.get(profile.provider_id)
if not cb.is_available:
continue
# Step 2: Check live health status from Redis
raw = await self.redis.get(f"gdl:provider_status:{profile.provider_id}")
if raw:
health_data = json.loads(raw)
if health_data["status"] == ProviderStatus.UNAVAILABLE.value:
continue
# Allow DEGRADED providers only for non-critical tasks
if health_data["status"] == ProviderStatus.DEGRADED.value:
if task.task_criticality == "critical":
continue
# Step 3: Validate capability requirements
if profile.max_context_tokens < task.required_context_tokens:
continue
if task.requires_function_calling and not profile.supports_function_calling:
continue
if task.requires_vision and not profile.supports_vision:
continue
if task.requires_streaming and not profile.supports_streaming:
continue
# Step 4: Score by capability tag overlap
tag_overlap = len(set(task.required_capability_tags) & set(profile.capability_tags))
score = (tag_overlap * 10) - profile.priority_rank + (1 / (profile.cost_per_1k_tokens + 0.001))
candidates.append((score, profile))
if not candidates:
return None # Total blackout: no provider available
# Return highest-scoring candidate
candidates.sort(key=lambda x: x[0], reverse=True)
return candidates[0][1]
The scoring function balances three factors: capability tag overlap (rewarding providers that are genuinely suited to the task), priority rank (preserving your preferred provider order when all else is equal), and cost efficiency (gently favoring cheaper options when scores are close). You can tune these weights for your organization's priorities.
Step 5: Implement the Task Context Preservation Buffer
Context preservation is the piece most tutorials skip, and it is the difference between a reroute that feels seamless and one that causes your agent to lose its train of thought mid-task:
import uuid
import json
@dataclass
class AgentTaskContext:
task_id: str
original_provider_id: str
system_prompt: str
conversation_history: List[dict] # OpenAI-style message list
tool_results: List[dict] # Accumulated tool call outputs
intermediate_state: dict # Framework-specific state blob
created_at: float
token_count_estimate: int
class TaskContextBuffer:
def __init__(self, redis_client, ttl_seconds: int = 3600):
self.redis = redis_client
self.ttl = ttl_seconds
async def checkpoint(self, context: AgentTaskContext) -> str:
"""Serialize and store task context. Returns checkpoint key."""
key = f"gdl:context:{context.task_id}"
payload = json.dumps({
"task_id": context.task_id,
"original_provider_id": context.original_provider_id,
"system_prompt": context.system_prompt,
"conversation_history": context.conversation_history,
"tool_results": context.tool_results,
"intermediate_state": context.intermediate_state,
"created_at": context.created_at,
"token_count_estimate": context.token_count_estimate
})
await self.redis.set(key, payload, ex=self.ttl)
return key
async def restore(self, task_id: str) -> Optional[AgentTaskContext]:
"""Retrieve a checkpointed context for rerouting."""
key = f"gdl:context:{task_id}"
raw = await self.redis.get(key)
if not raw:
return None
data = json.loads(raw)
return AgentTaskContext(**data)
async def build_resume_prompt(self, context: AgentTaskContext,
new_provider_profile: ModelCapabilityProfile) -> List[dict]:
"""
Construct a minimal conversation history that fits within the
new provider's context window, trimming oldest non-critical turns first.
"""
messages = [{"role": "system", "content": context.system_prompt}]
# Inject a reroute notice so the model understands it is resuming
messages.append({
"role": "system",
"content": (
"CONTEXT TRANSFER NOTICE: You are resuming an in-progress task. "
"The following conversation history represents work completed so far. "
"Continue from where this left off without restarting."
)
})
# Trim history to fit new provider's context window (reserve 20% for response)
available_tokens = int(new_provider_profile.max_context_tokens * 0.8)
trimmed_history = self._trim_to_token_budget(
context.conversation_history, available_tokens
)
messages.extend(trimmed_history)
return messages
def _trim_to_token_budget(self, history: List[dict], token_budget: int) -> List[dict]:
"""Naive token estimation: remove oldest turns until under budget."""
# Always preserve the last 3 turns (most recent context is most critical)
protected = history[-3:] if len(history) >= 3 else history
trimmable = history[:-3] if len(history) >= 3 else []
estimated_tokens = sum(len(m["content"].split()) * 1.3 for m in protected)
result = list(protected)
for msg in reversed(trimmable):
msg_tokens = len(msg["content"].split()) * 1.3
if estimated_tokens + msg_tokens <= token_budget:
result.insert(0, msg)
estimated_tokens += msg_tokens
else:
break
return result
Step 6: Wire It All Together in the GDL Orchestrator
Now you assemble all four pillars into a single GracefulDegradationLayer class that your agent workers call instead of hitting provider APIs directly:
class GracefulDegradationLayer:
def __init__(self, provider_registry, redis_url: str, api_keys: dict):
self.redis = None
self.redis_url = redis_url
self.api_keys = api_keys
self.provider_registry = provider_registry
self.circuit_breakers = CircuitBreakerRegistry()
self.context_buffer = None
self.router = None
self.health_monitor = None
async def initialize(self):
import redis.asyncio as aioredis
self.redis = await aioredis.from_url(self.redis_url)
self.context_buffer = TaskContextBuffer(self.redis)
self.router = CapabilityAwareFallbackRouter(
self.provider_registry, self.circuit_breakers, self.redis
)
self.health_monitor = ProviderHealthMonitor(self.redis_url, self.api_keys)
await self.health_monitor.start()
async def execute_with_resilience(self, task_context: AgentTaskContext,
task_requirements: TaskRequirements,
model_call_fn) -> dict:
"""
Main entry point for agent workers.
Args:
task_context: The current agent task state.
task_requirements: Capability requirements for this task.
model_call_fn: A callable(provider_id, messages) -> response dict.
Returns:
Model response dict, or raises GDLBlackoutError if all providers fail.
"""
max_attempts = len(self.provider_registry)
for attempt in range(max_attempts):
# Checkpoint context before every attempt
await self.context_buffer.checkpoint(task_context)
# Select best available provider
selected = await self.router.select_provider(task_requirements)
if selected is None:
raise GDLBlackoutError(
"All providers unavailable. Task checkpointed for recovery."
)
cb = self.circuit_breakers.get(selected.provider_id)
try:
messages = await self.context_buffer.build_resume_prompt(
task_context, selected
)
response = await model_call_fn(selected.provider_id, messages)
cb.record_success()
# Log reroute event for observability
if selected.provider_id != task_context.original_provider_id:
await self._log_reroute_event(task_context, selected)
return response
except (ProviderAPIError, httpx.HTTPStatusError) as e:
cb.record_failure()
# Update Redis health status immediately for other workers
await self.redis.set(
f"gdl:provider_status:{selected.provider_id}",
json.dumps({"status": "unavailable", "checked_at": time.time()}),
ex=60
)
continue # Try next provider
raise GDLBlackoutError("Exhausted all provider attempts.")
async def _log_reroute_event(self, context: AgentTaskContext,
new_provider: ModelCapabilityProfile):
event = {
"event": "gdl_reroute",
"task_id": context.task_id,
"from_provider": context.original_provider_id,
"to_provider": new_provider.provider_id,
"timestamp": time.time()
}
await self.redis.lpush("gdl:reroute_events", json.dumps(event))
await self.redis.ltrim("gdl:reroute_events", 0, 9999) # Keep last 10k events
class GDLBlackoutError(Exception):
"""Raised when all providers are unavailable and no fallback can be selected."""
pass
class ProviderAPIError(Exception):
pass
Step 7: Integrate with Your Agent Framework
Here is how you would integrate the GDL into a LangGraph-based agent worker. The pattern is similar for AutoGen and CrewAI, with minor framework-specific adaptations:
from langgraph.graph import StateGraph, END
import uuid
# Initialize GDL once at application startup
gdl = GracefulDegradationLayer(
provider_registry=PROVIDER_REGISTRY,
redis_url="redis://localhost:6379",
api_keys={
"openai-gpt4o": os.environ["OPENAI_API_KEY"],
"anthropic-claude4": os.environ["ANTHROPIC_API_KEY"],
"google-gemini2": os.environ["GOOGLE_API_KEY"],
}
)
async def resilient_agent_node(state: dict) -> dict:
task_context = AgentTaskContext(
task_id=state.get("task_id", str(uuid.uuid4())),
original_provider_id="openai-gpt4o",
system_prompt=state["system_prompt"],
conversation_history=state["messages"],
tool_results=state.get("tool_results", []),
intermediate_state=state.get("agent_state", {}),
created_at=time.time(),
token_count_estimate=state.get("token_estimate", 0)
)
task_requirements = TaskRequirements(
required_context_tokens=state.get("token_estimate", 4096),
requires_function_calling=True,
required_capability_tags=["reasoning", "document-analysis"],
task_criticality=state.get("criticality", "normal")
)
async def call_model(provider_id: str, messages: List[dict]) -> dict:
# Your existing model call logic, parameterized by provider_id
client = get_client_for_provider(provider_id)
return await client.complete(messages)
try:
response = await gdl.execute_with_resilience(
task_context, task_requirements, call_model
)
return {**state, "messages": state["messages"] + [response], "status": "completed"}
except GDLBlackoutError:
# All providers down: park the task for deferred processing
return {**state, "status": "parked_for_recovery"}
Step 8: Handle the "Total Blackout" Scenario
Even with three major providers in your registry, you need a plan for the unlikely but catastrophic scenario where all are simultaneously unavailable. Rather than failing hard, implement a deferred processing queue that parks tasks and replays them when any provider recovers:
class DeferredTaskQueue:
def __init__(self, redis_client):
self.redis = redis_client
self.queue_key = "gdl:deferred_tasks"
async def enqueue(self, task_context: AgentTaskContext,
task_requirements: TaskRequirements):
payload = json.dumps({
"task_id": task_context.task_id,
"requirements": {
"required_context_tokens": task_requirements.required_context_tokens,
"requires_function_calling": task_requirements.requires_function_calling,
"required_capability_tags": task_requirements.required_capability_tags,
"task_criticality": task_requirements.task_criticality
},
"enqueued_at": time.time()
})
await self.redis.rpush(self.queue_key, payload)
async def drain_when_recovered(self, gdl: GracefulDegradationLayer,
model_call_fn):
"""Called by the health monitor when any provider returns to HEALTHY."""
while True:
raw = await self.redis.lpop(self.queue_key)
if not raw:
break
data = json.loads(raw)
context = await gdl.context_buffer.restore(data["task_id"])
if context:
reqs = TaskRequirements(**data["requirements"])
await gdl.execute_with_resilience(context, reqs, model_call_fn)
Observability: Knowing When Your GDL Is Earning Its Keep
A GDL you cannot observe is a GDL you cannot trust. Wire these metrics into your existing observability stack (Datadog, Grafana, OpenTelemetry, whatever you use):
- gdl.reroute.count: Number of successful reroutes per time window. A spike here is your early warning that a primary provider is struggling.
- gdl.circuit_breaker.open.duration_seconds: How long each provider's circuit breaker stays open. Long durations indicate extended outages rather than transient blips.
- gdl.blackout.count: Number of tasks that hit total blackout and were parked. This should be near zero in normal operation.
- gdl.context_preservation.token_trim_ratio: How much context is being trimmed during reroutes. A high ratio means your fallback models have smaller context windows than your primary, which may affect output quality.
- gdl.deferred_queue.depth: Number of tasks waiting for provider recovery. Alert on this crossing your SLA-safe threshold.
Configuration Tuning for Peak Enterprise Windows
Your GDL should behave differently during peak processing windows versus off-peak hours. Consider a configuration profile system that tightens thresholds when stakes are highest:
- Reduce health check interval from 15 seconds to 5 seconds during peak windows so reroutes happen faster.
- Lower the circuit breaker failure threshold from 5 consecutive failures to 2 during peak, accepting more aggressive failover in exchange for faster recovery.
- Promote lower-priority providers to co-primary status during peak windows so load is pre-distributed before any outage occurs, not just after one strikes.
- Increase context buffer TTL from 1 hour to 4 hours during overnight batch runs, ensuring tasks that park during a blackout can still be recovered when the morning team arrives.
Conclusion: Resilience Is a Feature, Not a Fix
The multi-agent pipelines powering enterprise AI in 2026 are mission-critical infrastructure. They deserve the same resilience engineering discipline that we have applied to databases, message queues, and microservices for the past decade. A Graceful Degradation Layer is not a defensive measure you bolt on after your first production incident. It is a first-class architectural feature that earns its place in your system design from day one.
What you have built in this tutorial is a production-ready foundation: a health monitor that watches your providers continuously, circuit breakers that trip before failures cascade, a capability-aware router that makes intelligent model selection decisions, and a context preservation buffer that keeps your agents from losing their place when the ground shifts beneath them.
The next time a foundation model provider goes dark at 2:47 AM during your peak processing window, your pipeline will not page you. It will simply reroute, log the event, and keep working. That is what resilient agentic AI looks like in practice.
Start with Step 1 today. Even a basic provider registry and health monitor will put you miles ahead of teams that are still relying on retry loops and hoping for the best.
