Testing & reliability → Failure handling
What it is
AI systems depend on more unreliable external services than typical applications: model providers (rate limits, outages, latency spikes), vector stores, rerankers, embedding services, and every tool an agent calls. Failure handling turns those into degraded quality rather than errors.
The layers:
| Control | Purpose | Key parameter |
|---|---|---|
| Timeout | Bound the wait on every external call | Per-call, shorter than the caller's budget |
| Retry with backoff + jitter | Survive transient failures without stampeding | Bounded attempts, retriable errors only |
| Circuit breaker | Stop hammering a broken dependency | Failure threshold, open duration, half-open probe |
| Hedged requests | Cut tail latency | Fire a second request after p95, take the first response |
| Fallback | Keep serving when a dependency is down | Alternate provider/model/index, with quality tracked |
| Graceful degradation | Deliver partial value instead of an error | Sources without a summary; cached answer; scripted path |
| Load shedding | Protect the system under overload | Drop optional stages before required ones (capacity & limits) |
The principle that shapes all of them: degrade in a way the user can still use. The source file's example is exact — returning the retrieved sources with "I couldn't summarise these" is far better than a 500, because the user still gets the documents.
How to implement
- Timeout every external call, derived from the request's remaining budget (cost & step budgets) rather than a fixed constant, so a slow chain doesn't blow the overall SLO.
- Classify errors: retriable (429, 5xx, timeout, connection reset) vs terminal (400, 401, 403, schema, content policy). Never retry terminal errors.
- Exponential backoff with full jitter, bounded attempts, and respect for
Retry-After. - Circuit breaker per dependency (and per provider/region): open after N failures in a window, serve fallback while open, half-open probe to recover.
- Define a degradation ladder per stage: what to do when reranker / vector store / model / tool fails, in order of preference.
- Fallback models across providers with the eval suite run against them — a fallback that answers worse silently is a hidden regression (model selection & tradeoffs).
- Make degradation visible: tell the user what's reduced, and record
degraded: truewith a reason in the trace (observability & tracing).
Small example
import random, time
from enum import Enum
RETRIABLE = (429, 500, 502, 503, 504, 529)
class CircuitState(str, Enum):
CLOSED = "closed"; OPEN = "open"; HALF_OPEN = "half_open"
class CircuitBreaker:
def __init__(self, name, threshold=5, window_s=60, open_s=30):
self.name, self.threshold, self.window_s, self.open_s = name, threshold, window_s, open_s
self.failures, self.state, self.opened_at = [], CircuitState.CLOSED, None
def allow(self) -> bool:
now = time.time()
if self.state == CircuitState.OPEN:
if now - self.opened_at > self.open_s:
self.state = CircuitState.HALF_OPEN; return True # single probe
return False
return True
def record(self, ok: bool):
now = time.time()
if ok:
if self.state == CircuitState.HALF_OPEN:
self.state, self.failures = CircuitState.CLOSED, []
log_circuit(self.name, "closed")
return
self.failures = [t for t in self.failures if now - t < self.window_s] + [now]
if len(self.failures) >= self.threshold and self.state != CircuitState.OPEN:
self.state, self.opened_at = CircuitState.OPEN, now
log_circuit(self.name, "opened"); alert_circuit_open(self.name)
def call_with_resilience(fn, breaker: CircuitBreaker, budget, attempts=3, timeout_s=None):
if not breaker.allow():
raise CircuitOpen(breaker.name)
for i in range(attempts):
t = timeout_s or min(10.0, budget.remaining_seconds() * 0.5) # never exceed the request budget
try:
out = with_timeout(fn, t)
breaker.record(True); return out
except HTTPError as e:
if e.status not in RETRIABLE:
breaker.record(True); raise # terminal: do not retry
wait = float(e.headers.get("Retry-After", 0)) or random.uniform(0, 2 ** i) # full jitter
breaker.record(False)
if i == attempts - 1 or budget.remaining_seconds() < wait:
raise
time.sleep(wait)
except TimeoutError:
breaker.record(False)
if i == attempts - 1: raise
# --- degradation ladder: partial value beats an error ---
def answer_with_degradation(query, caller, budget):
degraded, notes = False, []
try:
pool = call_with_resilience(lambda: hybrid_retrieve(query, caller), BREAKERS["vector"], budget)
except Exception:
try: # dense down → keyword only
pool = call_with_resilience(lambda: bm25_only(query, caller), BREAKERS["bm25"], budget)
degraded, notes = True, notes + ["semantic search unavailable; keyword results only"]
except Exception:
return {"answer": None, "degraded": True,
"message": "Search is temporarily unavailable. Please try again shortly.",
"notes": ["retrieval unavailable"]}
try:
chunks = call_with_resilience(lambda: rerank_hosted(query, pool), BREAKERS["reranker"], budget, attempts=2)
except Exception:
chunks = pool[:6] # fusion order (reranking)
degraded, notes = True, notes + ["ranking degraded"]
for model in [TIER["balanced"], TIER["fast"], FALLBACK_PROVIDER_MODEL]: # cross-provider fallback
try:
out = call_with_resilience(lambda: generate_grounded(query, chunks, model=model),
BREAKERS[f"llm:{model}"], budget)
if model != TIER["balanced"]:
degraded, notes = True, notes + [f"answered with fallback model"]
return out | {"degraded": degraded, "notes": notes}
except Exception:
continue
# everything failed to summarise — STILL return the sources (the key pattern)
return {"answer": None, "degraded": True,
"message": "I found these sources but couldn't summarise them right now.",
"sources": [{"title": c["title"], "page": c.get("page"), "chunk_id": c["chunk_id"],
"excerpt": c["text"][:300]} for c in chunks[:5]],
"notes": notes + ["generation unavailable"]}
Pros
- Availability during partial outages — the system keeps delivering value when one dependency is down.
- Graceful degradation preserves user trust far better than an error page.
- Circuit breakers prevent cascading failure and stop wasting budget on a broken dependency.
- Backoff with jitter prevents thundering herds after an outage.
- Fallback providers reduce single-vendor risk (model selection & tradeoffs).
Cons
- Silent quality degradation is a real hazard: a fallback model or fusion-order ranking answers worse, and nobody notices without instrumentation.
- Retries multiply cost and latency — an unbounded retry on an expensive model is a budget incident.
- Complexity: every stage needs a defined failure path, tested.
- Fallbacks need their own evals, or you're shipping an untested system during your worst hour.
- Circuit breakers can flap and need tuning.
When to use
Every external dependency in a production system. The degradation ladder in particular is what separates a demo from a service.
When NOT to use
- Don't retry terminal errors (400s, content policy, schema) — it wastes budget and time and never succeeds.
- Don't fall back silently on high-stakes paths: for regulated or financial outputs, failing explicitly is better than answering with a degraded model.
Production at scale
Timeouts derived from the request budget, not fixed constants. A three-second reranker timeout inside a two-second remaining budget is a guaranteed SLO breach; compute per-call timeouts from what's left (cost & step budgets).
Full jitter, always. sleep(random.uniform(0, 2**attempt)) rather than fixed exponential — synchronised retries after a provider outage are how you turn one incident into two.
Track degraded responses as a first-class metric. degraded_rate by reason, on the dashboard next to error rate. Degradation is invisible in error rates and visible in quality metrics only if you slice by it (observability & tracing).
Evaluate the fallback path. Run the golden set against fallback models and degraded configurations so you know what quality you're serving during an incident (prompt evaluation). "We fall back to provider B" is not a plan until you've measured provider B.
Circuit breakers per provider and per region, with automatic failover. Provider outages are often regional; a breaker that opens globally when one region degrades over-reacts.
Hedged requests for tail latency: after p95 elapses, fire a duplicate to another replica/provider and take the first response. Effective on latency-sensitive paths, at the cost of extra spend — cap the hedge rate.
Idempotency makes retries safe (state & durability). Retrying a generation is harmless; retrying a tool that sends email is not, unless keyed.
Test the failure paths in CI with fault injection: simulate 429s, timeouts, and provider errors and assert the degradation ladder behaves (the testing pyramid). Untested failure paths fail when used.
Communicate degradation to users in plain language, and to operators via alerts. "Semantic search is temporarily unavailable; showing keyword results" preserves trust; a silently worse answer erodes it.
Prod checklist
- Timeout on every external call, derived from remaining request budget
- Retriable vs terminal error classification; terminal errors never retried
- Exponential backoff with full jitter;
Retry-Afterrespected; bounded attempts - Circuit breaker per dependency, provider, and region; alerts on open
- Degradation ladder defined per stage, ending in "sources without summary"
- Cross-provider fallback models, evaluated on the golden set
- Idempotency keys so retries of side-effecting calls are safe
-
degradedflag and reason recorded in the trace; degraded rate on the dashboard - Degradation communicated to the user in plain language
- Fault-injection tests in CI covering each failure path
Metrics to watch
| Metric | Why |
|---|---|
| Degraded-response rate by reason | Hidden quality loss |
| Retry rate and retry-induced cost | Budget impact |
| Circuit-breaker open events and duration | Dependency health |
| Fallback model usage and its quality delta | What you serve during incidents |
| p99 latency with and without hedging | Tail latency control |
Related
- Budgets — timeouts derive from them
- Idempotency — makes retries safe
- Capacity & limits — load shedding
- Rollout — fallbacks during deploys
- Observability — degradation must be visible