Testing & reliability → Failure handling

LLM Failure Handling Explained: Timeouts, Retries, Degradation

What it is

AI systems depend on more unreliable external services than typical applications: model providers (rate limits, outages, latency spikes), vector stores, rerankers, embedding services, and every tool an agent calls. Failure handling turns those into degraded quality rather than errors.

The layers:

ControlPurposeKey parameter
TimeoutBound the wait on every external callPer-call, shorter than the caller's budget
Retry with backoff + jitterSurvive transient failures without stampedingBounded attempts, retriable errors only
Circuit breakerStop hammering a broken dependencyFailure threshold, open duration, half-open probe
Hedged requestsCut tail latencyFire a second request after p95, take the first response
FallbackKeep serving when a dependency is downAlternate provider/model/index, with quality tracked
Graceful degradationDeliver partial value instead of an errorSources without a summary; cached answer; scripted path
Load sheddingProtect the system under overloadDrop optional stages before required ones (capacity & limits)

The principle that shapes all of them: degrade in a way the user can still use. The source file's example is exact — returning the retrieved sources with "I couldn't summarise these" is far better than a 500, because the user still gets the documents.

How to implement

  1. Timeout every external call, derived from the request's remaining budget (cost & step budgets) rather than a fixed constant, so a slow chain doesn't blow the overall SLO.
  2. Classify errors: retriable (429, 5xx, timeout, connection reset) vs terminal (400, 401, 403, schema, content policy). Never retry terminal errors.
  3. Exponential backoff with full jitter, bounded attempts, and respect for Retry-After.
  4. Circuit breaker per dependency (and per provider/region): open after N failures in a window, serve fallback while open, half-open probe to recover.
  5. Define a degradation ladder per stage: what to do when reranker / vector store / model / tool fails, in order of preference.
  6. Fallback models across providers with the eval suite run against them — a fallback that answers worse silently is a hidden regression (model selection & tradeoffs).
  7. Make degradation visible: tell the user what's reduced, and record degraded: true with a reason in the trace (observability & tracing).

Small example

import random, time
from enum import Enum

RETRIABLE = (429, 500, 502, 503, 504, 529)

class CircuitState(str, Enum):
    CLOSED = "closed"; OPEN = "open"; HALF_OPEN = "half_open"

class CircuitBreaker:
    def __init__(self, name, threshold=5, window_s=60, open_s=30):
        self.name, self.threshold, self.window_s, self.open_s = name, threshold, window_s, open_s
        self.failures, self.state, self.opened_at = [], CircuitState.CLOSED, None

    def allow(self) -> bool:
        now = time.time()
        if self.state == CircuitState.OPEN:
            if now - self.opened_at > self.open_s:
                self.state = CircuitState.HALF_OPEN; return True          # single probe
            return False
        return True

    def record(self, ok: bool):
        now = time.time()
        if ok:
            if self.state == CircuitState.HALF_OPEN:
                self.state, self.failures = CircuitState.CLOSED, []
                log_circuit(self.name, "closed")
            return
        self.failures = [t for t in self.failures if now - t < self.window_s] + [now]
        if len(self.failures) >= self.threshold and self.state != CircuitState.OPEN:
            self.state, self.opened_at = CircuitState.OPEN, now
            log_circuit(self.name, "opened"); alert_circuit_open(self.name)

def call_with_resilience(fn, breaker: CircuitBreaker, budget, attempts=3, timeout_s=None):
    if not breaker.allow():
        raise CircuitOpen(breaker.name)
    for i in range(attempts):
        t = timeout_s or min(10.0, budget.remaining_seconds() * 0.5)      # never exceed the request budget
        try:
            out = with_timeout(fn, t)
            breaker.record(True); return out
        except HTTPError as e:
            if e.status not in RETRIABLE:
                breaker.record(True); raise                                # terminal: do not retry
            wait = float(e.headers.get("Retry-After", 0)) or random.uniform(0, 2 ** i)   # full jitter
            breaker.record(False)
            if i == attempts - 1 or budget.remaining_seconds() < wait:
                raise
            time.sleep(wait)
        except TimeoutError:
            breaker.record(False)
            if i == attempts - 1: raise

# --- degradation ladder: partial value beats an error ---
def answer_with_degradation(query, caller, budget):
    degraded, notes = False, []

    try:
        pool = call_with_resilience(lambda: hybrid_retrieve(query, caller), BREAKERS["vector"], budget)
    except Exception:
        try:                                                               # dense down → keyword only
            pool = call_with_resilience(lambda: bm25_only(query, caller), BREAKERS["bm25"], budget)
            degraded, notes = True, notes + ["semantic search unavailable; keyword results only"]
        except Exception:
            return {"answer": None, "degraded": True,
                    "message": "Search is temporarily unavailable. Please try again shortly.",
                    "notes": ["retrieval unavailable"]}

    try:
        chunks = call_with_resilience(lambda: rerank_hosted(query, pool), BREAKERS["reranker"], budget, attempts=2)
    except Exception:
        chunks = pool[:6]                                                  # fusion order (reranking)
        degraded, notes = True, notes + ["ranking degraded"]

    for model in [TIER["balanced"], TIER["fast"], FALLBACK_PROVIDER_MODEL]:  # cross-provider fallback
        try:
            out = call_with_resilience(lambda: generate_grounded(query, chunks, model=model),
                                       BREAKERS[f"llm:{model}"], budget)
            if model != TIER["balanced"]:
                degraded, notes = True, notes + [f"answered with fallback model"]
            return out | {"degraded": degraded, "notes": notes}
        except Exception:
            continue

    # everything failed to summarise — STILL return the sources (the key pattern)
    return {"answer": None, "degraded": True,
            "message": "I found these sources but couldn't summarise them right now.",
            "sources": [{"title": c["title"], "page": c.get("page"), "chunk_id": c["chunk_id"],
                         "excerpt": c["text"][:300]} for c in chunks[:5]],
            "notes": notes + ["generation unavailable"]}

Pros

  • Availability during partial outages — the system keeps delivering value when one dependency is down.
  • Graceful degradation preserves user trust far better than an error page.
  • Circuit breakers prevent cascading failure and stop wasting budget on a broken dependency.
  • Backoff with jitter prevents thundering herds after an outage.
  • Fallback providers reduce single-vendor risk (model selection & tradeoffs).

Cons

  • Silent quality degradation is a real hazard: a fallback model or fusion-order ranking answers worse, and nobody notices without instrumentation.
  • Retries multiply cost and latency — an unbounded retry on an expensive model is a budget incident.
  • Complexity: every stage needs a defined failure path, tested.
  • Fallbacks need their own evals, or you're shipping an untested system during your worst hour.
  • Circuit breakers can flap and need tuning.

When to use

Every external dependency in a production system. The degradation ladder in particular is what separates a demo from a service.

When NOT to use

  • Don't retry terminal errors (400s, content policy, schema) — it wastes budget and time and never succeeds.
  • Don't fall back silently on high-stakes paths: for regulated or financial outputs, failing explicitly is better than answering with a degraded model.

Production at scale

Timeouts derived from the request budget, not fixed constants. A three-second reranker timeout inside a two-second remaining budget is a guaranteed SLO breach; compute per-call timeouts from what's left (cost & step budgets).

Full jitter, always. sleep(random.uniform(0, 2**attempt)) rather than fixed exponential — synchronised retries after a provider outage are how you turn one incident into two.

Track degraded responses as a first-class metric. degraded_rate by reason, on the dashboard next to error rate. Degradation is invisible in error rates and visible in quality metrics only if you slice by it (observability & tracing).

Evaluate the fallback path. Run the golden set against fallback models and degraded configurations so you know what quality you're serving during an incident (prompt evaluation). "We fall back to provider B" is not a plan until you've measured provider B.

Circuit breakers per provider and per region, with automatic failover. Provider outages are often regional; a breaker that opens globally when one region degrades over-reacts.

Hedged requests for tail latency: after p95 elapses, fire a duplicate to another replica/provider and take the first response. Effective on latency-sensitive paths, at the cost of extra spend — cap the hedge rate.

Idempotency makes retries safe (state & durability). Retrying a generation is harmless; retrying a tool that sends email is not, unless keyed.

Test the failure paths in CI with fault injection: simulate 429s, timeouts, and provider errors and assert the degradation ladder behaves (the testing pyramid). Untested failure paths fail when used.

Communicate degradation to users in plain language, and to operators via alerts. "Semantic search is temporarily unavailable; showing keyword results" preserves trust; a silently worse answer erodes it.

Prod checklist

  • Timeout on every external call, derived from remaining request budget
  • Retriable vs terminal error classification; terminal errors never retried
  • Exponential backoff with full jitter; Retry-After respected; bounded attempts
  • Circuit breaker per dependency, provider, and region; alerts on open
  • Degradation ladder defined per stage, ending in "sources without summary"
  • Cross-provider fallback models, evaluated on the golden set
  • Idempotency keys so retries of side-effecting calls are safe
  • degraded flag and reason recorded in the trace; degraded rate on the dashboard
  • Degradation communicated to the user in plain language
  • Fault-injection tests in CI covering each failure path

Metrics to watch

MetricWhy
Degraded-response rate by reasonHidden quality loss
Retry rate and retry-induced costBudget impact
Circuit-breaker open events and durationDependency health
Fallback model usage and its quality deltaWhat you serve during incidents
p99 latency with and without hedgingTail latency control