Agents → State & durability (checkpointing and idempotency)

AI Agent Checkpointing and Idempotency Keys Explained

What it is

An agent run is a long-lived, side-effecting computation. A 40-step run that crashes at step 38 has consumed real money and possibly performed real actions. Without durability, you restart — paying again and, worse, re-executing the emails you already sent.

Two mechanisms, always together:

Checkpointing — after each step, persist enough state to resume: messages, plan, step index, budget consumed, tool results, and any external references. A resumed run continues from step 38, not step 1.

Idempotency — every side-effecting tool call carries a stable key derived from the run and the call. The tool implementation (or the downstream service) records the key; a repeat with the same key returns the original result instead of acting again.

step 37 → checkpoint(37) ──► step 38 (send_email, key=run42:step38:hash) ──► CRASH after send, before checkpoint
                                                                              │
resume from checkpoint(37) ──► step 38 replays ──► same idempotency key ──► service returns the original
                                                    message_id, does NOT resend ──► checkpoint(38) → continue

The crash window between "action executed" and "checkpoint written" is exactly why idempotency is mandatory, not optional.

How to implement

  1. Define the state object: run_id, status, messages, plan, step, budget_consumed, caller_ref, created_at, updated_at, last_error. Keep it serialisable and bounded (large tool payloads go to blob storage by reference).
  2. Persist after every step (post-LLM-call and post-tool-execution) to a durable store — Postgres for transactional guarantees, or a workflow engine (Temporal, Step Functions, Restate) that provides this natively.
  3. Derive idempotency keys deterministically: sha256(run_id + step_index + tool_name + canonical(args)). Same logical call → same key on replay.
  4. Implement idempotency at the tool layer: check the key store before acting; record (key → result) atomically with the action where possible; return the stored result on repeat.
  5. Make resume the default recovery path: a worker picking up an interrupted run loads the checkpoint and continues, rather than a human restarting it.
  6. Bound retries and detect poison runs: a run that crashes repeatedly at the same step goes to a dead-letter state with its trace, not an infinite retry loop.
  7. Expire and clean up: completed runs' states are retained for audit (auditability) then purged; abandoned runs time out.

Small example

import hashlib, json
from enum import Enum

class RunStatus(str, Enum):
    RUNNING = "running"; PAUSED = "paused_for_approval"; DONE = "done"
    FAILED = "failed"; DEAD_LETTER = "dead_letter"

# --- durable state ---
"""
CREATE TABLE agent_runs (
  run_id TEXT PRIMARY KEY, status TEXT, step INT, caller_id TEXT,
  state JSONB,                      -- messages, plan, budget
  attempt_count INT DEFAULT 0, last_error TEXT,
  created_at TIMESTAMPTZ DEFAULT now(), updated_at TIMESTAMPTZ DEFAULT now());
CREATE TABLE idempotency (
  key TEXT PRIMARY KEY, run_id TEXT, tool TEXT,
  result JSONB, created_at TIMESTAMPTZ DEFAULT now());
"""

def checkpoint(conn, run):
    conn.execute("""INSERT INTO agent_runs(run_id, status, step, caller_id, state, updated_at)
                    VALUES (%s,%s,%s,%s,%s, now())
                    ON CONFLICT (run_id) DO UPDATE SET status=EXCLUDED.status, step=EXCLUDED.step,
                        state=EXCLUDED.state, updated_at=now()""",
                 (run.id, run.status, run.step, run.caller_id,
                  json.dumps({"messages": run.messages, "plan": run.plan, "budget": run.budget.to_dict()})))

def idem_key(run_id: str, step: int, tool: str, args: dict) -> str:
    canonical = json.dumps(args, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(f"{run_id}:{step}:{tool}:{canonical}".encode()).hexdigest()

def execute_idempotent(conn, run, tool_name, args, caller):
    key = idem_key(run.id, run.step, tool_name, args)
    prior = conn.execute("SELECT result FROM idempotency WHERE key=%s", (key,)).fetchone()
    if prior:
        log_idempotent_hit(run.id, tool_name, key)
        return prior[0]                                              # replay: return, do not re-execute

    with conn.transaction():                                          # atomic where the action is in the same DB
        result = TOOL_IMPLS[tool_name](args, caller, idempotency_key=key)
        conn.execute("INSERT INTO idempotency(key, run_id, tool, result) VALUES (%s,%s,%s,%s) "
                     "ON CONFLICT (key) DO NOTHING", (key, run.id, tool_name, json.dumps(result)))
    return result

# --- resume ---
def resume_interrupted_runs(conn, max_attempts=3, stale_after_minutes=10):
    rows = conn.execute("""SELECT run_id, state, step, attempt_count FROM agent_runs
                           WHERE status=%s AND updated_at < now() - interval '%s minutes'
                           FOR UPDATE SKIP LOCKED""", (RunStatus.RUNNING, stale_after_minutes)).fetchall()
    for run_id, state, step, attempts in rows:
        if attempts >= max_attempts:
            conn.execute("UPDATE agent_runs SET status=%s WHERE run_id=%s", (RunStatus.DEAD_LETTER, run_id))
            alert_dead_letter(run_id, step); continue
        conn.execute("UPDATE agent_runs SET attempt_count=attempt_count+1, updated_at=now() WHERE run_id=%s", (run_id,))
        run = RunState.from_json(run_id, state, step)
        log_resume(run_id, from_step=step, attempt=attempts + 1)
        continue_run(run)                                             # picks up at `step`, replays idempotently

For external services, prefer their native idempotency (Stripe Idempotency-Key, SES MessageDeduplicationId) over your own key store — it closes the crash window on their side too.

Pros

  • Crashes stop being catastrophic: resume from step 38 instead of paying for 38 steps again.
  • Double-sends become impossible — the failure that turns an agent incident into a customer incident.
  • Enables long-running and paused runs: human approval (human-in-the-loop), overnight batches, multi-day workflows.
  • Deploys stop being disruptive — you can restart workers mid-run.
  • The state store doubles as an operations surface: what's running, what's stuck, what's dead-lettered.

Cons

  • Real engineering: serialisation, schema migration of state, concurrency control, cleanup.
  • State size: message histories grow; needs compaction (compaction / summarisation of long history) and blob offloading.
  • Idempotency keys must be genuinely stable — including a timestamp or a random ID in the args silently breaks them, and you only find out when something double-sends.
  • Not all external services support idempotency; some need a local guard plus reconciliation.
  • Schema evolution: a state written by yesterday's code must be resumable by today's.

When to use

  • Any agent that performs side-effecting actions — mandatory, not optional.
  • Long runs (minutes+), batch fan-outs (fan-out / fan-in), and anything with human approval (human-in-the-loop).
  • Multi-tenant production systems where a deploy shouldn't kill in-flight work.

When NOT to use

  • Short, read-only, single-request agents where a failure just means the user retries and nothing was changed. Even then, log the trace.
  • Prototypes — but the moment a write tool appears, this becomes required.

Production at scale

Use a workflow engine if you can. Temporal, AWS Step Functions, Restate, and similar give durable execution, retries, and idempotency semantics as primitives. Hand-rolling checkpointing is a common and expensive detour; the engine is usually the right answer for anything beyond simple runs.

Idempotency key stability is the top bug. Never include timestamps, UUIDs generated at call time, or unsorted dict serialisation. Canonicalise arguments (sorted keys, fixed separators). Add a test: same logical call twice → same key.

Atomic action + key recording where possible (same database transaction). Where impossible (external API), record the key before the call with a pending status, then update with the result — so a crash mid-call leaves evidence, and the resume path can reconcile by querying the external service.

Reconciliation for the crash window: for critical actions (payments, sends), a periodic job checks pending idempotency records against the external service's records and resolves them. This is the difference between "we think we didn't double-charge" and knowing.

State schema versioning: state_version in the checkpoint; the resume path must handle the previous version or explicitly dead-letter runs it can't resume. Deploys happen mid-run.

Concurrency control: FOR UPDATE SKIP LOCKED or a lease with a TTL, so two workers never resume the same run concurrently. A double-resumed run is a double-execution risk even with idempotency (races on key insertion).

Blob offloading: large tool results in object storage with references in the state; the checkpoint stays small and fast to write.

Dead-letter with the trace: after N failed attempts, stop, alert, and preserve everything needed to diagnose. A run retrying forever burns budget silently.

Retention and privacy: agent state contains prompts, retrieved content, and possibly PII. Same retention, encryption, and erasure obligations as any other data (data residency & retention, PII).

Prod checklist

  • Durable state persisted after every step; state schema versioned
  • Deterministic idempotency keys from canonicalised args; no timestamps or random values
  • Idempotency enforced at the tool layer; external-service native keys used where available
  • Action and key recording atomic where possible; pending record + reconciliation where not
  • Resume as the default recovery; bounded attempts then dead-letter with alert
  • Concurrency control so one run resumes on one worker only
  • Large payloads offloaded to blob storage by reference
  • Workflow engine evaluated before hand-rolling
  • State retention, encryption, and erasure covered by policy
  • Test: same logical call twice produces the same key and executes once

Metrics to watch

MetricWhy
Resume rate and resume success rateDurability working
Idempotency-hit rate (replays caught)Crash-window protection
Dead-letter count and agePoison runs
Duplicate external actions detected by reconciliation (target 0)The failure that matters
Checkpoint write latency and sizeOverhead