Agents → Human-in-the-loop (approval gates and resume)
What it is
Some actions can't be undone. An agent that sends an email, issues a refund, deletes a record, or deploys code has crossed from "produces text" to "changes the world", and the failure modes change with it: a hallucinated recipient, an injected instruction (prompt-injection defence), a misread amount.
Human-in-the-loop inserts a deliberate stop:
agent → proposes action ──► policy check: does this need approval?
├── no → execute
└── yes → PAUSE: persist state, present the action to a human
├── approved → execute → resume the run
├── modified → execute the modified version → resume
└── rejected → feed the rejection back → agent replans or stops
Three distinct triggers:
- Action risk — irreversible, external-facing, financial, or destructive. Determined by the tool, not the model.
- Low confidence — the agent's own uncertainty, or a verifier's low score (faithfulness check, self-consistency).
- Policy — amount thresholds, first-time recipients, out-of-hours, specific tenants or regulated flows.
The hard engineering is the pause and resume: the run must survive the wait (minutes to days), and resuming must not re-execute completed steps (state & durability).
How to implement
- Classify every tool as auto / approval-required / never-automated (tool design
TOOL_METADATA). This lives in the runtime, not in the prompt. - Policy layer on top: thresholds and conditions evaluated in code with the caller's context (amount > X, recipient not in the address book, first action for this tenant).
- On a gated action, checkpoint the full agent state (messages, plan, budget, step index) durably and emit an approval request with: the exact action, its arguments, the reasoning, the evidence, and a diff/preview of the effect.
- Present it well — the reviewer needs to decide in seconds: what will happen, to whom, why the agent chose it, and what it's based on. Bad approval UX produces rubber-stamping, which is worse than no gate.
- Support approve / modify / reject with a reason. Modification is the most valuable option and the most commonly omitted.
- Resume from the checkpoint, executing the approved (possibly modified) action with an idempotency key, then continuing the loop.
- Timeout policy: what happens if nobody approves in N hours — expire, escalate, or auto-reject. Never leave a run hanging forever.
- Audit everything: who approved what, when, with what shown to them (auditability).
Small example
from enum import Enum
class ApprovalStatus(str, Enum):
PENDING = "pending"; APPROVED = "approved"; MODIFIED = "modified"; REJECTED = "rejected"; EXPIRED = "expired"
def needs_approval(tool_name: str, args: dict, caller, run_ctx) -> tuple[bool, str]:
meta = TOOL_METADATA[tool_name]
if meta["approval"] == "required": return True, "irreversible_action"
if tool_name == "issue_refund" and args["amount_usd"] > 500: return True, "amount_threshold"
if tool_name == "send_email" and not known_recipient(caller, args["to"]): return True, "new_recipient"
if run_ctx.get("confidence", 1.0) < 0.6: return True, "low_confidence"
if run_ctx.get("evidence_flagged_suspicious"): return True, "suspicious_source" # Prompt-injection defence
return False, ""
def gated_execute(block, caller, run_state, trace):
needed, reason = needs_approval(block.name, block.input, caller, run_state.ctx)
if not needed:
return execute_tool(block, caller, TOOL_IMPLS, trace, run_state.budget)
checkpoint_id = persist_run_state(run_state) # durable; survives process restart (state & durability)
request = {
"run_id": run_state.run_id, "checkpoint_id": checkpoint_id, "trace_id": trace.id,
"tool": block.name, "arguments": block.input, "reason": reason,
"rationale": run_state.last_reasoning[:800],
"evidence": [{"chunk_id": c, "title": title_of(c)} for c in run_state.cited_chunks],
"preview": preview_effect(block.name, block.input), # "Will email a@b.com, subject '...'"
"requested_by": caller.id, "approver_group": approver_for(block.name, caller),
"expires_at": now() + timedelta(hours=24), "status": ApprovalStatus.PENDING,
}
approval_store.create(request)
notify_approvers(request)
trace.log("paused_for_approval", tool=block.name, reason=reason)
raise PausedForApproval(checkpoint_id) # the run stops here, durably
def on_approval_decision(approval_id, decision: ApprovalStatus, modified_args=None, approver=None, note=None):
req = approval_store.get(approval_id)
audit_log(action="approval_decision", approval_id=approval_id, decision=decision,
approver=approver, shown=req["preview"], note=note) # Auditability
run_state = load_run_state(req["checkpoint_id"])
if decision == ApprovalStatus.REJECTED:
run_state.messages.append({"role": "user", "content": [
{"type": "tool_result", "tool_use_id": req["tool_use_id"],
"content": json.dumps({"status": "rejected_by_human", "reason": note or "not approved",
"guidance": "Do not retry this action. Propose an alternative or stop."})}]})
return resume_run(run_state)
args = modified_args if decision == ApprovalStatus.MODIFIED else req["arguments"]
key = idempotency_key(req["run_id"], req["tool"], args) # retried resumes cannot double-send
result = TOOL_IMPLS[req["tool"]](SCHEMAS[req["tool"]].model_validate(args), load_caller(req["requested_by"]), key)
run_state.messages.append({"role": "user", "content": [
{"type": "tool_result", "tool_use_id": req["tool_use_id"], "content": json.dumps(result)}]})
return resume_run(run_state)
def expire_stale_approvals():
for req in approval_store.pending_past(now()):
approval_store.update(req["id"], status=ApprovalStatus.EXPIRED)
st = load_run_state(req["checkpoint_id"])
st.messages.append({"role": "user", "content": [{"type": "tool_result", "tool_use_id": req["tool_use_id"],
"content": json.dumps({"status": "expired", "error": "Approval timed out; action not taken."})}]})
resume_run(st) # never leave a run hanging
Pros
- Prevents irreversible mistakes — the only reliable control against a hallucinated or injected destructive action.
- Enables deployment of agents that would otherwise be too risky: the gate is what makes "the agent can issue refunds" acceptable.
- Approval decisions are labelled data — rejections and modifications are the best training signal you'll get (data flywheel).
- Audit trail satisfies regulated-domain requirements (auditability).
- Modification lets humans fix small errors instead of rejecting whole runs.
Cons
- Latency: minutes to days. Only workable for async workflows, not interactive turns.
- Approval fatigue: too many gates and reviewers rubber-stamp, which is worse than no gate because it manufactures accountability without scrutiny.
- State persistence complexity — pausing and resuming across process restarts is real engineering (state & durability).
- Operational load: approval queues need owners, SLAs, and monitoring, or they rot.
- Poorly presented approvals lead to bad decisions; the UX is part of the control.
When to use
- Any irreversible external action: send, pay, refund, delete, deploy, provision, publish, sign.
- Financial thresholds, first-time counterparties, regulated flows.
- Low confidence on consequential decisions.
- When evidence was flagged as containing injected instructions (prompt-injection defence).
When NOT to use
- Read-only operations — gating searches destroys usability for zero risk.
- Actions that are cheaply reversible with a good undo (draft saved, label applied) — prefer undo over approval.
- Interactive latency-critical paths — restructure the task to be async instead.
- As a substitute for least privilege: an agent that can't do the dangerous thing needs no gate for it (excessive agency).
Production at scale
Least privilege first, gates second. The strongest control is that the agent's credentials cannot perform the action at all. Gates cover what must remain possible. A read-only agent with a read-only token cannot be talked into a delete, no matter what a document tells it.
Fight approval fatigue deliberately. Measure approval rate: if 98% are approved unchanged, the gate is theatre — either raise the threshold or automate that class. If 20% are rejected, the agent needs work. The healthy zone is a small number of meaningful decisions.
Batch related approvals: one review of 12 similar draft emails beats 12 separate interruptions. Show them as a list with per-item approve/reject.
Preview the effect, not the arguments. "Send to jane@acme.com, subject 'Renewal', body: …" with the rendered content, not a JSON blob. Include what the agent based it on — the evidence — so the reviewer can spot an injected instruction or a wrong document.
Idempotency on resume is non-negotiable (state & durability). Approvals arrive via webhooks that retry; a duplicate approval must not send twice.
Timeouts and escalation: every pending approval expires; expiry resumes the run with a clear "not taken" result. Track queue age; alert on breaches.
Approvers are a group, not a person — route by tool, amount, and tenant to an on-call group so runs aren't blocked by one person's holiday.
Audit what was shown, not just what was decided. In a dispute, "the approver saw this preview and this evidence at this timestamp" is the record that matters (auditability).
Rejection feedback goes back to the model as a tool result with guidance not to retry — otherwise the agent proposes the same action again and re-enters the queue.
Cost of the wait: a paused run holds a checkpoint, not a process. Never block a worker thread waiting for approval; the run resumes from storage when the decision arrives.
Prod checklist
- Every tool classified auto / approval / never-automated in the runtime
- Policy layer for thresholds, new counterparties, low confidence, flagged evidence
- Durable checkpoint before pausing; no worker blocked while waiting
- Approval request shows rendered preview, rationale, and the evidence used
- Approve / modify / reject with reason; rejection fed back with "do not retry" guidance
- Idempotency key on the executed action; duplicate approvals are safe
- Expiry policy with automatic resume; queue age monitored and alerted
- Approver groups with routing by tool, amount, tenant
- Full audit of what was shown, decided, by whom, when
- Approval-rate monitoring to detect both theatre and agent quality problems
Metrics to watch
| Metric | Why |
|---|---|
| Approval rate (approved / modified / rejected) | Gate value and agent quality |
| Time-to-decision; queue age | Operational health |
| Expiry rate | Under-staffed or over-gated |
| Modifications by field | Where the agent is systematically wrong |
| Actions executed without a required gate (must be 0) | Control integrity |
Related
- State & durability — pause/resume and idempotency
- Excessive agency — least privilege first
- Tool design — which tools are gated
- Injection — flagged evidence as a trigger
- Auditability