Agents → Cost & step budgets
What it is
A scripted pipeline's cost is roughly constant. An agent's is a distribution: most runs take 4 steps, some take 30, and a looping one takes as many as you let it. Without enforcement, the worst case is unbounded — and it's the worst case that appears on the invoice and in the incident review.
Four independent budgets, all enforced in code:
| Budget | Unit | Why it's separate |
|---|---|---|
| Steps / tool calls | count | Catches loops and thrashing |
| Tokens | in + out, including cached | Catches context bloat from large tool results |
| Wall-clock | seconds | Protects user experience and worker capacity |
| Dollars | computed from model pricing | The one leadership asks about; catches expensive-model escalations |
Budgets nest: per tool call → per subagent → per run → per user/day → per tenant/month. Each level must be checked at the level that can act on it.
The critical design rule from the source file: enforced by the runtime, not by asking the model nicely. "Be efficient and use at most 5 tool calls" in a prompt is a suggestion. if budget.steps >= max: force_finalize() is a control.
How to implement
- Budget object threaded through the run, checked before each LLM call and each tool execution (single agent loop).
- Charge every consumption: LLM usage (input, output, cache read/write, priced per model), tool costs (reranker calls, sandbox minutes, external API fees), and elapsed time.
- Exhaustion is an outcome, not an exception: force a final answer from what's been gathered, marked incomplete with the reason (single agent loop). Never return nothing; never crash.
- Nest budgets: allocate a run budget across subtasks before dispatch (task decomposition, fan-out / fan-in); a subagent gets a slice, and its overrun cannot exceed the parent's remaining.
- Pre-flight estimation: before a fan-out, estimate
subtasks × per-subtask costand reject or escalate for approval if it exceeds the ceiling (human-in-the-loop). - Per-user and per-tenant quotas enforced at the API edge, with graceful messaging when exhausted.
- Alert on distribution shifts, not just totals: a p99 that doubles overnight is a regression even if the mean looks fine.
Small example
import time
from dataclasses import dataclass, field
PRICES = { # $ per million tokens; keep in config, not code
"claude-haiku-4-5-20251001": {"in": 1.00, "out": 5.00, "cache_read": 0.10},
"claude-sonnet-5": {"in": 3.00, "out": 15.00, "cache_read": 0.30},
"claude-opus-5": {"in": 15.00,"out": 75.00, "cache_read": 1.50},
}
TOOL_COSTS = {"rerank": 0.002, "sandbox_minute": 0.01, "web_search": 0.005}
class BudgetExceeded(Exception):
def __init__(self, reason): self.reason = reason; super().__init__(reason)
@dataclass
class Budget:
max_steps: int = 12
max_tokens: int = 150_000
max_seconds: float = 90.0
max_usd: float = 0.75
steps: int = 0
tokens: int = 0
usd: float = 0.0
started: float = field(default_factory=time.time)
consumed: dict = field(default_factory=dict) # attribution by model/tool
def check(self):
if self.steps >= self.max_steps: raise BudgetExceeded("max_steps")
if self.tokens >= self.max_tokens: raise BudgetExceeded("max_tokens")
if time.time() - self.started >= self.max_seconds: raise BudgetExceeded("deadline")
if self.usd >= self.max_usd: raise BudgetExceeded("max_cost")
def charge_llm(self, usage, model):
p = PRICES[model]
cost = (usage.input_tokens * p["in"] + usage.output_tokens * p["out"]
+ getattr(usage, "cache_read_input_tokens", 0) * p["cache_read"]) / 1_000_000
self.tokens += usage.input_tokens + usage.output_tokens
self.usd += cost
self.consumed[model] = round(self.consumed.get(model, 0.0) + cost, 6)
def charge_tool(self, name, units=1):
cost = TOOL_COSTS.get(name, 0.0) * units
self.usd += cost
self.consumed[name] = round(self.consumed.get(name, 0.0) + cost, 6)
def remaining_usd(self): return max(self.max_usd - self.usd, 0.0)
def slice_for_subtask(self, n_subtasks: int, reserve_for_synthesis: float = 0.2) -> "Budget":
"""Child budget cannot exceed the parent's remaining share."""
available = self.remaining_usd() * (1 - reserve_for_synthesis)
return Budget(max_steps=max(self.max_steps // 2, 3),
max_tokens=self.max_tokens // max(n_subtasks, 1),
max_seconds=max(self.max_seconds / 2, 15),
max_usd=available / max(n_subtasks, 1))
def preflight(plan, spec, ceiling_usd, approval_threshold_usd):
est = len(plan.subtasks) * estimate_cost(spec)
if est > ceiling_usd:
return {"status": "rejected", "reason": f"estimated ${est:.2f} exceeds ceiling ${ceiling_usd:.2f}"}
if est > approval_threshold_usd:
return {"status": "needs_approval", "estimate": est} # Human-in-the-loop
return {"status": "ok", "estimate": est}
# per-tenant quota at the edge
def check_tenant_quota(tenant_id, estimated_usd):
spent = usage_store.month_to_date(tenant_id)
cap = tenant_config(tenant_id)["monthly_usd_cap"]
if spent + estimated_usd > cap:
raise QuotaExceeded(f"tenant {tenant_id} at ${spent:.2f} of ${cap:.2f}")
Pros
- Bounded worst case — the property that makes agents deployable at all.
- Cost attribution by model, tool, subagent type, and tenant shows where money goes and what to optimise.
- Predictable latency through wall-clock deadlines.
- Pre-flight estimates turn expensive plans into approval decisions rather than surprises (human-in-the-loop).
- Budget-exhaustion telemetry is a quality signal: which bound binds tells you what's wrong.
Cons
- Truncated runs produce worse answers — a budget that's too tight degrades quality invisibly unless you monitor exhaustion reasons.
- Estimation is approximate; agents are variable by nature, so pre-flight numbers are ranges.
- Nested allocation is fiddly: dividing a run budget across subtasks fairly, with reserve for synthesis.
- Four budgets means four thresholds to tune per workflow.
- Hard caps can cut off a run that was one step from success.
When to use
Every agent, every fan-out, every recursive retrieval loop, every reflection loop. There is no production case for an unbounded agent.
When NOT to use (tight caps)
- Deep-research modes where users explicitly accept long, expensive runs — raise the caps and show progress, but keep them finite.
- Offline batch jobs where wall-clock doesn't matter — relax the deadline, keep the dollar cap.
Production at scale
Which bound binds is diagnostic. Steps → looping or a task genuinely needing more hops. Tokens → tool results too large (per-source truncation budgets) or missing compaction (compaction / summarisation of long history). Deadline → slow tools. Dollars → escalating to an expensive model too often (routing & escalation). Dashboard exhaustion reasons and treat each as a different bug.
Set budgets from measured distributions, not intuition. Run the golden set, plot steps/tokens/cost, set the cap at p95–p99 of successful runs. A cap below p95 truncates legitimate work.
Reserve for finalisation. Always keep enough budget for the forced final answer; hitting the cap with nothing left to summarise wastes everything spent.
Per-tenant and per-user quotas at the edge, with clear messaging and an upgrade path. Without these, one tenant's runaway automation consumes the shared budget.
Cache accounting matters. With prompt caching (prompt caching of static prefixes), cached input tokens cost a fraction — budgets computed at full price will falsely trigger. Price cache reads separately, as in the example.
Cost per successful task is the metric that matters, not cost per run: a cheap agent that fails half the time is expensive. Track total_spend / successful_completions.
Kill switches: a per-tenant and a global circuit breaker that disables agentic paths and falls back to scripted pipelines (planning styles) when spend exceeds a threshold. Test it.
Attribution in every trace: model, tool, subagent type, tenant, run. "Which subagent type burns the most tokens" and "which tenant drives cost" are the questions optimisation starts from (observability & tracing).
Budget propagation across handoffs (multi-agent communication): the remaining budget travels in the task envelope and the receiving agent enforces it, or a chain of agents each spends "their" budget and the total is unbounded.
Alert on the tail. Mean cost hides everything. Alert on p99 cost per run, on exhaustion-rate increases, and on any single run exceeding a hard multiple of the median.
Prod checklist
- Four independent budgets (steps, tokens, wall-clock, dollars) checked before each step and each tool call
- All consumption charged: LLM (with cache pricing), tool fees, sandbox minutes
- Exhaustion produces a marked-incomplete final answer, never an exception or empty result
- Reserve held for forced finalisation
- Budgets nested: run → subtask slices that cannot exceed the parent's remaining
- Pre-flight estimation with rejection and approval thresholds for fan-outs
- Per-user and per-tenant quotas at the API edge with clear messaging
- Caps derived from measured p95–p99 of successful runs; revisited after model changes
- Exhaustion reason, cost attribution (model/tool/subagent/tenant) in every trace
- Kill switch to fall back to scripted pipelines; tested
- Budget propagated across agent handoffs
Metrics to watch
| Metric | Why |
|---|---|
| Cost per run: p50 / p95 / p99 | The tail is the risk |
| Cost per successful task | Real efficiency |
| Budget-exhaustion rate by reason | Which bound binds, and why |
| Spend by tenant, subagent type, model | Attribution and optimisation targets |
| Steps and tokens per run distribution | Loop and bloat detection |
Related
- Agent loop — where checks live
- Fan-out — pre-flight estimation and run ceilings
- Reflection — rounds consume budget
- Compaction / summarisation of long history / per-source truncation budgets — the levers when the token budget binds
- Cost levers, Capacity & limits