Production → Capacity & limits
What it is
AI systems have unusual capacity characteristics: the bottleneck is typically an external quota (tokens per minute, requests per minute) rather than your own CPU, and the cost of exceeding it is 429s that cascade into user-visible failures.
Four control layers:
| Layer | Purpose | Mechanism |
|---|---|---|
| Quota management | Stay within provider limits | Token-bucket per provider/model/region; track TPM and RPM separately |
| Per-tenant throttling | Prevent one tenant starving others | Per-tenant rate and concurrency limits, plus monthly spend caps |
| Queueing with backpressure | Absorb bursts without dropping work | Bounded queues; reject early when full rather than timing out late |
| Load shedding | Degrade deliberately under overload | Drop optional stages first, then non-critical traffic classes |
The essential design decision is the shed order — what you give up, in what sequence, when you can't serve everything:
1. Optional quality stages multi-query → HyDE → reranking depth → context size
2. Tier downgrade reasoning tier → balanced → fast
3. Non-interactive work background jobs, ingestion, evals, re-indexing
4. Low-priority tenants free tier before paid; batch users before interactive
5. Reject with a clear message + Retry-After
How to implement
- Know your quotas: TPM and RPM per provider, model, and region; per-tenant caps you've committed to; vector store and reranker limits.
- Client-side rate limiting with a token bucket per (provider, model), sized at ~85% of the quota to leave headroom for retries.
- Estimate tokens before sending so the bucket can reserve accurately; reconcile with actual usage afterwards.
- Per-tenant limits: concurrent requests, requests/minute, and monthly spend (cost & step budgets); enforced at ingress with a clear error and
Retry-After. - Bounded queues with early rejection: when the queue is full, reject immediately with a retry hint rather than accepting work you cannot finish.
- Priority classes: interactive > batch > background. Background work yields first.
- Shed-order configuration, applied automatically on pressure signals (queue depth, 429 rate, latency SLO breach) and reversed when pressure clears.
Small example
import time, threading, random
class TokenBucket:
"""Client-side limiter for both TPM and RPM — the two provider quotas."""
def __init__(self, tokens_per_min, requests_per_min, headroom=0.85):
self.tpm_cap = tokens_per_min * headroom
self.rpm_cap = requests_per_min * headroom
self.tokens = self.tpm_cap; self.requests = self.rpm_cap
self.last = time.time(); self.lock = threading.Lock()
def _refill(self):
now = time.time(); elapsed = now - self.last
self.tokens = min(self.tpm_cap, self.tokens + self.tpm_cap * elapsed / 60)
self.requests = min(self.rpm_cap, self.requests + self.rpm_cap * elapsed / 60)
self.last = now
def try_acquire(self, estimated_tokens: int) -> tuple[bool, float]:
with self.lock:
self._refill()
if self.tokens >= estimated_tokens and self.requests >= 1:
self.tokens -= estimated_tokens; self.requests -= 1
return True, 0.0
wait = max((estimated_tokens - self.tokens) / (self.tpm_cap / 60),
(1 - self.requests) / (self.rpm_cap / 60))
return False, wait
def reconcile(self, estimated: int, actual: int):
with self.lock:
self.tokens -= max(actual - estimated, 0) # correct under-estimates
BUCKETS = {("anthropic", "claude-sonnet-5"): TokenBucket(400_000, 4_000)}
# --- per-tenant limits at ingress ---
class TenantLimiter:
def __init__(self, cfg): self.cfg, self.inflight, self.rpm = cfg, {}, {}
def check(self, tenant_id) -> tuple[bool, str, int]:
c = self.cfg[tenant_id]
if self.inflight.get(tenant_id, 0) >= c["max_concurrent"]:
return False, "tenant_concurrency", 5
if self.rpm.get(tenant_id, 0) >= c["rpm"]:
return False, "tenant_rate", 30
if spend_month_to_date(tenant_id) >= c["monthly_usd_cap"]:
return False, "tenant_quota_exhausted", 0 # no retry: needs an upgrade
return True, "", 0
# --- pressure-driven load shedding ---
SHED_LADDER = [
("multi_query", lambda cfg: cfg.update(multi_query=False)),
("hyde", lambda cfg: cfg.update(hyde=False)),
("rerank_depth", lambda cfg: cfg.update(pool_k=min(cfg["pool_k"], 20))),
("context_size", lambda cfg: cfg.update(final_k=min(cfg["final_k"], 4))),
("tier_downgrade", lambda cfg: cfg.update(tier="fast")),
("background_jobs", lambda cfg: pause_background_jobs()),
("free_tier", lambda cfg: cfg.update(reject_classes=["free"])),
]
def apply_shedding(pressure: dict, cfg: dict) -> list[str]:
"""pressure: {queue_depth, rate_limit_429s, ttft_p95_ms}"""
level = 0
if pressure["queue_depth"] > 50 or pressure["rate_limit_429s"] > 10: level = 2
if pressure["queue_depth"] > 150 or pressure["ttft_p95_ms"] > 3000: level = 4
if pressure["queue_depth"] > 400: level = 6
applied = []
for name, action in SHED_LADDER[:level]:
action(cfg); applied.append(name)
if applied:
log_shedding(applied, pressure) # visible: degradation must never be silent
return applied
def admit(request, tenant_limiter, queue, cfg):
ok, reason, retry_after = tenant_limiter.check(request.tenant_id)
if not ok:
return reject(reason, retry_after) # fail fast with a clear reason
if queue.qsize() >= queue.maxsize: # bounded queue, early rejection
return reject("system_busy", retry_after=10)
if request.priority == "background" and cfg.get("reject_classes"):
return reject("shedding_background", retry_after=60)
queue.put(request); return None
def call_model(provider, model, messages, estimated_tokens, deadline_s):
bucket = BUCKETS[(provider, model)]
deadline = time.time() + deadline_s
while time.time() < deadline:
ok, wait = bucket.try_acquire(estimated_tokens)
if ok:
r = client.messages.create(model=model, messages=messages)
bucket.reconcile(estimated_tokens, r.usage.input_tokens + r.usage.output_tokens)
return r
time.sleep(min(wait + random.uniform(0, 0.5), max(deadline - time.time(), 0)))
raise CapacityExceeded("could not acquire quota within deadline")
Pros
- Client-side limiting avoids 429 storms, which are far more damaging than a short local wait.
- Per-tenant limits give fairness and predictable multi-tenant behaviour — one tenant's automation can't starve everyone.
- Graceful degradation preserves availability during spikes, at reduced quality.
- Bounded queues fail fast, which is a much better user experience than accepting work and timing out.
- Spend caps prevent runaway cost from a bug or an abusive workload (cost & step budgets).
Cons
- Client-side estimation is imprecise; conservative headroom wastes quota, aggressive settings cause 429s.
- Shedding degrades quality silently unless instrumented and surfaced.
- Per-tenant limits need commercial alignment — the limit is a product decision as much as a technical one.
- Queueing adds latency and complexity; bounded queues mean rejecting real work.
- Distributed rate limiting across replicas needs shared state (Redis) and adds a dependency.
When to use
Any multi-tenant production system, and any system whose traffic can spike (support tools after an incident, internal assistants at Monday-morning peak, batch ingestion colliding with interactive traffic).
When NOT to use
- Single-user internal tools with predictable load — provider limits alone suffice.
- Don't shed on high-stakes paths (compliance answers, financial calculations): reject explicitly rather than serving a degraded answer.
Production at scale
Distributed rate limiting is the real problem. Per-process buckets over-admit by the number of replicas. Use a shared limiter (Redis token bucket, or a sidecar gateway like Envoy/LiteLLM) that enforces the quota fleet-wide, with local buckets as a fast path.
Separate interactive from batch capacity. Ingestion, re-indexing (versioning & migration), contextual retrieval generation (contextual retrieval), and eval runs should use batch endpoints and a separate quota pool, so a backfill can never degrade interactive traffic. This is one of the most common production incidents in RAG systems.
Shed order is a product decision — write it down and get agreement before you need it. The sequence above (optional stages → tier → background → low-priority tenants → reject) is a sensible default, but the tenant-priority part in particular needs commercial sign-off.
Make shedding visible: log it, alert on it, expose a degraded flag in the response (failure handling), and show a banner if quality is reduced. Silent degradation destroys trust in evaluation metrics too, since online quality drops with no code change.
Reserve headroom for retries. Running at 100% of quota means every retry fails. 80–85% utilisation is the practical ceiling.
Multiple providers/regions multiply capacity and provide failover (failure handling). Route by health and quota headroom, not round-robin.
Watch the leading indicators: queue depth and 429 rate rise before latency does. Alert on those, not on user-visible failures.
Capacity planning needs token forecasts, not request forecasts — a feature that adds 3k tokens of context to every request consumes quota like a 3× traffic increase.
Prod checklist
- Provider TPM and RPM quotas documented per model and region
- Fleet-wide distributed rate limiting at ~85% of quota, with local fast-path buckets
- Token estimation before send; reconciliation after
- Per-tenant concurrency, rate, and monthly spend limits enforced at ingress
- Bounded queues with early rejection and
Retry-After - Priority classes: interactive > batch > background; background pauses first
- Separate quota pool and batch endpoints for ingestion/backfill/evals
- Documented, agreed shed ladder applied automatically on pressure signals
- Shedding logged, alerted, and surfaced as
degradedin responses - Alerts on queue depth and 429 rate, not just on failures
- Capacity forecasting in tokens, not requests
Metrics to watch
| Metric | Why |
|---|---|
| 429 rate per provider/model | Quota pressure |
| Queue depth and rejection rate | Leading indicators |
| Shed-ladder level and duration | How often you're degrading |
| Per-tenant consumption (tokens, spend) | Fairness and cost |
| Quota utilisation vs cap | Headroom for retries |
Related
- Failure handling — retries, breakers, degradation
- Latency levers — shedding optional stages
- Budgets — per-request and per-tenant caps
- Cost levers — spend caps
- Re-indexing — the classic capacity collision