Production → Capacity & limits

LLM Rate Limits and Capacity Planning Explained: Quota Is Your Ceiling

What it is

AI systems have unusual capacity characteristics: the bottleneck is typically an external quota (tokens per minute, requests per minute) rather than your own CPU, and the cost of exceeding it is 429s that cascade into user-visible failures.

Four control layers:

LayerPurposeMechanism
Quota managementStay within provider limitsToken-bucket per provider/model/region; track TPM and RPM separately
Per-tenant throttlingPrevent one tenant starving othersPer-tenant rate and concurrency limits, plus monthly spend caps
Queueing with backpressureAbsorb bursts without dropping workBounded queues; reject early when full rather than timing out late
Load sheddingDegrade deliberately under overloadDrop optional stages first, then non-critical traffic classes

The essential design decision is the shed order — what you give up, in what sequence, when you can't serve everything:

1. Optional quality stages   multi-query → HyDE → reranking depth → context size
2. Tier downgrade            reasoning tier → balanced → fast
3. Non-interactive work      background jobs, ingestion, evals, re-indexing
4. Low-priority tenants      free tier before paid; batch users before interactive
5. Reject with a clear message + Retry-After

How to implement

  1. Know your quotas: TPM and RPM per provider, model, and region; per-tenant caps you've committed to; vector store and reranker limits.
  2. Client-side rate limiting with a token bucket per (provider, model), sized at ~85% of the quota to leave headroom for retries.
  3. Estimate tokens before sending so the bucket can reserve accurately; reconcile with actual usage afterwards.
  4. Per-tenant limits: concurrent requests, requests/minute, and monthly spend (cost & step budgets); enforced at ingress with a clear error and Retry-After.
  5. Bounded queues with early rejection: when the queue is full, reject immediately with a retry hint rather than accepting work you cannot finish.
  6. Priority classes: interactive > batch > background. Background work yields first.
  7. Shed-order configuration, applied automatically on pressure signals (queue depth, 429 rate, latency SLO breach) and reversed when pressure clears.

Small example

import time, threading, random

class TokenBucket:
    """Client-side limiter for both TPM and RPM — the two provider quotas."""
    def __init__(self, tokens_per_min, requests_per_min, headroom=0.85):
        self.tpm_cap = tokens_per_min * headroom
        self.rpm_cap = requests_per_min * headroom
        self.tokens = self.tpm_cap; self.requests = self.rpm_cap
        self.last = time.time(); self.lock = threading.Lock()

    def _refill(self):
        now = time.time(); elapsed = now - self.last
        self.tokens = min(self.tpm_cap, self.tokens + self.tpm_cap * elapsed / 60)
        self.requests = min(self.rpm_cap, self.requests + self.rpm_cap * elapsed / 60)
        self.last = now

    def try_acquire(self, estimated_tokens: int) -> tuple[bool, float]:
        with self.lock:
            self._refill()
            if self.tokens >= estimated_tokens and self.requests >= 1:
                self.tokens -= estimated_tokens; self.requests -= 1
                return True, 0.0
            wait = max((estimated_tokens - self.tokens) / (self.tpm_cap / 60),
                       (1 - self.requests) / (self.rpm_cap / 60))
            return False, wait

    def reconcile(self, estimated: int, actual: int):
        with self.lock:
            self.tokens -= max(actual - estimated, 0)            # correct under-estimates

BUCKETS = {("anthropic", "claude-sonnet-5"): TokenBucket(400_000, 4_000)}

# --- per-tenant limits at ingress ---
class TenantLimiter:
    def __init__(self, cfg): self.cfg, self.inflight, self.rpm = cfg, {}, {}
    def check(self, tenant_id) -> tuple[bool, str, int]:
        c = self.cfg[tenant_id]
        if self.inflight.get(tenant_id, 0) >= c["max_concurrent"]:
            return False, "tenant_concurrency", 5
        if self.rpm.get(tenant_id, 0) >= c["rpm"]:
            return False, "tenant_rate", 30
        if spend_month_to_date(tenant_id) >= c["monthly_usd_cap"]:
            return False, "tenant_quota_exhausted", 0            # no retry: needs an upgrade
        return True, "", 0

# --- pressure-driven load shedding ---
SHED_LADDER = [
    ("multi_query",     lambda cfg: cfg.update(multi_query=False)),
    ("hyde",            lambda cfg: cfg.update(hyde=False)),
    ("rerank_depth",    lambda cfg: cfg.update(pool_k=min(cfg["pool_k"], 20))),
    ("context_size",    lambda cfg: cfg.update(final_k=min(cfg["final_k"], 4))),
    ("tier_downgrade",  lambda cfg: cfg.update(tier="fast")),
    ("background_jobs", lambda cfg: pause_background_jobs()),
    ("free_tier",       lambda cfg: cfg.update(reject_classes=["free"])),
]

def apply_shedding(pressure: dict, cfg: dict) -> list[str]:
    """pressure: {queue_depth, rate_limit_429s, ttft_p95_ms}"""
    level = 0
    if pressure["queue_depth"] > 50 or pressure["rate_limit_429s"] > 10: level = 2
    if pressure["queue_depth"] > 150 or pressure["ttft_p95_ms"] > 3000:  level = 4
    if pressure["queue_depth"] > 400:                                     level = 6
    applied = []
    for name, action in SHED_LADDER[:level]:
        action(cfg); applied.append(name)
    if applied:
        log_shedding(applied, pressure)                          # visible: degradation must never be silent
    return applied

def admit(request, tenant_limiter, queue, cfg):
    ok, reason, retry_after = tenant_limiter.check(request.tenant_id)
    if not ok:
        return reject(reason, retry_after)                       # fail fast with a clear reason
    if queue.qsize() >= queue.maxsize:                           # bounded queue, early rejection
        return reject("system_busy", retry_after=10)
    if request.priority == "background" and cfg.get("reject_classes"):
        return reject("shedding_background", retry_after=60)
    queue.put(request); return None

def call_model(provider, model, messages, estimated_tokens, deadline_s):
    bucket = BUCKETS[(provider, model)]
    deadline = time.time() + deadline_s
    while time.time() < deadline:
        ok, wait = bucket.try_acquire(estimated_tokens)
        if ok:
            r = client.messages.create(model=model, messages=messages)
            bucket.reconcile(estimated_tokens, r.usage.input_tokens + r.usage.output_tokens)
            return r
        time.sleep(min(wait + random.uniform(0, 0.5), max(deadline - time.time(), 0)))
    raise CapacityExceeded("could not acquire quota within deadline")

Pros

  • Client-side limiting avoids 429 storms, which are far more damaging than a short local wait.
  • Per-tenant limits give fairness and predictable multi-tenant behaviour — one tenant's automation can't starve everyone.
  • Graceful degradation preserves availability during spikes, at reduced quality.
  • Bounded queues fail fast, which is a much better user experience than accepting work and timing out.
  • Spend caps prevent runaway cost from a bug or an abusive workload (cost & step budgets).

Cons

  • Client-side estimation is imprecise; conservative headroom wastes quota, aggressive settings cause 429s.
  • Shedding degrades quality silently unless instrumented and surfaced.
  • Per-tenant limits need commercial alignment — the limit is a product decision as much as a technical one.
  • Queueing adds latency and complexity; bounded queues mean rejecting real work.
  • Distributed rate limiting across replicas needs shared state (Redis) and adds a dependency.

When to use

Any multi-tenant production system, and any system whose traffic can spike (support tools after an incident, internal assistants at Monday-morning peak, batch ingestion colliding with interactive traffic).

When NOT to use

  • Single-user internal tools with predictable load — provider limits alone suffice.
  • Don't shed on high-stakes paths (compliance answers, financial calculations): reject explicitly rather than serving a degraded answer.

Production at scale

Distributed rate limiting is the real problem. Per-process buckets over-admit by the number of replicas. Use a shared limiter (Redis token bucket, or a sidecar gateway like Envoy/LiteLLM) that enforces the quota fleet-wide, with local buckets as a fast path.

Separate interactive from batch capacity. Ingestion, re-indexing (versioning & migration), contextual retrieval generation (contextual retrieval), and eval runs should use batch endpoints and a separate quota pool, so a backfill can never degrade interactive traffic. This is one of the most common production incidents in RAG systems.

Shed order is a product decision — write it down and get agreement before you need it. The sequence above (optional stages → tier → background → low-priority tenants → reject) is a sensible default, but the tenant-priority part in particular needs commercial sign-off.

Make shedding visible: log it, alert on it, expose a degraded flag in the response (failure handling), and show a banner if quality is reduced. Silent degradation destroys trust in evaluation metrics too, since online quality drops with no code change.

Reserve headroom for retries. Running at 100% of quota means every retry fails. 80–85% utilisation is the practical ceiling.

Multiple providers/regions multiply capacity and provide failover (failure handling). Route by health and quota headroom, not round-robin.

Watch the leading indicators: queue depth and 429 rate rise before latency does. Alert on those, not on user-visible failures.

Capacity planning needs token forecasts, not request forecasts — a feature that adds 3k tokens of context to every request consumes quota like a 3× traffic increase.

Prod checklist

  • Provider TPM and RPM quotas documented per model and region
  • Fleet-wide distributed rate limiting at ~85% of quota, with local fast-path buckets
  • Token estimation before send; reconciliation after
  • Per-tenant concurrency, rate, and monthly spend limits enforced at ingress
  • Bounded queues with early rejection and Retry-After
  • Priority classes: interactive > batch > background; background pauses first
  • Separate quota pool and batch endpoints for ingestion/backfill/evals
  • Documented, agreed shed ladder applied automatically on pressure signals
  • Shedding logged, alerted, and surfaced as degraded in responses
  • Alerts on queue depth and 429 rate, not just on failures
  • Capacity forecasting in tokens, not requests

Metrics to watch

MetricWhy
429 rate per provider/modelQuota pressure
Queue depth and rejection rateLeading indicators
Shed-ladder level and durationHow often you're degrading
Per-tenant consumption (tokens, spend)Fairness and cost
Quota utilisation vs capHeadroom for retries