Foundations → Model selection & tradeoffs
What it is
An AI system makes many different kinds of LLM calls: classify the intent, rewrite the query, extract fields, rerank candidates, generate the answer, critique the answer, summarise the history. They have wildly different requirements. Treating "the model" as one global choice means either overpaying (a frontier model classifying yes/no) or underdelivering (a tiny model writing a legal summary).
The practical split from the source file:
| Task type | Model class | Why |
|---|---|---|
| Classification, routing, intent | Cheap, fast (Haiku-class, small open models) | Short output, low ambiguity, called on every request |
| Query rewriting, HyDE, decomposition | Cheap, fast | Latency is on the critical path |
| Field extraction, structured output | Cheap, fast, with schema enforcement | Deterministic-ish; validate + retry |
| Reranking-lite, relevance grading | Cheap, or a dedicated cross-encoder | Many calls per request |
| Final answer generation | Reasoning model (Sonnet/Opus-class) | Quality is what users see |
| Planning, critique, complex agent steps | Reasoning model | Errors compound |
Selection axes: capability (on your eval set), cost ($/1M input + output tokens, cache discounts), latency (time-to-first-token, tokens/s), context window (and how quality holds at long context), plus deployment constraints (region, data residency, self-host vs API), and stability (versioning, deprecation policy).
How to implement
- Inventory the calls. List every LLM call site with: purpose, input size, output size, latency budget, quality bar, call volume.
- Assign a model tier per call site, not a model:
tier=fast,tier=balanced,tier=reasoning. Map tiers to concrete pinned model IDs in one config file. Swapping a model is a config change. - Evaluate per call site on the golden set (golden dataset first): the cheapest tier that meets the quality bar wins. Don't assume; measure — small models often pass classification at 99% of the big model's accuracy at 5% of the cost.
- Compute the cost model:
calls/day × (input_tokens × price_in + output_tokens × price_out)per site; sum; that's your $/task and $/user/month (cost levers). - Set latency budgets per site and verify TTFT/throughput under load, not in a notebook.
- Pin versions; treat model upgrades as releases with eval gates (versioning & rollout).
- Abstract the provider behind a thin client interface (messages in, structured response out, with retries/timeouts/telemetry) so a fallback provider is a config change (failure handling).
Small example
# models.yaml — one place, pinned IDs
"""
tiers:
fast: {provider: anthropic, model: claude-haiku-4-5-20251001, max_tokens: 512}
balanced: {provider: anthropic, model: claude-sonnet-5, max_tokens: 2048}
reasoning: {provider: anthropic, model: claude-opus-5, max_tokens: 4096}
call_sites:
intent_classifier: fast
query_rewrite: fast
field_extract: fast
answer_generate: balanced
answer_critique: reasoning
plan_step: reasoning
"""
import yaml, anthropic, time
CFG = yaml.safe_load(open("models.yaml"))
client = anthropic.Anthropic()
def llm(call_site: str, system: str, messages: list, **kw):
tier = CFG["tiers"][CFG["call_sites"][call_site]]
t0 = time.time()
resp = client.messages.create(model=tier["model"], max_tokens=kw.get("max_tokens", tier["max_tokens"]),
system=system, messages=messages, temperature=kw.get("temperature", 0))
log_call(call_site, tier["model"], resp.usage.input_tokens, resp.usage.output_tokens, time.time() - t0)
return resp
# usage
r = llm("intent_classifier", "Classify the user's request as one of: search, chit_chat, action. Answer with the label only.",
[{"role": "user", "content": "what's our parental leave policy?"}])
Pros
- Cost drops dramatically when the majority of calls (classification, rewriting) run on fast models.
- Latency improves on the critical path.
- Quality is protected where it matters (final answer, planning).
- Config-driven tiers make model swaps, A/B tests, and provider fallbacks routine.
Cons
- More models = more evals to maintain, more behaviours to understand.
- Small models are less robust to prompt drift and adversarial input; guardrails matter more.
- Provider heterogeneity (different tokenizers, formats, tool-calling semantics) leaks through abstractions.
- Over-optimising for cost early can hide quality issues; start with a strong model to establish the ceiling, then step down where evals allow.
When to use (tiering)
- Any system with more than one type of LLM call — i.e. every real system.
- High-volume products where per-request cost matters.
When NOT to use (prematurely)
- Prototype phase: use one strong model everywhere to find the quality ceiling; tier down after you have evals.
- Don't tier down a call site without a golden-set comparison; "it looked fine" is how regressions ship.
Production at scale
$/task is the number. Instrument every call with call_site, model, tokens_in, tokens_out, cached_tokens, latency. Roll up to $/request and $/user/month (cost levers). Leadership will ask; you should already know.
Latency under load ≠ latency in a notebook. Provider TTFT and throughput vary with load and region; measure p50/p95 per model per hour; plan critical-path calls with headroom. Streaming (latency levers) hides generation latency but not TTFT.
Context window vs quality. A 200k-token window does not mean quality holds at 150k. Long contexts degrade attention ("lost in the middle", "Lost in the middle") and cost more per call. Prefer better retrieval over bigger context.
Pin and stage. Model IDs pinned in config; new versions go through the eval suite, then a canary (versioning & rollout). Providers deprecate models on schedules — track deprecation dates as tickets.
Provider fallback. Map each tier to a primary and a fallback (different provider or self-hosted). Fallback triggers on error rate/latency SLO breach (failure handling). Verify the fallback on the eval suite too — a fallback that produces worse answers silently is a hidden regression.
Self-host vs API. Self-hosting (self-hosted serving) wins at high, steady volume with predictable prompts, or for residency; APIs win on frontier capability and elasticity. Many systems use both: self-hosted small models for fast-tier calls, API for reasoning-tier.
Region and residency are selection criteria, not afterthoughts (data residency & retention): some tenants require in-region inference and zero retention; encode as per-tenant tier overrides.
Escalation ladders (routing & escalation) add the dynamic version of tiering: try fast, escalate on low confidence.
Prod checklist
- Call-site inventory with purpose, sizes, latency budget, quality bar, volume
- Tier config with pinned model IDs; call sites map to tiers
- Per-call-site golden-set eval; cheapest passing tier chosen
- Cost model per site; $/task and $/user/month dashboards
- Latency SLOs per site measured under load
- Provider abstraction with retries, timeouts, telemetry, fallback
- Deprecation dates tracked; upgrade = eval gate + canary
- Per-tenant residency overrides
Metrics to watch
| Metric | Why |
|---|---|
| $/request by call site | Cost attribution |
| Quality on golden set by tier | Tier-down safety |
| p95 TTFT and tokens/s by model | Latency SLOs |
| Fallback activation rate | Provider health |
Related
- Routing/escalation — dynamic tiering
- Cost levers, Latency levers, Self-hosted serving
- Versioning & rollout — pinned models
- Failure handling — fallbacks
- Prompt evaluation — per-site evals