Foundations → Model selection & tradeoffs

LLM Model Selection: Cheap vs Reasoning Models Per Call

What it is

An AI system makes many different kinds of LLM calls: classify the intent, rewrite the query, extract fields, rerank candidates, generate the answer, critique the answer, summarise the history. They have wildly different requirements. Treating "the model" as one global choice means either overpaying (a frontier model classifying yes/no) or underdelivering (a tiny model writing a legal summary).

The practical split from the source file:

Task typeModel classWhy
Classification, routing, intentCheap, fast (Haiku-class, small open models)Short output, low ambiguity, called on every request
Query rewriting, HyDE, decompositionCheap, fastLatency is on the critical path
Field extraction, structured outputCheap, fast, with schema enforcementDeterministic-ish; validate + retry
Reranking-lite, relevance gradingCheap, or a dedicated cross-encoderMany calls per request
Final answer generationReasoning model (Sonnet/Opus-class)Quality is what users see
Planning, critique, complex agent stepsReasoning modelErrors compound

Selection axes: capability (on your eval set), cost ($/1M input + output tokens, cache discounts), latency (time-to-first-token, tokens/s), context window (and how quality holds at long context), plus deployment constraints (region, data residency, self-host vs API), and stability (versioning, deprecation policy).

How to implement

  1. Inventory the calls. List every LLM call site with: purpose, input size, output size, latency budget, quality bar, call volume.
  2. Assign a model tier per call site, not a model: tier=fast, tier=balanced, tier=reasoning. Map tiers to concrete pinned model IDs in one config file. Swapping a model is a config change.
  3. Evaluate per call site on the golden set (golden dataset first): the cheapest tier that meets the quality bar wins. Don't assume; measure — small models often pass classification at 99% of the big model's accuracy at 5% of the cost.
  4. Compute the cost model: calls/day × (input_tokens × price_in + output_tokens × price_out) per site; sum; that's your $/task and $/user/month (cost levers).
  5. Set latency budgets per site and verify TTFT/throughput under load, not in a notebook.
  6. Pin versions; treat model upgrades as releases with eval gates (versioning & rollout).
  7. Abstract the provider behind a thin client interface (messages in, structured response out, with retries/timeouts/telemetry) so a fallback provider is a config change (failure handling).

Small example

# models.yaml — one place, pinned IDs
"""
tiers:
  fast:      {provider: anthropic, model: claude-haiku-4-5-20251001, max_tokens: 512}
  balanced:  {provider: anthropic, model: claude-sonnet-5,          max_tokens: 2048}
  reasoning: {provider: anthropic, model: claude-opus-5,            max_tokens: 4096}
call_sites:
  intent_classifier: fast
  query_rewrite:     fast
  field_extract:     fast
  answer_generate:   balanced
  answer_critique:   reasoning
  plan_step:         reasoning
"""
import yaml, anthropic, time

CFG = yaml.safe_load(open("models.yaml"))
client = anthropic.Anthropic()

def llm(call_site: str, system: str, messages: list, **kw):
    tier = CFG["tiers"][CFG["call_sites"][call_site]]
    t0 = time.time()
    resp = client.messages.create(model=tier["model"], max_tokens=kw.get("max_tokens", tier["max_tokens"]),
                                  system=system, messages=messages, temperature=kw.get("temperature", 0))
    log_call(call_site, tier["model"], resp.usage.input_tokens, resp.usage.output_tokens, time.time() - t0)
    return resp

# usage
r = llm("intent_classifier", "Classify the user's request as one of: search, chit_chat, action. Answer with the label only.",
        [{"role": "user", "content": "what's our parental leave policy?"}])

Pros

  • Cost drops dramatically when the majority of calls (classification, rewriting) run on fast models.
  • Latency improves on the critical path.
  • Quality is protected where it matters (final answer, planning).
  • Config-driven tiers make model swaps, A/B tests, and provider fallbacks routine.

Cons

  • More models = more evals to maintain, more behaviours to understand.
  • Small models are less robust to prompt drift and adversarial input; guardrails matter more.
  • Provider heterogeneity (different tokenizers, formats, tool-calling semantics) leaks through abstractions.
  • Over-optimising for cost early can hide quality issues; start with a strong model to establish the ceiling, then step down where evals allow.

When to use (tiering)

  • Any system with more than one type of LLM call — i.e. every real system.
  • High-volume products where per-request cost matters.

When NOT to use (prematurely)

  • Prototype phase: use one strong model everywhere to find the quality ceiling; tier down after you have evals.
  • Don't tier down a call site without a golden-set comparison; "it looked fine" is how regressions ship.

Production at scale

$/task is the number. Instrument every call with call_site, model, tokens_in, tokens_out, cached_tokens, latency. Roll up to $/request and $/user/month (cost levers). Leadership will ask; you should already know.

Latency under load ≠ latency in a notebook. Provider TTFT and throughput vary with load and region; measure p50/p95 per model per hour; plan critical-path calls with headroom. Streaming (latency levers) hides generation latency but not TTFT.

Context window vs quality. A 200k-token window does not mean quality holds at 150k. Long contexts degrade attention ("lost in the middle", "Lost in the middle") and cost more per call. Prefer better retrieval over bigger context.

Pin and stage. Model IDs pinned in config; new versions go through the eval suite, then a canary (versioning & rollout). Providers deprecate models on schedules — track deprecation dates as tickets.

Provider fallback. Map each tier to a primary and a fallback (different provider or self-hosted). Fallback triggers on error rate/latency SLO breach (failure handling). Verify the fallback on the eval suite too — a fallback that produces worse answers silently is a hidden regression.

Self-host vs API. Self-hosting (self-hosted serving) wins at high, steady volume with predictable prompts, or for residency; APIs win on frontier capability and elasticity. Many systems use both: self-hosted small models for fast-tier calls, API for reasoning-tier.

Region and residency are selection criteria, not afterthoughts (data residency & retention): some tenants require in-region inference and zero retention; encode as per-tenant tier overrides.

Escalation ladders (routing & escalation) add the dynamic version of tiering: try fast, escalate on low confidence.

Prod checklist

  • Call-site inventory with purpose, sizes, latency budget, quality bar, volume
  • Tier config with pinned model IDs; call sites map to tiers
  • Per-call-site golden-set eval; cheapest passing tier chosen
  • Cost model per site; $/task and $/user/month dashboards
  • Latency SLOs per site measured under load
  • Provider abstraction with retries, timeouts, telemetry, fallback
  • Deprecation dates tracked; upgrade = eval gate + canary
  • Per-tenant residency overrides

Metrics to watch

MetricWhy
$/request by call siteCost attribution
Quality on golden set by tierTier-down safety
p95 TTFT and tokens/s by modelLatency SLOs
Fallback activation rateProvider health