Production → Versioning & rollout

LLM Versioning and Rollout Explained: Pin, Canary, Roll Back

What it is

An AI system's behaviour is determined by a version vector, not a single code version:

app_sha · prompt_versions{} · model_id · model_params · retrieval_config
        · index_version · embedding_model · parser_version · chunking_config
        · reranker_version · tool_schema_versions{} · guardrail_config

Any of these changing alters behaviour. Three implications:

  1. Everything is pinned in environment-specific config; production never resolves "latest".
  2. Every change goes through the same gate: eval suite (prompt evaluation) → canary → online comparison → promote or roll back.
  3. Every trace carries the full vector (observability & tracing) so a quality change maps to the artifact that moved.

The rollout mechanism differs by artifact:

ArtifactRolloutRollback
PromptConfig pin flip, % trafficFlip pin (seconds)
ModelConfig pin, canary %Flip pin (seconds)
Retrieval config (k, weights, thresholds)Config, canary %Flip config
Index / embedding modelDual-index build → cutover flag (versioning & migration)Flip back to old index (kept for a window)
Parser / chunking configRe-index wave (change detection) → cutoverSame as index
Tool schemaAdditive deploy; consumers pinRevert deploy

How to implement

  1. Config-driven pins per environment (dev/staging/prod), changed by PR, applied without a code deploy where possible.
  2. Eval gate before rollout (the testing pyramid): the candidate must not regress previously-passing cases; safety gates are absolute.
  3. Canary by percentage or cohort, starting with internal users, then 1%, 5%, 25%, 100%, with a hold at each step.
  4. Compare online, not just offline: quality proxies (faithfulness, citation validity, abstention), user signals (retry, edit distance, escalation), cost, and latency, sliced by version (online eval).
  5. Automatic rollback triggers on hard metrics: error rate, guardrail violations, cost per request, latency SLO.
  6. Keep the previous version warm: old index retained for a rollback window; old prompt pin one flip away.
  7. Track deprecations: provider model sunsets are scheduled work, not surprises.

Small example

# config/versions.prod.yaml
"""
prompts:   {answer_generate: 7, query_rewrite: 3, self_query: 2}
model:     {balanced: claude-sonnet-5, fast: claude-haiku-4-5-20251001, reasoning: claude-opus-5}
retrieval: {config: hybrid_rrf_k50@v3, pool_k: 50, final_k: 6, rerank: rerank-v3.5}
index:     {live: chunks_v2, candidate: null, embedding_model: "e5-acme-ft-v1"}
parser:    {version: "docling@2.14", chunking: "parent_child@v4"}
canary:    {enabled: true, percent: 5, cohort: internal, overrides: {prompts: {answer_generate: 8}}}
"""

import hashlib

def version_vector(cfg, canary: bool) -> dict:
    v = {"app_sha": os.getenv("GIT_SHA"), **{f"prompt.{k}": val for k, val in cfg["prompts"].items()},
         "model": cfg["model"]["balanced"], "retrieval": cfg["retrieval"]["config"],
         "index": cfg["index"]["live"], "embedding_model": cfg["index"]["embedding_model"],
         "parser": cfg["parser"]["version"], "chunking": cfg["parser"]["chunking"],
         "variant": "canary" if canary else "stable"}
    if canary:
        for k, val in cfg["canary"].get("overrides", {}).get("prompts", {}).items():
            v[f"prompt.{k}"] = val
    return v

def in_canary(user_id: str, cfg) -> bool:
    if not cfg["canary"]["enabled"]:
        return False
    if cfg["canary"].get("cohort") == "internal" and not is_internal(user_id):
        return False
    bucket = int(hashlib.sha256(user_id.encode()).hexdigest()[:8], 16) % 100    # sticky per user
    return bucket < cfg["canary"]["percent"]

def resolve_config(user_id, cfg):
    canary = in_canary(user_id, cfg)
    resolved = deep_merge(cfg, cfg["canary"]["overrides"]) if canary else cfg
    return resolved, version_vector(cfg, canary)

# --- automatic rollback on hard metrics ---
ROLLBACK_RULES = [
    ("error_rate",           lambda c, s: c > max(s * 2, 0.02)),
    ("guardrail_violations", lambda c, s: c > max(s * 2, 0.01)),
    ("faithfulness",         lambda c, s: c < s - 0.05),
    ("cost_per_request",     lambda c, s: c > s * 1.5),
    ("p95_latency_ms",       lambda c, s: c > s * 1.5),
    ("citation_valid_rate",  lambda c, s: c < s - 0.03),
]

def canary_health(canary_metrics, stable_metrics, min_n=200):
    if canary_metrics["n"] < min_n:
        return {"decision": "wait", "reason": f"n={canary_metrics['n']} below {min_n}"}   # Statistical hygiene
    breaches = [name for name, rule in ROLLBACK_RULES
                if rule(canary_metrics[name], stable_metrics[name])]
    if breaches:
        return {"decision": "rollback", "breaches": breaches}
    if all(canary_metrics[m] >= stable_metrics[m] - 0.01
           for m in ("faithfulness", "citation_valid_rate")):
        return {"decision": "promote"}
    return {"decision": "hold"}

def rollback(artifact: str, cfg_path: str):
    """One command, seconds, no deploy."""
    cfg = load(cfg_path)
    cfg["canary"]["enabled"] = False                      # prompts/models/retrieval: flip the pin
    if artifact == "index":
        cfg["index"]["live"], cfg["index"]["candidate"] = cfg["index"]["candidate"], cfg["index"]["live"]
    save(cfg, cfg_path); publish_config()
    alert_rollback(artifact, actor="auto")

Pros

  • Regressions are caught on 1–5% of traffic rather than all of it.
  • Rollback in seconds because behaviour lives in config, not in a deploy.
  • Attribution: the version vector on every trace makes "what changed on Tuesday" answerable (observability & tracing).
  • Pinned models eliminate silent behaviour drift from provider-side upgrades.
  • Canary comparison catches what offline evals miss: real query distribution, real corpus, real users.

Cons

  • Canary needs volume and time to reach statistical significance; low-traffic systems can't canary meaningfully (statistical hygiene).
  • Index and parser changes can't be canaried cheaply — dual-index doubles storage for the window (versioning & migration).
  • Config sprawl: many pinned artifacts across environments to keep coherent.
  • Interaction effects: two simultaneous canaries confound each other; serialise them.
  • Sticky bucketing by user means small cohorts drift from the population.

When to use

Every behaviour-affecting change in a production system. Especially model upgrades, which alter every behaviour at once.

When NOT to use (full canary)

  • Low-traffic internal tools: run the eval suite, ship to everyone, monitor, and keep rollback ready.
  • Emergency security fixes: ship immediately with monitoring; the risk calculus is different.

Production at scale

Pin models, always. An unpinned model alias is a behaviour change scheduled by someone else. Track deprecation dates as planned work; migrating under a sunset deadline with no eval time is how quality regressions ship (model selection & tradeoffs).

One change at a time. Two concurrent canaries (a prompt and a model) make attribution impossible. Serialise, or use factorial assignment with enough volume to separate effects.

Model upgrades run everything: full eval suite, red-team (prompt-injection defence), tool-selection accuracy (tool design), schema-violation rates (structured outputs & tool use), position-sensitivity probe ("Lost in the middle"), and grounding adherence (grounded prompting). A newer model is not automatically better on your task.

Index and parser cutovers reuse the dual-index pattern (versioning & migration): build alongside, evaluate, flip a flag, keep the old index for a rollback window, then delete. Same machinery for embedding models, chunking configs, and parser upgrades.

Automatic rollback on hard metrics, human decision on soft ones. Error rate, guardrail violations, cost, and latency roll back automatically. Quality judgements (faithfulness deltas within noise) go to a human with the comparison data.

Minimum sample sizes before deciding (statistical hygiene): a canary at 1% on a low-traffic system may take days to reach significance. Compute it in advance and set the hold duration accordingly.

Sticky assignment by user, not per request, so a user doesn't see the old and new behaviour alternately within a conversation.

Practise the rollback. A rollback that has never been executed is a hypothesis. Run a drill; measure time-to-rollback; it should be under five minutes.

Config changes are reviewed like code: PR, approval, audit trail. A prompt pin flip is a production behaviour change (prompt versioning & evaluation, auditability).

Prod checklist

  • All behaviour-affecting artifacts pinned per environment; prod never resolves "latest"
  • Eval gate + absolute safety gate before any rollout (the testing pyramid)
  • Canary by sticky user bucket, staged internal → 1% → 5% → 25% → 100% with holds
  • Online comparison by version: quality proxies, user signals, cost, latency
  • Automatic rollback rules on hard metrics; human decision on quality deltas
  • Minimum sample size per canary stage computed in advance
  • One change canaried at a time
  • Index/parser/embedding changes via dual-index cutover with a rollback window
  • Model deprecation dates tracked as scheduled work
  • Rollback drilled; time-to-rollback measured and under target
  • Config changes reviewed and audited like code

Metrics to watch

MetricWhy
Quality/cost/latency by version vectorAttribution
Canary vs stable deltas with sample sizeDecision input
Time-to-rollback (drilled and actual)Operational readiness
Unpinned or unknown versions in prod traces (target 0)Discipline
Rollbacks per quarter and their causesRelease health