— Production cutover playbook
Back to GPUResizing.md. Code: 11_examples.md.
Phase A — Diagnose (hours, not a ticket titled “need A100s”)
- Split metrics per pool (generation / embed / rerank / OCR / train).
- Identify the binder: VRAM OOM, KV utilisation + preemption, queue depth, TTFT, TPOT, SM util, CPU raster, or host disk.
- Confirm it is not software: context bloat, missing prefix cache, wrong
max-num-seqs, embed job on gen GPUs, tenant without concurrency cap. - Write the target: e.g. “p95 TTFT ≤ 800 ms at 40 concurrent 8k-ctx generations, with 20% KV headroom.”
Phase B — Size the new shape on paper
- Recompute VRAM (weights + KV at p95 context × target concurrency + headroom).
- Pick axis: S / H / V / W (02_decision_tree.md).
- Estimate tokens/s per GPU from a similar SKU or a 15-minute benchmark, then
replicas = ceil(peak_tokens_per_s / tokens_per_s_per_gpu)plus N+1. - Cost:
$/hour × 24 × 30vs$/1M tokensvs API overflow. Include overlap during cutover.
Phase C — Prove it off to the side
- Stand up a canary node group with the new SKU/flags. Do not mutate prod.
- Correctness: golden set at the new quant/TP/engine/CUDA.
- Load test with production prompt-length histogram, not 128-token synthetic. Find the concurrency knee (TTFT/TPOT vs
max-num-seqs). - Watch KV preemption, GPU memory, TTFT, tokens/s, error rate, and quality.
Phase D — Cut over
- Pin image, engine, CUDA, weights hash, flags.
- Shift traffic 5% → 25% → 50% → 100% with automatic rollback on TTFT, error rate, or eval drift.
- Keep API or old-SKU overflow until the new fleet survives a peak.
- Drain old nodes; only then cancel old reservations / scale old ASG to 0.
- Recalibrate HPA/KEDA: queue depth, not GPU util. Update runbooks.
Phase E — After
- Recompute
$/1M tokensand$/task. If VRAM is half empty at peak, you over-bought — pack or downsize. - Record SKU, TP, flags, and the reason in traces / change log.
Change management
A GPU SKU change is a production deploy: golden set, canary, rollback. Reservations should follow a stable SKU — do not convert to 1-year reserved H100s until model size and TP topology have been stable through two releases.
Overflow is part of resize. API or a second region is the elastic tier; self-hosted GPUs are the baseline. Circuit-break to overflow when queue exceeds a hard limit.
Checklist: 14_checklist_metrics.md.