GPU resizing for production AI applications

GPU Resizing for LLM Inference Explained: Replace the Fleet

1. The point

On CPUs you often change vCPU/RAM on a running node. On GPUs in production, the card is the unit of capacity, and most clouds cannot hot-swap GPU type or GPU count on a live instance. A “resize” is almost always:

decide new shape → provision new capacity → load model(s) → shift traffic → drain old → terminate

That sequence is minutes to tens of minutes, not seconds. Model weights must land in VRAM, CUDA contexts initialise, and KV-cache capacity only exists after the replica is warm. Treat every resize as a controlled fleet replacement, not a live edit.

Four axes people lump together as “resize.” They are not interchangeable:

AxisWhat you changeTypical reasonDetail
S — Software packingQuantisation, max context, max seqs, batch, MIGFit more work on the same metal03_axes
H — HorizontalReplica / node count of the same GPU SKUTraffic volume, availability03_axes
V — Vertical (SKU)GPU generation / VRAMModel no longer fits; decode too slow; KV too small03_axes
W — Width (parallelism)GPUs per replica (tensor / pipeline / expert parallel)Weights + KV exceed one card03_axes

Default order in production: exhaust S, then H, then V, then W. Bigger/newer GPUs and multi-GPU replicas are expensive and sticky. Software packing and more identical replicas are reversible.


2. How to approach it

Ask in this order. Stop at the first “yes.” Full tree: 02_decision_tree.

1. Is the model / quantisation / max context the real constraint?
   YES → change S. Do not buy a bigger GPU yet.
2. Is latency SLO breached while queue depth is growing, and tokens/s per GPU is healthy?
   YES → add H (same SKU) if the model still fits.
3. Are you OOM, KV-preempting, or unable to reach target concurrency on this SKU?
   YES → V (more VRAM / bandwidth) or S (smaller KV).
4. Do weights + KV at target concurrency exceed the largest single card you can get?
   YES → W (tensor parallel), shrink the model / context, or keep that model on an API.
5. Is utilisation low and cost high?
   YES → scale H down, consolidate pools, or stop self-hosting that workload.

Do not resize because GPU utilisation is high. vLLM sitting at 95% SM util can be healthy. A GPU at 60% util with KV full and TTFT climbing is already failing.

A production GPU replica is limited by three coupled resources — VRAM, memory bandwidth, and FLOPS. Resize to the one that is actually binding. See 01_mental_model and 04_resources_and_sizing.

Rule of thumb:

VRAM ≈ weights + KV(context × concurrent seqs) + activations + 10–20% headroom

At production concurrency, KV usually dominates. Vertical resize (more VRAM) buys concurrency and/or context. A newer generation at similar VRAM buys tokens/s and TTFT. They are different purchases.


3. Considerations

Never resize “the GPU fleet” as one thing. A prod AI app is several pools with different shapes — generation, embeddings, rerank, OCR/ingest, training. Co-locating them looks efficient and destroys SLOs. Detail: 05_workload_pools.

Before you change a SKU, walk this list (full notes: 06_considerations):

AreaWhat to decide
SLOWhich number is failing — TTFT, TPOT, E2E p95, ingest lag, train wall time? Each maps to a different axis.
Context mixSize for p95 context, not p99 on every replica. Long context is a pool, not a flag.
QualitySKU / quant / TP / engine changes behaviour. Gate with a golden set, not only a load test.
Serving flagsmax-num-seqs, max-model-len, gpu-memory-utilization, TP size, KV dtype must be re-derived for the new card.
TopologyPrefer many one-GPU replicas over one wide TP replica unless the model cannot fit.
Time constantsVM provision 2–15+ min; model load 30 s–10 min. Autoscale from queue + TTFT, never GPU util. Interactive generation: never scale to zero.
CostCompare $/1M tokens after packing, not $/GPU-hour. Reservations lock SKU. Overlap during cutover means you pay for two fleets.
IsolationStaging SKU must predict prod VRAM. Per-tenant concurrency caps often beat a bigger GPU.

Cloud/K8s mechanics (no in-place GPU type change, MIG vs exclusive, drivers): 07_cloud_kubernetes. Serving knobs: 08_serving_knobs.


4. Trade-offs

Full matrices: 09_trade_offs.

MoveWinsLosesBest when
S — Pack better$ down, often latency downQuality risk; eval costAlways first
H — More same GPUsLinear-ish throughput; simple HAIdle $; cannot fit a too-big modelSLO miss, model fits, tokens/s/GPU OK
V — Bigger/newer SKUFit, concurrency, TTFT$, reservation lock-in, driver churn, scarcityOOM/KV-bound or decode-bound on current gen
W — TP > 1Serve models that do not fitNCCL tax; worse blast radiusWeights + KV > one card after quant
API overflowElasticity, no GPU opsUnit $ at high volume; residencyBurst, frontier quality, small team

Other forks you will hit:

  • One 80 GB card vs several 24 GB cards — large cards for 70B + KV; small cards for 8B, embed, rerank. Do not put embeddings on H100s.
  • Exclusive vs MIG vs time-slice — exclusive for interactive generation; MIG for small models on leftover slices; time-slice is for batch/dev, not chat.
  • Warm minReplicas vs scale-to-zero — chat stays warm; OCR/embed may scale down.
  • Rolling vs blue/green — H-axis can roll; V/W is a new node group with overlap cost.

5. Cutover (short)

Detail and code: 10_cutover_playbook, 11_examples.

  1. Diagnose the binder per pool (OOM/KV vs queue vs TPOT vs idle).
  2. Size on paper — VRAM math, axis, replica count + N+1, cost including overlap.
  3. Prove off to the side — canary node group, golden set, load test on the production prompt-length histogram.
  4. Shift traffic 5% → 25% → 50% → 100% with abort on TTFT, errors, eval drop. Keep API/old-SKU overflow through the first peak.
  5. Drain the old fleet; only then drop reservations. Recalibrate HPA on queue depth. Recheck $/1M tokens.

Worked patterns (hybrid 8B, 70B residency, OCR spike, training isolation, overnight scale-down): 12_worked_patterns.


6. When to resize vs not

Resize when the model does not fit at required context × concurrency after packing; when queue + TTFT grow at the measured tokens/s knee; when TPOT misses SLO on an old generation; when weights exceed one card; or when KV util is low and you are overpaying.

Do not resize to fix quality, because GPU util is high, because one tenant ran 100k-token jobs (cap them), to absorb ingest/eval/train on the chat pool, or before caching/tiering/context trim if the goal is cost.


7. Finer details index

FileWhat is in it
01_mental_model.mdWhat a resize is; why GPUs are not CPUs
02_decision_tree.mdStep-by-step “whether / which axis”
03_axes_h_v_w_s.mdHorizontal, vertical, width, software packing
04_resources_and_sizing.mdVRAM / bandwidth / FLOPS; KV math
05_workload_pools.mdGen vs embed vs rerank vs OCR vs train
06_considerations.mdSLO, flags, topology, time, cost, tenancy
07_cloud_kubernetes.mdNode groups, MIG, drivers, probes, HPA
08_serving_knobs.mdvLLM flags, prefix cache, GQA, multi-LoRA
09_trade_offs.mdDecision matrices
10_cutover_playbook.mdDiagnose → canary → flip → FinOps
11_examples.mdPython sizing + Kubernetes sketch
12_worked_patterns.mdCommon prod AI shapes
13_gotchas.mdDetails that cause outages
14_checklist_metrics.mdProd checklist and metrics