— Trade-off matrices

Back to GPUResizing.md.


Horizontal vs vertical vs width vs software

MoveWinsLosesBest when
S — Pack better (quant, ctx cap, prefix cache, FP8 KV, split pools)$ down, often latency downQuality risk; eng time; eval costAlways first
H — More same GPUsLinear-ish throughput; simple; good HAIdle $; still cannot fit a too-big modelSLO miss, model fits, tokens/s/GPU OK
V — Bigger/newer SKUFit, concurrency, TTFT, future headroom$; reservation lock-in; driver/CUDA churn; capacity scarcityOOM/KV-bound or decode-bound on current gen
W — TP > 1Serve models that do not fitNCCL tax; worse blast radius; slower rolling updatesWeights + KV > one card after quant
API overflow / stay on APIElasticity, no GPU opsUnit $ at high volume; residencyBurst, frontier quality, small team

One large GPU vs many small

One 80 GB (A100/H100)Several 24 GB (L4/A10G)
Fits70B 4-bit + real concurrency; long context7–13B (or 70B only with TP or heavy quant)
PackingCan MIG-split leftover for embed/rerankSimple 1:1 model↔card
FailureOne card down is a large capacity holeMany small holes; easier N+1
Cost curveHigh idle costBetter for many small models
BandwidthHigh; good decodeAda L4 is strong $ / inference; T4 is usually a trap for LLM decode

Practical default: L4/A10G for embed + rerank + 8B fast tier; A100/H100 only for the generation model that needs it. Do not put embeddings on H100s.

MIG vs exclusive vs time-slice

See 07_cloud_kubernetes.md. Short: exclusive for interactive LLM; MIG for small models; time-slice not for chat.

Warm capacity vs autoscale-from-zero

Keep warm minReplicasScale to zero
TTFT after idleSLO-safeMinutes of ice-cold load
CostPay for idleAttractive for nights/weekends
UseUser-facing generationBatch pools, internal tools, staging

Headroom

  • VRAM 10–20% after weights + target KV. gpu-memory-utilization 0.98 turns fragmentation into random OOM.
  • Replica N+1 (or N+2 across AZs) so one death or one rolling update does not breach SLO.
  • Regional GPU quota headroom. The limiter is often not your Helm chart.

Paying for two fleets

During V/W you pay for both fleets. Budget the overlap window. Drain with connection draining longer than max decode. Do not cancel old reservations until the new fleet survives a real peak.