— Four axes: S, H, V, W

Back to GPUResizing.md.


Default order: S → H → V → W.

S — Software packing

Change how work uses the same metal.

LeverEffectRisk
Quantisation (AWQ/GPTQ/FP8/INT8)2–4× less weight VRAM; more room for KVQuality; must eval
FP8 / lower-precision KVLarge concurrency winQuality on long context
Lower max-model-lenLinear drop in KV per seqProduct: refuse or route long jobs
Lower max-num-seqsStops preemption; can improve TTFTThroughput cap
Prefix / automatic prefix cachingPrefill and KV reuseHit rate drops if prompts are unique
Split pools (embed/rerank/OCR off gen)Stops SLO collisionsMore node groups
Multi-LoRA vs many full modelsOne base + adaptersActive adapter count still costs VRAM
MIG slices for small modelsDensity on A100/H100 leftoversNot for bandwidth-hungry decode

Exhaust S before a purchase. Flags: 08_serving_knobs.md.

H — Horizontal (more identical replicas)

Add (or remove) copies of the same GPU SKU and the same serving unit (usually TP=1).

  • Wins: throughput roughly linear; blast radius small; rolling replace is ordinary.
  • Loses: idle dollars; prefix-cache hit rate can fall (each replica has its own cache); does not help if one replica cannot fit the model.
  • Autoscale: queue depth and TTFT, plus a warm minReplicas. Never scale interactive generation to zero.

N+1 (or N+2 across AZs) is part of H, not optional spare.

V — Vertical (different SKU)

Change GPU generation and/or VRAM: T4 → L4/A10G → A100 40/80 → H100/H200/B200.

Two different vertical buys:

BuyWhat you getWhen
More VRAM (40 → 80 GB, same-ish gen)More KV and/or a larger model at TP=1OOM, preemption, concurrency shortfall
Newer gen, similar VRAM (A10 → L4, A100 → H100)Bandwidth + FLOPS → tokens/s, TTFTDecode/prefill slow at the concurrency knee

V implies a new node group, new drivers possibly, new flags, overlap cost, and often reservation churn. Treat as a production deploy.

W — Width (more GPUs per replica)

Tensor / pipeline / expert parallelism so one request spans N GPUs.

ModeRole in servingTax
Tensor parallel (TP)Split layers across GPUs when weights do not fitNCCL on every layer; needs NVLink for large TP
Pipeline parallel (PP)Stage layers; rare for latency-sensitive servingPipeline bubbles
Expert parallel (EP)MoE expert placementRouting imbalance
Data parallelThis is H, not WNone in the request path

Prefer 8× one-GPU replicas over 1× eight-GPU replica unless the model cannot fit. TP a model that already fits “for speed” is often slower and always more expensive.

TP size must match nvidia.com/gpu on the pod, divide model heads correctly, and live on a SKU family with the interconnect you assumed.

Trade-off tables: 09_trade_offs.md.