— Mental model: what GPU resize actually is

Back to GPUResizing.md.


On CPUs you often change vCPU/RAM on a running node. On GPUs in production, the card is the unit of capacity, and most clouds cannot hot-swap GPU type or GPU count on a live instance. A “resize” is almost always:

decide new shape → provision new capacity → load model(s) → shift traffic → drain old → terminate

That sequence is minutes to tens of minutes, not seconds. Model weights must land in VRAM, CUDA contexts initialise, and KV-cache capacity only exists after the replica is warm. Treat every resize as a controlled fleet replacement, not a live edit.

Why GPUs are not CPUs

CPU serviceGPU serving
Capacity ≈ cores × clockCapacity ≈ VRAM pages + memory bandwidth + FLOPS, coupled
Utilisation high → add replicasUtilisation high is normal for continuous batching
Scale in secondsWarm replica takes minutes (provision + weight load)
Vertical resize often in placeSKU change = new VM / node / endpoint variant
Memory is shared RAMWeights + KV must fit in this card's VRAM or you OOM

What people call “resize”

Four different operations. Using the wrong word buys the wrong hardware.

People sayThey usually meanAxis
“We need more GPUs”More copies of the same serving unitH
“We need bigger GPUs”More VRAM or a newer generationV
“The model doesn’t fit”More GPUs inside one replica (TP) or a larger SKUW or V
“We’re wasting the cards”Quantise, cap context, split pools, scale downS or H down

See 03_axes_h_v_w_s.md.

Implications that follow from the mental model

  1. Readiness is “weights in VRAM,” not “process started.” Load balancers must wait.
  2. Drain time is max decode time, not HTTP idle timeout.
  3. You pay for overlap during any SKU change — old fleet + new fleet.
  4. Autoscaling is predictive or heavily buffered. GPU VMs are not Lambda.
  5. A SKU/quant/TP change is a behaviour change. Treat it like a model deploy.

Next: 02_decision_tree.md.