— Decision tree: whether to resize, and which axis

Back to GPUResizing.md.


Ask in this order. Stop at the first “yes.” Identify the binder per pool (generation vs embed vs OCR vs train) — see 05_workload_pools.md.

1. Is the model / quantisation / max context the real constraint?
   YES → change S (quantise, cap context, split workloads). Do not buy a bigger GPU yet.

2. Is latency SLO breached while GPU util is high AND queue depth is growing?
   YES → add H (same SKU replicas) if the model still fits and tokens/s per GPU is healthy.

3. Are you OOM, KV-preempting, or unable to reach target concurrency on this SKU?
   YES → V (more VRAM / bandwidth) or S (smaller KV: shorter ctx, fewer seqs, GQA/MLA, FP8 KV).

4. Do weights + KV at target concurrency exceed the largest single card you can get?
   YES → W (tensor parallel), or shrink the model / context, or keep that model on an API.

5. Is utilisation low and cost high?
   YES → scale H down, consolidate pools, or stop self-hosting that workload.

Signals that look like “need GPUs” but are not

SymptomOften actuallyFirst move
GPU util ~95%Healthy vLLMIgnore util; look at queue + TTFT + KV
TTFT up, KV full, preemptionContext/concurrency too high for VRAMCap max-model-len / max-num-seqs, or V
TTFT up, KV empty, queue deepNot enough replicas or prefill-bound SKUH if tokens/s/GPU at knee; else V
OOM on one tenantNo per-tenant capTenant concurrency / context cap
Chat p95 dies during ingestShared poolSplit OCR/embed off generation
High $ , empty queue at nightOver-provisioned HScheduled scale-down to minReplicas
Quality complaints after “upgrade”Quant/TP/engine changeGolden set; rollback SKU/flags

Target you must write down before buying

Example: “p95 TTFT ≤ 800 ms at 40 concurrent 8k-ctx generations, with 20% KV headroom, N+1 replicas.”

Without concurrency × context × SLO, every SKU is defensible and none is.

Binder identification in code: 11_examples.md. Sizing math: 04_resources_and_sizing.md.