— Serving knobs that must change with the GPU

Back to GPUResizing.md.


When GPU memory, count, or generation changes, re-derive these. Copying flags from the old SKU is a usual outage.

KnobWhy it must change
--gpu-memory-utilization0.90 on 40 GB ≠ 0.90 on 80 GB; leave room for CUDA graphs / fragmentation
--max-num-seqs / max batchCeiling is VRAM − weights, not “what we used on A10G”
--max-model-lenKV grows linearly; a 128k window on a small card starves concurrency
--tensor-parallel-sizeMust divide hidden size / heads; must match GPU count on the pod
Quantisation / dtypeFP16 → FP8/AWQ changes quality and KV bytes
--kv-cache-dtypeFP8 KV is a large concurrency win with a quality check
Prefix / automatic prefix cachingHit rate changes with batch mix after you add replicas
Speculative decoding draft modelExtra VRAM; may no longer fit after shrinking SKU

Prefix cache vs replica count

More replicas (H) reduce prefix-cache hit rate because each cache is local unless you run a disaggregated KV store. Scale-out can worsen TTFT for shared-system-prompt workloads. Measure cache hit rate after H changes.

Long-context is a pool, not a flag

Raising max-model-len on every replica to 128k can cut concurrency by 8–16×. Route long jobs to a dedicated high-VRAM pool.

Multi-LoRA

Adapters are small; active adapter count and extra KV still matter. A SKU that fit one adapter may OOM at ten concurrent adapters.

Disaggregated prefill / decode

Prefill pool + decode pool is a topology resize: GPUs specialise. Worth it at large scale when prefill and decode fight for SM/bandwidth. Not a first-year move.

CUDA graphs

Graphs capture extra memory. After a V-axis upgrade, teams raise utilisation to “use the card” and OOM on the first spike. Keep 10–20% headroom.

Sizing math: 04_resources_and_sizing.md. Gotchas: 13_gotchas.md.