— Three resources and the sizing math

Back to GPUResizing.md.


A production GPU replica is limited by three coupled resources. Resize to the one that is actually binding.

ResourceDominated byFailure modeUser-visible effect
VRAMWeights + KV + activations + fragmentationOOM, KV preemption, refused seqs503, truncated context, recompute spikes
Memory bandwidthDecode (token-by-token), KV readTokens/s ceilingSlow streaming, high TPOT
Compute (FLOPS)Prefill, embeddings, rerank, OCR, trainingQueue of prefill-heavy jobsHigh TTFT, ingest backlog

Buying H100s to fix a KV problem caused by 32k RAG prompts is often worse than capping context, enabling prefix cache, or splitting long jobs to a dedicated pool. Buying more T4s to fix a 70B model that does not fit is wasted — you need V or W.

VRAM equation

VRAM ≈ weights + KV cache + activations + overhead

weights   ≈ params × bytes_per_param
            70B @ FP16 = 140 GB; 70B @ AWQ-4bit ≈ 40 GB
            8B @ FP16 ≈ 16 GB; 8B @ 4-bit ≈ 4 GB

KV cache  ≈ 2 × layers × kv_heads × head_dim × bytes × context_len × concurrent_seqs
            (K and V; bytes = 2 for FP16 KV, 1 for FP8 KV)

activations ≈ ~1–2 GB per concurrent batch (model dependent)
overhead    = CUDA graphs, allocator fragmentation, engine workspace
headroom    = leave 10–20% (do not run gpu-memory-utilization at 0.98)

At production concurrency, KV usually dominates. It scales linearly with both context length and concurrent sequences — which is why long-context RAG is expensive to self-host.

GQA / MLA: KV size uses kv_heads, not query heads. A 70B with GQA is cheaper to serve than the parameter count suggests. Do not size KV from param count alone.

Concurrency from leftover VRAM

usable = VRAM × headroom − weights − overhead
max_seqs ≈ usable / kv_per_sequence(context)

If max_seqs at your p95 context is below the SLO concurrency, you are VRAM/KV-bound: S (shorter ctx, quant, FP8 KV) or V (more VRAM), not more tiny replicas of the same SKU.

If max_seqs is fine but tokens/s per GPU cannot clear the queue: H or a newer-gen V.

Throughput to replica count

replicas ≈ ceil(peak_output_tokens_per_s / measured_tokens_per_s_per_gpu) + N+1

Measure tokens/s on the real prompt-length histogram, not 128-token synthetic prompts. Prefill-heavy RAG traffic has a different knee than short chat.

Worked functions: 11_examples.md.