— Workload pools: never resize one fleet

Back to GPUResizing.md.


A prod AI application is several GPU shapes, not one. Resize per pool.

PoolProfileBinding resourceResize instinct
Generation (LLM)Small batch, long KV, latency SLOVRAM (KV) + bandwidthH on queue/TTFT; V if KV-bound; W if model > 1 GPU
EmbeddingsHuge batch, short seq, throughputFLOPS + host I/OH on queue depth; smaller/cheaper GPUs often win
Rerankers / NLIMedium batch, short seqFLOPSSame as embeddings; do not co-locate with generation
OCR / layout / VLM ingestPage-parallel, bursty, minutes-scaleFLOPS + CPU raster bottleneckSeparate queue; autoscale on page queue, not chat QPS
Training / LoRA jobsOccupies a card for hoursAll of VRAM, often multi-GPUIsolated node group; never share with interactive serving

Why co-location fails

A generation replica with a full KV cache cannot absorb a 50k-chunk embed job. An OCR backfill will raise TTFT for interactive chat even if “average GPU util” looks fine. Training nvidia.com/gpu: 8 on the serving node group evicts or starves chat.

Shared node groups with time-slicing are for dev, not interactive prod.

Practical default for a RAG app

  • L4 / A10G class: embeddings, rerank, 8B fast tier.
  • A100 / H100 class: only the generation model that actually needs the VRAM/bandwidth.
  • Spot / scale-toward-zero: OCR, embed backfill, training — not user-facing decode.

See patterns: 12_worked_patterns.md.