— Worked patterns for a prod AI app
Back to GPUResizing.md.
A — RAG chat, self-hosted 8B + API reasoning (common hybrid)
Generation: 1× L4/A10G per replica, max-model-len 8–16k, HPA on queue. Embeddings: separate 1× L4 with large batch. Rerank: same or MIG slice.
Resize path: first cap context and prefix-cache (S), then add 8B replicas (H). Do not jump to H100.
B — Self-hosted 70B for residency
Weights ~40 GB at 4-bit → 80 GB card for KV, or TP=2 on 40 GB. Vertical to 80 GB usually beats TP=2 on 40 GB for latency. Keep an 8B pool for rewrite/classify so the 70B fleet stays small.
C — Ingest OCR spike
Do not add chat GPUs. Page queue → OCR pool with scale-from-zero or scheduled scale before known backfills. If GPU util is low, CPU raster is often the bottleneck — pre-stage images on CPU workers.
D — Fine-tune weekly, serve always
Training node group (spot OK) distinct from serving (on-demand). Never “borrow” serving GPUs for an emergency train run without an explicit shed of non-interactive traffic.
E — Over-provisioned overnight
Scheduled H scale-down to minReplicas that still hold SLO + N+1. Not scale-to-zero for interactive. Cron-based scale-up 30+ minutes before known peaks (GPU provision time).
At larger scale
- Pools > SKUs > replicas. Org chart of GPUs matches workload profiles, not a single
ml-gpunode group. - N+1 across AZs, not just replica count.
- FinOps monthly:
$/1M tokensby pool, idle %, KV p95, reservation utilisation. Vertical overbuy shows up as idle VRAM, not idle SM.