Production → Self-hosted serving (vLLM/TGI, quantisation, GPU sizing)
What it is
Self-hosting means running open-weight models on your own GPUs instead of calling an API. The machinery that makes it viable:
| Component | What it does | Why it matters |
|---|---|---|
| Continuous batching | New requests join the batch as others finish, instead of waiting for a full batch | 5–20× throughput over naive batching |
| PagedAttention (vLLM) | KV cache in fixed-size pages, like OS virtual memory | Removes fragmentation; 2–4× more concurrent sequences |
| Prefix caching | Reuses KV for shared prompt prefixes across requests | The self-hosted analogue of prompt caching (prompt caching of static prefixes) |
| Quantisation (AWQ, GPTQ, FP8, INT8) | Smaller weights, faster memory-bound decode | 2–4× memory reduction, modest quality loss |
| Tensor / pipeline parallelism | Splits a model across GPUs | Required when weights exceed one card |
| Speculative decoding | Small draft model proposes, big model verifies | 1.5–3× decode speedup on predictable text |
GPU memory budget is the sizing equation:
VRAM = weights + KV cache + activations + overhead
weights ≈ params × bytes_per_param (70B @ FP16 = 140 GB; @ AWQ-4bit ≈ 40 GB)
KV cache ≈ 2 × layers × kv_heads × head_dim × bytes × context_len × concurrent_seqs
activations ≈ ~1-2 GB per concurrent batch (model dependent)
The KV cache usually dominates at production concurrency, and it scales linearly with both context length and concurrency — which is why long-context RAG prompts are expensive to self-host.
How to implement
- Justify it first: compute break-even against API pricing at your actual volume, including GPU cost, engineering time, and on-call. Below roughly steady multi-million tokens/day, APIs usually win.
- Pick the model and quantisation, then validate quality on your golden set (golden dataset first) — quantisation is a quality change and must be evaluated, not assumed.
- Size the GPU: compute weights + KV for your target concurrency and p95 context length; leave 10–20% headroom.
- Deploy vLLM (or TGI/SGLang) with continuous batching, prefix caching, and appropriate parallelism.
- Load-test at realistic prompt/response lengths and concurrency; measure TTFT, inter-token latency, and throughput; find the concurrency knee.
- Autoscale on queue depth, not GPU utilisation — utilisation stays high while latency degrades.
- Run the same evals as for the API path; keep an API fallback for overflow and incidents (failure handling).
Small example
# --- Sizing: does it fit, and how many concurrent requests? ---
def vram_estimate(params_b, bytes_per_param, layers, kv_heads, head_dim,
context_len, concurrent, kv_bytes=2):
weights_gb = params_b * 1e9 * bytes_per_param / 1e9
kv_per_seq_gb = 2 * layers * kv_heads * head_dim * kv_bytes * context_len / 1e9 # K and V
return {"weights_gb": round(weights_gb, 1),
"kv_per_seq_gb": round(kv_per_seq_gb, 3),
"kv_total_gb": round(kv_per_seq_gb * concurrent, 1),
"total_gb": round(weights_gb + kv_per_seq_gb * concurrent + 2, 1)}
# Llama-3.1-8B, FP16, 8k context, 64 concurrent → ~16 GB weights + ~16 GB KV ≈ 34 GB (fits an A100-40GB)
print(vram_estimate(8, 2, layers=32, kv_heads=8, head_dim=128, context_len=8192, concurrent=64))
# Same model AWQ-4bit → ~4 GB weights, leaving far more room for concurrency
print(vram_estimate(8, 0.5, layers=32, kv_heads=8, head_dim=128, context_len=8192, concurrent=192))
def max_concurrency(vram_gb, weights_gb, kv_per_seq_gb, overhead_gb=2, headroom=0.85):
return int(((vram_gb * headroom) - weights_gb - overhead_gb) / kv_per_seq_gb)
# --- Serving with vLLM ---
"""
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--quantization awq --dtype half \
--max-model-len 8192 \
--gpu-memory-utilization 0.90 \
--enable-prefix-caching \ # shared system prompts reuse KV (prompt caching of static prefixes analogue)
--max-num-seqs 128 \ # continuous batching width
--tensor-parallel-size 1 \
--served-model-name internal-fast
"""
# --- Break-even against API pricing ---
def breakeven(daily_input_tokens, daily_output_tokens,
api_in_per_m, api_out_per_m, gpu_hourly, n_gpus, eng_fte_monthly=0):
api_monthly = 30 * (daily_input_tokens * api_in_per_m + daily_output_tokens * api_out_per_m) / 1e6
self_monthly = gpu_hourly * n_gpus * 24 * 30 + eng_fte_monthly
return {"api_monthly_usd": round(api_monthly), "self_hosted_monthly_usd": round(self_monthly),
"self_hosted_cheaper": self_monthly < api_monthly,
"breakeven_daily_input_tokens": round(self_monthly / 30 * 1e6 / max(api_in_per_m, 1e-9))}
# 50M in + 5M out per day vs 2×A100 at $2/hr + 0.3 FTE
print(breakeven(50e6, 5e6, api_in_per_m=1.00, api_out_per_m=5.00, gpu_hourly=2.0, n_gpus=2, eng_fte_monthly=6000))
# --- Autoscale on queue depth, not utilisation ---
def scaling_decision(metrics, target_queue_depth=4, ttft_slo_ms=800):
if metrics["queue_depth"] > target_queue_depth * 2 or metrics["ttft_p95_ms"] > ttft_slo_ms:
return "scale_up"
if metrics["queue_depth"] < 1 and metrics["gpu_util"] < 0.4 and metrics["replicas"] > 1:
return "scale_down"
return "hold"
Pros
- Unit cost at high steady volume can be far below API pricing, especially for high input-token workloads (RAG).
- Data residency and zero retention by construction — often the actual reason (data residency & retention).
- Latency control: no provider queueing, no noisy neighbours, tunable batching for your traffic shape.
- Model choice and fine-tuning: serve your own fine-tuned adapters (PEFT / LoRA / QLoRA) without vendor support.
- No rate limits beyond your own capacity.
Cons
- Operational burden: GPU capacity, driver/CUDA versions, OOM debugging, upgrades, on-call — a genuine platform team commitment.
- Idle cost: GPUs bill continuously; bursty traffic is expensive to serve.
- Capability gap: open-weight models trail frontier models on hard reasoning; often the right pattern is self-hosted fast tier + API reasoning tier (model selection & tradeoffs).
- Quantisation is a quality change that must be evaluated per task (prompt evaluation).
- Long-context RAG is KV-cache hungry, which caps concurrency and drives GPU count.
- Scaling is slow: GPU acquisition and model load times are minutes, not seconds.
When to use
- High, steady volume where break-even is clearly passed (typically tens of millions of tokens/day sustained).
- Residency, air-gap, or zero-retention requirements that APIs cannot meet (data residency & retention).
- Fast-tier workloads — classification, rewriting, extraction, reranking, embeddings — where small models are sufficient and volume is high.
- Fine-tuned models you need to serve (PEFT / LoRA / QLoRA).
When NOT to use
- Bursty or low volume — idle GPUs destroy the economics.
- Frontier-capability needs, where open-weight models don't reach the quality bar.
- Small teams without platform/GPU operations capacity.
- Before instrumenting cost: many teams self-host to fix a cost problem that prompt caching and tiering would have solved (cost levers).
Production at scale
Hybrid is the common production shape: self-hosted small models for the fast tier (classification, rewriting, extraction, embeddings, reranking — which are high-volume and latency-critical) and an API for the reasoning tier. This captures most of the cost saving with a fraction of the operational risk.
Prefix caching is as important as it is on the API side. Enable it in vLLM and structure prompts stable-first (prompt structure, prompt caching of static prefixes); shared system prompts and tool definitions then cost almost nothing in prefill.
Autoscale on queue depth and TTFT, not GPU utilisation. Utilisation sits near 100% while latency degrades; queue depth is the leading indicator. Keep warm replicas — cold model load is minutes.
Serve embeddings and rerankers separately from generation. They have different batch profiles (large batches, short sequences) and pack far more efficiently on their own GPUs (model choice, reranking).
Evaluate quantisation per task. AWQ/GPTQ 4-bit is usually near-lossless for extraction and classification, and more noticeable on complex reasoning. Run the golden set at each quantisation level before committing (prompt evaluation).
Keep an API fallback for overflow, incidents, and capacity gaps, with a circuit breaker and a documented quality delta (failure handling).
Watch the KV cache, not just VRAM totals. Preemption and recomputation events in vLLM mean concurrency exceeds cache capacity — reduce max-num-seqs or max-model-len, or add GPUs.
Version and pin everything: model weights hash, quantisation method, serving engine version, CUDA version. A silent engine upgrade can change outputs (versioning & rollout).
Total cost of ownership includes GPUs, idle time, engineers, on-call, and the eval work — compare that against API spend honestly before committing.
Prod checklist
- Break-even computed including GPU, idle, engineering, and on-call costs
- Quality validated on the golden set at the chosen quantisation
- VRAM sized from weights + KV at target concurrency and p95 context, with headroom
- vLLM/TGI with continuous batching and prefix caching enabled
- Load tested at realistic lengths and concurrency; concurrency knee identified
- Autoscaling on queue depth and TTFT; warm replicas maintained
- Embeddings/rerankers served on separate replicas from generation
- API fallback with circuit breaker and documented quality delta
- KV preemption/recompute events monitored
- Weights, quantisation, engine, and CUDA versions pinned and logged
Metrics to watch
| Metric | Why |
|---|---|
| TTFT and inter-token latency p95 | User-facing latency |
| Tokens/sec throughput per GPU | Capacity and unit cost |
| KV cache utilisation; preemption/recompute events | Concurrency ceiling |
| Queue depth | Autoscaling signal |
| $/1M tokens self-hosted vs API | Whether it's still worth it |
| Quality delta vs API model on golden set | The hidden trade |
Related
- Cost levers — try caching and tiering before self-hosting
- Model selection — hybrid tiering
- Prompt caching — prefix caching is the same idea
- Embeddings, Reranking — the best self-host candidates
- Data residency — often the real driver
- Failure handling — API fallback