Production → Self-hosted serving (vLLM/TGI, quantisation, GPU sizing)

Self-Hosted LLM Serving with vLLM Explained: When It Pays Off

What it is

Self-hosting means running open-weight models on your own GPUs instead of calling an API. The machinery that makes it viable:

ComponentWhat it doesWhy it matters
Continuous batchingNew requests join the batch as others finish, instead of waiting for a full batch5–20× throughput over naive batching
PagedAttention (vLLM)KV cache in fixed-size pages, like OS virtual memoryRemoves fragmentation; 2–4× more concurrent sequences
Prefix cachingReuses KV for shared prompt prefixes across requestsThe self-hosted analogue of prompt caching (prompt caching of static prefixes)
Quantisation (AWQ, GPTQ, FP8, INT8)Smaller weights, faster memory-bound decode2–4× memory reduction, modest quality loss
Tensor / pipeline parallelismSplits a model across GPUsRequired when weights exceed one card
Speculative decodingSmall draft model proposes, big model verifies1.5–3× decode speedup on predictable text

GPU memory budget is the sizing equation:

VRAM = weights + KV cache + activations + overhead

weights   ≈ params × bytes_per_param        (70B @ FP16 = 140 GB; @ AWQ-4bit ≈ 40 GB)
KV cache  ≈ 2 × layers × kv_heads × head_dim × bytes × context_len × concurrent_seqs
activations ≈ ~1-2 GB per concurrent batch (model dependent)

The KV cache usually dominates at production concurrency, and it scales linearly with both context length and concurrency — which is why long-context RAG prompts are expensive to self-host.

How to implement

  1. Justify it first: compute break-even against API pricing at your actual volume, including GPU cost, engineering time, and on-call. Below roughly steady multi-million tokens/day, APIs usually win.
  2. Pick the model and quantisation, then validate quality on your golden set (golden dataset first) — quantisation is a quality change and must be evaluated, not assumed.
  3. Size the GPU: compute weights + KV for your target concurrency and p95 context length; leave 10–20% headroom.
  4. Deploy vLLM (or TGI/SGLang) with continuous batching, prefix caching, and appropriate parallelism.
  5. Load-test at realistic prompt/response lengths and concurrency; measure TTFT, inter-token latency, and throughput; find the concurrency knee.
  6. Autoscale on queue depth, not GPU utilisation — utilisation stays high while latency degrades.
  7. Run the same evals as for the API path; keep an API fallback for overflow and incidents (failure handling).

Small example

# --- Sizing: does it fit, and how many concurrent requests? ---
def vram_estimate(params_b, bytes_per_param, layers, kv_heads, head_dim,
                  context_len, concurrent, kv_bytes=2):
    weights_gb = params_b * 1e9 * bytes_per_param / 1e9
    kv_per_seq_gb = 2 * layers * kv_heads * head_dim * kv_bytes * context_len / 1e9   # K and V
    return {"weights_gb": round(weights_gb, 1),
            "kv_per_seq_gb": round(kv_per_seq_gb, 3),
            "kv_total_gb": round(kv_per_seq_gb * concurrent, 1),
            "total_gb": round(weights_gb + kv_per_seq_gb * concurrent + 2, 1)}

# Llama-3.1-8B, FP16, 8k context, 64 concurrent  → ~16 GB weights + ~16 GB KV ≈ 34 GB (fits an A100-40GB)
print(vram_estimate(8, 2, layers=32, kv_heads=8, head_dim=128, context_len=8192, concurrent=64))
# Same model AWQ-4bit → ~4 GB weights, leaving far more room for concurrency
print(vram_estimate(8, 0.5, layers=32, kv_heads=8, head_dim=128, context_len=8192, concurrent=192))

def max_concurrency(vram_gb, weights_gb, kv_per_seq_gb, overhead_gb=2, headroom=0.85):
    return int(((vram_gb * headroom) - weights_gb - overhead_gb) / kv_per_seq_gb)

# --- Serving with vLLM ---
"""
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --quantization awq --dtype half \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \                # shared system prompts reuse KV (prompt caching of static prefixes analogue)
  --max-num-seqs 128 \                     # continuous batching width
  --tensor-parallel-size 1 \
  --served-model-name internal-fast
"""

# --- Break-even against API pricing ---
def breakeven(daily_input_tokens, daily_output_tokens,
              api_in_per_m, api_out_per_m, gpu_hourly, n_gpus, eng_fte_monthly=0):
    api_monthly = 30 * (daily_input_tokens * api_in_per_m + daily_output_tokens * api_out_per_m) / 1e6
    self_monthly = gpu_hourly * n_gpus * 24 * 30 + eng_fte_monthly
    return {"api_monthly_usd": round(api_monthly), "self_hosted_monthly_usd": round(self_monthly),
            "self_hosted_cheaper": self_monthly < api_monthly,
            "breakeven_daily_input_tokens": round(self_monthly / 30 * 1e6 / max(api_in_per_m, 1e-9))}

# 50M in + 5M out per day vs 2×A100 at $2/hr + 0.3 FTE
print(breakeven(50e6, 5e6, api_in_per_m=1.00, api_out_per_m=5.00, gpu_hourly=2.0, n_gpus=2, eng_fte_monthly=6000))

# --- Autoscale on queue depth, not utilisation ---
def scaling_decision(metrics, target_queue_depth=4, ttft_slo_ms=800):
    if metrics["queue_depth"] > target_queue_depth * 2 or metrics["ttft_p95_ms"] > ttft_slo_ms:
        return "scale_up"
    if metrics["queue_depth"] < 1 and metrics["gpu_util"] < 0.4 and metrics["replicas"] > 1:
        return "scale_down"
    return "hold"

Pros

  • Unit cost at high steady volume can be far below API pricing, especially for high input-token workloads (RAG).
  • Data residency and zero retention by construction — often the actual reason (data residency & retention).
  • Latency control: no provider queueing, no noisy neighbours, tunable batching for your traffic shape.
  • Model choice and fine-tuning: serve your own fine-tuned adapters (PEFT / LoRA / QLoRA) without vendor support.
  • No rate limits beyond your own capacity.

Cons

  • Operational burden: GPU capacity, driver/CUDA versions, OOM debugging, upgrades, on-call — a genuine platform team commitment.
  • Idle cost: GPUs bill continuously; bursty traffic is expensive to serve.
  • Capability gap: open-weight models trail frontier models on hard reasoning; often the right pattern is self-hosted fast tier + API reasoning tier (model selection & tradeoffs).
  • Quantisation is a quality change that must be evaluated per task (prompt evaluation).
  • Long-context RAG is KV-cache hungry, which caps concurrency and drives GPU count.
  • Scaling is slow: GPU acquisition and model load times are minutes, not seconds.

When to use

  • High, steady volume where break-even is clearly passed (typically tens of millions of tokens/day sustained).
  • Residency, air-gap, or zero-retention requirements that APIs cannot meet (data residency & retention).
  • Fast-tier workloads — classification, rewriting, extraction, reranking, embeddings — where small models are sufficient and volume is high.
  • Fine-tuned models you need to serve (PEFT / LoRA / QLoRA).

When NOT to use

  • Bursty or low volume — idle GPUs destroy the economics.
  • Frontier-capability needs, where open-weight models don't reach the quality bar.
  • Small teams without platform/GPU operations capacity.
  • Before instrumenting cost: many teams self-host to fix a cost problem that prompt caching and tiering would have solved (cost levers).

Production at scale

Hybrid is the common production shape: self-hosted small models for the fast tier (classification, rewriting, extraction, embeddings, reranking — which are high-volume and latency-critical) and an API for the reasoning tier. This captures most of the cost saving with a fraction of the operational risk.

Prefix caching is as important as it is on the API side. Enable it in vLLM and structure prompts stable-first (prompt structure, prompt caching of static prefixes); shared system prompts and tool definitions then cost almost nothing in prefill.

Autoscale on queue depth and TTFT, not GPU utilisation. Utilisation sits near 100% while latency degrades; queue depth is the leading indicator. Keep warm replicas — cold model load is minutes.

Serve embeddings and rerankers separately from generation. They have different batch profiles (large batches, short sequences) and pack far more efficiently on their own GPUs (model choice, reranking).

Evaluate quantisation per task. AWQ/GPTQ 4-bit is usually near-lossless for extraction and classification, and more noticeable on complex reasoning. Run the golden set at each quantisation level before committing (prompt evaluation).

Keep an API fallback for overflow, incidents, and capacity gaps, with a circuit breaker and a documented quality delta (failure handling).

Watch the KV cache, not just VRAM totals. Preemption and recomputation events in vLLM mean concurrency exceeds cache capacity — reduce max-num-seqs or max-model-len, or add GPUs.

Version and pin everything: model weights hash, quantisation method, serving engine version, CUDA version. A silent engine upgrade can change outputs (versioning & rollout).

Total cost of ownership includes GPUs, idle time, engineers, on-call, and the eval work — compare that against API spend honestly before committing.

Prod checklist

  • Break-even computed including GPU, idle, engineering, and on-call costs
  • Quality validated on the golden set at the chosen quantisation
  • VRAM sized from weights + KV at target concurrency and p95 context, with headroom
  • vLLM/TGI with continuous batching and prefix caching enabled
  • Load tested at realistic lengths and concurrency; concurrency knee identified
  • Autoscaling on queue depth and TTFT; warm replicas maintained
  • Embeddings/rerankers served on separate replicas from generation
  • API fallback with circuit breaker and documented quality delta
  • KV preemption/recompute events monitored
  • Weights, quantisation, engine, and CUDA versions pinned and logged

Metrics to watch

MetricWhy
TTFT and inter-token latency p95User-facing latency
Tokens/sec throughput per GPUCapacity and unit cost
KV cache utilisation; preemption/recompute eventsConcurrency ceiling
Queue depthAutoscaling signal
$/1M tokens self-hosted vs APIWhether it's still worth it
Quality delta vs API model on golden setThe hidden trade