Fine-tuning → PEFT / LoRA / QLoRA
What it is
Full fine-tuning updates every parameter — for a 70B model that means ~280 GB of optimiser state and a multi-GPU cluster. LoRA (Low-Rank Adaptation) instead freezes the base weights and learns a small pair of low-rank matrices per target layer:
frozen W (d × k) learned: A (d × r) and B (r × k), r ≈ 8–64
output = W·x + (B·A)·x · (α / r)
trainable parameters ≈ 0.1–1% of the model
QLoRA adds 4-bit quantisation of the frozen base (NF4) plus paged optimisers, so a 7B model trains in ~10 GB of VRAM and a 70B in ~48 GB — one GPU instead of a cluster, at close to LoRA quality.
Why this is the production default:
| Property | Consequence |
|---|---|
| Adapter is 10–200 MB | Cheap to store, version, and ship |
| Base weights unchanged | One base model serves many adapters |
| Multi-adapter serving | vLLM/S-LoRA swap adapters per request — one GPU, many tenants or tasks |
| Fast training | Hours on one GPU for typical datasets |
| Easy rollback | Detach the adapter |
Key hyperparameters: r (rank, 8–64; higher = more capacity and more overfitting risk), alpha (scaling, commonly 2×r), dropout (0.05–0.1), and target_modules (attention projections at minimum; adding MLP layers increases capacity and cost).
How to implement
- Start with QLoRA unless you have spare multi-GPU capacity — quality difference is small, cost difference is large.
- Target modules:
q_proj, k_proj, v_proj, o_projas a baseline; addgate_proj, up_proj, down_projwhen the task needs more capacity (measure, don't assume). - Rank: start at r=16, alpha=32. Raise to 32–64 only if the training loss plateaus above your target and validation isn't overfitting.
- Learning rate around 1e-4 to 2e-4 (higher than full fine-tuning), cosine schedule with warmup, 2–4 epochs.
- Watch validation loss per epoch; stop when it turns up — small datasets overfit within 2–3 epochs.
- Merge or serve separately: merge into base weights for a single-purpose deployment; keep separate for multi-adapter serving (the usual production choice).
- Evaluate on the held-out set and the safety suite (SFT, prompt evaluation) before shipping.
Small example
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training, PeftModel
from trl import SFTTrainer, SFTConfig
BASE = "meta-llama/Llama-3.1-8B-Instruct"
# ---- QLoRA: 4-bit frozen base + trainable adapters ----
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True)
model = AutoModelForCausalLM.from_pretrained(BASE, quantization_config=bnb, device_map="auto",
attn_implementation="flash_attention_2")
model = prepare_model_for_kbit_training(model, use_gradient_checkpointing=True)
lora = LoraConfig(
r=16, lora_alpha=32, lora_dropout=0.05, bias="none", task_type="CAUSAL_LM",
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"], # + MLP layers if capacity is short
)
model = get_peft_model(model, lora)
model.print_trainable_parameters() # e.g. "trainable: 20,971,520 || all: 8,051,232,768 || 0.26%"
trainer = SFTTrainer(
model=model, train_dataset=train_ds, eval_dataset=eval_ds,
args=SFTConfig(output_dir="out/adapter-soap-v1", num_train_epochs=3,
per_device_train_batch_size=4, gradient_accumulation_steps=4,
learning_rate=2e-4, lr_scheduler_type="cosine", warmup_ratio=0.03,
bf16=True, max_seq_length=4096, logging_steps=10,
eval_strategy="epoch", save_strategy="epoch",
load_best_model_at_end=True, metric_for_best_model="eval_loss"),
)
trainer.train()
trainer.model.save_pretrained("adapters/soap-v1") # ~40 MB, not 16 GB
# ---- Multi-adapter serving: one base, many tasks/tenants ----
"""
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--enable-lora --max-loras 8 --max-lora-rank 32 \
--lora-modules soap=adapters/soap-v1 extract=adapters/extract-v3 tenant_a=adapters/tenant-a-v2
# request-time: {"model": "soap", ...} routes to the right adapter on the same GPU
"""
def serve_with_adapter(base_model, adapter_path):
"""Single-purpose deployment: merge for slightly lower latency."""
m = PeftModel.from_pretrained(base_model, adapter_path)
return m.merge_and_unload() # base + adapter folded into one set of weights
# ---- overfitting guard on small datasets ----
def check_overfitting(log_history):
evals = [e["eval_loss"] for e in log_history if "eval_loss" in e]
if len(evals) >= 2 and evals[-1] > min(evals) * 1.05:
return {"overfitting": True, "best_epoch": evals.index(min(evals)) + 1,
"advice": "use the best checkpoint; reduce epochs or rank, or add data"}
return {"overfitting": False}
Pros
- Trains on one GPU — QLoRA puts 7–13B models within reach of a single 24 GB card and 70B on a single 80 GB card.
- Adapters are tiny: store, version, and ship dozens for the cost of one base model.
- Multi-adapter serving: one GPU serves many tasks or tenants with per-request adapter selection.
- Fast iteration: hours per training run means you can actually iterate on data.
- Trivial rollback — detach the adapter, revert to base behaviour.
- Reduced catastrophic forgetting versus full fine-tuning, since base weights are frozen.
Cons
- Slight quality gap versus full fine-tuning on hard tasks (usually small; measure on your eval).
- QLoRA adds inference-time considerations: training in 4-bit and serving in 16-bit can shift behaviour; evaluate in the serving configuration.
- Multi-adapter serving has overhead — throughput drops with many concurrent distinct adapters.
- Rank and target-module choices need tuning; too little capacity underfits, too much overfits small datasets.
- Adapter–base coupling: an adapter is valid only for the exact base version and quantisation it was trained against.
When to use
Essentially always, when fine-tuning at all. Full fine-tuning is justified only for large datasets (100k+ examples), major domain shifts, or when you're training a model you'll own outright.
When NOT to use
- When the base model is served by an API you don't control and doesn't support adapters — your options are the provider's fine-tuning service or a different base.
- For knowledge injection (when not to fine-tune, prompting vs RAG vs fine-tuning).
- Extremely latency-critical single-task serving where merged weights and a dedicated deployment are simpler.
Production at scale
Multi-adapter serving is the operational win. One base model in memory, adapters swapped per request, lets you serve per-tenant or per-task models on the same GPU (self-hosted serving). Measure throughput degradation with your realistic adapter count — it grows with the number of distinct adapters in flight, not the number registered.
Adapters are versioned artifacts bound to (base_model_id, base_revision, quantisation, adapter_version, dataset_hash). A base model upgrade invalidates every adapter — plan retraining as part of any base migration (fine-tuning ops, versioning & rollout).
Evaluate in the serving configuration. Training with a 4-bit base and serving with a 16-bit base changes behaviour; run the eval suite against the exact deployment setup (prompt evaluation).
Rank selection by measurement: r=16 handles most format/style tasks. If training loss plateaus high, raise rank or add MLP target modules; if validation loss diverges early, lower rank or add data.
Watch for early overfitting — with a few hundred examples, models overfit within 2–3 epochs. Always keep the best checkpoint by validation loss, not the last.
Per-tenant adapters are attractive for multi-tenant products but multiply the operational surface (training pipelines, storage, evals per tenant). Start with one shared adapter and split only when evals show a tenant-specific need.
Storage and registry: adapters in object storage with a manifest (base, revision, dataset hash, eval scores, training config), promoted through environments like any other artifact (environments).
Cost model: QLoRA training for a typical dataset is single-digit GPU-hours — cheap enough that data curation dominates the project cost (SFT).
Prod checklist
- QLoRA as the default; full fine-tuning justified explicitly if used
-
r,alpha,target_moduleschosen by measurement; start r=16/alpha=32 - Validation loss monitored per epoch; best checkpoint retained, not the last
- Evaluation run in the exact serving configuration (quantisation, adapter vs merged)
- Adapter bound to base model ID, revision, and quantisation in its manifest
- Multi-adapter serving throughput measured at realistic concurrency
- Adapter registry with dataset hash, training config, and eval scores
- Base-model upgrade plan includes adapter retraining
- Rollback = detach adapter, tested
Metrics to watch
| Metric | Why |
|---|---|
| Held-out quality vs base and vs full fine-tune (if measured) | Whether the adapter earns its place |
| Validation loss curve | Overfitting |
| GPU-hours per training run | Iteration cost |
| Serving throughput with N concurrent adapters | Multi-adapter overhead |
| Adapter/base version mismatches (must be 0) | Coupling discipline |
Related
- SFT — what you're training
- Preference tuning, Distillation — same adapter machinery
- Fine-tuning ops — registry and reproducibility
- Self-hosted serving — where adapters are served
- Versioning — base upgrades invalidate adapters