Fine-tuning → PEFT / LoRA / QLoRA

LoRA vs QLoRA Fine-Tuning Explained: Tiny Adapters, One Base

What it is

Full fine-tuning updates every parameter — for a 70B model that means ~280 GB of optimiser state and a multi-GPU cluster. LoRA (Low-Rank Adaptation) instead freezes the base weights and learns a small pair of low-rank matrices per target layer:

frozen W  (d × k)          learned:  A (d × r)  and  B (r × k),  r ≈ 8–64
output = W·x + (B·A)·x · (α / r)

trainable parameters ≈ 0.1–1% of the model

QLoRA adds 4-bit quantisation of the frozen base (NF4) plus paged optimisers, so a 7B model trains in ~10 GB of VRAM and a 70B in ~48 GB — one GPU instead of a cluster, at close to LoRA quality.

Why this is the production default:

PropertyConsequence
Adapter is 10–200 MBCheap to store, version, and ship
Base weights unchangedOne base model serves many adapters
Multi-adapter servingvLLM/S-LoRA swap adapters per request — one GPU, many tenants or tasks
Fast trainingHours on one GPU for typical datasets
Easy rollbackDetach the adapter

Key hyperparameters: r (rank, 8–64; higher = more capacity and more overfitting risk), alpha (scaling, commonly 2×r), dropout (0.05–0.1), and target_modules (attention projections at minimum; adding MLP layers increases capacity and cost).

How to implement

  1. Start with QLoRA unless you have spare multi-GPU capacity — quality difference is small, cost difference is large.
  2. Target modules: q_proj, k_proj, v_proj, o_proj as a baseline; add gate_proj, up_proj, down_proj when the task needs more capacity (measure, don't assume).
  3. Rank: start at r=16, alpha=32. Raise to 32–64 only if the training loss plateaus above your target and validation isn't overfitting.
  4. Learning rate around 1e-4 to 2e-4 (higher than full fine-tuning), cosine schedule with warmup, 2–4 epochs.
  5. Watch validation loss per epoch; stop when it turns up — small datasets overfit within 2–3 epochs.
  6. Merge or serve separately: merge into base weights for a single-purpose deployment; keep separate for multi-adapter serving (the usual production choice).
  7. Evaluate on the held-out set and the safety suite (SFT, prompt evaluation) before shipping.

Small example

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training, PeftModel
from trl import SFTTrainer, SFTConfig

BASE = "meta-llama/Llama-3.1-8B-Instruct"

# ---- QLoRA: 4-bit frozen base + trainable adapters ----
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_compute_dtype=torch.bfloat16,
                         bnb_4bit_use_double_quant=True)

model = AutoModelForCausalLM.from_pretrained(BASE, quantization_config=bnb, device_map="auto",
                                             attn_implementation="flash_attention_2")
model = prepare_model_for_kbit_training(model, use_gradient_checkpointing=True)

lora = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05, bias="none", task_type="CAUSAL_LM",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],          # + MLP layers if capacity is short
)
model = get_peft_model(model, lora)
model.print_trainable_parameters()          # e.g. "trainable: 20,971,520 || all: 8,051,232,768 || 0.26%"

trainer = SFTTrainer(
    model=model, train_dataset=train_ds, eval_dataset=eval_ds,
    args=SFTConfig(output_dir="out/adapter-soap-v1", num_train_epochs=3,
                   per_device_train_batch_size=4, gradient_accumulation_steps=4,
                   learning_rate=2e-4, lr_scheduler_type="cosine", warmup_ratio=0.03,
                   bf16=True, max_seq_length=4096, logging_steps=10,
                   eval_strategy="epoch", save_strategy="epoch",
                   load_best_model_at_end=True, metric_for_best_model="eval_loss"),
)
trainer.train()
trainer.model.save_pretrained("adapters/soap-v1")     # ~40 MB, not 16 GB

# ---- Multi-adapter serving: one base, many tasks/tenants ----
"""
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-lora --max-loras 8 --max-lora-rank 32 \
  --lora-modules soap=adapters/soap-v1 extract=adapters/extract-v3 tenant_a=adapters/tenant-a-v2
# request-time: {"model": "soap", ...} routes to the right adapter on the same GPU
"""

def serve_with_adapter(base_model, adapter_path):
    """Single-purpose deployment: merge for slightly lower latency."""
    m = PeftModel.from_pretrained(base_model, adapter_path)
    return m.merge_and_unload()                        # base + adapter folded into one set of weights

# ---- overfitting guard on small datasets ----
def check_overfitting(log_history):
    evals = [e["eval_loss"] for e in log_history if "eval_loss" in e]
    if len(evals) >= 2 and evals[-1] > min(evals) * 1.05:
        return {"overfitting": True, "best_epoch": evals.index(min(evals)) + 1,
                "advice": "use the best checkpoint; reduce epochs or rank, or add data"}
    return {"overfitting": False}

Pros

  • Trains on one GPU — QLoRA puts 7–13B models within reach of a single 24 GB card and 70B on a single 80 GB card.
  • Adapters are tiny: store, version, and ship dozens for the cost of one base model.
  • Multi-adapter serving: one GPU serves many tasks or tenants with per-request adapter selection.
  • Fast iteration: hours per training run means you can actually iterate on data.
  • Trivial rollback — detach the adapter, revert to base behaviour.
  • Reduced catastrophic forgetting versus full fine-tuning, since base weights are frozen.

Cons

  • Slight quality gap versus full fine-tuning on hard tasks (usually small; measure on your eval).
  • QLoRA adds inference-time considerations: training in 4-bit and serving in 16-bit can shift behaviour; evaluate in the serving configuration.
  • Multi-adapter serving has overhead — throughput drops with many concurrent distinct adapters.
  • Rank and target-module choices need tuning; too little capacity underfits, too much overfits small datasets.
  • Adapter–base coupling: an adapter is valid only for the exact base version and quantisation it was trained against.

When to use

Essentially always, when fine-tuning at all. Full fine-tuning is justified only for large datasets (100k+ examples), major domain shifts, or when you're training a model you'll own outright.

When NOT to use

  • When the base model is served by an API you don't control and doesn't support adapters — your options are the provider's fine-tuning service or a different base.
  • For knowledge injection (when not to fine-tune, prompting vs RAG vs fine-tuning).
  • Extremely latency-critical single-task serving where merged weights and a dedicated deployment are simpler.

Production at scale

Multi-adapter serving is the operational win. One base model in memory, adapters swapped per request, lets you serve per-tenant or per-task models on the same GPU (self-hosted serving). Measure throughput degradation with your realistic adapter count — it grows with the number of distinct adapters in flight, not the number registered.

Adapters are versioned artifacts bound to (base_model_id, base_revision, quantisation, adapter_version, dataset_hash). A base model upgrade invalidates every adapter — plan retraining as part of any base migration (fine-tuning ops, versioning & rollout).

Evaluate in the serving configuration. Training with a 4-bit base and serving with a 16-bit base changes behaviour; run the eval suite against the exact deployment setup (prompt evaluation).

Rank selection by measurement: r=16 handles most format/style tasks. If training loss plateaus high, raise rank or add MLP target modules; if validation loss diverges early, lower rank or add data.

Watch for early overfitting — with a few hundred examples, models overfit within 2–3 epochs. Always keep the best checkpoint by validation loss, not the last.

Per-tenant adapters are attractive for multi-tenant products but multiply the operational surface (training pipelines, storage, evals per tenant). Start with one shared adapter and split only when evals show a tenant-specific need.

Storage and registry: adapters in object storage with a manifest (base, revision, dataset hash, eval scores, training config), promoted through environments like any other artifact (environments).

Cost model: QLoRA training for a typical dataset is single-digit GPU-hours — cheap enough that data curation dominates the project cost (SFT).

Prod checklist

  • QLoRA as the default; full fine-tuning justified explicitly if used
  • r, alpha, target_modules chosen by measurement; start r=16/alpha=32
  • Validation loss monitored per epoch; best checkpoint retained, not the last
  • Evaluation run in the exact serving configuration (quantisation, adapter vs merged)
  • Adapter bound to base model ID, revision, and quantisation in its manifest
  • Multi-adapter serving throughput measured at realistic concurrency
  • Adapter registry with dataset hash, training config, and eval scores
  • Base-model upgrade plan includes adapter retraining
  • Rollback = detach adapter, tested

Metrics to watch

MetricWhy
Held-out quality vs base and vs full fine-tune (if measured)Whether the adapter earns its place
Validation loss curveOverfitting
GPU-hours per training runIteration cost
Serving throughput with N concurrent adaptersMulti-adapter overhead
Adapter/base version mismatches (must be 0)Coupling discipline