Fine-tuning → Distillation
What it is
Distillation transfers behaviour from a large "teacher" model to a small "student":
teacher (frontier model) ──► generates outputs on YOUR inputs ──► filtered/verified
│
SFT/LoRA on the student
│
small model, ~teacher quality on THIS task,
5-50× cheaper, 3-10× faster
Two flavours:
| Type | Mechanism | Requirement |
|---|---|---|
| Response distillation (black-box) | Train the student on the teacher's outputs | Only API access to the teacher — the practical default |
| Logit distillation (white-box) | Match the teacher's full output distribution (KL divergence) | Access to teacher logits; only for open-weight teachers, but more sample-efficient |
The economics are the point. A frontier model doing classification at scale costs 10–50× what a fine-tuned 8B model costs, at similar accuracy on that specific task. In a pipeline with many high-volume narrow steps — routing, extraction, rewriting, grading, reranking-lite (model selection & tradeoffs) — distillation is how the fast tier gets good enough to use.
The essential discipline: the student inherits the teacher's mistakes, so unfiltered teacher output is not a dataset. Verification is what separates a working distillation from an amplified-error one.
How to implement
- Pick a narrow, stable task with high volume — the same criteria as SFT (SFT), plus volume that justifies the effort.
- Collect real inputs from production traffic (online eval), covering the actual distribution including edge cases, not synthetic uniform samples.
- Generate teacher outputs with your best prompt, at temperature 0, using the batch API (half price, no latency requirement).
- Verify and filter — the critical step: deterministic checks (schema, citations, format), self-consistency (sample the teacher n times and keep only agreeing outputs, self-consistency), and human review of a sample. Discard anything that fails.
- Train the student with LoRA/QLoRA on the filtered pairs (PEFT / LoRA / QLoRA).
- Evaluate against the teacher on held-out cases: accuracy delta, cost delta, latency delta.
- Deploy with escalation: student handles the volume, escalating to the teacher on low confidence (routing & escalation) — this recovers most of the remaining quality gap.
Small example
import asyncio, json
from collections import Counter
# ---- 1. Generate teacher outputs on REAL production inputs, verified ----
async def generate_teacher_data(inputs: list[str], teacher="claude-opus-5", n_samples=3):
"""Self-consistency filtering: keep only what the teacher agrees with itself on (self-consistency)."""
dataset, stats = [], Counter()
for chunk in batched(inputs, 50):
results = await asyncio.gather(*[
asyncio.gather(*[teacher_call(teacher, x, temperature=0.3) for _ in range(n_samples)])
for x in chunk])
for x, samples in zip(chunk, results):
parsed = [safe_parse(s) for s in samples]
valid = [p for p in parsed if p is not None and validates_schema(p)]
if len(valid) < n_samples:
stats["invalid_output"] += 1; continue
counts = Counter(canonical(p) for p in valid)
top, votes = counts.most_common(1)[0]
if votes < n_samples: # teacher disagreed with itself → drop
stats["inconsistent"] += 1; continue
dataset.append({"input": x, "output": top, "teacher_agreement": votes / n_samples})
stats["kept"] += 1
return dataset, dict(stats)
def verify_dataset(dataset, human_sample_frac=0.02):
"""Deterministic checks on everything, human review on a sample."""
verified, rejected = [], Counter()
for ex in dataset:
if not validates_schema(ex["output"]): rejected["schema"] += 1; continue
if not references_valid(ex["input"], ex["output"]): rejected["bad_reference"] += 1; continue
verified.append(ex)
sample = random.sample(verified, max(int(len(verified) * human_sample_frac), 20))
review = human_review(sample) # the accuracy ceiling of your student
return {"dataset": verified, "rejected": dict(rejected),
"human_verified_accuracy": review["accuracy"],
"gate": review["accuracy"] >= 0.95} # don't distil from a flawed teacher
# ---- 2. Train the student (PEFT / LoRA / QLoRA) ----
"""
SFTTrainer(model=qlora_student, train_dataset=teacher_ds, peft_config=lora,
args=SFTConfig(num_train_epochs=3, learning_rate=2e-4, ...)).train()
"""
# ---- 3. Compare student against teacher on the axes that matter ----
def compare_student_teacher(cases, student, teacher):
s = [student(c.input) for c in cases]; t = [teacher(c.input) for c in cases]
agree = sum(canonical(a) == canonical(b) for a, b in zip(s, t)) / len(cases)
return {
"agreement_with_teacher": agree,
"accuracy_vs_ground_truth": {"student": accuracy(s, cases), "teacher": accuracy(t, cases)},
"cost_per_1k": {"student": cost_1k(STUDENT_MODEL), "teacher": cost_1k(TEACHER_MODEL)},
"p95_latency_ms": {"student": p95_latency(student, cases), "teacher": p95_latency(teacher, cases)},
"verdict": "deploy" if accuracy(s, cases) >= accuracy(t, cases) - 0.02 else "escalation_required",
}
# ---- 4. Deploy with escalation: recovers most of the residual gap (routing & escalation) ----
def distilled_with_escalation(x, student, teacher, confidence_floor=0.85):
out = student(x, return_confidence=True)
if out["confidence"] >= confidence_floor and validates_schema(out["result"]):
log_tier("student"); return out["result"]
log_tier("escalated_to_teacher")
return teacher(x) # ~5-15% of traffic; blended cost stays near the student's
Pros
- Large cost and latency reductions — typically 5–50× cheaper and several times faster on narrow tasks.
- Enables self-hosting of the fast tier (self-hosted serving), with residency and rate-limit benefits.
- Teacher data is cheap to produce with batch APIs and requires no human labelling for the bulk of it.
- Escalation recovers the quality gap: student for volume, teacher for the hard cases (routing & escalation).
- Compounds with the flywheel: as more production inputs accumulate, the student improves (data flywheel).
Cons
- The student inherits the teacher's errors — and amplifies them, since it has no capacity to hedge. Unfiltered teacher output is the classic mistake.
- Narrow by construction: excellent on the distilled task, worse than the base model at anything else.
- Teacher terms of service may restrict using outputs to train competing models — check them.
- Distribution shift: a student trained on last quarter's inputs degrades as usage changes; needs retraining.
- Extra pipeline: generation, verification, training, evaluation, serving — all to maintain (fine-tuning ops).
When to use
- High-volume, narrow, stable tasks: classification, routing, extraction, query rewriting, grading, simple summarisation.
- When the fast tier's quality is the blocker to tiering (model selection & tradeoffs) — this is the fix.
- When you need self-hosted inference for cost or residency and open-weight quality falls short.
When NOT to use
- Open-ended, high-variance tasks — final answer generation, complex reasoning, planning. The student will be confidently worse.
- Low-volume tasks where the pipeline costs more than the API bill it saves.
- When the teacher itself isn't reliably correct on the task — verify before distilling (a 90%-accurate teacher yields at best a 90%-accurate student).
- When the teacher's terms prohibit it.
Production at scale
Verification is the whole project. Self-consistency filtering (drop anything the teacher doesn't agree with itself on across n samples) plus deterministic schema/reference checks plus a human-reviewed sample. Teams that skip this ship a small model that reproduces the teacher's failure modes with more confidence and no hedging.
Use real production inputs, sampled to match the live distribution including the long tail. A student trained on clean, uniform synthetic inputs fails on the messy ones that dominate production.
Batch API for generation — half price, no latency constraint, and you can generate hundreds of thousands of examples cheaply (cost levers).
Deploy with escalation, not as a replacement. Student handles 85–95% of traffic; low confidence or schema failure escalates to the teacher (routing & escalation). Blended cost stays near the student's while quality stays near the teacher's, and the escalation rate is a live quality metric.
Monitor drift: student–teacher agreement on a sampled slice of live traffic, weekly. Falling agreement means the input distribution moved and it's time to regenerate and retrain (data flywheel).
Legal check on teacher terms before starting — some providers restrict using outputs to train models, particularly competing ones. This has stopped projects late.
Combine with the flywheel: production inputs → teacher outputs → verification → retraining on a cadence. Each cycle narrows the gap on the current distribution.
Cost accounting must include the pipeline: generation, verification, training, and serving infrastructure. For genuinely high volume the payback is fast; for moderate volume it may never arrive.
Prod checklist
- Task is narrow, stable, and high-volume; teacher accuracy verified on it first
- Teacher terms of service reviewed for training restrictions
- Real production inputs sampled to match the live distribution, including the tail
- Teacher outputs generated via batch API at low temperature
- Self-consistency filtering (n samples must agree) plus deterministic checks
- Human review of a sample; accuracy gate before training
- Student trained with LoRA/QLoRA; evaluated against teacher and ground truth
- Deployed with confidence-based escalation to the teacher
- Weekly student–teacher agreement monitoring on live traffic
- Retraining cadence tied to drift, with the full pipeline cost accounted
Metrics to watch
| Metric | Why |
|---|---|
| Student accuracy vs teacher on held-out cases | The quality gap |
| Escalation rate | Residual gap and blended cost |
| Student–teacher agreement on live traffic | Distribution drift |
| Blended $/request and p95 latency vs teacher-only | The payoff |
| Teacher-data rejection rate during verification | Teacher reliability |
Related
- SFT, LoRA — the training mechanism
- Model selection, Escalation — where the student is deployed
- Self-consistency — the filtering method
- Cost levers, Self-hosted serving — the payoff
- Data flywheel — the retraining loop