Fine-tuning → Distillation

LLM Distillation Explained: Cheap Student, Verified Teacher

What it is

Distillation transfers behaviour from a large "teacher" model to a small "student":

teacher (frontier model) ──► generates outputs on YOUR inputs ──► filtered/verified
                                                                        │
                                                            SFT/LoRA on the student
                                                                        │
                                                 small model, ~teacher quality on THIS task,
                                                 5-50× cheaper, 3-10× faster

Two flavours:

TypeMechanismRequirement
Response distillation (black-box)Train the student on the teacher's outputsOnly API access to the teacher — the practical default
Logit distillation (white-box)Match the teacher's full output distribution (KL divergence)Access to teacher logits; only for open-weight teachers, but more sample-efficient

The economics are the point. A frontier model doing classification at scale costs 10–50× what a fine-tuned 8B model costs, at similar accuracy on that specific task. In a pipeline with many high-volume narrow steps — routing, extraction, rewriting, grading, reranking-lite (model selection & tradeoffs) — distillation is how the fast tier gets good enough to use.

The essential discipline: the student inherits the teacher's mistakes, so unfiltered teacher output is not a dataset. Verification is what separates a working distillation from an amplified-error one.

How to implement

  1. Pick a narrow, stable task with high volume — the same criteria as SFT (SFT), plus volume that justifies the effort.
  2. Collect real inputs from production traffic (online eval), covering the actual distribution including edge cases, not synthetic uniform samples.
  3. Generate teacher outputs with your best prompt, at temperature 0, using the batch API (half price, no latency requirement).
  4. Verify and filter — the critical step: deterministic checks (schema, citations, format), self-consistency (sample the teacher n times and keep only agreeing outputs, self-consistency), and human review of a sample. Discard anything that fails.
  5. Train the student with LoRA/QLoRA on the filtered pairs (PEFT / LoRA / QLoRA).
  6. Evaluate against the teacher on held-out cases: accuracy delta, cost delta, latency delta.
  7. Deploy with escalation: student handles the volume, escalating to the teacher on low confidence (routing & escalation) — this recovers most of the remaining quality gap.

Small example

import asyncio, json
from collections import Counter

# ---- 1. Generate teacher outputs on REAL production inputs, verified ----
async def generate_teacher_data(inputs: list[str], teacher="claude-opus-5", n_samples=3):
    """Self-consistency filtering: keep only what the teacher agrees with itself on (self-consistency)."""
    dataset, stats = [], Counter()
    for chunk in batched(inputs, 50):
        results = await asyncio.gather(*[
            asyncio.gather(*[teacher_call(teacher, x, temperature=0.3) for _ in range(n_samples)])
            for x in chunk])
        for x, samples in zip(chunk, results):
            parsed = [safe_parse(s) for s in samples]
            valid = [p for p in parsed if p is not None and validates_schema(p)]
            if len(valid) < n_samples:
                stats["invalid_output"] += 1; continue
            counts = Counter(canonical(p) for p in valid)
            top, votes = counts.most_common(1)[0]
            if votes < n_samples:                      # teacher disagreed with itself → drop
                stats["inconsistent"] += 1; continue
            dataset.append({"input": x, "output": top, "teacher_agreement": votes / n_samples})
            stats["kept"] += 1
    return dataset, dict(stats)

def verify_dataset(dataset, human_sample_frac=0.02):
    """Deterministic checks on everything, human review on a sample."""
    verified, rejected = [], Counter()
    for ex in dataset:
        if not validates_schema(ex["output"]):     rejected["schema"] += 1; continue
        if not references_valid(ex["input"], ex["output"]):  rejected["bad_reference"] += 1; continue
        verified.append(ex)
    sample = random.sample(verified, max(int(len(verified) * human_sample_frac), 20))
    review = human_review(sample)                                   # the accuracy ceiling of your student
    return {"dataset": verified, "rejected": dict(rejected),
            "human_verified_accuracy": review["accuracy"],
            "gate": review["accuracy"] >= 0.95}                     # don't distil from a flawed teacher

# ---- 2. Train the student (PEFT / LoRA / QLoRA) ----
"""
SFTTrainer(model=qlora_student, train_dataset=teacher_ds, peft_config=lora,
           args=SFTConfig(num_train_epochs=3, learning_rate=2e-4, ...)).train()
"""

# ---- 3. Compare student against teacher on the axes that matter ----
def compare_student_teacher(cases, student, teacher):
    s = [student(c.input) for c in cases]; t = [teacher(c.input) for c in cases]
    agree = sum(canonical(a) == canonical(b) for a, b in zip(s, t)) / len(cases)
    return {
        "agreement_with_teacher": agree,
        "accuracy_vs_ground_truth": {"student": accuracy(s, cases), "teacher": accuracy(t, cases)},
        "cost_per_1k": {"student": cost_1k(STUDENT_MODEL), "teacher": cost_1k(TEACHER_MODEL)},
        "p95_latency_ms": {"student": p95_latency(student, cases), "teacher": p95_latency(teacher, cases)},
        "verdict": "deploy" if accuracy(s, cases) >= accuracy(t, cases) - 0.02 else "escalation_required",
    }

# ---- 4. Deploy with escalation: recovers most of the residual gap (routing & escalation) ----
def distilled_with_escalation(x, student, teacher, confidence_floor=0.85):
    out = student(x, return_confidence=True)
    if out["confidence"] >= confidence_floor and validates_schema(out["result"]):
        log_tier("student"); return out["result"]
    log_tier("escalated_to_teacher")
    return teacher(x)                                   # ~5-15% of traffic; blended cost stays near the student's

Pros

  • Large cost and latency reductions — typically 5–50× cheaper and several times faster on narrow tasks.
  • Enables self-hosting of the fast tier (self-hosted serving), with residency and rate-limit benefits.
  • Teacher data is cheap to produce with batch APIs and requires no human labelling for the bulk of it.
  • Escalation recovers the quality gap: student for volume, teacher for the hard cases (routing & escalation).
  • Compounds with the flywheel: as more production inputs accumulate, the student improves (data flywheel).

Cons

  • The student inherits the teacher's errors — and amplifies them, since it has no capacity to hedge. Unfiltered teacher output is the classic mistake.
  • Narrow by construction: excellent on the distilled task, worse than the base model at anything else.
  • Teacher terms of service may restrict using outputs to train competing models — check them.
  • Distribution shift: a student trained on last quarter's inputs degrades as usage changes; needs retraining.
  • Extra pipeline: generation, verification, training, evaluation, serving — all to maintain (fine-tuning ops).

When to use

  • High-volume, narrow, stable tasks: classification, routing, extraction, query rewriting, grading, simple summarisation.
  • When the fast tier's quality is the blocker to tiering (model selection & tradeoffs) — this is the fix.
  • When you need self-hosted inference for cost or residency and open-weight quality falls short.

When NOT to use

  • Open-ended, high-variance tasks — final answer generation, complex reasoning, planning. The student will be confidently worse.
  • Low-volume tasks where the pipeline costs more than the API bill it saves.
  • When the teacher itself isn't reliably correct on the task — verify before distilling (a 90%-accurate teacher yields at best a 90%-accurate student).
  • When the teacher's terms prohibit it.

Production at scale

Verification is the whole project. Self-consistency filtering (drop anything the teacher doesn't agree with itself on across n samples) plus deterministic schema/reference checks plus a human-reviewed sample. Teams that skip this ship a small model that reproduces the teacher's failure modes with more confidence and no hedging.

Use real production inputs, sampled to match the live distribution including the long tail. A student trained on clean, uniform synthetic inputs fails on the messy ones that dominate production.

Batch API for generation — half price, no latency constraint, and you can generate hundreds of thousands of examples cheaply (cost levers).

Deploy with escalation, not as a replacement. Student handles 85–95% of traffic; low confidence or schema failure escalates to the teacher (routing & escalation). Blended cost stays near the student's while quality stays near the teacher's, and the escalation rate is a live quality metric.

Monitor drift: student–teacher agreement on a sampled slice of live traffic, weekly. Falling agreement means the input distribution moved and it's time to regenerate and retrain (data flywheel).

Legal check on teacher terms before starting — some providers restrict using outputs to train models, particularly competing ones. This has stopped projects late.

Combine with the flywheel: production inputs → teacher outputs → verification → retraining on a cadence. Each cycle narrows the gap on the current distribution.

Cost accounting must include the pipeline: generation, verification, training, and serving infrastructure. For genuinely high volume the payback is fast; for moderate volume it may never arrive.

Prod checklist

  • Task is narrow, stable, and high-volume; teacher accuracy verified on it first
  • Teacher terms of service reviewed for training restrictions
  • Real production inputs sampled to match the live distribution, including the tail
  • Teacher outputs generated via batch API at low temperature
  • Self-consistency filtering (n samples must agree) plus deterministic checks
  • Human review of a sample; accuracy gate before training
  • Student trained with LoRA/QLoRA; evaluated against teacher and ground truth
  • Deployed with confidence-based escalation to the teacher
  • Weekly student–teacher agreement monitoring on live traffic
  • Retraining cadence tied to drift, with the full pipeline cost accounted

Metrics to watch

MetricWhy
Student accuracy vs teacher on held-out casesThe quality gap
Escalation rateResidual gap and blended cost
Student–teacher agreement on live trafficDistribution drift
Blended $/request and p95 latency vs teacher-onlyThe payoff
Teacher-data rejection rate during verificationTeacher reliability