Fine-tuning → Ops (datasets, checkpoints, reproducibility)
What it is
Fine-tuning adds a model artifact to a system that previously had only code and config. That artifact needs the same lifecycle rigour as everything else, plus some specifics:
| Artifact | Must record |
|---|---|
| Dataset | Content hash, row count, source provenance, curation rules version, split assignment, PII treatment |
| Training run | Base model + revision, hyperparameters, seed, library versions, hardware, dataset hash, duration, loss curves |
| Checkpoint / adapter | Base coupling (model ID + revision + quantisation), dataset hash, run ID, eval scores, approval status |
| Serving config | Which adapter, merged or separate, quantisation, serving engine version |
Three properties to guarantee:
- Reproducibility — the same dataset hash, config, and seed produce an equivalent model.
- Traceability — from a production answer, back to the adapter, the run, the dataset, and the examples that shaped it.
- Rollback — reverting to the previous adapter (or to the base model) is a config change, not a retraining project (versioning & rollout).
How to implement
- Content-hash the dataset (sorted, canonicalised rows) and store it immutably with a manifest: sources, counts by category, curation rules version, split assignment.
- Freeze the test split by input hash on day one, and never train on it — enforce with an assertion at train time.
- Record the full training config: base model and revision, LoRA parameters, learning rate, epochs, seed, library and CUDA versions, hardware.
- Model registry entry per checkpoint with lineage (dataset hash, run ID, base), eval results, and an approval state (
experimental→staging→production). - Run the same eval suite as prompts (prompt evaluation) plus safety and abstention cases; gate promotion on it (the testing pyramid).
- Deploy via config pin with staged rollout and instant rollback (versioning & rollout).
- Track base-model deprecations — every dependent adapter needs retraining before the sunset (model & supply-chain hygiene).
Small example
import hashlib, json
from dataclasses import dataclass, asdict
from datetime import datetime
def dataset_hash(examples: list[dict]) -> str:
canonical = json.dumps(sorted(json.dumps(e, sort_keys=True) for e in examples), separators=(",", ":"))
return hashlib.sha256(canonical.encode()).hexdigest()[:16]
@dataclass
class DatasetManifest:
dataset_id: str; hash: str; created: str
n_train: int; n_test: int
sources: dict # {"golden_set": 210, "production_accepted": 640, "expert": 95}
coverage: dict # counts by doc_type / language / edge case
curation_rules_version: str
pii_treatment: str # "scrubbed" | "synthetic" | "in-boundary"
test_split_key_hashes: list[str] # frozen forever
@dataclass
class TrainingRun:
run_id: str; started: str
base_model: str; base_revision: str; quantisation: str
dataset_hash: str
hyperparameters: dict # lora r/alpha/dropout/targets, lr, epochs, batch, seed
library_versions: dict # transformers, peft, trl, torch, cuda
hardware: str
final_train_loss: float; best_eval_loss: float; best_epoch: int
@dataclass
class ModelArtifact:
adapter_id: str; run_id: str; dataset_hash: str
base_model: str; base_revision: str; serving_quantisation: str
eval_results: dict # quality, safety, format-violation, cost
approval_state: str # experimental | staging | production | retired
created: str; approved_by: str | None = None
class ModelRegistry:
def register(self, artifact: ModelArtifact, run: TrainingRun, manifest: DatasetManifest):
assert artifact.dataset_hash == run.dataset_hash == manifest.hash, "lineage mismatch"
self.store.put(f"adapters/{artifact.adapter_id}/manifest.json",
json.dumps({"artifact": asdict(artifact), "run": asdict(run),
"dataset": asdict(manifest)}, indent=2))
def promote(self, adapter_id: str, to_state: str, eval_report: dict, approver: str):
a = self.get(adapter_id)
if to_state == "production":
assert eval_report["quality_delta"] >= -0.01, "quality regression"
assert eval_report["safety_delta"] >= -0.01, "safety regression" # SFT
assert eval_report["format_violation_rate"] <= 0.01
assert not eval_report["regressions"], f"case regressions: {eval_report['regressions'][:5]}"
a.approval_state, a.approved_by = to_state, approver
audit_log("model_promoted", adapter=adapter_id, state=to_state, approver=approver, evals=eval_report)
self.save(a)
# ---- guards that prevent the classic mistakes ----
def assert_no_test_leakage(train_examples, manifest: DatasetManifest):
train_keys = {hashlib.sha256(normalise(e["messages"][1]["content"]).encode()).hexdigest()[:16]
for e in train_examples}
leaked = train_keys & set(manifest.test_split_key_hashes)
assert not leaked, f"test examples in training data: {len(leaked)}"
def assert_base_compatibility(artifact: ModelArtifact, serving_cfg: dict):
assert artifact.base_model == serving_cfg["base_model"], "adapter/base model mismatch"
assert artifact.base_revision == serving_cfg["base_revision"], "adapter/base revision mismatch"
def reproducibility_check(run: TrainingRun, registry) -> dict:
"""Retrain from the recorded config and compare — run periodically, not on every change."""
rerun = train_from_config(asdict(run))
delta = abs(rerun["best_eval_loss"] - run.best_eval_loss)
return {"reproducible": delta < 0.02, "loss_delta": round(delta, 4)}
def deprecation_watch(registry, days=90):
"""Every adapter is coupled to a base model that will eventually sunset (model & supply-chain hygiene)."""
at_risk = []
for a in registry.list(state="production"):
if (d := base_deprecation_date(a.base_model)) and (d - date.today()).days < days:
at_risk.append({"adapter": a.adapter_id, "base": a.base_model, "sunset": str(d),
"action": "retrain on a successor base and re-evaluate"})
return at_risk
Pros
- Reproducibility: a model can be rebuilt, debugged, and safely retrained months later.
- Traceability: a bad production output traces to the adapter, the run, and the training examples that shaped it.
- Safe promotion: eval gates prevent quality and safety regressions from reaching users (the testing pyramid).
- Rollback in seconds via config pin (versioning & rollout).
- Base-deprecation tracking turns a forced migration into scheduled work.
Cons
- Real MLOps overhead: registry, storage, manifests, promotion workflow.
- Dataset storage and governance: training data contains customer content and falls under retention and erasure policy (data residency & retention).
- Reproducibility is approximate — GPU non-determinism means "equivalent", not "identical"; use loss and eval tolerances.
- Coupling burden: every base-model upgrade invalidates every adapter (PEFT / LoRA / QLoRA).
- Multiple adapters per tenant or task multiply everything.
When to use
The moment a fine-tuned model reaches staging. Ad-hoc training in notebooks is fine for exploration and disastrous in production.
When NOT to use (heavy process)
- Pure exploration and hyperparameter search — record enough to reproduce the winner, not every run.
- One-off experiments that will never be served.
Production at scale
The dataset is the artifact that matters most. Models can be retrained; a lost or unreproducible dataset cannot. Store it immutably with a hash, a manifest, and provenance for every example (which production interaction, which reviewer). This also makes erasure possible when a customer requests it (data residency & retention).
Freeze the test split by input hash on day one and assert at train time that no training example shares a key with it. Leakage is silent and inflates every number you report.
Same eval suite as prompts, plus safety. The temptation is to evaluate a fine-tune only on its own task; the risk is eroded refusal behaviour and general capability (SFT, preference tuning). Gate promotion on the full suite (the testing pyramid).
Adapters are pinned like models in environment config, with staged rollout and canary comparison (versioning & rollout). Rollback is a pin flip.
Training data is customer data. Apply retention, PII treatment, tenant isolation, and erasure coverage; a fine-tuning dataset assembled from production is one of the easiest places to accidentally retain personal data indefinitely (data residency & retention, PII).
Track base deprecations across all adapters (model & supply-chain hygiene): a base sunset invalidates every dependent adapter, and each needs retraining plus re-evaluation. Give it a lead time of months, not weeks.
Periodic reproducibility checks: retrain from a recorded config quarterly and compare eval loss. Drift usually indicates an unpinned library or a mutated dataset.
Cost tracking per model iteration: generation (for distillation), curation, training, and evaluation. This makes the retraining cadence an informed decision rather than a habit.
Prod checklist
- Dataset content-hashed and stored immutably with a manifest (sources, coverage, curation rules, PII treatment)
- Test split frozen by input hash on day one; leakage assertion at train time
- Full training config recorded: base + revision, hyperparameters, seed, library and CUDA versions, hardware
- Model registry with lineage (dataset hash, run ID, base coupling) and approval states
- Promotion gated on the prompt eval suite plus safety and abstention cases
- Adapter/base compatibility asserted at serving time
- Deployment via config pin with staged rollout and tested rollback
- Training datasets covered by retention, PII, and erasure policy
- Base-model deprecation watch across all production adapters
- Periodic reproducibility check from recorded configs
Metrics to watch
| Metric | Why |
|---|---|
| Adapters in production without complete lineage (target 0) | Traceability |
| Test-leakage assertion failures | Inflated evals |
| Time from base-deprecation notice to retrained adapter | Migration readiness |
| Reproducibility check pass rate | Config/library drift |
| Cost per model iteration | Retraining cadence economics |
Related
- SFT, LoRA, Distillation — what produces these artifacts
- Golden dataset first / prompt evaluation — the eval suite gating promotion
- Versioning & rollout — pins, canaries, rollback
- Retention, Supply chain — dataset governance and base deprecation
- Observability — model version in the trace