Fine-tuning → Ops (datasets, checkpoints, reproducibility)

Fine-Tuning Reproducibility Explained: Datasets, Checkpoints, Lineage

What it is

Fine-tuning adds a model artifact to a system that previously had only code and config. That artifact needs the same lifecycle rigour as everything else, plus some specifics:

ArtifactMust record
DatasetContent hash, row count, source provenance, curation rules version, split assignment, PII treatment
Training runBase model + revision, hyperparameters, seed, library versions, hardware, dataset hash, duration, loss curves
Checkpoint / adapterBase coupling (model ID + revision + quantisation), dataset hash, run ID, eval scores, approval status
Serving configWhich adapter, merged or separate, quantisation, serving engine version

Three properties to guarantee:

  1. Reproducibility — the same dataset hash, config, and seed produce an equivalent model.
  2. Traceability — from a production answer, back to the adapter, the run, the dataset, and the examples that shaped it.
  3. Rollback — reverting to the previous adapter (or to the base model) is a config change, not a retraining project (versioning & rollout).

How to implement

  1. Content-hash the dataset (sorted, canonicalised rows) and store it immutably with a manifest: sources, counts by category, curation rules version, split assignment.
  2. Freeze the test split by input hash on day one, and never train on it — enforce with an assertion at train time.
  3. Record the full training config: base model and revision, LoRA parameters, learning rate, epochs, seed, library and CUDA versions, hardware.
  4. Model registry entry per checkpoint with lineage (dataset hash, run ID, base), eval results, and an approval state (experimental → staging → production).
  5. Run the same eval suite as prompts (prompt evaluation) plus safety and abstention cases; gate promotion on it (the testing pyramid).
  6. Deploy via config pin with staged rollout and instant rollback (versioning & rollout).
  7. Track base-model deprecations — every dependent adapter needs retraining before the sunset (model & supply-chain hygiene).

Small example

import hashlib, json
from dataclasses import dataclass, asdict
from datetime import datetime

def dataset_hash(examples: list[dict]) -> str:
    canonical = json.dumps(sorted(json.dumps(e, sort_keys=True) for e in examples), separators=(",", ":"))
    return hashlib.sha256(canonical.encode()).hexdigest()[:16]

@dataclass
class DatasetManifest:
    dataset_id: str; hash: str; created: str
    n_train: int; n_test: int
    sources: dict                       # {"golden_set": 210, "production_accepted": 640, "expert": 95}
    coverage: dict                      # counts by doc_type / language / edge case
    curation_rules_version: str
    pii_treatment: str                  # "scrubbed" | "synthetic" | "in-boundary"
    test_split_key_hashes: list[str]    # frozen forever

@dataclass
class TrainingRun:
    run_id: str; started: str
    base_model: str; base_revision: str; quantisation: str
    dataset_hash: str
    hyperparameters: dict               # lora r/alpha/dropout/targets, lr, epochs, batch, seed
    library_versions: dict              # transformers, peft, trl, torch, cuda
    hardware: str
    final_train_loss: float; best_eval_loss: float; best_epoch: int

@dataclass
class ModelArtifact:
    adapter_id: str; run_id: str; dataset_hash: str
    base_model: str; base_revision: str; serving_quantisation: str
    eval_results: dict                  # quality, safety, format-violation, cost
    approval_state: str                 # experimental | staging | production | retired
    created: str; approved_by: str | None = None

class ModelRegistry:
    def register(self, artifact: ModelArtifact, run: TrainingRun, manifest: DatasetManifest):
        assert artifact.dataset_hash == run.dataset_hash == manifest.hash, "lineage mismatch"
        self.store.put(f"adapters/{artifact.adapter_id}/manifest.json",
                       json.dumps({"artifact": asdict(artifact), "run": asdict(run),
                                   "dataset": asdict(manifest)}, indent=2))

    def promote(self, adapter_id: str, to_state: str, eval_report: dict, approver: str):
        a = self.get(adapter_id)
        if to_state == "production":
            assert eval_report["quality_delta"] >= -0.01,     "quality regression"
            assert eval_report["safety_delta"] >= -0.01,      "safety regression"      # SFT
            assert eval_report["format_violation_rate"] <= 0.01
            assert not eval_report["regressions"],            f"case regressions: {eval_report['regressions'][:5]}"
        a.approval_state, a.approved_by = to_state, approver
        audit_log("model_promoted", adapter=adapter_id, state=to_state, approver=approver, evals=eval_report)
        self.save(a)

# ---- guards that prevent the classic mistakes ----
def assert_no_test_leakage(train_examples, manifest: DatasetManifest):
    train_keys = {hashlib.sha256(normalise(e["messages"][1]["content"]).encode()).hexdigest()[:16]
                  for e in train_examples}
    leaked = train_keys & set(manifest.test_split_key_hashes)
    assert not leaked, f"test examples in training data: {len(leaked)}"

def assert_base_compatibility(artifact: ModelArtifact, serving_cfg: dict):
    assert artifact.base_model == serving_cfg["base_model"], "adapter/base model mismatch"
    assert artifact.base_revision == serving_cfg["base_revision"], "adapter/base revision mismatch"

def reproducibility_check(run: TrainingRun, registry) -> dict:
    """Retrain from the recorded config and compare — run periodically, not on every change."""
    rerun = train_from_config(asdict(run))
    delta = abs(rerun["best_eval_loss"] - run.best_eval_loss)
    return {"reproducible": delta < 0.02, "loss_delta": round(delta, 4)}

def deprecation_watch(registry, days=90):
    """Every adapter is coupled to a base model that will eventually sunset (model & supply-chain hygiene)."""
    at_risk = []
    for a in registry.list(state="production"):
        if (d := base_deprecation_date(a.base_model)) and (d - date.today()).days < days:
            at_risk.append({"adapter": a.adapter_id, "base": a.base_model, "sunset": str(d),
                            "action": "retrain on a successor base and re-evaluate"})
    return at_risk

Pros

  • Reproducibility: a model can be rebuilt, debugged, and safely retrained months later.
  • Traceability: a bad production output traces to the adapter, the run, and the training examples that shaped it.
  • Safe promotion: eval gates prevent quality and safety regressions from reaching users (the testing pyramid).
  • Rollback in seconds via config pin (versioning & rollout).
  • Base-deprecation tracking turns a forced migration into scheduled work.

Cons

  • Real MLOps overhead: registry, storage, manifests, promotion workflow.
  • Dataset storage and governance: training data contains customer content and falls under retention and erasure policy (data residency & retention).
  • Reproducibility is approximate — GPU non-determinism means "equivalent", not "identical"; use loss and eval tolerances.
  • Coupling burden: every base-model upgrade invalidates every adapter (PEFT / LoRA / QLoRA).
  • Multiple adapters per tenant or task multiply everything.

When to use

The moment a fine-tuned model reaches staging. Ad-hoc training in notebooks is fine for exploration and disastrous in production.

When NOT to use (heavy process)

  • Pure exploration and hyperparameter search — record enough to reproduce the winner, not every run.
  • One-off experiments that will never be served.

Production at scale

The dataset is the artifact that matters most. Models can be retrained; a lost or unreproducible dataset cannot. Store it immutably with a hash, a manifest, and provenance for every example (which production interaction, which reviewer). This also makes erasure possible when a customer requests it (data residency & retention).

Freeze the test split by input hash on day one and assert at train time that no training example shares a key with it. Leakage is silent and inflates every number you report.

Same eval suite as prompts, plus safety. The temptation is to evaluate a fine-tune only on its own task; the risk is eroded refusal behaviour and general capability (SFT, preference tuning). Gate promotion on the full suite (the testing pyramid).

Adapters are pinned like models in environment config, with staged rollout and canary comparison (versioning & rollout). Rollback is a pin flip.

Training data is customer data. Apply retention, PII treatment, tenant isolation, and erasure coverage; a fine-tuning dataset assembled from production is one of the easiest places to accidentally retain personal data indefinitely (data residency & retention, PII).

Track base deprecations across all adapters (model & supply-chain hygiene): a base sunset invalidates every dependent adapter, and each needs retraining plus re-evaluation. Give it a lead time of months, not weeks.

Periodic reproducibility checks: retrain from a recorded config quarterly and compare eval loss. Drift usually indicates an unpinned library or a mutated dataset.

Cost tracking per model iteration: generation (for distillation), curation, training, and evaluation. This makes the retraining cadence an informed decision rather than a habit.

Prod checklist

  • Dataset content-hashed and stored immutably with a manifest (sources, coverage, curation rules, PII treatment)
  • Test split frozen by input hash on day one; leakage assertion at train time
  • Full training config recorded: base + revision, hyperparameters, seed, library and CUDA versions, hardware
  • Model registry with lineage (dataset hash, run ID, base coupling) and approval states
  • Promotion gated on the prompt eval suite plus safety and abstention cases
  • Adapter/base compatibility asserted at serving time
  • Deployment via config pin with staged rollout and tested rollback
  • Training datasets covered by retention, PII, and erasure policy
  • Base-model deprecation watch across all production adapters
  • Periodic reproducibility check from recorded configs

Metrics to watch

MetricWhy
Adapters in production without complete lineage (target 0)Traceability
Test-leakage assertion failuresInflated evals
Time from base-deprecation notice to retrained adapterMigration readiness
Reproducibility check pass rateConfig/library drift
Cost per model iterationRetraining cadence economics