Prompt engineering → Prompt versioning & evaluation (prompts are code)

Prompt Versioning and Evaluation: Treat Prompts Like Code

What it is

A prompt is a program that runs on a stochastic interpreter. Treat it accordingly:

  • Stored in git as files (templates + examples + schemas), not in a database field someone edits in the console at 5 pm on a Friday.
  • Versioned: semantic version or content hash per prompt template; every deploy pins versions.
  • Reviewed: PRs with diffs; reviewers see exactly what changed.
  • Tested: the eval suite (prompt evaluation) runs on every change; regressions block merge (the testing pyramid).
  • Traced: every LLM call logs prompt_id, prompt_version, model, schema_version (observability & tracing); dashboards break down quality/cost by version.
  • Rolled out safely: canary or A/B by version; rollback = config change (versioning & rollout).

The alternative — "we tweaked the prompt and it seemed better" — is how silent regressions ship and how nobody can explain why answers changed last Tuesday.

How to implement

  1. Prompt files: prompts/<call_site>/<name>.md (or .j2) with front-matter: id, version, model_tier, schema, owner, changelog. Examples and schemas in adjacent files.
  2. Loader that renders templates with named slots (prompt structure) and returns (text, prompt_id, version_hash).
  3. Registry/config mapping call sites → prompt versions for each environment (dev/staging/prod).
  4. Eval hook: CI job runs the golden set for the affected call sites on any change under prompts/; reports metrics vs the current version; fails on regression beyond tolerance.
  5. Telemetry: prompt_id/version on every call record; dashboards by version.
  6. Rollout: new version to canary traffic; compare online metrics (online eval); promote or roll back.
  7. Changelog discipline: each version records why (failure case IDs) — this is the institutional memory.

Small example

# prompts/answer_generate/v3.md
"""
---
id: answer_generate
version: 3
model_tier: balanced
schema: answer_v2
owner: rag-team
changelog: "v3: restate citation rule at end (fixes GOLD-142, GOLD-158)"
---
<instructions>
...
</instructions>
"""

import hashlib, re, yaml, pathlib

class PromptRegistry:
    def __init__(self, root="prompts", pins=None):
        self.root, self.pins = pathlib.Path(root), pins or {}          # pins: {call_site: version} per environment
        self._cache = {}

    def load(self, call_site):
        v = self.pins[call_site]
        path = self.root / call_site / f"v{v}.md"
        if path not in self._cache:
            raw = path.read_text()
            fm, body = re.match(r"---\n(.*?)\n---\n(.*)", raw, re.S).groups()
            meta = yaml.safe_load(fm)
            meta["hash"] = hashlib.sha256(body.encode()).hexdigest()[:12]
            self._cache[path] = (meta, body)
        return self._cache[path]

    def render(self, call_site, **slots):
        meta, body = self.load(call_site)
        return body.format(**slots), {"prompt_id": meta["id"], "prompt_version": meta["version"], "prompt_hash": meta["hash"],
                                      "schema": meta["schema"], "model_tier": meta["model_tier"]}

registry = PromptRegistry(pins=yaml.safe_load(open("config/prompt_pins.prod.yaml")))
text, tags = registry.render("answer_generate", context=ctx, question=q)
resp = llm(tags["model_tier"], text)
log_call(**tags, model=resp.model, tokens_in=resp.usage.input_tokens, tokens_out=resp.usage.output_tokens, trace_id=trace_id)

CI (pseudo): on: pull_request paths: prompts/** → run evals --call-sites $(changed) --compare-to main → fail if delta < -tolerance.

Pros

  • Traceability: any answer → prompt version → diff → author → reason.
  • Regression prevention: evals in CI catch what "looked fine".
  • Safe rollout and instant rollback.
  • Institutional memory via changelogs tied to failure cases.
  • Enables A/B tests and per-tenant prompt variants without chaos.

Cons

  • Process overhead: PRs and eval runs for a one-word change. (The one-word change is exactly the one that breaks citations.)
  • Eval cost: running the golden set on every change costs tokens/time; mitigate with targeted evals per call site and cached prefixes.
  • Template complexity if slots and includes proliferate.
  • Multi-tenant prompt variants multiply versions to track.

When to use

Always, from the first production prompt. Retrofitting versioning after a year of console edits means reverse-engineering what's live.

When NOT to use (heavy process)

  • Throwaway prototypes — but keep prompts in files anyway; the prototype becomes the product.

Production at scale

Pin per environment. prompt_pins.{dev,staging,prod}.yaml; promotion is a pin change via PR. Prod never loads "latest".

Composite versioning. Behaviour depends on prompt + model + schema + retrieval config + examples. Log all of them per call; a "version vector" lets you attribute a change to the right component (component vs end-to-end).

Eval scoping. Map call sites → golden subsets; a change to answer_generate runs the generation evals, not the parser evals. Full-suite runs nightly and before releases.

Online monitoring by version (online eval): thumbs, abstention rate, schema violations, cost, latency — split by prompt_version. Canary compares versions on the same traffic slice.

Prompt registries/tools (Langfuse, LangSmith, PromptLayer, Braintrust, Weights & Biases) give UI + history + evals; still keep the source of truth in git and pin by hash.

Tenant/locale variants as overlays on a base prompt (base version + overlay version), not forks, or you'll maintain 200 diverging copies.

Security review: prompt changes that touch tool descriptions, permissions language, or injection defences get a security reviewer (model & supply-chain hygiene, prompt-injection defence).

Change freeze and rollback drill: practise a rollback; it should be a pin change and a deploy, under five minutes.

Prod checklist

  • Prompts in git with front-matter (id, version, tier, schema, owner, changelog)
  • Registry with per-environment pins; prod never loads latest
  • Every call logs prompt id/version/hash + model + schema + retrieval config
  • CI evals on prompt changes scoped by call site; regression blocks merge
  • Canary/A/B by version; rollback = pin change
  • Dashboards split by prompt version
  • Overlay pattern for tenant/locale variants
  • Security review for prompts touching tools/permissions/injection defences

Metrics to watch

MetricWhy
Golden-set score per prompt version (CI)Regression gate
Online quality/cost/latency per versionCanary decisions
Time from regression detected → rolled backOps readiness
Unpinned or unknown prompt versions in prod logs (should be 0)Discipline