Prompt engineering → Prompt versioning & evaluation (prompts are code)
What it is
A prompt is a program that runs on a stochastic interpreter. Treat it accordingly:
- Stored in git as files (templates + examples + schemas), not in a database field someone edits in the console at 5 pm on a Friday.
- Versioned: semantic version or content hash per prompt template; every deploy pins versions.
- Reviewed: PRs with diffs; reviewers see exactly what changed.
- Tested: the eval suite (prompt evaluation) runs on every change; regressions block merge (the testing pyramid).
- Traced: every LLM call logs
prompt_id,prompt_version,model,schema_version(observability & tracing); dashboards break down quality/cost by version. - Rolled out safely: canary or A/B by version; rollback = config change (versioning & rollout).
The alternative — "we tweaked the prompt and it seemed better" — is how silent regressions ship and how nobody can explain why answers changed last Tuesday.
How to implement
- Prompt files:
prompts/<call_site>/<name>.md(or.j2) with front-matter:id, version, model_tier, schema, owner, changelog. Examples and schemas in adjacent files. - Loader that renders templates with named slots (prompt structure) and returns
(text, prompt_id, version_hash). - Registry/config mapping call sites → prompt versions for each environment (dev/staging/prod).
- Eval hook: CI job runs the golden set for the affected call sites on any change under
prompts/; reports metrics vs the current version; fails on regression beyond tolerance. - Telemetry:
prompt_id/versionon every call record; dashboards by version. - Rollout: new version to canary traffic; compare online metrics (online eval); promote or roll back.
- Changelog discipline: each version records why (failure case IDs) — this is the institutional memory.
Small example
# prompts/answer_generate/v3.md
"""
---
id: answer_generate
version: 3
model_tier: balanced
schema: answer_v2
owner: rag-team
changelog: "v3: restate citation rule at end (fixes GOLD-142, GOLD-158)"
---
<instructions>
...
</instructions>
"""
import hashlib, re, yaml, pathlib
class PromptRegistry:
def __init__(self, root="prompts", pins=None):
self.root, self.pins = pathlib.Path(root), pins or {} # pins: {call_site: version} per environment
self._cache = {}
def load(self, call_site):
v = self.pins[call_site]
path = self.root / call_site / f"v{v}.md"
if path not in self._cache:
raw = path.read_text()
fm, body = re.match(r"---\n(.*?)\n---\n(.*)", raw, re.S).groups()
meta = yaml.safe_load(fm)
meta["hash"] = hashlib.sha256(body.encode()).hexdigest()[:12]
self._cache[path] = (meta, body)
return self._cache[path]
def render(self, call_site, **slots):
meta, body = self.load(call_site)
return body.format(**slots), {"prompt_id": meta["id"], "prompt_version": meta["version"], "prompt_hash": meta["hash"],
"schema": meta["schema"], "model_tier": meta["model_tier"]}
registry = PromptRegistry(pins=yaml.safe_load(open("config/prompt_pins.prod.yaml")))
text, tags = registry.render("answer_generate", context=ctx, question=q)
resp = llm(tags["model_tier"], text)
log_call(**tags, model=resp.model, tokens_in=resp.usage.input_tokens, tokens_out=resp.usage.output_tokens, trace_id=trace_id)
CI (pseudo): on: pull_request paths: prompts/** → run evals --call-sites $(changed) --compare-to main → fail if delta < -tolerance.
Pros
- Traceability: any answer → prompt version → diff → author → reason.
- Regression prevention: evals in CI catch what "looked fine".
- Safe rollout and instant rollback.
- Institutional memory via changelogs tied to failure cases.
- Enables A/B tests and per-tenant prompt variants without chaos.
Cons
- Process overhead: PRs and eval runs for a one-word change. (The one-word change is exactly the one that breaks citations.)
- Eval cost: running the golden set on every change costs tokens/time; mitigate with targeted evals per call site and cached prefixes.
- Template complexity if slots and includes proliferate.
- Multi-tenant prompt variants multiply versions to track.
When to use
Always, from the first production prompt. Retrofitting versioning after a year of console edits means reverse-engineering what's live.
When NOT to use (heavy process)
- Throwaway prototypes — but keep prompts in files anyway; the prototype becomes the product.
Production at scale
Pin per environment. prompt_pins.{dev,staging,prod}.yaml; promotion is a pin change via PR. Prod never loads "latest".
Composite versioning. Behaviour depends on prompt + model + schema + retrieval config + examples. Log all of them per call; a "version vector" lets you attribute a change to the right component (component vs end-to-end).
Eval scoping. Map call sites → golden subsets; a change to answer_generate runs the generation evals, not the parser evals. Full-suite runs nightly and before releases.
Online monitoring by version (online eval): thumbs, abstention rate, schema violations, cost, latency — split by prompt_version. Canary compares versions on the same traffic slice.
Prompt registries/tools (Langfuse, LangSmith, PromptLayer, Braintrust, Weights & Biases) give UI + history + evals; still keep the source of truth in git and pin by hash.
Tenant/locale variants as overlays on a base prompt (base version + overlay version), not forks, or you'll maintain 200 diverging copies.
Security review: prompt changes that touch tool descriptions, permissions language, or injection defences get a security reviewer (model & supply-chain hygiene, prompt-injection defence).
Change freeze and rollback drill: practise a rollback; it should be a pin change and a deploy, under five minutes.
Prod checklist
- Prompts in git with front-matter (id, version, tier, schema, owner, changelog)
- Registry with per-environment pins; prod never loads latest
- Every call logs prompt id/version/hash + model + schema + retrieval config
- CI evals on prompt changes scoped by call site; regression blocks merge
- Canary/A/B by version; rollback = pin change
- Dashboards split by prompt version
- Overlay pattern for tenant/locale variants
- Security review for prompts touching tools/permissions/injection defences
Metrics to watch
| Metric | Why |
|---|---|
| Golden-set score per prompt version (CI) | Regression gate |
| Online quality/cost/latency per version | Canary decisions |
| Time from regression detected → rolled back | Ops readiness |
| Unpinned or unknown prompt versions in prod logs (should be 0) | Discipline |
Related
- Prompt structure — the template
- Prompt evaluation — the eval suite
- Testing pyramid — CI gates
- Observability — logging versions
- Versioning & rollout — canary/rollback