Production → Environments
What it is
AI systems have more stateful, expensive-to-rebuild components than typical applications: vector indexes, chunk registries, document stores, memory stores, eval datasets, and trace stores. Environment separation applies to all of them, and the vector index is the one teams most often share "just to save money".
| Component | Separation requirement |
|---|---|
| Vector index | Per environment, always. Dev writes must never reach the production index |
| Chunk registry / document store | Per environment; prod data subsetted and anonymised for lower environments |
| Memory store | Per environment; never seed dev with real user memories (memory security) |
| Trace store | Per environment; prod traces contain user data (data residency & retention) |
| Eval datasets | Shared and versioned, but run against environment-appropriate indexes |
| Model access | Separate API keys/quotas per environment so dev experiments can't exhaust prod quota (capacity & limits) |
| Feature flags / config | Per environment, pinned (versioning & rollout) |
Two failure modes drive this. Contamination: an eval or a dev experiment writing chunks into the production index, corrupting retrieval for real users. Invalidity: evaluating against a production index that is concurrently being re-indexed, producing results that can't be reproduced and don't reflect either state.
How to implement
- Physically separate indexes (different clusters, or at minimum different collections with distinct credentials) with production credentials unavailable to dev/CI.
- Snapshot production for staging: a reproducible, versioned snapshot of the corpus with an anonymisation/subsetting step, refreshed on a schedule.
- Evaluate against a pinned snapshot, not a live index, so results are reproducible and unaffected by concurrent ingestion (golden dataset first:
corpus_version). - Separate API keys and quota pools per environment (capacity & limits) so a runaway dev job can't exhaust production capacity.
- Write-protect production from lower environments: network policy, IAM, and credentials that make the wrong write impossible rather than merely discouraged.
- Environment stamped in every trace and every eval result (observability & tracing) so nobody misreads a staging number as production.
- Seed lower environments realistically: enough scale and document variety that retrieval behaviour is representative.
Small example
# config/environments.yaml
"""
dev:
vector: {url: "http://qdrant-dev:6333", collection: "chunks_dev", write: true}
registry: postgres://dev-db/chunks
llm: {key_env: ANTHROPIC_KEY_DEV, tpm_quota: 50_000}
corpus: {source: "snapshot", snapshot: "2024-06-01-anonymised-10pct"}
staging:
vector: {url: "http://qdrant-staging:6333", collection: "chunks_staging", write: true}
llm: {key_env: ANTHROPIC_KEY_STAGING, tpm_quota: 200_000}
corpus: {source: "snapshot", snapshot: "2024-06-01-full-anonymised"}
prod:
vector: {url: "http://qdrant-prod:6333", collection: "chunks_v2", write: true}
llm: {key_env: ANTHROPIC_KEY_PROD, tpm_quota: 2_000_000}
corpus: {source: "live"}
eval: # read-only replica, pinned snapshot: reproducible
vector: {url: "http://qdrant-eval:6333", collection: "chunks_snapshot_2024_06_01", write: false}
"""
ENV = os.environ["APP_ENV"]
CFG = load_environments()[ENV]
class GuardedVectorClient:
"""Structural protection: the wrong write is impossible, not merely discouraged."""
def __init__(self, cfg, env):
self.client, self.collection, self.env = connect(cfg["url"]), cfg["collection"], env
self.write_allowed = cfg.get("write", False)
if env != "prod" and "prod" in cfg["url"]:
raise RuntimeError(f"env={env} configured against a production endpoint — refusing to start")
def upsert(self, *a, **kw):
if not self.write_allowed:
raise PermissionError(f"writes disabled for env={self.env} (collection={self.collection})")
return self.client.upsert(collection_name=self.collection, *a, **kw)
def search(self, *a, **kw):
return self.client.search(collection_name=self.collection, *a, **kw)
def run_eval(suite, env="eval"):
"""Evals always run against a pinned, read-only snapshot."""
cfg = load_environments()[env]
assert not cfg["vector"].get("write", False), "eval must not run against a writable index"
results = suite.run(vector=GuardedVectorClient(cfg["vector"], env))
return results | {"environment": env, "corpus_snapshot": cfg["vector"]["collection"],
"version_vector": version_vector(CFG, canary=False)}
def create_snapshot(prod_registry, target: str, sample_pct=100, anonymise=True):
"""Reproducible lower-environment corpus, safe to share."""
docs = prod_registry.sample(pct=sample_pct, stratify_by=["source", "doc_type", "tenant_tier"])
for d in docs:
if anonymise:
d = anonymise_document(d) # PII scrubbed, tenants remapped (PII)
snapshot_store.put(target, d)
manifest = {"created": now(), "n_docs": len(docs), "sample_pct": sample_pct,
"anonymised": anonymise, "source_corpus_version": prod_registry.version}
snapshot_store.put_manifest(target, manifest)
return manifest
# CI safety net
def test_no_prod_credentials_in_ci():
for var in ("ANTHROPIC_KEY_PROD", "QDRANT_PROD_URL", "PROD_DB_URL"):
assert var not in os.environ, f"{var} must not be available in CI"
Pros
- Prevents corruption of production retrieval by experiments and eval runs — the failure this exists to stop.
- Reproducible evals against a pinned snapshot; results comparable across weeks.
- Quota isolation stops dev jobs from consuming production capacity (capacity & limits).
- Safe experimentation: chunking, parser, and embedding changes can be tried at realistic scale without risk.
- Anonymised snapshots let more engineers work with realistic data safely (PII).
Cons
- Cost: multiple vector indexes and document stores, potentially several times the storage.
- Snapshot staleness: lower environments drift from production, so behaviour differs.
- Anonymisation is work and can change retrieval characteristics (names and identifiers matter for BM25, hybrid search).
- Full-scale staging is expensive; sampled corpora may not reproduce scale-dependent behaviour (ANN recall under filtering, filtered search).
- More configuration and more ways to point at the wrong thing.
When to use
Always for production systems, and from the start — retrofitting separation after an incident is worse than building it.
When NOT to use (full-scale copies)
- Full-size staging indexes for very large corpora may not be justified; use a stratified sample large enough to reproduce the behaviours you care about, and validate scale-dependent properties (index recall, latency) against production read replicas instead.
Production at scale
Make the wrong write structurally impossible. Credentials that don't exist in CI, network policies that block prod endpoints from dev, and a client that refuses to start if the environment and endpoint disagree. Convention alone fails eventually.
Evals run against a read-only pinned snapshot. This is the single most important rule here: it makes results reproducible and removes any chance of contaminating live data. Tie each eval result to a corpus_snapshot identifier (golden dataset first).
Stratified sampling for lower environments: sample by source, document type, language, and tenant tier so retrieval behaviour is representative. A dev corpus of 500 clean English PDFs will not reveal the problems your production corpus has.
Anonymisation must preserve retrieval characteristics. Replacing every name with "PERSON_1" destroys BM25 behaviour and dedupe behaviour. Use consistent pseudonymisation (same entity → same fake name) so identifier and name matching still behave realistically (PII).
Production read replicas for scale-dependent testing: ANN recall under filters, latency at concurrency, and index memory behaviour need production-scale data. A read-only replica gives you that without write risk (index types: Flat, HNSW, IVF-PQ).
Separate trace stores per environment. Production traces contain user queries and document content and fall under retention and erasure policy (data residency & retention); dev traces should never mix with them.
Refresh snapshots on a schedule and record the corpus version in eval results, so a shifted baseline is visible rather than mysterious.
Cost control: lower-environment indexes can use cheaper configurations (smaller instances, higher ef for accuracy over speed, disk-based indexes) since they don't serve users.
Prod checklist
- Physically separate vector indexes, registries, memory stores, and trace stores per environment
- Production credentials unavailable in dev/CI; client refuses env/endpoint mismatches
- Evals run against a read-only, pinned corpus snapshot; snapshot ID recorded in results
- Separate API keys and quota pools per environment
- Stratified, consistently pseudonymised snapshots for lower environments, refreshed on a schedule
- Production read replica available for scale-dependent testing
- Environment stamped on every trace and eval result
- CI test asserting production credentials are absent
- Lower-environment cost controls (smaller instances, sampled corpora)
Metrics to watch
| Metric | Why |
|---|---|
| Writes to prod index by non-prod identities (must be 0) | Contamination |
| Snapshot age per environment | Drift |
| Eval reproducibility (same snapshot → same results) | Trustworthy evals |
| Non-prod quota consumption vs prod | Capacity isolation |
| Storage cost by environment | Justifying the copies |
Related
- Golden dataset — corpus_version pinning
- Embedding versioning — dual-index is a per-environment concern too
- Capacity & limits — quota isolation
- Data residency & retention / PII — anonymisation and trace-store governance
- Versioning & rollout — config per environment