Security → Data residency & retention

Data Residency and Retention for LLM Systems Explained

What it is

An AI system spreads data across more places than teams expect. Every one is a residency and retention question:

StoreContainsTypical retention question
Vector indexEmbeddings (and often chunk text in payload)Are embeddings personal data? (Increasingly treated as yes)
Chunk registry / document storeFull document textMirrors source retention
Trace storePrompts, retrieved content, answers (observability & tracing)Usually the longest-lived copy of sensitive content
Eval datasetsSampled production queries and documents (golden dataset first)Often forgotten in erasure
Memory storeUser facts and past interactions (memory)Subject-access and erasure requests
CachesSemantic cache answers, retrieval caches (latency levers)Short TTL, but must be invalidated on erasure
Provider-sidePrompts sent to the model APIZero-retention endpoints; training opt-out
BackupsEverything aboveThe hardest erasure case

Three commitments a customer contract typically requires:

  1. Residency: data is processed and stored in a named region and does not leave it.
  2. Zero retention / no training: the model provider does not store prompts beyond the request and does not train on them.
  3. Retention and erasure: data is deleted after a stated period, and on request within a stated SLA.

How to implement

  1. Inventory every store that touches customer data, with its region, retention period, encryption, and access controls. This document is the basis of every security review you'll face.
  2. Per-tenant policy configuration: region, allowed model providers/endpoints, retention periods, training opt-out, PII handling (PII).
  3. Route by tenant policy: regional model endpoints, regional vector clusters, regional object storage. Enforce in the client so a misconfiguration fails rather than silently routing cross-region.
  4. Use zero-retention endpoints where required, and record the provider's data-processing terms per model.
  5. Retention jobs per store with defined periods, plus an erase(subject) operation covering all of them (how to forget).
  6. Backup policy documented: usually "backups age out within N days" rather than surgical deletion — write it down and have legal approve it.
  7. Evidence: log erasure operations with verification results; produce a completion record per request (auditability).

Small example

from dataclasses import dataclass

@dataclass(frozen=True)
class TenantPolicy:
    tenant_id: str
    region: str                              # "eu-central-1" | "us-east-1" | "ap-southeast-2"
    allowed_providers: frozenset[str]        # {"anthropic-eu"} for EU-only tenants
    zero_retention_required: bool
    training_opt_out: bool
    retention_days: dict                     # {"traces": 30, "memories": 365, "documents": None}
    pii_action: str                          # "mask" | "block" | "allow"
    self_hosted_only: bool = False

REGION_ENDPOINTS = {
    "eu-central-1": {"llm": "https://api.eu.anthropic.com", "vector": "qdrant-eu.internal",
                     "objects": "s3://acme-eu-documents"},
    "us-east-1":    {"llm": "https://api.anthropic.com",     "vector": "qdrant-us.internal",
                     "objects": "s3://acme-us-documents"},
}

def clients_for(policy: TenantPolicy):
    """Routing enforced structurally: a cross-region call cannot be constructed."""
    ep = REGION_ENDPOINTS[policy.region]
    if policy.self_hosted_only:
        llm = SelfHostedClient(endpoint=INTERNAL_VLLM[policy.region])           # Self-hosted serving
    else:
        provider = next(iter(policy.allowed_providers))
        assert provider.endswith(policy.region.split("-")[0]) or provider in REGION_PROVIDERS[policy.region], \
            f"provider {provider} not permitted in {policy.region}"
        llm = APIClient(base_url=ep["llm"],
                        headers={"anthropic-beta": "zero-retention"} if policy.zero_retention_required else {})
    return {"llm": llm, "vector": connect(ep["vector"]), "objects": s3(ep["objects"])}

# ---- retention enforcement per store ----
RETENTION_JOBS = {
    "traces":    lambda days: trace_store.delete_older_than(days),
    "memories":  lambda days: memory_store.expire_older_than(days),
    "eval_samples": lambda days: eval_store.delete_samples_older_than(days),
    "caches":    lambda days: cache.purge_older_than(min(days, 1)),
    "documents": lambda days: doc_store.delete_soft_deleted_older_than(days),     # Document versioning
}

def enforce_retention(policy: TenantPolicy):
    results = {}
    for store, days in policy.retention_days.items():
        if days is None:                                       # indefinite (e.g. contractual documents)
            results[store] = "retained"; continue
        results[store] = RETENTION_JOBS[store](days)
    audit_log("retention_enforced", tenant=policy.tenant_id, results=results)
    return results

# ---- subject erasure across EVERY store ----
ERASURE_TARGETS = ["documents", "chunks", "vectors", "parents", "page_images",
                   "memories", "traces", "eval_samples", "caches", "quarantine"]

def erase_subject(subject_id: str, tenant_id: str, reason="gdpr_request") -> dict:
    op_id = str(uuid.uuid4())
    audit_log("erasure_start", op_id=op_id, subject=subject_id, tenant=tenant_id, reason=reason)
    outcomes = {t: ERASERS[t](subject_id, tenant_id) for t in ERASURE_TARGETS}
    verification = {t: VERIFIERS[t](subject_id, tenant_id) for t in ERASURE_TARGETS}   # prove it, don't assume
    clean = all(verification.values())
    audit_log("erasure_complete", op_id=op_id, outcomes=outcomes, verified=clean,
              backup_note="Backups age out within 35 days per retention policy DP-07")
    if not clean:
        alert_erasure_incomplete(op_id, [t for t, ok in verification.items() if not ok])
    return {"op_id": op_id, "verified": clean, "targets": outcomes}

def test_no_cross_region_calls():
    """CI gate: an EU tenant must be unable to reach a US endpoint."""
    eu = TenantPolicy("t_eu", "eu-central-1", frozenset({"anthropic-eu"}), True, True,
                      {"traces": 30}, "mask")
    c = clients_for(eu)
    assert "eu" in c["llm"].base_url and "eu" in str(c["vector"])

Pros

  • Unlocks regulated and EU/public-sector customers — often the difference between winning a deal and not.
  • Bounded exposure: shorter retention means less data at risk in a breach.
  • Zero-retention endpoints remove the largest third-party concern in most security reviews.
  • Documented inventory and policy turns each security questionnaire from a research project into a lookup.
  • Per-tenant policy allows one platform to serve varied compliance needs.

Cons

  • Regional deployments multiply infrastructure cost and operational surface — separate indexes, separate model endpoints, separate storage per region.
  • Model availability differs by region: the best model may not be available where a tenant requires processing.
  • Short trace retention hurts debugging and eval-set growth (data flywheel) — a genuine trade-off.
  • Erasure across derived data is hard: embeddings, summaries, caches, eval sets, and backups all hold copies (how to forget).
  • Backups are usually handled by policy rather than surgical deletion, which some customers challenge.

When to use

Any system serving EU, UK, healthcare, financial, or public-sector customers — and any system whose contracts mention data processing, which is most B2B systems.

When NOT to use (full regionalisation)

  • Single-region, single-jurisdiction internal tools: apply retention and zero-retention, skip multi-region infrastructure until a customer requires it.

Production at scale

The data inventory is the foundational artifact. Every store, its region, its retention, its encryption, its access controls, and its erasure procedure. Security reviews, DPAs, and incident response all start here, and teams that lack it discover their trace store has been retaining prompts indefinitely.

Trace stores are the sleeper risk. They contain queries, retrieved document content, and answers — often the richest concentration of sensitive data you hold — and they're typically configured with generous retention for debugging. Set an explicit period, sample rather than retain everything (observability & tracing), and include them in erasure.

Embeddings are increasingly treated as personal data, since they can be partially inverted. Assume they're in scope for residency and erasure; don't argue the point in a customer review.

Zero-retention and training opt-out per provider and model — record which endpoints and terms apply, and re-verify on provider changes. This is a common question and the answer must be precise, not "we think so".

Enforce routing structurally: the client construction should make a cross-region call impossible, and CI should test it. Configuration alone drifts.

Erasure needs verification, not just execution (how to forget): query every store afterwards and assert absence; log the verification. "We ran the delete" is not evidence.

Retention shortens the eval flywheel. If traces expire in 30 days, sample into a longer-lived, anonymised eval store within that window (online eval, data flywheel) — otherwise short retention silently starves your evaluation.

Sub-processor list: model providers, vector store vendors, observability tools, and reranker APIs are all sub-processors that must appear in your DPA. Keep the list current as tools change (model & supply-chain hygiene).

Prod checklist

  • Complete data inventory: every store, region, retention, encryption, access, erasure procedure
  • Per-tenant policy: region, providers, zero-retention, training opt-out, retention periods, PII action
  • Routing enforced in client construction; cross-region calls impossible; CI test
  • Zero-retention endpoints and training opt-out configured and documented per provider
  • Retention jobs per store, running and monitored
  • erase_subject covering documents, chunks, vectors, parents, images, memories, traces, eval samples, caches, quarantine
  • Erasure verification and completion records with SLA tracking
  • Backup policy documented and legally reviewed
  • Trace retention explicitly set; anonymised eval sampling within the window
  • Sub-processor list current and reflected in the DPA

Metrics to watch

MetricWhy
Erasure request → verified completion timeContractual/legal SLA
Erasure verification failures (must be 0)Incomplete deletion
Cross-region calls detected (must be 0)Residency compliance
Data older than its retention period, per storePolicy enforcement
Sub-processor list driftDPA accuracy