Security → Model & supply-chain hygiene
What it is
AI systems have the usual software supply chain — packages, containers, base images — plus three novel surfaces:
| Surface | Risk | Why it's different |
|---|---|---|
| Model weights | Backdoored or trojaned open-weight models; pickle deserialisation RCE in .bin checkpoints | Weights are opaque; you cannot read them for malice |
| Third-party tool servers (MCP) | Tool descriptions are instructions your model follows; results enter your context | A malicious description is a prompt injection you installed deliberately (prompt-injection defence) |
| Prompt/agent packages | Community prompts, agent templates, and skill files containing hidden instructions | Text executed by a model, not a compiler |
| Model provider changes | Unpinned aliases changing behaviour; deprecations forcing rushed migrations | Behaviour changes without a deploy (versioning & rollout) |
| Data poisoning | Adversarial content in ingested corpora | Persistent, and reaches every future query (source ingestion, prompt-injection defence) |
The rule that captures the novelty: anything that contributes text to a prompt is executable content. Tool descriptions, system-prompt fragments, retrieved documents, and MCP server metadata all get read and acted on by the model.
How to implement
- Pin everything: model IDs (never aliases), weight file hashes, serving-engine versions, package versions with lockfiles, container digests (not tags).
- Prefer safetensors over pickle-based checkpoints; verify SHA-256 against the publisher's published hash; scan for unexpected pickle imports if you must load them.
- Review third-party MCP servers before enabling: read every tool description, check what data it accesses and what network calls it makes, pin the version, and re-review on upgrade.
- Run untrusted tool servers with least privilege: separate process or container, no credentials, egress allow-list (sandboxing, excessive agency).
- Treat tool descriptions as prompt assets under version control and review (prompt versioning & evaluation, tool design); diff them on every dependency update.
- SBOM and vulnerability scanning for containers and packages, plus an AI-specific inventory: models, adapters, embedding models, rerankers, and their sources.
- Track deprecations and provider changes as scheduled work, with eval runs on every change (versioning & rollout).
Small example
import hashlib, json, subprocess
# ---- Pin and verify model artefacts ----
MODEL_MANIFEST = {
"generation": {"provider": "anthropic", "model_id": "claude-sonnet-5", # exact ID, never an alias
"deprecation_date": "2026-03-01", "terms": "zero-retention-eu"},
"embedding": {"source": "huggingface", "repo": "intfloat/multilingual-e5-large",
"revision": "ab10c1a7f42e2b0c", # commit, not "main"
"sha256": "9f2e...c41a", "format": "safetensors"},
"reranker": {"source": "huggingface", "repo": "BAAI/bge-reranker-v2-m3",
"revision": "5f8d2c9e1b34", "sha256": "3b7d...80fe", "format": "safetensors"},
}
def verify_model_artifact(path: str, expected_sha: str, fmt: str):
if fmt != "safetensors":
raise ValueError("refusing pickle-format weights: use safetensors") # pickle load = RCE
h = hashlib.sha256()
with open(path, "rb") as f:
for block in iter(lambda: f.read(1 << 20), b""):
h.update(block)
if h.hexdigest() != expected_sha:
raise SecurityError(f"weight hash mismatch for {path}")
return True
# ---- MCP server review gate ----
APPROVED_MCP_SERVERS = {
"acme-knowledge": {"version": "1.4.2", "reviewed": "2024-06-10", "reviewer": "sec-team",
"descriptions_hash": "b31f...9a2c", "egress": [], "credentials": "caller_delegated"},
}
INSTRUCTION_LIKE = re.compile(
r"(ignore\s+(previous|all)|you\s+must\s+always|system\s*:|do\s+not\s+tell\s+the\s+user|"
r"send\s+(this|the\s+\w+)\s+to\s+https?://)", re.I)
def vet_mcp_server(name: str, manifest: dict) -> dict:
"""Run before enabling ANY third-party tool server."""
findings = []
approved = APPROVED_MCP_SERVERS.get(name)
if not approved:
findings.append("server not in the approved list")
elif manifest["version"] != approved["version"]:
findings.append(f"version drift: {manifest['version']} != approved {approved['version']}")
desc_blob = json.dumps([t["description"] for t in manifest["tools"]], sort_keys=True)
desc_hash = hashlib.sha256(desc_blob.encode()).hexdigest()
if approved and desc_hash != approved["descriptions_hash"]:
findings.append("tool descriptions changed since review — re-review required")
for tool in manifest["tools"]: # descriptions are text the model OBEYS
if INSTRUCTION_LIKE.search(tool["description"]):
findings.append(f"instruction-like content in description of {tool['name']}")
if tool.get("input_schema", {}).get("properties", {}).keys() & {"tenant_id", "user_id", "acl_groups"}:
findings.append(f"{tool['name']} accepts identity parameters — must come from session (authentication & authorization)")
return {"approved": not findings, "findings": findings, "descriptions_hash": desc_hash}
def enable_mcp_server(name, manifest, caller_context):
v = vet_mcp_server(name, manifest)
if not v["approved"]:
audit_log("mcp_server_rejected", server=name, findings=v["findings"])
raise SecurityError(f"MCP server {name} rejected: {v['findings']}")
return MCPClient(name, credentials=None, # no ambient credentials (excessive agency)
identity=caller_context, # delegated, per request
egress_allowlist=APPROVED_MCP_SERVERS[name]["egress"])
# ---- CI: dependency and description drift ----
def test_no_unpinned_models():
for role, m in MODEL_MANIFEST.items():
assert "latest" not in str(m.get("model_id", "")) and m.get("revision") != "main", \
f"{role} is unpinned"
def test_deprecations_not_imminent(days=60):
for role, m in MODEL_MANIFEST.items():
if d := m.get("deprecation_date"):
assert (date.fromisoformat(d) - date.today()).days > days, \
f"{role} deprecates in <{days} days — migration must be scheduled (versioning & rollout)"
Pros
- Prevents behaviour changes you didn't authorise — the most common real incident, from an unpinned model alias.
- Description review catches installed injections before they reach production.
- Hash verification and safetensors eliminate the weight-loading RCE class.
- Deprecation tracking turns forced migrations into planned work (versioning & rollout).
- Least-privilege tool servers bound the damage from a compromised dependency.
Cons
- Review effort per dependency, and the MCP ecosystem moves fast enough that pinning means falling behind.
- Pinning creates upgrade debt; security patches still need applying.
- Description review doesn't scale to dozens of servers without tooling.
- Open-weight provenance is hard to verify beyond publisher hashes — you cannot inspect weights for backdoors.
- Vetting slows adoption of genuinely useful third-party tools.
When to use
Any production system. The MCP-review gate specifically applies the moment you enable a tool server you did not write.
When NOT to use (heavy vetting)
- First-party tools in your own codebase go through normal code review; the extra gate is for third-party and community-sourced servers, prompts, and models.
Production at scale
Tool descriptions are prompt assets. Store them in version control, hash them, and diff on every dependency update. A dependency upgrade that changes a tool description is a prompt change and must go through eval and review (prompt versioning & evaluation, tool design).
No ambient credentials in tool servers. A third-party server should receive delegated, per-request identity and hold nothing of its own (excessive agency, authentication & authorization). If it needs a credential, it needs a review that treats it as a system with production access.
Egress allow-lists for tool servers and sandboxes (sandboxing). The exfiltration path is the risk; a server that cannot reach the internet cannot send your data anywhere.
Reject identity parameters in tool schemas. A tool that accepts tenant_id or user_id as an argument lets the model choose whose data to access. Identity comes from the authenticated session, never from arguments — check this in vetting.
Maintain an AI-specific inventory alongside your SBOM: models, adapters, embedding models, rerankers, tool servers, and prompt packages, each with source, version, hash, licence, and review date. Security reviews will ask for it.
Licence review for open-weight models — some prohibit commercial use or impose conditions. This is a real blocker discovered late by many teams.
Provider change monitoring: subscribe to model deprecation notices, track dates as tickets, and run the full eval suite on any provider-announced change (versioning & rollout).
Data poisoning is a supply-chain risk too: content ingested from external sources can carry injections that persist in your index (prompt-injection defence). Screen at ingest, track the source, and be able to purge everything from a compromised connector.
Prod checklist
- All model IDs pinned exactly; no aliases, no
mainrevisions - Weight hashes verified; safetensors required; pickle checkpoints refused
- Third-party MCP servers reviewed before enabling: descriptions read, egress checked, version pinned
- Tool-description hashes tracked; drift triggers re-review
- Tool schemas rejected if they accept identity parameters
- Tool servers run with delegated identity, no ambient credentials, egress allow-lists
- SBOM plus AI-specific inventory (models, adapters, rerankers, servers, prompts) with licences
- Deprecation dates tracked; CI fails when a sunset is imminent
- Container digests pinned; images scanned and rebuilt on a schedule
- Ingest-source tracking enabling purge of a compromised connector
Metrics to watch
| Metric | Why |
|---|---|
| Unpinned dependencies/models (target 0) | Uncontrolled behaviour change |
| Tool-description drift events | Installed-injection risk |
| Days to nearest model deprecation | Migration runway |
| Third-party servers enabled without review (must be 0) | Gate integrity |
| Vulnerability findings in images/packages | Ordinary supply chain |
Related
- Tool design, MCP — where third-party tools enter
- Prompt injection — descriptions are an injection vector
- Excessive agency — no ambient credentials
- Versioning & rollout — pinning and deprecations
- Sandboxing — isolating untrusted execution