Security → Threat checklist (OWASP LLM Top 10)
What it is
The OWASP Top 10 for LLM Applications is the standard threat taxonomy for AI systems. Its value is as a structured review checklist: for each item, name the control, name the test, and name the owner.
| # | Threat | Concrete form in a RAG/agent system | Primary control |
|---|---|---|---|
| 1 | Prompt injection | Poisoned document steers an agent with a user's authority | Policy check outside the model; delimiters; trust tiers (prompt-injection defence) |
| 2 | Insecure output handling | Model output rendered as HTML (XSS), interpolated into SQL, passed to a shell | Encode/parameterise at every boundary (output guardrails) |
| 3 | Training-data poisoning | Adversarial content in the ingested corpus or in fine-tuning data | Source tracking, ingest screening, dataset provenance (source ingestion, fine-tuning ops) |
| 4 | Model denial of service | Unbounded context, recursive agent loops, expensive queries driving cost or exhausting quota | Budgets, rate limits, load shedding (cost & step budgets, capacity & limits) |
| 5 | Supply chain | Malicious MCP server, backdoored weights, unpinned model alias | Pinning, hash verification, server vetting (model & supply-chain hygiene) |
| 6 | Sensitive information disclosure | Cross-tenant retrieval; PII in answers; secrets in context | Permission-filtered retrieval; PII handling; output checks (authentication & authorization, PII, output guardrails) |
| 7 | Insecure plugin/tool design | Tool accepting tenant_id as an argument; over-broad write tool | Typed schemas, identity from session, least privilege (tool design, excessive agency) |
| 8 | Excessive agency | Agent with a broad service account that can delete | Least-privilege credentials, approval gates (excessive agency, human-in-the-loop) |
| 9 | Overreliance | Users acting on ungrounded or unverifiable answers | Citations, faithfulness checks, abstention, UI framing (generation & grounding) |
| 10 | Model theft | Weight exfiltration; behaviour extraction via bulk querying | Access control on weights; rate limits; egress control (self-hosted serving, capacity & limits) |
How to implement
- Run it as a review, once per significant architecture change and at least annually: for each item, document the control, the test that proves it, and the owner.
- Map each threat to existing controls in this reference rather than inventing new ones — most are already covered by permission-filtered retrieval, least privilege, budgets, and output encoding.
- Write the test: a CI check, a red-team case, or a canary for each item (the testing pyramid).
- Record accepted risks explicitly, with a rationale and a review date. "We accept overreliance risk for internal analysts because outputs are always cited and reviewed" is a valid position; silence is not.
- Re-run on model changes — susceptibility to injection and jailbreaks varies by model (versioning & rollout).
- Feed incidents back: every security incident maps to an item and adds a test.
Small example
# security/owasp_llm_review.py — the review, as executable checks
THREAT_CONTROLS = {
"LLM01_prompt_injection": {
"controls": ["delimited untrusted content (delimiters: instructions vs data)", "trust tiers", "tainted-context write suppression",
"authorisation computed in code (prompt-injection defence/08.5)"],
"tests": ["test_injection_corpus_no_forbidden_action", "test_tainted_context_blocks_writes"],
"owner": "platform-security", "residual_risk": "medium — no complete defence exists",
},
"LLM02_insecure_output_handling": {
"controls": ["HTML escaping", "parameterised SQL", "shlex.quote for shell", "schema validation (output guardrails)"],
"tests": ["test_output_html_escaped", "test_sql_parameterised", "test_no_shell_interpolation"],
"owner": "app-team", "residual_risk": "low",
},
"LLM04_model_dos": {
"controls": ["per-request token/step/cost budgets (cost & step budgets)", "tenant rate limits (capacity & limits)",
"bounded queues", "load shedding"],
"tests": ["test_budget_enforced_before_step", "test_tenant_rate_limit", "test_recursive_loop_bounded"],
"owner": "platform", "residual_risk": "low",
},
"LLM06_sensitive_disclosure": {
"controls": ["permission-filtered retrieval (authentication & authorization)", "PII masking at boundary (PII)",
"output leakage checks (output guardrails)", "non-disclosing refusals"],
"tests": ["test_cross_tenant_canary", "test_permission_fixtures", "test_pii_not_in_output"],
"owner": "platform-security", "residual_risk": "low",
},
"LLM08_excessive_agency": {
"controls": ["role-scoped tool catalogue", "caller-delegated credentials",
"approval gates (human-in-the-loop)", "no delete tool in catalogue"],
"tests": ["test_viewer_cannot_send_email", "test_readonly_role_cannot_write",
"test_write_requires_approval"],
"owner": "platform-security", "residual_risk": "low",
},
"LLM09_overreliance": {
"controls": ["mandatory citations (citations)", "faithfulness gate (faithfulness check)",
"abstention path (refusal / no-answer path)", "UI shows sources and confidence"],
"tests": ["test_uncited_claims_blocked", "test_abstention_on_low_score"],
"owner": "product", "residual_risk": "medium — mitigated by UI framing and review workflow",
},
# ... LLM03, LLM05, LLM07, LLM10 similarly
}
def review_report() -> dict:
"""Generate the artefact a security reviewer actually asks for."""
rows, gaps = [], []
for threat, spec in THREAT_CONTROLS.items():
results = {t: run_test(t) for t in spec["tests"]}
passing = all(results.values())
if not passing or not spec["controls"]:
gaps.append({"threat": threat, "failing": [t for t, ok in results.items() if not ok]})
rows.append({"threat": threat, "controls": spec["controls"], "owner": spec["owner"],
"tests_passing": passing, "residual_risk": spec["residual_risk"]})
return {"generated": now().isoformat(), "app_version": os.getenv("GIT_SHA"),
"model": MODEL_ID, "rows": rows, "gaps": gaps,
"verdict": "PASS" if not gaps else "GAPS"}
def test_owasp_coverage():
"""CI gate: every threat has at least one control and one passing test."""
report = review_report()
assert not report["gaps"], f"OWASP LLM coverage gaps: {report['gaps']}"
# Example concrete tests referenced above
def test_output_html_escaped():
out = {"answer": "<img src=x onerror=alert(1)>"}
assert "<img" not in render_for_html(out["answer"])
def test_recursive_loop_bounded():
b = Budget(max_steps=3)
result = recursive_retrieve("q", CALLER, max_hops=99, budget=b) # runtime bound wins over the argument
assert result["hops"] <= 3
Pros
- Structured coverage — the checklist catches the threat you didn't think about, which is the point.
- Common language with security teams, auditors, and customers; a completed review answers most questionnaires.
- Maps cleanly to existing controls: most items are already addressed by permission filtering, least privilege, budgets, and output encoding.
- Executable review (tests per threat) keeps coverage from decaying.
- Explicit residual-risk records make acceptance decisions visible.
Cons
- Checklists breed false confidence — passing all ten doesn't mean secure, particularly for prompt injection, where no complete defence exists.
- Some items are barely actionable for application teams (model theft, training-data poisoning for API models).
- Evolves with the field; the list is periodically revised and your mapping needs updating.
- Doesn't capture domain-specific threats — you still need threat modelling for your particular system.
When to use
- At design review for any new AI feature with data access or tools.
- Before a security review, penetration test, or customer questionnaire.
- After any security incident, to check whether related items are covered.
- Annually, and on major model or architecture changes.
When NOT to use
- As a substitute for threat modelling your specific system — the checklist is a floor, not a ceiling.
- As a compliance box-tick without the tests behind it.
Production at scale
Turn the checklist into tests. A review document decays; a CI job that asserts every threat has a control and a passing test does not (the testing pyramid). Generate the review report from the tests, not the other way around.
Prompt injection has no complete defence — say so explicitly in the residual-risk column. What you can claim is containment: a successful injection reaches only data the caller could already access and cannot take an irreversible action without approval (prompt-injection defence, excessive agency, human-in-the-loop). That statement, backed by tests, is the honest and defensible position.
Insecure output handling is the most-missed item by AI teams because it's a classic web vulnerability that arrives through a new door. Model output rendered as HTML, interpolated into SQL, or passed to a shell is an injection regardless of the model's behaviour. Test it explicitly.
Model DoS is also a cost attack. In AI systems, the denial is often financial rather than availability-based — an attacker (or a buggy integration) driving your token spend. Budgets and per-tenant quotas are the control (cost & step budgets, capacity & limits).
Overreliance is a product control, not a technical one: citations, visible sources, confidence signals, abstention, and a UI that frames output as needing review. Own it with the product team, not only with engineering.
Re-run on model changes. Injection and jailbreak susceptibility differ across models and versions; a review passed on the previous model is not evidence for the current one (versioning & rollout).
Keep residual-risk statements current with review dates. Auditors and customers accept documented, owned residual risk far more readily than an unqualified "we're secure".
Prod checklist
- Each of the ten threats mapped to named controls, tests, and an owner
- Tests executable in CI; review report generated from test results
- Gaps tracked as issues with owners and dates; residual risks documented with review dates
- Insecure output handling tested at every boundary (HTML, SQL, shell, downstream APIs)
- Injection containment statement backed by least-privilege and approval-gate tests
- Cost-based DoS covered by budgets and per-tenant quotas
- Overreliance controls owned by product (citations, sources, abstention, UI framing)
- Review re-run on model changes and major architecture changes
- Incidents mapped back to threats, adding tests
Metrics to watch
| Metric | Why |
|---|---|
| Threats with passing tests / 10 | Coverage |
| Open gaps and their age | Remediation discipline |
| Red-team injection success rate per model | The item with no complete fix |
| Security incidents mapped to threats | Whether the taxonomy is catching reality |
| Review age (days since last run) | Currency |
Related
- Injection, Excessive agency, Output guardrails
- AuthZ, PII, Supply chain
- Budgets, Capacity — DoS controls
- Generation & grounding Grounding and citations — overreliance
- Testing pyramid — where the tests live