Security → Threat checklist (OWASP LLM Top 10)

OWASP LLM Top 10 Checklist Explained: Control, Test, Owner

What it is

The OWASP Top 10 for LLM Applications is the standard threat taxonomy for AI systems. Its value is as a structured review checklist: for each item, name the control, name the test, and name the owner.

#ThreatConcrete form in a RAG/agent systemPrimary control
1Prompt injectionPoisoned document steers an agent with a user's authorityPolicy check outside the model; delimiters; trust tiers (prompt-injection defence)
2Insecure output handlingModel output rendered as HTML (XSS), interpolated into SQL, passed to a shellEncode/parameterise at every boundary (output guardrails)
3Training-data poisoningAdversarial content in the ingested corpus or in fine-tuning dataSource tracking, ingest screening, dataset provenance (source ingestion, fine-tuning ops)
4Model denial of serviceUnbounded context, recursive agent loops, expensive queries driving cost or exhausting quotaBudgets, rate limits, load shedding (cost & step budgets, capacity & limits)
5Supply chainMalicious MCP server, backdoored weights, unpinned model aliasPinning, hash verification, server vetting (model & supply-chain hygiene)
6Sensitive information disclosureCross-tenant retrieval; PII in answers; secrets in contextPermission-filtered retrieval; PII handling; output checks (authentication & authorization, PII, output guardrails)
7Insecure plugin/tool designTool accepting tenant_id as an argument; over-broad write toolTyped schemas, identity from session, least privilege (tool design, excessive agency)
8Excessive agencyAgent with a broad service account that can deleteLeast-privilege credentials, approval gates (excessive agency, human-in-the-loop)
9OverrelianceUsers acting on ungrounded or unverifiable answersCitations, faithfulness checks, abstention, UI framing (generation & grounding)
10Model theftWeight exfiltration; behaviour extraction via bulk queryingAccess control on weights; rate limits; egress control (self-hosted serving, capacity & limits)

How to implement

  1. Run it as a review, once per significant architecture change and at least annually: for each item, document the control, the test that proves it, and the owner.
  2. Map each threat to existing controls in this reference rather than inventing new ones — most are already covered by permission-filtered retrieval, least privilege, budgets, and output encoding.
  3. Write the test: a CI check, a red-team case, or a canary for each item (the testing pyramid).
  4. Record accepted risks explicitly, with a rationale and a review date. "We accept overreliance risk for internal analysts because outputs are always cited and reviewed" is a valid position; silence is not.
  5. Re-run on model changes — susceptibility to injection and jailbreaks varies by model (versioning & rollout).
  6. Feed incidents back: every security incident maps to an item and adds a test.

Small example

# security/owasp_llm_review.py — the review, as executable checks
THREAT_CONTROLS = {
    "LLM01_prompt_injection": {
        "controls": ["delimited untrusted content (delimiters: instructions vs data)", "trust tiers", "tainted-context write suppression",
                     "authorisation computed in code (prompt-injection defence/08.5)"],
        "tests": ["test_injection_corpus_no_forbidden_action", "test_tainted_context_blocks_writes"],
        "owner": "platform-security", "residual_risk": "medium — no complete defence exists",
    },
    "LLM02_insecure_output_handling": {
        "controls": ["HTML escaping", "parameterised SQL", "shlex.quote for shell", "schema validation (output guardrails)"],
        "tests": ["test_output_html_escaped", "test_sql_parameterised", "test_no_shell_interpolation"],
        "owner": "app-team", "residual_risk": "low",
    },
    "LLM04_model_dos": {
        "controls": ["per-request token/step/cost budgets (cost & step budgets)", "tenant rate limits (capacity & limits)",
                     "bounded queues", "load shedding"],
        "tests": ["test_budget_enforced_before_step", "test_tenant_rate_limit", "test_recursive_loop_bounded"],
        "owner": "platform", "residual_risk": "low",
    },
    "LLM06_sensitive_disclosure": {
        "controls": ["permission-filtered retrieval (authentication & authorization)", "PII masking at boundary (PII)",
                     "output leakage checks (output guardrails)", "non-disclosing refusals"],
        "tests": ["test_cross_tenant_canary", "test_permission_fixtures", "test_pii_not_in_output"],
        "owner": "platform-security", "residual_risk": "low",
    },
    "LLM08_excessive_agency": {
        "controls": ["role-scoped tool catalogue", "caller-delegated credentials",
                     "approval gates (human-in-the-loop)", "no delete tool in catalogue"],
        "tests": ["test_viewer_cannot_send_email", "test_readonly_role_cannot_write",
                  "test_write_requires_approval"],
        "owner": "platform-security", "residual_risk": "low",
    },
    "LLM09_overreliance": {
        "controls": ["mandatory citations (citations)", "faithfulness gate (faithfulness check)",
                     "abstention path (refusal / no-answer path)", "UI shows sources and confidence"],
        "tests": ["test_uncited_claims_blocked", "test_abstention_on_low_score"],
        "owner": "product", "residual_risk": "medium — mitigated by UI framing and review workflow",
    },
    # ... LLM03, LLM05, LLM07, LLM10 similarly
}

def review_report() -> dict:
    """Generate the artefact a security reviewer actually asks for."""
    rows, gaps = [], []
    for threat, spec in THREAT_CONTROLS.items():
        results = {t: run_test(t) for t in spec["tests"]}
        passing = all(results.values())
        if not passing or not spec["controls"]:
            gaps.append({"threat": threat, "failing": [t for t, ok in results.items() if not ok]})
        rows.append({"threat": threat, "controls": spec["controls"], "owner": spec["owner"],
                     "tests_passing": passing, "residual_risk": spec["residual_risk"]})
    return {"generated": now().isoformat(), "app_version": os.getenv("GIT_SHA"),
            "model": MODEL_ID, "rows": rows, "gaps": gaps,
            "verdict": "PASS" if not gaps else "GAPS"}

def test_owasp_coverage():
    """CI gate: every threat has at least one control and one passing test."""
    report = review_report()
    assert not report["gaps"], f"OWASP LLM coverage gaps: {report['gaps']}"

# Example concrete tests referenced above
def test_output_html_escaped():
    out = {"answer": "<img src=x onerror=alert(1)>"}
    assert "<img" not in render_for_html(out["answer"])

def test_recursive_loop_bounded():
    b = Budget(max_steps=3)
    result = recursive_retrieve("q", CALLER, max_hops=99, budget=b)     # runtime bound wins over the argument
    assert result["hops"] <= 3

Pros

  • Structured coverage — the checklist catches the threat you didn't think about, which is the point.
  • Common language with security teams, auditors, and customers; a completed review answers most questionnaires.
  • Maps cleanly to existing controls: most items are already addressed by permission filtering, least privilege, budgets, and output encoding.
  • Executable review (tests per threat) keeps coverage from decaying.
  • Explicit residual-risk records make acceptance decisions visible.

Cons

  • Checklists breed false confidence — passing all ten doesn't mean secure, particularly for prompt injection, where no complete defence exists.
  • Some items are barely actionable for application teams (model theft, training-data poisoning for API models).
  • Evolves with the field; the list is periodically revised and your mapping needs updating.
  • Doesn't capture domain-specific threats — you still need threat modelling for your particular system.

When to use

  • At design review for any new AI feature with data access or tools.
  • Before a security review, penetration test, or customer questionnaire.
  • After any security incident, to check whether related items are covered.
  • Annually, and on major model or architecture changes.

When NOT to use

  • As a substitute for threat modelling your specific system — the checklist is a floor, not a ceiling.
  • As a compliance box-tick without the tests behind it.

Production at scale

Turn the checklist into tests. A review document decays; a CI job that asserts every threat has a control and a passing test does not (the testing pyramid). Generate the review report from the tests, not the other way around.

Prompt injection has no complete defence — say so explicitly in the residual-risk column. What you can claim is containment: a successful injection reaches only data the caller could already access and cannot take an irreversible action without approval (prompt-injection defence, excessive agency, human-in-the-loop). That statement, backed by tests, is the honest and defensible position.

Insecure output handling is the most-missed item by AI teams because it's a classic web vulnerability that arrives through a new door. Model output rendered as HTML, interpolated into SQL, or passed to a shell is an injection regardless of the model's behaviour. Test it explicitly.

Model DoS is also a cost attack. In AI systems, the denial is often financial rather than availability-based — an attacker (or a buggy integration) driving your token spend. Budgets and per-tenant quotas are the control (cost & step budgets, capacity & limits).

Overreliance is a product control, not a technical one: citations, visible sources, confidence signals, abstention, and a UI that frames output as needing review. Own it with the product team, not only with engineering.

Re-run on model changes. Injection and jailbreak susceptibility differ across models and versions; a review passed on the previous model is not evidence for the current one (versioning & rollout).

Keep residual-risk statements current with review dates. Auditors and customers accept documented, owned residual risk far more readily than an unqualified "we're secure".

Prod checklist

  • Each of the ten threats mapped to named controls, tests, and an owner
  • Tests executable in CI; review report generated from test results
  • Gaps tracked as issues with owners and dates; residual risks documented with review dates
  • Insecure output handling tested at every boundary (HTML, SQL, shell, downstream APIs)
  • Injection containment statement backed by least-privilege and approval-gate tests
  • Cost-based DoS covered by budgets and per-tenant quotas
  • Overreliance controls owned by product (citations, sources, abstention, UI framing)
  • Review re-run on model changes and major architecture changes
  • Incidents mapped back to threats, adding tests

Metrics to watch

MetricWhy
Threats with passing tests / 10Coverage
Open gaps and their ageRemediation discipline
Red-team injection success rate per modelThe item with no complete fix
Security incidents mapped to threatsWhether the taxonomy is catching reality
Review age (days since last run)Currency