Testing & reliability → The testing pyramid

AI Testing Pyramid Explained: What to Unit Test, Mock and Eval

What it is

AI systems have deterministic parts and probabilistic parts, and they need different test strategies. The pyramid layers them by speed, determinism, and cost:

        ┌──────────────────────────┐
        │  Red-team / adversarial  │  slow, manual+automated, run on model changes and releases
        ├──────────────────────────┤
        │  End-to-end evals        │  minutes, statistical, pre-release + nightly
        ├──────────────────────────┤
        │  Integration (mocked)    │  seconds, deterministic, every PR
        ├──────────────────────────┤
        │  Component evals         │  seconds-minutes, per artefact changed
        ├──────────────────────────┤
        │  Unit tests              │  milliseconds, fully deterministic, every commit
        └──────────────────────────┘
LayerWhat it testsDeterminismRuns on
UnitParsers, chunkers, tokenizers, schema validators, filter builders, idempotency keys, retry logicFully deterministicEvery commit
Component evalsParsing accuracy (test-time validation against a golden sample), chunk recall, retrieval metrics (retrieval evals), prompt behaviour (prompt evaluation)Statistical, temp 0The artefact that changed
IntegrationAgent flows with mocked tools, pipeline wiring, error paths, budget enforcement, approval gatesDeterministic with stubsEvery PR
End-to-end evalsFull system on the golden set (golden dataset first), with cost and latencyStatisticalPre-release, nightly
Red-teamPrompt injection, jailbreaks, PII leakage, ACL bypass, excessive agencyAdversarial, partly manualModel changes, releases, quarterly

The rule from the source file: evals run in CI on every prompt/model/index change and block the merge on regression. Not "we run evals before big releases" — every change to a behaviour-affecting artefact.

How to implement

  1. Make the deterministic parts genuinely deterministic and test them hard: ACL filter construction, chunk ID generation, idempotency keys, citation validation, budget enforcement, schema validators. These carry the security and correctness invariants and cost nothing to test.
  2. Map artefacts to eval suites (component vs end-to-end): prompts/** → prompt evals; chunking/** → chunk recall + retrieval; parsers/** → parsing evals; model config → full suite.
  3. Mock the LLM in integration tests with recorded or scripted responses so agent flows, error handling, and gates are tested deterministically and fast.
  4. Gate on regressions, not aggregates (prompt evaluation): any previously-passing case that now fails blocks the merge.
  5. Hard gates for safety properties: ACL leakage, forbidden-tool calls, unapproved writes, PII in output — any failure blocks regardless of other metrics.
  6. Nightly full suite including end-to-end with live retrieval, plus the red-team suite.
  7. Report a diff in the PR: metrics before/after, regressed cases, cost/latency delta.

Small example

# ============ Layer 1: unit — deterministic invariants, every commit ============
def test_acl_filter_never_empty():
    caller = Caller(id="u1", tenant_id="t1", groups=set())
    flt = build_acl_filter(caller)
    assert any(c["key"] == "tenant_id" for c in flt["must"])
    assert any(c["key"] == "acl_groups" for c in flt["must"])          # never absent, even with no groups

def test_chunk_ids_deterministic():
    a = fixed_size_chunks("doc1", 3, TEXT); b = fixed_size_chunks("doc1", 3, TEXT)
    assert [c["chunk_id"] for c in a] == [c["chunk_id"] for c in b]    # re-runs upsert, never duplicate (fixed-size + overlap)

def test_idempotency_key_stable():
    args = {"to": ["a@b.com"], "subject": "Hi", "body": "x"}
    assert idem_key("run1", 3, "send_email", args) == idem_key("run1", 3, "send_email", dict(reversed(list(args.items()))))

def test_citation_validation_rejects_unretrieved_id():
    out = {"claims": [{"text": "x", "citations": ["c_never_retrieved"]}]}
    v = validate_citations(out, retrieved=[{"chunk_id": "c1", "text": "..."}])
    assert v["citation_issues"] and v["claims"][0]["citations"] == []

def test_budget_blocks_before_execution():
    b = Budget(max_steps=1); b.steps = 1
    with pytest.raises(BudgetExceeded): b.check()

# ============ Layer 2: integration with mocked tools — every PR ============
class MockLLM:
    def __init__(self, script): self.script, self.i = script, 0
    def create(self, **kw):
        r = self.script[self.i]; self.i += 1; return r

def test_agent_recovers_from_tool_error():
    llm = MockLLM([tool_use("search_documents", {"query": "x"}),
                   tool_use("search_documents", {"query": "narrower x"}),
                   final_answer("Found it [c1]")])
    tools = {"search_documents": lambda a, c, k: (_ for _ in ()).throw(TimeoutError())
             if a["query"] == "x" else {"status": "ok", "passages": [{"id": "c1"}]}}
    out = run_agent("q", CALLER, TOOL_DEFS, tools, SYS, llm=llm)
    assert out["answer"].startswith("Found it")                        # error text fed back, agent recovered (single agent loop)

def test_write_tool_requires_approval():
    llm = MockLLM([tool_use("send_email", {"to": ["a@b.com"], "subject": "s", "body": "b",
                                           "idempotency_key": "k1"})])
    with pytest.raises(PausedForApproval):
        run_agent("send it", CALLER, TOOL_DEFS, TOOL_IMPLS, SYS, llm=llm)   # Human-in-the-loop

def test_approval_resume_is_idempotent():
    aid = start_and_pause(); on_approval_decision(aid, ApprovalStatus.APPROVED)
    on_approval_decision(aid, ApprovalStatus.APPROVED)                 # duplicate webhook
    assert sent_email_count() == 1                                     # State & durability

# ============ Layer 3: component + e2e evals — on the artefact that changed ============
# .github/workflows/eval.yml (pseudo)
"""
on: pull_request
jobs:
  evals:
    steps:
      - id: changed
        run: echo "areas=$(python ci/map_changes_to_suites.py)" >> $GITHUB_OUTPUT
      - run: python -m evals.run --suites ${{ steps.changed.outputs.areas }} --compare-to main --fail-on-regression
      - run: python -m evals.run --suite safety --fail-on-any            # hard gate, always
      - run: python ci/comment_diff.py                                   # posts metric diff to the PR
"""

def ci_gate(report, safety_report):
    if safety_report["failures"]:
        raise SystemExit(f"BLOCKED (safety): {safety_report['failures']}")           # never negotiable
    if report["regressions"]:
        raise SystemExit(f"BLOCKED: {len(report['regressions'])} regressions: {report['regressions'][:5]}")
    print(f"PASS: {report['fixes']} fixes, cost delta {report['metrics']['cost_per_query']['delta']:+.4f}")

Pros

  • Fast feedback where it's cheap: unit and integration tests catch most bugs in seconds.
  • Regressions caught before users — the entire purpose of the golden set (golden dataset first).
  • Safety properties become non-negotiable rather than best-effort.
  • Mocked integration tests make agent flows testable, including error paths and gates that are hard to trigger in production.
  • Scoped eval runs keep CI fast enough that people don't route around it.

Cons

  • Eval runs cost money and time on every PR — mitigated by scoping and caching, but real.
  • Statistical layers are flaky by nature; a genuine regression and noise look similar on small sets (statistical hygiene).
  • Mocks drift from real tool behaviour; integration tests can pass while production breaks.
  • Red-team coverage is never complete and requires ongoing human effort.
  • Maintaining fixtures, mocks, and golden data is continuous work.

When to use

Every production AI system. The unit and integration layers especially — they're cheap, deterministic, and they carry the security invariants.

When NOT to use (heavy layers)

  • Prototypes: unit tests on the deterministic parts only; skip eval CI until there's something to regress.
  • Don't gate PRs on expensive full-suite evals if the scoped runs cover the change; run the full suite nightly.

Production at scale

The deterministic layer carries your security guarantees. ACL filter construction, tenant isolation, citation validation, budget enforcement, idempotency — these are ordinary code with ordinary tests, and they're where the leaks and double-sends actually get prevented. Prioritise them over elaborate eval infrastructure.

Scope eval runs by changed artefact or CI becomes slow enough that people bypass it. A mapping file from paths to suites, plus a nightly full run, is the working pattern.

Safety gates are separate and absolute: cross-tenant retrieval, PII in output, forbidden-tool calls, unapproved writes, injection-following. Any failure blocks, regardless of quality improvements elsewhere (prompt-injection defence, excessive agency, authentication & authorization).

Flakiness management: quarantine flaky eval cases rather than letting them erode trust in the gate. Re-run once on failure at temperature 0; if a case flips repeatedly, it's either a genuinely marginal case (move it out of the gate) or a real instability (fix it).

Cache aggressively: pinned retrieval results, cached prompt prefixes (prompt caching of static prefixes), and provider batch endpoints keep CI eval cost manageable.

Model upgrades run everything: full suite, red-team, prompt evals, tool-selection accuracy, schema violations. A model change alters every behaviour (versioning & rollout).

Red-team on a cadence and on every model change: injection corpora, jailbreak prompts, ACL probes, PII extraction attempts. Automate what you can; keep a manual exploratory session per release (prompt-injection defence).

Post-incident: every production incident adds a test at the lowest layer that could have caught it — usually a unit test, sometimes a golden case (data flywheel).

Prod checklist

  • Unit tests for ACL filters, chunk IDs, idempotency keys, citation validation, budget enforcement, schema validators
  • Integration tests with mocked LLM and tools covering error paths, approval gates, resume, and budgets
  • Artefact-to-suite mapping; scoped eval runs on PRs; full suite nightly
  • CI blocks on any previously-passing case that now fails
  • Separate absolute safety gate (ACL, PII, injection, forbidden tools, unapproved writes)
  • PR comment with metric diff, regressions, cost and latency delta
  • Flaky-case quarantine process
  • Full suite + red-team on every model/provider change
  • Every incident adds a test at the lowest layer that would have caught it

Metrics to watch

MetricWhy
CI eval runtime and cost per PRWhether people route around it
Regressions caught pre-merge vs incidents in productionGate effectiveness
Flaky-case rateTrust in the gate
Safety-gate failuresMust be zero at merge
Coverage of deterministic invariantsWhere the real guarantees live