Testing & reliability → The testing pyramid
What it is
AI systems have deterministic parts and probabilistic parts, and they need different test strategies. The pyramid layers them by speed, determinism, and cost:
┌──────────────────────────┐
│ Red-team / adversarial │ slow, manual+automated, run on model changes and releases
├──────────────────────────┤
│ End-to-end evals │ minutes, statistical, pre-release + nightly
├──────────────────────────┤
│ Integration (mocked) │ seconds, deterministic, every PR
├──────────────────────────┤
│ Component evals │ seconds-minutes, per artefact changed
├──────────────────────────┤
│ Unit tests │ milliseconds, fully deterministic, every commit
└──────────────────────────┘
| Layer | What it tests | Determinism | Runs on |
|---|---|---|---|
| Unit | Parsers, chunkers, tokenizers, schema validators, filter builders, idempotency keys, retry logic | Fully deterministic | Every commit |
| Component evals | Parsing accuracy (test-time validation against a golden sample), chunk recall, retrieval metrics (retrieval evals), prompt behaviour (prompt evaluation) | Statistical, temp 0 | The artefact that changed |
| Integration | Agent flows with mocked tools, pipeline wiring, error paths, budget enforcement, approval gates | Deterministic with stubs | Every PR |
| End-to-end evals | Full system on the golden set (golden dataset first), with cost and latency | Statistical | Pre-release, nightly |
| Red-team | Prompt injection, jailbreaks, PII leakage, ACL bypass, excessive agency | Adversarial, partly manual | Model changes, releases, quarterly |
The rule from the source file: evals run in CI on every prompt/model/index change and block the merge on regression. Not "we run evals before big releases" — every change to a behaviour-affecting artefact.
How to implement
- Make the deterministic parts genuinely deterministic and test them hard: ACL filter construction, chunk ID generation, idempotency keys, citation validation, budget enforcement, schema validators. These carry the security and correctness invariants and cost nothing to test.
- Map artefacts to eval suites (component vs end-to-end):
prompts/**→ prompt evals;chunking/**→ chunk recall + retrieval;parsers/**→ parsing evals; model config → full suite. - Mock the LLM in integration tests with recorded or scripted responses so agent flows, error handling, and gates are tested deterministically and fast.
- Gate on regressions, not aggregates (prompt evaluation): any previously-passing case that now fails blocks the merge.
- Hard gates for safety properties: ACL leakage, forbidden-tool calls, unapproved writes, PII in output — any failure blocks regardless of other metrics.
- Nightly full suite including end-to-end with live retrieval, plus the red-team suite.
- Report a diff in the PR: metrics before/after, regressed cases, cost/latency delta.
Small example
# ============ Layer 1: unit — deterministic invariants, every commit ============
def test_acl_filter_never_empty():
caller = Caller(id="u1", tenant_id="t1", groups=set())
flt = build_acl_filter(caller)
assert any(c["key"] == "tenant_id" for c in flt["must"])
assert any(c["key"] == "acl_groups" for c in flt["must"]) # never absent, even with no groups
def test_chunk_ids_deterministic():
a = fixed_size_chunks("doc1", 3, TEXT); b = fixed_size_chunks("doc1", 3, TEXT)
assert [c["chunk_id"] for c in a] == [c["chunk_id"] for c in b] # re-runs upsert, never duplicate (fixed-size + overlap)
def test_idempotency_key_stable():
args = {"to": ["a@b.com"], "subject": "Hi", "body": "x"}
assert idem_key("run1", 3, "send_email", args) == idem_key("run1", 3, "send_email", dict(reversed(list(args.items()))))
def test_citation_validation_rejects_unretrieved_id():
out = {"claims": [{"text": "x", "citations": ["c_never_retrieved"]}]}
v = validate_citations(out, retrieved=[{"chunk_id": "c1", "text": "..."}])
assert v["citation_issues"] and v["claims"][0]["citations"] == []
def test_budget_blocks_before_execution():
b = Budget(max_steps=1); b.steps = 1
with pytest.raises(BudgetExceeded): b.check()
# ============ Layer 2: integration with mocked tools — every PR ============
class MockLLM:
def __init__(self, script): self.script, self.i = script, 0
def create(self, **kw):
r = self.script[self.i]; self.i += 1; return r
def test_agent_recovers_from_tool_error():
llm = MockLLM([tool_use("search_documents", {"query": "x"}),
tool_use("search_documents", {"query": "narrower x"}),
final_answer("Found it [c1]")])
tools = {"search_documents": lambda a, c, k: (_ for _ in ()).throw(TimeoutError())
if a["query"] == "x" else {"status": "ok", "passages": [{"id": "c1"}]}}
out = run_agent("q", CALLER, TOOL_DEFS, tools, SYS, llm=llm)
assert out["answer"].startswith("Found it") # error text fed back, agent recovered (single agent loop)
def test_write_tool_requires_approval():
llm = MockLLM([tool_use("send_email", {"to": ["a@b.com"], "subject": "s", "body": "b",
"idempotency_key": "k1"})])
with pytest.raises(PausedForApproval):
run_agent("send it", CALLER, TOOL_DEFS, TOOL_IMPLS, SYS, llm=llm) # Human-in-the-loop
def test_approval_resume_is_idempotent():
aid = start_and_pause(); on_approval_decision(aid, ApprovalStatus.APPROVED)
on_approval_decision(aid, ApprovalStatus.APPROVED) # duplicate webhook
assert sent_email_count() == 1 # State & durability
# ============ Layer 3: component + e2e evals — on the artefact that changed ============
# .github/workflows/eval.yml (pseudo)
"""
on: pull_request
jobs:
evals:
steps:
- id: changed
run: echo "areas=$(python ci/map_changes_to_suites.py)" >> $GITHUB_OUTPUT
- run: python -m evals.run --suites ${{ steps.changed.outputs.areas }} --compare-to main --fail-on-regression
- run: python -m evals.run --suite safety --fail-on-any # hard gate, always
- run: python ci/comment_diff.py # posts metric diff to the PR
"""
def ci_gate(report, safety_report):
if safety_report["failures"]:
raise SystemExit(f"BLOCKED (safety): {safety_report['failures']}") # never negotiable
if report["regressions"]:
raise SystemExit(f"BLOCKED: {len(report['regressions'])} regressions: {report['regressions'][:5]}")
print(f"PASS: {report['fixes']} fixes, cost delta {report['metrics']['cost_per_query']['delta']:+.4f}")
Pros
- Fast feedback where it's cheap: unit and integration tests catch most bugs in seconds.
- Regressions caught before users — the entire purpose of the golden set (golden dataset first).
- Safety properties become non-negotiable rather than best-effort.
- Mocked integration tests make agent flows testable, including error paths and gates that are hard to trigger in production.
- Scoped eval runs keep CI fast enough that people don't route around it.
Cons
- Eval runs cost money and time on every PR — mitigated by scoping and caching, but real.
- Statistical layers are flaky by nature; a genuine regression and noise look similar on small sets (statistical hygiene).
- Mocks drift from real tool behaviour; integration tests can pass while production breaks.
- Red-team coverage is never complete and requires ongoing human effort.
- Maintaining fixtures, mocks, and golden data is continuous work.
When to use
Every production AI system. The unit and integration layers especially — they're cheap, deterministic, and they carry the security invariants.
When NOT to use (heavy layers)
- Prototypes: unit tests on the deterministic parts only; skip eval CI until there's something to regress.
- Don't gate PRs on expensive full-suite evals if the scoped runs cover the change; run the full suite nightly.
Production at scale
The deterministic layer carries your security guarantees. ACL filter construction, tenant isolation, citation validation, budget enforcement, idempotency — these are ordinary code with ordinary tests, and they're where the leaks and double-sends actually get prevented. Prioritise them over elaborate eval infrastructure.
Scope eval runs by changed artefact or CI becomes slow enough that people bypass it. A mapping file from paths to suites, plus a nightly full run, is the working pattern.
Safety gates are separate and absolute: cross-tenant retrieval, PII in output, forbidden-tool calls, unapproved writes, injection-following. Any failure blocks, regardless of quality improvements elsewhere (prompt-injection defence, excessive agency, authentication & authorization).
Flakiness management: quarantine flaky eval cases rather than letting them erode trust in the gate. Re-run once on failure at temperature 0; if a case flips repeatedly, it's either a genuinely marginal case (move it out of the gate) or a real instability (fix it).
Cache aggressively: pinned retrieval results, cached prompt prefixes (prompt caching of static prefixes), and provider batch endpoints keep CI eval cost manageable.
Model upgrades run everything: full suite, red-team, prompt evals, tool-selection accuracy, schema violations. A model change alters every behaviour (versioning & rollout).
Red-team on a cadence and on every model change: injection corpora, jailbreak prompts, ACL probes, PII extraction attempts. Automate what you can; keep a manual exploratory session per release (prompt-injection defence).
Post-incident: every production incident adds a test at the lowest layer that could have caught it — usually a unit test, sometimes a golden case (data flywheel).
Prod checklist
- Unit tests for ACL filters, chunk IDs, idempotency keys, citation validation, budget enforcement, schema validators
- Integration tests with mocked LLM and tools covering error paths, approval gates, resume, and budgets
- Artefact-to-suite mapping; scoped eval runs on PRs; full suite nightly
- CI blocks on any previously-passing case that now fails
- Separate absolute safety gate (ACL, PII, injection, forbidden tools, unapproved writes)
- PR comment with metric diff, regressions, cost and latency delta
- Flaky-case quarantine process
- Full suite + red-team on every model/provider change
- Every incident adds a test at the lowest layer that would have caught it
Metrics to watch
| Metric | Why |
|---|---|
| CI eval runtime and cost per PR | Whether people route around it |
| Regressions caught pre-merge vs incidents in production | Gate effectiveness |
| Flaky-case rate | Trust in the gate |
| Safety-gate failures | Must be zero at merge |
| Coverage of deterministic invariants | Where the real guarantees live |
Related
- Golden dataset, Prompt evals, Component vs end-to-end
- Injection, Excessive agency — the red-team layer
- Prompt versioning — what triggers eval runs
- Versioning & rollout — the layer after CI
- Data flywheel — incidents become tests