Context engineering → Failure modes: poisoning, distraction, confusion, clash
What it is
| Failure | What happens | Typical source | Symptom |
|---|---|---|---|
| Poisoning | A wrong or fabricated fact enters the context (a hallucinated tool result, a bad summary, an injected instruction) and is treated as truth in later steps; summaries carry it forward | Hallucination early in an agent run; compaction (compaction / summarisation of long history) preserving an error; memory (memory security); injection (prompt-injection defence) | Agent confidently pursues a wrong plan; answers repeat a false premise across turns |
| Distraction | Irrelevant or marginally relevant content competes for attention; the model anchors on it or rambles | Large k, weak reranking, verbose tool output, long history | Answers cite the wrong passage; quality falls as context grows even though the right chunk is present |
| Confusion | Too many tools, too many similar options, contradictory instructions; the model picks the wrong tool or misuses arguments | Big tool catalogues; overlapping tool descriptions; stacked instructions from multiple layers | Wrong tool calls; schema errors; asking for clarification when it shouldn't |
| Clash | Two sources disagree (old vs new policy, two documents, tool vs memory) and nothing tells the model which wins | Versioning gaps (document versioning); multi-source retrieval; stale memory | Inconsistent answers across runs; silently picking one side |
These are context-engineering failures: the model behaved reasonably given a bad window.
How to implement (detect + prevent)
Poisoning
- Provenance on every fact (chunk ID, tool call ID, timestamp); the model is told to prefer sourced facts.
- Validate tool outputs before they enter the window (schema, plausibility) (structured outputs & tool use).
- Compaction keeps
facts_established[].source; on contradiction, supersede (compaction / summarisation of long history, memory conflict resolution). - Faithfulness checks on intermediate conclusions in long runs (faithfulness check), not just the final answer.
- Reset: when a run goes off the rails, restart the subagent with a clean window (context isolation) rather than "correcting" in place.
Distraction
- Rerank + small k + relevance-ordered placement (reranking, "Lost in the middle"); score thresholds (score thresholding & abstention).
- Per-source budgets (per-source truncation budgets); compaction of history (compaction / summarisation of long history).
- Measure quality vs context size on the golden set and stop adding.
Confusion
- Fewer, well-named, non-overlapping tools; explicit "when not to use" in descriptions (tool design).
- Tool routing: expose only the tools relevant to the current step/subagent (context isolation).
- One instruction source of truth; avoid stacking contradictory rules from system, tenant, and user layers.
Clash
- Version filters so only current documents are retrieved (document versioning).
- Dates and source authority in metadata; instruct the model to surface conflicts with dates rather than pick (conflict handling).
- Memory conflict resolution with
valid_from(memory conflict resolution). - Detect contradictions with a cheap NLI pass over the top-k when the query is policy/date-sensitive.
Small example
# 1) Provenance-tagged facts + supersede-on-contradiction (poisoning / clash)
def add_fact(state, fact, source, ts):
for f in state["facts"]:
if same_subject(f["fact"], fact) and f["fact"] != fact:
f["superseded_by"] = source; f["active"] = False # don't delete: keep the audit trail
state["facts"].append({"fact": fact, "source": source, "ts": ts, "active": True})
# 2) Contradiction detection over top-k (clash) using a small NLI model
from transformers import pipeline
nli = pipeline("text-classification", model="cross-encoder/nli-deberta-v3-small")
def find_clashes(chunks, max_pairs=15):
clashes = []
for i in range(len(chunks)):
for j in range(i + 1, min(len(chunks), i + 4)):
r = nli(f"{chunks[i]['text']} </s></s> {chunks[j]['text']}", truncation=True)[0]
if r["label"].lower() == "contradiction" and r["score"] > 0.8:
clashes.append((chunks[i]["id"], chunks[j]["id"], r["score"]))
if len(clashes) >= max_pairs: return clashes
return clashes
def build_with_clash_note(chunks, question):
clashes = find_clashes(chunks)
note = ""
if clashes:
pairs = "; ".join(f"[{a}] vs [{b}]" for a, b, _ in clashes)
note = f"\nNote: these passages appear to conflict: {pairs}. If relevant, present both with their dates/sources instead of choosing."
return build_body(chunks, question) + note
# 3) Tool exposure per step (confusion)
def tools_for_step(step_kind):
return {"research": [SEARCH, READ], "write": [DRAFT, SAVE_DRAFT], "send": [SEND_EMAIL]}[step_kind] # never all 14 at once
Pros (of naming and handling them)
- Targeted fixes: each failure has a specific lever; "make the prompt better" doesn't.
- Monitorable: each has a proxy metric.
- Safer agents: poisoning and confusion are the mechanisms behind most "the agent did something weird" incidents.
Cons
- Detection costs: NLI passes, faithfulness checks, and provenance bookkeeping add latency/tokens.
- False positives: contradiction detectors flag paraphrases; too-aggressive tool hiding blocks legitimate paths.
- Requires discipline across ingestion (versioning), retrieval (rerank), and agents (contracts) — it's a systems property, not one component.
When to use
- Every agent system (poisoning, confusion) and every multi-source RAG system (distraction, clash).
- After incidents where "the model was given the right information and still failed" — usually one of these four.
When NOT to use (heavy detectors)
- Single-source, short-context Q&A: distraction and clash are rare; don't add NLI passes there.
Production at scale
Proxy metrics per failure mode (observability & tracing):
- Poisoning: faithfulness failures on intermediate steps; rate of facts without provenance; agent runs restarted.
- Distraction: quality vs number of included chunks; citation rank distribution; tool-output tokens per step.
- Confusion: wrong-tool rate (judge or human sample); schema violations by tool; clarification requests.
- Clash: contradiction-detector hit rate; answers that changed between runs on the same query; version-filter bypasses (should be 0).
Triage buckets (feedback capture & triage): add these four to the failure taxonomy; they route to context-assembly work rather than prompt or model work.
Design rules that prevent most of it: version-filtered retrieval; rerank + small k; per-source budgets; structured compaction with provenance; tool exposure per step; subagent isolation; supersede-not-append for facts.
Agent restart policy: after N consecutive failed verifications or a detected poisoned fact, restart from the last good checkpoint (state & durability) with a clean context rather than continuing.
Injection is poisoning with intent — the same provenance and "data not instructions" defences apply (prompt-injection defence), plus policy gates on actions (excessive agency).
Prod checklist
- Provenance on facts, tool results, and summary entries; supersede-on-contradiction
- Version-filtered retrieval; dates/authority in metadata; conflict-surfacing instruction
- Rerank + capped k + budgets + compaction (distraction controls)
- Tool exposure per step/subagent; non-overlapping tool descriptions
- Contradiction detection for date/policy-sensitive queries
- Intermediate faithfulness checks in long agent runs; restart-from-checkpoint policy
- Proxy metrics per failure mode; failure taxonomy updated
Metrics to watch
| Metric | Why |
|---|---|
| Intermediate faithfulness failure rate (agents) | Poisoning |
| Quality vs context size; citation rank | Distraction |
| Wrong-tool rate; tool schema violations | Confusion |
| Contradiction hits; cross-run answer variance | Clash |
Related
- Compaction, Ordering, Budgets, Isolation — the controls
- Conflict handling, Memory conflicts — clash
- Tool design — confusion
- Injection — poisoning with intent
- Triage