Context engineering → Failure modes: poisoning, distraction, confusion, clash

LLM Context Failure Modes: Poisoning, Distraction, Confusion, Clash

What it is

FailureWhat happensTypical sourceSymptom
PoisoningA wrong or fabricated fact enters the context (a hallucinated tool result, a bad summary, an injected instruction) and is treated as truth in later steps; summaries carry it forwardHallucination early in an agent run; compaction (compaction / summarisation of long history) preserving an error; memory (memory security); injection (prompt-injection defence)Agent confidently pursues a wrong plan; answers repeat a false premise across turns
DistractionIrrelevant or marginally relevant content competes for attention; the model anchors on it or ramblesLarge k, weak reranking, verbose tool output, long historyAnswers cite the wrong passage; quality falls as context grows even though the right chunk is present
ConfusionToo many tools, too many similar options, contradictory instructions; the model picks the wrong tool or misuses argumentsBig tool catalogues; overlapping tool descriptions; stacked instructions from multiple layersWrong tool calls; schema errors; asking for clarification when it shouldn't
ClashTwo sources disagree (old vs new policy, two documents, tool vs memory) and nothing tells the model which winsVersioning gaps (document versioning); multi-source retrieval; stale memoryInconsistent answers across runs; silently picking one side

These are context-engineering failures: the model behaved reasonably given a bad window.

How to implement (detect + prevent)

Poisoning

Distraction

Confusion

  • Fewer, well-named, non-overlapping tools; explicit "when not to use" in descriptions (tool design).
  • Tool routing: expose only the tools relevant to the current step/subagent (context isolation).
  • One instruction source of truth; avoid stacking contradictory rules from system, tenant, and user layers.

Clash

  • Version filters so only current documents are retrieved (document versioning).
  • Dates and source authority in metadata; instruct the model to surface conflicts with dates rather than pick (conflict handling).
  • Memory conflict resolution with valid_from (memory conflict resolution).
  • Detect contradictions with a cheap NLI pass over the top-k when the query is policy/date-sensitive.

Small example

# 1) Provenance-tagged facts + supersede-on-contradiction (poisoning / clash)
def add_fact(state, fact, source, ts):
    for f in state["facts"]:
        if same_subject(f["fact"], fact) and f["fact"] != fact:
            f["superseded_by"] = source; f["active"] = False           # don't delete: keep the audit trail
    state["facts"].append({"fact": fact, "source": source, "ts": ts, "active": True})

# 2) Contradiction detection over top-k (clash) using a small NLI model
from transformers import pipeline
nli = pipeline("text-classification", model="cross-encoder/nli-deberta-v3-small")

def find_clashes(chunks, max_pairs=15):
    clashes = []
    for i in range(len(chunks)):
        for j in range(i + 1, min(len(chunks), i + 4)):
            r = nli(f"{chunks[i]['text']} </s></s> {chunks[j]['text']}", truncation=True)[0]
            if r["label"].lower() == "contradiction" and r["score"] > 0.8:
                clashes.append((chunks[i]["id"], chunks[j]["id"], r["score"]))
            if len(clashes) >= max_pairs: return clashes
    return clashes

def build_with_clash_note(chunks, question):
    clashes = find_clashes(chunks)
    note = ""
    if clashes:
        pairs = "; ".join(f"[{a}] vs [{b}]" for a, b, _ in clashes)
        note = f"\nNote: these passages appear to conflict: {pairs}. If relevant, present both with their dates/sources instead of choosing."
    return build_body(chunks, question) + note

# 3) Tool exposure per step (confusion)
def tools_for_step(step_kind):
    return {"research": [SEARCH, READ], "write": [DRAFT, SAVE_DRAFT], "send": [SEND_EMAIL]}[step_kind]   # never all 14 at once

Pros (of naming and handling them)

  • Targeted fixes: each failure has a specific lever; "make the prompt better" doesn't.
  • Monitorable: each has a proxy metric.
  • Safer agents: poisoning and confusion are the mechanisms behind most "the agent did something weird" incidents.

Cons

  • Detection costs: NLI passes, faithfulness checks, and provenance bookkeeping add latency/tokens.
  • False positives: contradiction detectors flag paraphrases; too-aggressive tool hiding blocks legitimate paths.
  • Requires discipline across ingestion (versioning), retrieval (rerank), and agents (contracts) — it's a systems property, not one component.

When to use

  • Every agent system (poisoning, confusion) and every multi-source RAG system (distraction, clash).
  • After incidents where "the model was given the right information and still failed" — usually one of these four.

When NOT to use (heavy detectors)

  • Single-source, short-context Q&A: distraction and clash are rare; don't add NLI passes there.

Production at scale

Proxy metrics per failure mode (observability & tracing):

  • Poisoning: faithfulness failures on intermediate steps; rate of facts without provenance; agent runs restarted.
  • Distraction: quality vs number of included chunks; citation rank distribution; tool-output tokens per step.
  • Confusion: wrong-tool rate (judge or human sample); schema violations by tool; clarification requests.
  • Clash: contradiction-detector hit rate; answers that changed between runs on the same query; version-filter bypasses (should be 0).

Triage buckets (feedback capture & triage): add these four to the failure taxonomy; they route to context-assembly work rather than prompt or model work.

Design rules that prevent most of it: version-filtered retrieval; rerank + small k; per-source budgets; structured compaction with provenance; tool exposure per step; subagent isolation; supersede-not-append for facts.

Agent restart policy: after N consecutive failed verifications or a detected poisoned fact, restart from the last good checkpoint (state & durability) with a clean context rather than continuing.

Injection is poisoning with intent — the same provenance and "data not instructions" defences apply (prompt-injection defence), plus policy gates on actions (excessive agency).

Prod checklist

  • Provenance on facts, tool results, and summary entries; supersede-on-contradiction
  • Version-filtered retrieval; dates/authority in metadata; conflict-surfacing instruction
  • Rerank + capped k + budgets + compaction (distraction controls)
  • Tool exposure per step/subagent; non-overlapping tool descriptions
  • Contradiction detection for date/policy-sensitive queries
  • Intermediate faithfulness checks in long agent runs; restart-from-checkpoint policy
  • Proxy metrics per failure mode; failure taxonomy updated

Metrics to watch

MetricWhy
Intermediate faithfulness failure rate (agents)Poisoning
Quality vs context size; citation rankDistraction
Wrong-tool rate; tool schema violationsConfusion
Contradiction hits; cross-run answer varianceClash