Generation & grounding → Conflict handling (when sources disagree)
What it is
Real corpora contradict themselves constantly:
| Conflict type | Example | Right resolution |
|---|---|---|
| Temporal | 2022 policy says 10 days, 2024 policy says 15 | Newest wins — but only if versions are correctly dated |
| Jurisdictional | German entity vs US entity leave policy | Both are correct; the question needs scoping |
| Authority | Signed contract vs a draft vs an internal FAQ | System of record wins; others are noted |
| Genuine inconsistency | Two current policies that disagree | Surface it — this is a finding, not a nuisance |
| Apparent | Same fact, different phrasing or units | Not a conflict; don't flag it |
If the pipeline doesn't handle this, the model picks whichever passage it attended to most — which means the answer depends on retrieval order, and the same question answered twice can differ. That's worse than either answer, because it destroys trust in both.
The primary defence is upstream: version filtering (document versioning) removes superseded documents from the candidate pool entirely. Conflict handling at generation time deals with what version filtering can't: jurisdictional scope, authority hierarchies, and genuine corpus inconsistency.
How to implement
- Fix it upstream first. Only current versions in the index (document versioning); dates, jurisdiction, entity, and document authority as metadata on every chunk (metadata on every chunk). Most "conflicts" are stale documents that should never have been retrieved.
- Surface metadata in the context block: every passage carries
effective_date,source,authority_tier, and scope fields, visible to the model (delimiters: instructions vs data). - Detect conflicts among top-k for sensitive query classes: a cheap NLI contradiction pass over pairs (failure modes: poisoning, distraction, confusion, clash), or ask the generator to report conflicts as a structured field.
- Instruct the resolution policy explicitly in the prompt: prefer the most recent effective date; prefer the higher authority tier; if scopes differ (country, entity, employee class), present both scoped; if a genuine conflict remains, present both with dates and sources and say they disagree.
- Structured output:
{"answer", "conflicts": [{"claim_a", "source_a", "date_a", "claim_b", "source_b", "date_b", "resolution": "recency"|"authority"|"scope"|"unresolved"}]}. - Render conflicts visibly — a comparison, not a footnote. "The 2024 policy says 15 days; the 2022 version said 10."
- Route unresolved conflicts into a data-quality queue: a genuine contradiction between two current documents is a corpus bug someone should fix (feedback capture & triage).
Small example
CONFLICT_SYS = """Answer from the passages. Each passage has an effective_date, source, and authority tier.
Conflict rules, in order:
1. If passages differ only in scope (country, entity, employee class), they are not in conflict — answer per scope and say which applies to what.
2. If passages state different values for the same scope, prefer the most recent effective_date, and say that an earlier version stated otherwise.
3. If dates are equal or missing, prefer the higher authority tier (contract > policy > FAQ > email).
4. If you cannot resolve it, present both with their dates and sources and state plainly that the sources disagree. Never pick silently.
Output JSON:
{"answer": string,
"claims": [{"text": string, "citations": [string]}],
"conflicts": [{"summary": string, "a": {"cid": string, "date": string, "value": string},
"b": {"cid": string, "date": string, "value": string},
"resolution": "recency"|"authority"|"scope"|"unresolved"}]}"""
def render_with_provenance(chunks):
return "\n".join(
f'<passage id="{c["chunk_id"]}" source="{html.escape(c["title"])}" '
f'effective_date="{c.get("effective_date","unknown")}" '
f'authority="{c.get("authority_tier","policy")}" '
f'scope="{c.get("scope","global")}">{html.escape(c["text"], quote=False)}</passage>'
for c in chunks)
def detect_contradictions(chunks, max_pairs=12):
pairs, out = [], []
for i in range(len(chunks)):
for j in range(i + 1, min(len(chunks), i + 4)):
pairs.append((i, j))
for i, j in pairs[:max_pairs]:
r = nli(f"{chunks[i]['text'][:900]} [SEP] {chunks[j]['text'][:900]}", truncation=True)[0]
if "contradict" in r["label"].lower() and r["score"] > 0.8:
out.append({"a": chunks[i]["chunk_id"], "b": chunks[j]["chunk_id"], "score": float(r["score"])})
return out
def answer_with_conflicts(question, chunks):
detected = detect_contradictions(chunks)
note = ""
if detected:
note = "\nNote: these passages may conflict: " + "; ".join(f"[{d['a']}] vs [{d['b']}]" for d in detected)
out = json.loads(llm("answer_conflicts", CONFLICT_SYS, [{"role": "user", "content":
f"<context>\n{render_with_provenance(chunks)}\n</context>\n<question>{question}</question>{note}"}],
max_tokens=1200).text)
for c in out.get("conflicts", []):
log_conflict(question, c)
if c["resolution"] == "unresolved":
data_quality_queue.push({"question": question, "conflict": c, "detected_at": now()})
return out
Pros
- Deterministic answers: the resolution rule, not retrieval order, decides — so the same question gets the same answer.
- Honest: users see that the corpus disagrees, which is usually information they need.
- Surfaces real data-quality problems that would otherwise stay hidden behind a plausible answer.
- Scope handling prevents a whole class of wrong answers (German employee told the US policy).
- Auditable: the resolution rule and the losing source are both recorded.
Cons
- Metadata dependency: recency and authority resolution require reliable
effective_dateand authority tiers. Missing or wrong dates make the rule actively harmful — you'll confidently prefer the wrong document. - Contradiction detection is noisy: paraphrases and different granularity get flagged; too many false conflicts make answers hedged and unreadable.
- Verbose answers when conflicts are surfaced; users sometimes just want the number.
- Cost: NLI pairs or a longer generation prompt.
- Authority tiers are a judgement call that must be maintained per corpus.
When to use
- Corpora with versioned policies, multiple jurisdictions, or multiple entities — HR, legal, compliance, finance.
- Any system where documents from different sources (contract, wiki, email) cover the same topic.
- Regulated domains where presenting a superseded rule as current is a compliance issue.
When NOT to use
- Single-source-of-truth corpora with strict versioning where conflicts genuinely can't occur — rely on the version filter.
- Low-stakes conversational surfaces where hedging hurts more than the rare conflict helps.
- Don't run pairwise NLI on every query; gate it to policy/date-sensitive query classes.
Production at scale
Version filtering is the real fix. Before building conflict handling, verify that superseded versions are actually excluded (document versioning). Most conflicts in production RAG are stale chunks that failed to be marked non-current — a versioning bug wearing a conflict-handling costume.
Metadata quality gates the whole feature. Measure effective_date coverage per source. If it's 60%, recency-based resolution will pick wrong on a large fraction. Fix ingestion before shipping the rule.
Authority tiers as config, per corpus: an ordered list (contract > signed_policy > policy > handbook > faq > email > chat) attached at ingest from the source and document type. Version it; changing it changes answers.
Scope is not conflict — and it's the most common resolution. Encode scope dimensions (country, legal entity, employee class, product line) as metadata and let self-query (self-query) filter on them when the user's scope is known. A user profile that supplies scope automatically eliminates most apparent conflicts before generation.
Gate detection by query class. Run pairwise contradiction detection only for date- or policy-sensitive queries (from routing, routing). Pairwise NLI over top-8 is 28 pairs; cap and prioritise adjacent pairs.
Unresolved conflicts are tickets. Route them to a data-quality queue with the question, the two chunk IDs, and the detection score. Closing these is a corpus improvement with compounding value (data flywheel). Track the queue's age.
Answer format matters. Lead with the resolved answer, then note the disagreement briefly: "15 days [c22, effective 2024-01-01]. Note: an earlier 2022 version stated 10 days [c9]." Burying the answer under a discussion of conflict is a worse experience than picking wrong.
Cross-run consistency as a metric: run the same golden questions repeatedly and measure answer variance. High variance on questions with known conflicts means the resolution rule isn't firing.
Prod checklist
- Version filtering verified — superseded chunks are not retrievable
-
effective_date,authority_tier, and scope fields on every chunk; coverage monitored per source - Authority tier ordering in versioned config
- Scope-based filtering from user profile where available (eliminates apparent conflicts)
- Contradiction detection gated to sensitive query classes; pair count capped
- Explicit ordered resolution rules in the prompt; structured
conflictsoutput - Answer leads with the resolution, notes the disagreement briefly
- Unresolved conflicts routed to a data-quality queue with ownership and age tracking
- Cross-run answer variance measured on conflict-bearing golden questions
Metrics to watch
| Metric | Why |
|---|---|
| Conflicts detected per 1k queries, by resolution type | Corpus health |
| Unresolved-conflict queue size and age | Data-quality debt |
effective_date coverage per source | Whether recency resolution is safe |
| Cross-run answer variance on conflict questions | Determinism |
| Stale-version retrieval rate (should be ~0) | Upstream bug detection |
Related
- Document versioning — the upstream fix
- Chunk metadata — dates, authority, scope
- Context clash — the failure mode this addresses
- Self-query — scope filtering
- Triage — where unresolved conflicts go