Evaluation → Component vs end-to-end
What it is
A RAG answer passes through six or more stages, each of which can be the failure point:
parse → chunk → embed/index → retrieve → rerank → assemble → generate
If end-to-end accuracy drops from 82% to 74%, the end-to-end number tells you nothing about which stage moved. Component evals localise it:
| Stage | Metric | Failure signature |
|---|---|---|
| Parse | CER/WER, table-cell accuracy, reading order (test-time validation against a golden sample) | Text garbled, tables flattened, headings lost |
| Chunk | Chunk-level recall on golden queries | Right doc retrieved, answer span not in any chunk |
| Embed/index | ANN recall vs flat (index types: Flat, HNSW, IVF-PQ) | Retrieval degrades after an index or model change |
| Retrieve | recall@pool_k (retrieval evals) | Answer never enters the candidate pool |
| Rerank | nDCG@final_k (retrieval evals) | Answer in the pool but ranked below the cut |
| Assemble | Truncation/drop rate of the top chunk (context engineering) | Right chunk retrieved, then evicted by budget |
| Generate | Faithfulness, correctness, behaviour (generation evals) | Context sufficient, answer still wrong |
The practical form is a diagnostic ladder: run the cheap component checks in order and stop at the first one that fails. Most "the model is bad" reports resolve at stage 1 or 4.
How to implement
- One golden set, many views. The same cases (golden dataset first) carry ground truth for several stages: expected parsed text, relevant spans, relevant chunks, expected answer.
- Component harnesses that can run a stage in isolation with fixed inputs: parse a known PDF, chunk known text, retrieve with a pinned index, generate with pinned context.
- Pin everything upstream when testing a stage, so the stage under test is the only variable.
- Run the ladder on every eval: parse → chunk → retrieve → rerank → assemble → generate, producing a diagnosis line naming the first failing stage.
- End-to-end run in the same job, with cost and latency, as the release gate.
- On production incidents, run the ladder against the failing query to localise before debugging.
- Wire component gates into CI for the artefacts that change independently: parser version, chunking config, embedding model, prompt (the testing pyramid).
Small example
def diagnostic_ladder(case, system, artefacts) -> dict:
"""Run cheap component checks in order; report the first failure."""
report, caller = {}, load_caller(case.caller)
# 1. Parse: is the answer text even present in the parsed corpus?
parsed = artefacts.parsed_text(case.source_doc_id)
report["parse_contains_answer"] = normalise(case.answer_span) in normalise(parsed)
if not report["parse_contains_answer"]:
return report | {"diagnosis": "PARSE: answer text absent from parsed document (validation)"}
# 2. Chunk: is the answer span wholly inside a single chunk?
chunks = artefacts.chunks_for(case.source_doc_id)
report["chunk_contains_span"] = any(normalise(case.answer_span) in normalise(ch["text"]) for ch in chunks)
if not report["chunk_contains_span"]:
return report | {"diagnosis": "CHUNK: answer span split across chunks (chunking) — try parent-child"}
# 3. Retrieve: does the answer chunk enter the candidate pool?
pool = system.retrieve_pool(case.query, caller, k=50)
pool_ids = [h["chunk_id"] for h in pool]
report["recall@50"] = recall_at_k(pool_ids, case.relevant_chunk_ids, 50)
if (report["recall@50"] or 0) < 1.0:
return report | {"diagnosis": "RETRIEVE: answer not in the pool (hybrid search, embeddings, query understanding)"}
# 4. Rerank: does it survive to the final k?
final = system.rerank(case.query, pool)[:6]
final_ids = [h["chunk_id"] for h in final]
report["recall@6"] = recall_at_k(final_ids, case.relevant_chunk_ids, 6)
if (report["recall@6"] or 0) < 1.0:
return report | {"diagnosis": "RERANK: answer in pool but ranked out (reranking)"}
# 5. Assemble: does it survive the context budget?
assembled, meta = system.assemble(case.query, final, caller)
report["survived_assembly"] = any(cid in assembled for cid in case.relevant_chunk_ids)
if not report["survived_assembly"]:
return report | {"diagnosis": f"ASSEMBLE: chunk dropped/truncated by budget (context engineering/03.4): {meta}"}
# 6. Generate: context was sufficient — the failure is generation
out = system.generate(case.query, assembled)
report["faithfulness"] = faithfulness_score(out, final)
report["correct"] = judge(case, out)["overall"] >= 4
if not report["correct"]:
return report | {"diagnosis": "GENERATE: context sufficient but answer wrong (grounded prompting, prompt engineering)"}
return report | {"diagnosis": "PASS"}
def aggregate_diagnoses(cases, system, artefacts):
"""Where is the system actually losing? This histogram drives the roadmap."""
from collections import Counter
diagnoses = [diagnostic_ladder(c, system, artefacts)["diagnosis"].split(":")[0] for c in cases]
return Counter(diagnoses).most_common()
# e.g. [("PASS", 61), ("RETRIEVE", 14), ("GENERATE", 12), ("CHUNK", 8), ("PARSE", 5)]
Pros
- Localises failures in minutes instead of days of speculative debugging.
- The diagnosis histogram is a roadmap: it tells you exactly where to spend engineering effort.
- Component checks are cheap and deterministic — parse, chunk, and retrieval checks need no LLM calls.
- Prevents fixing the wrong thing — the most common waste in RAG projects is prompt-tuning a retrieval failure.
- Each component can be gated independently in CI when its artefact changes.
Cons
- Requires richer ground truth: answer spans, expected chunks, expected parsed text — more labelling than end-to-end alone.
- Component harnesses are code to maintain, with fixtures and pinned artefacts.
- Components can pass individually and fail in combination — interaction effects (a chunking change that's fine in isolation but interacts badly with the reranker) need the end-to-end run.
- Pinning upstream stages can mask variance the live system has.
When to use
Always, as the standard eval structure. The ladder costs little once ground truth exists and pays back on the first incident.
When NOT to use (component-only)
- Never rely on component evals alone: every component passing does not guarantee the assembled system works. Always run end-to-end as the release gate.
Production at scale
The diagnosis histogram is the single most useful artefact this reference produces for planning. Run it monthly; the distribution tells you whether to invest in parsing, chunking, retrieval, or generation. Teams routinely discover that 20% of failures are parsing problems they never suspected (validation).
Ground truth at the span level (golden dataset first, retrieval evals) is what makes the chunk-level check possible. Label "the answer is in document D, characters 4120–4260"; everything else derives from it and survives re-chunking.
Pin artefacts by version — parser version, chunking config, embedding model, index snapshot (versioning & migration). A component eval that silently ran against a different index is worse than no eval.
Gate per artefact in CI (the testing pyramid): a parser change runs parsing evals; a chunking change runs chunk-recall; an embedding change runs retrieval evals; a prompt change runs generation evals. Full ladder nightly and pre-release.
Interaction effects need the end-to-end pass. Keep a small set of cases that always run fully, with live retrieval and no pinning, to catch combinations component tests miss.
Incident workflow: when a user reports a bad answer, run the ladder on that exact query and caller. The diagnosis line goes in the incident record, and the case goes into the golden set (data flywheel). This converts every complaint into a localised, permanently-tested fix.
Attribution across versions: log the version vector (parser, chunker, embedder, index, prompt, model) with every eval result so a regression can be bisected to the component that changed (prompt versioning & evaluation, observability & tracing).
Prod checklist
- Ground truth includes parsed text, answer spans, relevant chunks, and expected answers
- Component harnesses with pinned, versioned upstream artefacts
- Diagnostic ladder producing a first-failing-stage diagnosis per case
- Diagnosis histogram produced and reviewed regularly
- Per-artefact CI gates; full ladder nightly and pre-release
- End-to-end unpinned pass to catch interaction effects
- Incident workflow runs the ladder on the failing query; result recorded
- Version vector logged with every eval result for bisection
Metrics to watch
| Metric | Why |
|---|---|
| Diagnosis histogram (failures by stage) | Where to invest |
| Per-stage pass rates over time | Component health trends |
| End-to-end vs sum-of-components divergence | Interaction effects |
| Time from incident to localised stage | Debugging efficiency |
Related
- Golden dataset — span-level ground truth
- Validation Parsing validation, Retrieval evals, Generation evals
- Testing pyramid — per-artefact gates
- Triage — the diagnosis histogram feeds bucket assignment
- Observability — version vectors