Reference architecture (end to end)
What it is
┌──────────────── Offline / batch ─────────────────┐
sources ──► ingest ──► parse ──► chunk ──► embed ──► index (vector + BM25 + graph)
(S3, SP, dedupe, route parent/ version metadata + ACL + tenant
Confluence, hash, per page child, pinned
DB, web) ACLs validate contextual
┌──────────────── Online / request ────────────────┐
user ──► authn/z ──► guardrails(in) ──► query understanding ──► route
(rewrite, multi-query, │
decompose, filters) │
▼
hybrid retrieve (BM25 + vector, ACL-filtered)
│
▼
rerank (cross-encoder) ──► top-k
│
▼
context assembly (budget, order, dedupe, cite)
│
▼
LLM / agent loop (tools, HITL gates)
│
▼
grounding check ──► guardrails(out) ──► stream to user
│
▼
traces, cost, feedback ──► eval set ──► flywheel
Every box maps to a file:
| Stage | Files |
|---|---|
| Ingest | Connectors connectors · change detection change detection · deduplication dedupe · document versioning versioning · ACL |
| Parse | Per-page classification and routing routing · OCR / digitization parsers · structure extraction structure · validation validation · bounded escalation with visible failure retry ladder |
| Chunk | Chunking |
| Embed | Embeddings |
| Index | Indexing & vector stores |
| AuthN/Z + guardrails in | Authentication & authorization · input guardrails |
| Query understanding + route | Query understanding |
| Retrieve + rerank | Retrieval & ranking |
| Context assembly | Context engineering |
| Generate / agent | Prompt engineering · generation & grounding · agents · memory |
| Grounding + guardrails out | Faithfulness check · output guardrails |
| Observe + evaluate | Observability & tracing · evaluation |
| Operate | Production, cost & latency · failure handling · security, privacy & governance |
| Improve | Data flywheel · fine-tuning |
How to implement (build order)
- Golden set first (golden dataset first) — before any pipeline code. It defines what "correct" means and gates everything after.
- Thin vertical slice: one connector → one parser → recursive chunking → embeddings → a vector index → simple retrieval → a grounded prompt with citations. Measure on the golden set.
- Security from day one: tenant and ACL capture at ingest (tenant / ACL capture at ingest), permission-filtered retrieval (authentication & authorization). Retrofitting is a re-ingest.
- Add the high-ROI quality layers in order: hybrid search (hybrid search) → reranking (reranking) → parent–child chunking (parent–child) → query rewriting (query rewriting) → abstention (score thresholding & abstention).
- Add the production layers: observability (observability & tracing), budgets (per-source truncation budgets, cost & step budgets), guardrails (input guardrails, output guardrails), failure handling (failure handling), caching (prompt caching of static prefixes).
- Close the loop: online signals (online eval) → triage (feedback capture & triage) → golden cases (data flywheel) → CI gates (the testing pyramid) → canary (versioning & rollout).
- Only then consider agents (agents), advanced RAG (advanced RAG patterns), and fine-tuning (fine-tuning) — each justified by a measured gap.
Small example — the online path, assembled
def handle_request(raw_query: str, request, cfg) -> dict:
trace_id = new_trace_id()
with span("app.request", trace_id=trace_id, **version_vector(cfg)): # Observability & tracing
caller = authenticate(request) # Authentication & authorization
guard = input_guardrails(raw_query, caller, policy_for(caller)) # Input guardrails
if not guard.allow:
return refusal(guard.reason, trace_id)
understanding = understand_query(guard.masked_text, caller, history()) # Query rewriting/.4/.5/.6
if understanding.route != "vector":
return HANDLERS[understanding.route](understanding, caller, trace_id) # Routing, structured + unstructured hybrid
budget = Budget(max_tokens=cfg["budgets"]["tokens"], max_usd=cfg["budgets"]["usd"],
max_seconds=cfg["budgets"]["seconds"]) # Cost & step budgets
pool = hybrid_retrieve(understanding.query, caller, # Hybrid search (ACL-filtered, authentication & authorization)
k=cfg["retrieval"]["pool_k"], filters=understanding.filters)
ranked = rerank(understanding.query, pool, top_n=cfg["retrieval"]["final_k"]) # Reranking
ranked = mmr_select(ranked, understanding.query_vec, k=cfg["retrieval"]["final_k"]) # MMR / diversity
conf = retrieval_confidence(ranked) # Score thresholding & abstention
if conf["top"] < cfg["thresholds"]["tau_low"]:
return refuse(RefusalReason.NO_RELEVANT, {...}, trace_id) # Refusal / no-answer path
parents = collapse_to_parents(ranked) # Parent–child
memory = retrieve_memory(caller, understanding.task_type, understanding.query) # When to retrieve memory
context, meta = assemble(understanding.query, parents, memory, history(), # Context engineering
budget=cfg["budgets"]["context_tokens"])
out = generate_grounded(understanding.query, context, budget=budget) # Prompt structure, grounded prompting
out = validate_citations(out, parents) # Citations
out = faithfulness_gate(out, parents) # Faithfulness check
verdict = output_guardrails(out, parents, caller, policy_for(caller)) # Output guardrails
record_outcome(trace_id, out, ranked, meta, budget) # Observability & tracing → online eval → data flywheel
audit.record("ai.query_answered", caller, understanding, out, ranked) # Auditability
return {"answer": verdict.text, "citations": render_citations(out, parents),
"trace_id": trace_id, "degraded": out.get("degraded", False)}
Pros
- Every stage has a named owner, a metric, and a failure mode — the architecture is also the operations model.
- Each layer is independently evaluable (component vs end-to-end), so problems localise.
- Offline/online separation lets ingestion scale independently of serving.
- The flywheel is part of the architecture, not an afterthought, which is what makes the system improve.
Cons
- It's a lot of machinery — a naive prototype is five components, this is thirty. Build incrementally against measured gaps.
- Every stage is a failure point; without observability the composite is hard to debug.
- Cost and latency accumulate across stages; each needs a budget.
- Over-engineering is the common failure: teams build advanced RAG patterns before measuring whether hybrid + rerank suffices.
When to use
As the map for any document-grounded AI system. Use it to locate what you're building, and what you're missing.
When NOT to use (all of it)
- A prototype needs the thin slice, not the full diagram.
- Not every system needs agents, graphs, multimodal retrieval, or fine-tuning — each is justified by a measured gap, not by the diagram's completeness.
Production at scale
The order of investment is not the order of the diagram. After a working thin slice, the highest-return additions are consistently: reranking (reranking), hybrid search (hybrid search), parent–child chunking (parent–child), and abstention (score thresholding & abstention). Advanced RAG patterns (advanced RAG patterns), agents (agents), and fine-tuning (fine-tuning) come far later, and only when evals show the specific gap they close.
Security and versioning cannot be retrofitted cheaply. ACL capture at ingest (tenant / ACL capture at ingest) and document versioning (document versioning) touch every chunk; adding them later means re-ingesting the corpus. Build them into the thin slice.
Observability is a prerequisite, not a phase. Without retrieved chunk IDs and scores in the trace (observability & tracing), every debugging session is guesswork and the flywheel cannot start.
The offline path usually costs more than the online path in document-heavy systems: parsing, embedding, and contextual enrichment dominate. Change detection (change detection) and dedupe (deduplication) are the largest cost levers most teams haven't pulled (cost levers).
Budgets at every stage: context tokens (per-source truncation budgets), agent steps and dollars (cost & step budgets), per-tenant quotas (capacity & limits). Unbudgeted stages are where cost incidents originate.
Each stage needs a degraded mode (failure handling): reranker down → fusion order; dense down → BM25; generation down → return sources. The composite availability of ten dependencies without degradation is poor.
Keep the version vector everywhere (versioning & rollout, observability & tracing) — it's what makes a thirty-component system debuggable at all.
Prod checklist
- Golden set exists before the pipeline
- Thin vertical slice working end to end, measured
- ACL capture and document versioning built in from the start
- Hybrid + rerank + parent–child + abstention added and measured in that order
- Observability with chunk IDs, scores, and the version vector on every request
- Budgets at every stage; per-tenant quotas
- Degraded mode defined for each stage
- CI eval gates and canary rollout wired
- Flywheel running: signals → triage → golden cases → fixes
- Advanced components (agents, graph, multimodal, fine-tuning) justified by measured gaps
Metrics to watch
| Metric | Why |
|---|---|
| Per-stage latency and cost | Where the budget goes |
| Failure-bucket histogram (feedback capture & triage) | Which stage to invest in |
| End-to-end quality on the golden set | The headline |
| Degraded-response rate by stage | Composite reliability |
| Ingest cost per 1k documents | The usually-dominant line item |
Related
Every file in this reference maps to a stage in the diagram above; see the table in What it is.