Reference architecture (end to end)

RAG Reference Architecture Explained: Two Pipelines and a Loop

What it is

                          ┌──────────────── Offline / batch ─────────────────┐
 sources ──► ingest ──► parse ──► chunk ──► embed ──► index (vector + BM25 + graph)
 (S3, SP,     dedupe,   route     parent/   version    metadata + ACL + tenant
  Confluence, hash,     per page  child,    pinned
  DB, web)    ACLs      validate  contextual

                          ┌──────────────── Online / request ────────────────┐
 user ──► authn/z ──► guardrails(in) ──► query understanding ──► route
                                          (rewrite, multi-query,   │
                                           decompose, filters)     │
                                                                   ▼
                                       hybrid retrieve (BM25 + vector, ACL-filtered)
                                                    │
                                                    ▼
                                          rerank (cross-encoder) ──► top-k
                                                    │
                                                    ▼
                              context assembly (budget, order, dedupe, cite)
                                                    │
                                                    ▼
                                    LLM / agent loop (tools, HITL gates)
                                                    │
                                                    ▼
                             grounding check ──► guardrails(out) ──► stream to user
                                                    │
                                                    ▼
                              traces, cost, feedback ──► eval set ──► flywheel

Every box maps to a file:

How to implement (build order)

  1. Golden set first (golden dataset first) — before any pipeline code. It defines what "correct" means and gates everything after.
  2. Thin vertical slice: one connector → one parser → recursive chunking → embeddings → a vector index → simple retrieval → a grounded prompt with citations. Measure on the golden set.
  3. Security from day one: tenant and ACL capture at ingest (tenant / ACL capture at ingest), permission-filtered retrieval (authentication & authorization). Retrofitting is a re-ingest.
  4. Add the high-ROI quality layers in order: hybrid search (hybrid search) → reranking (reranking) → parent–child chunking (parent–child) → query rewriting (query rewriting) → abstention (score thresholding & abstention).
  5. Add the production layers: observability (observability & tracing), budgets (per-source truncation budgets, cost & step budgets), guardrails (input guardrails, output guardrails), failure handling (failure handling), caching (prompt caching of static prefixes).
  6. Close the loop: online signals (online eval) → triage (feedback capture & triage) → golden cases (data flywheel) → CI gates (the testing pyramid) → canary (versioning & rollout).
  7. Only then consider agents (agents), advanced RAG (advanced RAG patterns), and fine-tuning (fine-tuning) — each justified by a measured gap.

Small example — the online path, assembled

def handle_request(raw_query: str, request, cfg) -> dict:
    trace_id = new_trace_id()
    with span("app.request", trace_id=trace_id, **version_vector(cfg)):          # Observability & tracing

        caller = authenticate(request)                                            # Authentication & authorization
        guard = input_guardrails(raw_query, caller, policy_for(caller))           # Input guardrails
        if not guard.allow:
            return refusal(guard.reason, trace_id)

        understanding = understand_query(guard.masked_text, caller, history())    # Query rewriting/.4/.5/.6
        if understanding.route != "vector":
            return HANDLERS[understanding.route](understanding, caller, trace_id) # Routing, structured + unstructured hybrid

        budget = Budget(max_tokens=cfg["budgets"]["tokens"], max_usd=cfg["budgets"]["usd"],
                        max_seconds=cfg["budgets"]["seconds"])                    # Cost & step budgets

        pool = hybrid_retrieve(understanding.query, caller,                        # Hybrid search (ACL-filtered, authentication & authorization)
                               k=cfg["retrieval"]["pool_k"], filters=understanding.filters)
        ranked = rerank(understanding.query, pool, top_n=cfg["retrieval"]["final_k"])  # Reranking
        ranked = mmr_select(ranked, understanding.query_vec, k=cfg["retrieval"]["final_k"])  # MMR / diversity

        conf = retrieval_confidence(ranked)                                        # Score thresholding & abstention
        if conf["top"] < cfg["thresholds"]["tau_low"]:
            return refuse(RefusalReason.NO_RELEVANT, {...}, trace_id)              # Refusal / no-answer path

        parents = collapse_to_parents(ranked)                                      # Parent–child
        memory = retrieve_memory(caller, understanding.task_type, understanding.query)  # When to retrieve memory
        context, meta = assemble(understanding.query, parents, memory, history(),  # Context engineering
                                 budget=cfg["budgets"]["context_tokens"])

        out = generate_grounded(understanding.query, context, budget=budget)       # Prompt structure, grounded prompting
        out = validate_citations(out, parents)                                     # Citations
        out = faithfulness_gate(out, parents)                                      # Faithfulness check
        verdict = output_guardrails(out, parents, caller, policy_for(caller))      # Output guardrails

        record_outcome(trace_id, out, ranked, meta, budget)                        # Observability & tracing → online eval → data flywheel
        audit.record("ai.query_answered", caller, understanding, out, ranked)      # Auditability
        return {"answer": verdict.text, "citations": render_citations(out, parents),
                "trace_id": trace_id, "degraded": out.get("degraded", False)}

Pros

  • Every stage has a named owner, a metric, and a failure mode — the architecture is also the operations model.
  • Each layer is independently evaluable (component vs end-to-end), so problems localise.
  • Offline/online separation lets ingestion scale independently of serving.
  • The flywheel is part of the architecture, not an afterthought, which is what makes the system improve.

Cons

  • It's a lot of machinery — a naive prototype is five components, this is thirty. Build incrementally against measured gaps.
  • Every stage is a failure point; without observability the composite is hard to debug.
  • Cost and latency accumulate across stages; each needs a budget.
  • Over-engineering is the common failure: teams build advanced RAG patterns before measuring whether hybrid + rerank suffices.

When to use

As the map for any document-grounded AI system. Use it to locate what you're building, and what you're missing.

When NOT to use (all of it)

  • A prototype needs the thin slice, not the full diagram.
  • Not every system needs agents, graphs, multimodal retrieval, or fine-tuning — each is justified by a measured gap, not by the diagram's completeness.

Production at scale

The order of investment is not the order of the diagram. After a working thin slice, the highest-return additions are consistently: reranking (reranking), hybrid search (hybrid search), parent–child chunking (parent–child), and abstention (score thresholding & abstention). Advanced RAG patterns (advanced RAG patterns), agents (agents), and fine-tuning (fine-tuning) come far later, and only when evals show the specific gap they close.

Security and versioning cannot be retrofitted cheaply. ACL capture at ingest (tenant / ACL capture at ingest) and document versioning (document versioning) touch every chunk; adding them later means re-ingesting the corpus. Build them into the thin slice.

Observability is a prerequisite, not a phase. Without retrieved chunk IDs and scores in the trace (observability & tracing), every debugging session is guesswork and the flywheel cannot start.

The offline path usually costs more than the online path in document-heavy systems: parsing, embedding, and contextual enrichment dominate. Change detection (change detection) and dedupe (deduplication) are the largest cost levers most teams haven't pulled (cost levers).

Budgets at every stage: context tokens (per-source truncation budgets), agent steps and dollars (cost & step budgets), per-tenant quotas (capacity & limits). Unbudgeted stages are where cost incidents originate.

Each stage needs a degraded mode (failure handling): reranker down → fusion order; dense down → BM25; generation down → return sources. The composite availability of ten dependencies without degradation is poor.

Keep the version vector everywhere (versioning & rollout, observability & tracing) — it's what makes a thirty-component system debuggable at all.

Prod checklist

  • Golden set exists before the pipeline
  • Thin vertical slice working end to end, measured
  • ACL capture and document versioning built in from the start
  • Hybrid + rerank + parent–child + abstention added and measured in that order
  • Observability with chunk IDs, scores, and the version vector on every request
  • Budgets at every stage; per-tenant quotas
  • Degraded mode defined for each stage
  • CI eval gates and canary rollout wired
  • Flywheel running: signals → triage → golden cases → fixes
  • Advanced components (agents, graph, multimodal, fine-tuning) justified by measured gaps

Metrics to watch

MetricWhy
Per-stage latency and costWhere the budget goes
Failure-bucket histogram (feedback capture & triage)Which stage to invest in
End-to-end quality on the golden setThe headline
Degraded-response rate by stageComposite reliability
Ingest cost per 1k documentsThe usually-dominant line item

Every file in this reference maps to a stage in the diagram above; see the table in What it is.