Foundations → Build vs buy (managed retrieval vs owning the pipeline)

Build vs buy — managed knowledge base or your own RAG pipeline

What it is

Buy (managed): a service ingests your documents, parses, chunks, embeds, indexes, retrieves, and often generates, behind one API. You configure; you don't operate.

Build (own): you assemble connectors, parsers, chunkers, embedders, a vector/BM25 store, rerankers, and the generation layer from components (open-source and/or managed building blocks like a vector DB or an embedding API). You operate the pipeline.

The middle is where most production systems land: own the pipeline, buy the components — managed vector DB, hosted embeddings/rerankers, API models — but keep control of parsing, chunking, metadata, ACL, retrieval logic, and evaluation.

NeedManaged end-to-endOwned pipeline
Demo this week✅❌
Structure-aware/parent–child/contextual chunkingLimited / opaque✅
Custom parsers (tables, formulas, VLM captioning)Limited✅
Hybrid search + cross-encoder rerank tuned to your dataSometimes✅
Per-chunk ACL from source permissionsVaries; often coarse✅
Retrieval-level evals and debugging (which chunks, what scores)Limited visibility✅
Embedding-model/parser migrations under your control❌ (vendor's schedule)✅
Ops burdenLowHigh
Cost at scalePer-query/per-doc pricing can dominateInfra + people

How to implement (the decision)

  1. Write down the non-negotiables: ACL fidelity, data residency, citation granularity, table/figure support, latency, cost per query at target volume, eval visibility.
  2. Prototype on managed to validate the use case and produce a first golden set (golden dataset first). This is cheap and fast.
  3. Run the golden set against managed: if retrieval recall and answer quality clear the bar and ACL/residency are satisfied, ship managed and revisit at scale.
  4. Identify the first blocker (usually one of: table questions, ACLs, cost per query, a specific source's parsing). Estimate the cost of owning that stage only — hybrid designs are common (own parsing/chunking, feed a managed index).
  5. When owning, buy the components: managed vector DB, hosted embeddings/reranker, API models; build parsing, chunking, metadata, retrieval logic, evals, and observability.
  6. Keep the interface stable (retrieve(query, caller, filters) → chunks) so switching a component (or a vendor) doesn't touch the application.

Small example — a stable retrieval interface that hides the choice

from typing import Protocol

class Retriever(Protocol):
    def retrieve(self, query: str, caller, k: int = 10, filters: dict | None = None) -> list[dict]: ...

class ManagedKBRetriever:                       # e.g. Bedrock Knowledge Base
    def __init__(self, client, kb_id): self.c, self.kb = client, kb_id
    def retrieve(self, query, caller, k=10, filters=None):
        r = self.c.retrieve(knowledgeBaseId=self.kb, retrievalQuery={"text": query},
                            retrievalConfiguration={"vectorSearchConfiguration": {"numberOfResults": k,
                                "filter": {"equals": {"key": "tenant_id", "value": caller.tenant_id}}}})
        return [{"chunk_id": x["metadata"].get("chunk_id"), "text": x["content"]["text"],
                 "score": x["score"], "source": x["location"]} for x in r["retrievalResults"]]

class OwnedHybridRetriever:                     # your pipeline: BM25 + vector + rerank + ACL
    def __init__(self, vec, bm25, reranker): self.vec, self.bm25, self.rr = vec, bm25, reranker
    def retrieve(self, query, caller, k=10, filters=None):
        flt = build_acl_filter(caller) | (filters or {})
        cands = rrf(self.vec.search(query, 50, flt), self.bm25.search(query, 50, flt))
        return self.rr.rerank(query, cands)[:k]

# app code never changes:
def answer(q, caller, retriever: Retriever):
    chunks = retriever.retrieve(q, caller, k=8)
    return generate(q, chunks)

Pros

Buy: speed to first value; low ops; vendor handles scaling and upgrades; good enough for many internal tools. Build: full control of quality levers; ACL/residency fidelity; debuggability; cost control at volume; no vendor lock on the data model.

Cons

Buy: opaque chunking/retrieval; coarse or absent ACLs; limited eval visibility ("why did it retrieve that?"); pricing that scales with queries/docs; vendor-driven model changes; hard to fix table/figure failures. Build: months of engineering; ongoing ops (ingestion, index, upgrades, on-call); easy to build badly (no evals, no ACLs) and end up worse than managed.

When to buy

  • Internal tools, low stakes, single tenant, mostly digital text documents, modest volume.
  • Proof-of-value phase for any use case.
  • Teams without ML/infra capacity.

When to build (own the pipeline)

  • ACLs from source permissions must be exact (multi-tenant SaaS, regulated data).
  • Tables, forms, scans, figures are where the answers live.
  • Retrieval quality must be tuned (hybrid, rerank, parent–child, contextual) and measured at chunk level.
  • Cost per query at scale exceeds owned infra + people.
  • Residency/zero-retention constraints the managed service can't meet.
  • You need to control migration timing (embedding model, parser).

Production at scale

Most enterprises end up hybrid. Own parsing, chunking, metadata, ACL, and evals (the quality and security levers); buy the vector DB, embeddings, reranker, and models (the commodity infrastructure). This captures most of the control at a fraction of the ops.

Exit costs. Managed services own your index format and often your chunk IDs. Keep your own document/chunk registry (change detection, metadata on every chunk) even when using a managed index, so migrating off is a re-index, not a re-ingest.

Eval visibility is the deciding factor more often than features. If you can't see which chunks were retrieved with what scores, you can't debug quality (observability & tracing). Some managed services expose this; verify before committing.

ACL fidelity test. Before choosing managed, build a permission test (tenant / ACL capture at ingest): tenant fixtures, callers, forbidden docs. Run it. Many managed KBs support metadata filters but not source-permission sync; you'd be building the sync anyway.

Cost model at target volume. Managed pricing (per query, per GB indexed, per document processed) vs owned (vector DB nodes, embedding calls, GPU for parsing, 1–3 engineers). Cross-over is often in the tens of thousands of queries/day or millions of documents; compute it for your numbers.

Vendor model changes. Managed services upgrade embedding/generation models on their schedule; that can shift quality overnight. Ask for version pinning; if unavailable, budget for continuous eval monitoring (online eval) to catch it.

Don't build without evals. An owned pipeline with no golden set and no retrieval metrics is worse than managed. The first artifact of "build" is the eval harness (golden dataset first, retrieval evals), not the vector DB.

Prod checklist

  • Non-negotiables written down (ACL, residency, citations, tables, cost, eval visibility)
  • Managed prototype + golden set built first
  • Permission test and table-question test run against the managed option
  • Cost cross-over computed at target volume
  • If owning: stable Retriever interface; buy commodity components; evals first
  • Own document/chunk registry regardless of index vendor
  • Vendor model-change policy known; continuous eval monitoring in place

Metrics to watch

MetricWhy
Golden-set retrieval recall / answer quality (managed vs owned)The comparison
Permission-test pass rateACL fidelity
$/query and $/document at current and projected volumeCross-over
Time-to-diagnose a bad answerEval visibility