Foundations → Build vs buy (managed retrieval vs owning the pipeline)
What it is
Buy (managed): a service ingests your documents, parses, chunks, embeds, indexes, retrieves, and often generates, behind one API. You configure; you don't operate.
Build (own): you assemble connectors, parsers, chunkers, embedders, a vector/BM25 store, rerankers, and the generation layer from components (open-source and/or managed building blocks like a vector DB or an embedding API). You operate the pipeline.
The middle is where most production systems land: own the pipeline, buy the components — managed vector DB, hosted embeddings/rerankers, API models — but keep control of parsing, chunking, metadata, ACL, retrieval logic, and evaluation.
| Need | Managed end-to-end | Owned pipeline |
|---|---|---|
| Demo this week | ✅ | ❌ |
| Structure-aware/parent–child/contextual chunking | Limited / opaque | ✅ |
| Custom parsers (tables, formulas, VLM captioning) | Limited | ✅ |
| Hybrid search + cross-encoder rerank tuned to your data | Sometimes | ✅ |
| Per-chunk ACL from source permissions | Varies; often coarse | ✅ |
| Retrieval-level evals and debugging (which chunks, what scores) | Limited visibility | ✅ |
| Embedding-model/parser migrations under your control | ❌ (vendor's schedule) | ✅ |
| Ops burden | Low | High |
| Cost at scale | Per-query/per-doc pricing can dominate | Infra + people |
How to implement (the decision)
- Write down the non-negotiables: ACL fidelity, data residency, citation granularity, table/figure support, latency, cost per query at target volume, eval visibility.
- Prototype on managed to validate the use case and produce a first golden set (golden dataset first). This is cheap and fast.
- Run the golden set against managed: if retrieval recall and answer quality clear the bar and ACL/residency are satisfied, ship managed and revisit at scale.
- Identify the first blocker (usually one of: table questions, ACLs, cost per query, a specific source's parsing). Estimate the cost of owning that stage only — hybrid designs are common (own parsing/chunking, feed a managed index).
- When owning, buy the components: managed vector DB, hosted embeddings/reranker, API models; build parsing, chunking, metadata, retrieval logic, evals, and observability.
- Keep the interface stable (
retrieve(query, caller, filters) → chunks) so switching a component (or a vendor) doesn't touch the application.
Small example — a stable retrieval interface that hides the choice
from typing import Protocol
class Retriever(Protocol):
def retrieve(self, query: str, caller, k: int = 10, filters: dict | None = None) -> list[dict]: ...
class ManagedKBRetriever: # e.g. Bedrock Knowledge Base
def __init__(self, client, kb_id): self.c, self.kb = client, kb_id
def retrieve(self, query, caller, k=10, filters=None):
r = self.c.retrieve(knowledgeBaseId=self.kb, retrievalQuery={"text": query},
retrievalConfiguration={"vectorSearchConfiguration": {"numberOfResults": k,
"filter": {"equals": {"key": "tenant_id", "value": caller.tenant_id}}}})
return [{"chunk_id": x["metadata"].get("chunk_id"), "text": x["content"]["text"],
"score": x["score"], "source": x["location"]} for x in r["retrievalResults"]]
class OwnedHybridRetriever: # your pipeline: BM25 + vector + rerank + ACL
def __init__(self, vec, bm25, reranker): self.vec, self.bm25, self.rr = vec, bm25, reranker
def retrieve(self, query, caller, k=10, filters=None):
flt = build_acl_filter(caller) | (filters or {})
cands = rrf(self.vec.search(query, 50, flt), self.bm25.search(query, 50, flt))
return self.rr.rerank(query, cands)[:k]
# app code never changes:
def answer(q, caller, retriever: Retriever):
chunks = retriever.retrieve(q, caller, k=8)
return generate(q, chunks)
Pros
Buy: speed to first value; low ops; vendor handles scaling and upgrades; good enough for many internal tools. Build: full control of quality levers; ACL/residency fidelity; debuggability; cost control at volume; no vendor lock on the data model.
Cons
Buy: opaque chunking/retrieval; coarse or absent ACLs; limited eval visibility ("why did it retrieve that?"); pricing that scales with queries/docs; vendor-driven model changes; hard to fix table/figure failures. Build: months of engineering; ongoing ops (ingestion, index, upgrades, on-call); easy to build badly (no evals, no ACLs) and end up worse than managed.
When to buy
- Internal tools, low stakes, single tenant, mostly digital text documents, modest volume.
- Proof-of-value phase for any use case.
- Teams without ML/infra capacity.
When to build (own the pipeline)
- ACLs from source permissions must be exact (multi-tenant SaaS, regulated data).
- Tables, forms, scans, figures are where the answers live.
- Retrieval quality must be tuned (hybrid, rerank, parent–child, contextual) and measured at chunk level.
- Cost per query at scale exceeds owned infra + people.
- Residency/zero-retention constraints the managed service can't meet.
- You need to control migration timing (embedding model, parser).
Production at scale
Most enterprises end up hybrid. Own parsing, chunking, metadata, ACL, and evals (the quality and security levers); buy the vector DB, embeddings, reranker, and models (the commodity infrastructure). This captures most of the control at a fraction of the ops.
Exit costs. Managed services own your index format and often your chunk IDs. Keep your own document/chunk registry (change detection, metadata on every chunk) even when using a managed index, so migrating off is a re-index, not a re-ingest.
Eval visibility is the deciding factor more often than features. If you can't see which chunks were retrieved with what scores, you can't debug quality (observability & tracing). Some managed services expose this; verify before committing.
ACL fidelity test. Before choosing managed, build a permission test (tenant / ACL capture at ingest): tenant fixtures, callers, forbidden docs. Run it. Many managed KBs support metadata filters but not source-permission sync; you'd be building the sync anyway.
Cost model at target volume. Managed pricing (per query, per GB indexed, per document processed) vs owned (vector DB nodes, embedding calls, GPU for parsing, 1–3 engineers). Cross-over is often in the tens of thousands of queries/day or millions of documents; compute it for your numbers.
Vendor model changes. Managed services upgrade embedding/generation models on their schedule; that can shift quality overnight. Ask for version pinning; if unavailable, budget for continuous eval monitoring (online eval) to catch it.
Don't build without evals. An owned pipeline with no golden set and no retrieval metrics is worse than managed. The first artifact of "build" is the eval harness (golden dataset first, retrieval evals), not the vector DB.
Prod checklist
- Non-negotiables written down (ACL, residency, citations, tables, cost, eval visibility)
- Managed prototype + golden set built first
- Permission test and table-question test run against the managed option
- Cost cross-over computed at target volume
- If owning: stable
Retrieverinterface; buy commodity components; evals first - Own document/chunk registry regardless of index vendor
- Vendor model-change policy known; continuous eval monitoring in place
Metrics to watch
| Metric | Why |
|---|---|
| Golden-set retrieval recall / answer quality (managed vs owned) | The comparison |
| Permission-test pass rate | ACL fidelity |
| $/query and $/document at current and projected volume | Cross-over |
| Time-to-diagnose a bad answer | Eval visibility |
Related
- ACL capture — the usual first blocker
- Tables — the usual second blocker
- Chunking Chunking, retrieval & ranking Retrieval — the quality levers you gain by owning
- Golden dataset — the first thing to build either way
- Cost levers