Source ingestion → Tenant / ACL capture at ingest
What it is
Who is allowed to see this chunk? The only system that knows is the source: SharePoint's permission list, Confluence's space and page restrictions, Drive's sharing settings, the database's row-level policy. The connector is the only component that talks to the source. So the connector must capture the permission information and it must travel with the document through parse → chunk → embed → index, ending up as filterable metadata on every chunk.
Then at query time, the caller's identity (tenant + groups) becomes a hard filter on the ANN search (authentication & authorization). The model never sees a chunk the user can't see, so there is nothing for it to leak.
Two dimensions:
- Tenant isolation — hard boundary between customers/business units. Chunks from tenant A must never be candidates for tenant B's query. Enforce with a mandatory filter and, where possible, physical separation (namespace/collection per tenant).
- ACL within a tenant — groups/roles that may read a document. Enforce with a filter on
acl_groupsintersected with the caller's groups.
How to implement
- Schema: required fields
tenant_id,acl_groups: list[str],visibility: enum,owner,source_permission_ref(an opaque ID you can re-resolve). No defaults. - Connector resolves permissions to groups, not to user lists. Use the source's group model (AD/Entra groups, Google groups, Confluence groups). Expand "everyone in org" to a well-known group
tenant:<id>:all. - Normalise group IDs to a canonical form (
idp:<group_id>), never display names — names change, IDs don't. - Propagate through the pipeline unchanged: parser passes it through, chunker copies it onto every chunk, embedder ignores it, indexer writes it as payload and creates a payload index.
- Validate at index write: reject chunks with empty
tenant_idor emptyacl_groups. Fail the document loudly. - Query-time resolution: authn → caller identity → group memberships (from IdP, cached with short TTL) → filter
tenant_id == caller.tenant AND acl_groups ∩ caller.groups ≠ ∅. - ACL sync path: permission changes at the source often don't change content. Poll permission change feeds or re-fetch ACLs on schedule; update chunk payloads without re-embedding (change detection).
Small example
# Ingest side: connector → normalised ACL
def resolve_acl(sharepoint_item, graph_client, tenant_id) -> dict:
perms = graph_client.get(f"/drives/{sharepoint_item.drive}/items/{sharepoint_item.id}/permissions")
groups = set()
for p in perms["value"]:
if "grantedToV2" in p and "group" in p["grantedToV2"]:
groups.add(f"entra:{p['grantedToV2']['group']['id']}")
elif "grantedToV2" in p and "user" in p["grantedToV2"]:
groups.add(f"user:{p['grantedToV2']['user']['id']}") # rare; direct user grants
elif p.get("link", {}).get("scope") == "organization":
groups.add(f"tenant:{tenant_id}:all")
if not groups:
raise ValueError(f"no readable ACL for {sharepoint_item.id}") # never default to public
return {"tenant_id": tenant_id, "acl_groups": sorted(groups),
"visibility": "internal", "owner": sharepoint_item.created_by}
# Query side: mandatory filter built from the caller, never from the prompt
def build_filter(caller):
return {"must": [
{"key": "tenant_id", "match": {"value": caller.tenant_id}},
{"key": "is_current", "match": {"value": True}},
{"key": "acl_groups", "match": {"any": list(caller.groups | {f"user:{caller.id}", f"tenant:{caller.tenant_id}:all"})}},
]}
hits = vector_store.search(qvec, limit=30, query_filter=build_filter(caller))
Pros
- Security is structural, not behavioural. No prompt can talk the model into reading a chunk that was never retrieved.
- Filtering is cheap with payload indexes; ACL-aware retrieval adds milliseconds.
- Auditable: every chunk states who may read it; every query states who asked.
- Multi-tenant SaaS becomes possible on a shared index (with the right store).
Cons
- Permission models are messy. Inheritance, deny rules, link-sharing, external guests, nested groups. Perfect fidelity is hard; err on the side of more restrictive.
- ACL drift: source permissions change without content changes; you need a separate sync path.
- Group explosion: a document shared with 3,000 individual users has a 3,000-element ACL list; payload indexes struggle. Need a group-ID indirection.
- Retrofit pain is real: an index built without ACLs must be fully re-ingested; there's no shortcut.
When to use
Always, from the first document. Even a single-tenant internal tool has "HR only" and "everyone" documents.
When NOT to use
- Only if the entire corpus is genuinely public to all callers (public docs site). Even then, keep
tenant_idand setvisibility=public; scope creeps.
Production at scale
Physical vs logical tenant isolation. Logical (shared collection + mandatory filter) is cheaper and simpler; a single missed filter is a cross-tenant leak. Physical (collection/namespace per tenant, or per-tenant DB) is safer and lets you scale/tune per tenant; thousands of small tenants make it operationally heavy. Common pattern: physical isolation for large/regulated tenants, logical for the long tail — and the mandatory filter in both (defence in depth).
Filter at the store, never post-hoc. Post-filtering top-100 by ACL can return zero results for a user who can see 1% of the corpus (filtered search). Use native filtered ANN, and set k high enough or use the store's guaranteed-k filtered search.
Group-ID indirection for large ACLs. If the source grants to many individuals, create a synthetic group doc-acl:<hash of member set> per unique member set, store that ID on chunks, and maintain the membership map in a side table. Query-time: expand caller → groups → synthetic groups (cached). Payload stays small.
Caller group resolution. Cache IdP group lookups per user with a short TTL (1–5 min). Invalidate on explicit revocation events where available. Log the group set used for every query (auditability) — it's your audit answer to "why did they see that".
Deny rules. Most vector stores only do allow-list matching. Model a deny as removing the group from the allow list at ingest (resolve allow − deny at the connector), not as a query-time negation.
ACL sync SLA. Define it: "a permission revocation at the source is reflected in retrieval within X minutes". Poll change feeds (Graph delta, Drive changes) for sensitive sources; otherwise scheduled ACL re-fetch. Alert when ACL sync lags.
Never default open. The failure mode is always the same: a new connector, a new field, a migration script — and acl_groups is null. Null must mean nobody, enforced by the validator, and a nightly scan must find any chunk with empty security fields.
Testing. Permission tests are part of the eval suite: for each tenant fixture, a set of (caller, query) pairs with expected forbidden documents. Run in CI on every retrieval or connector change (the testing pyramid).
Observability. Per tenant: chunks with empty ACL (must be 0), ACL sync lag, filtered-search zero-result rate (a signal that filters are too restrictive or k too small), cross-tenant results in canary queries (must be 0; run synthetic canaries continuously).
Prod checklist
- Required
tenant_id,acl_groups,visibilityon every chunk; validator rejects empties - Connector resolves source permissions → canonical group IDs; deny resolved at ingest
- Synthetic group indirection for large member sets
- Mandatory filter built from caller identity in one shared retrieval function
- Native filtered ANN; no post-filtering
- Physical isolation for large/regulated tenants + logical filter everywhere
- ACL sync path independent of content changes, with SLA and alert
- Caller group cache with short TTL; group set logged per query
- Permission regression tests in CI; continuous cross-tenant canaries
- Nightly scan for empty security fields
Metrics to watch
| Metric | Why |
|---|---|
Chunks with empty tenant_id / acl_groups | Must be zero |
| ACL sync lag (source change → index) | Revocation SLA |
| Cross-tenant canary hits | Must be zero |
| Filtered zero-result rate | Over-restriction / k too small |
| ACL list size p95 | Need for indirection |
Related
- Connectors — where permissions are read
- Chunk metadata — the schema
- Filtered search — pre- vs post-filter
- AuthN/AuthZ — query-time enforcement
- Auditability