Source ingestion → Tenant / ACL capture at ingest

RAG Access Control Explained: Capture Tenant and ACL at Ingest

What it is

Who is allowed to see this chunk? The only system that knows is the source: SharePoint's permission list, Confluence's space and page restrictions, Drive's sharing settings, the database's row-level policy. The connector is the only component that talks to the source. So the connector must capture the permission information and it must travel with the document through parse → chunk → embed → index, ending up as filterable metadata on every chunk.

Then at query time, the caller's identity (tenant + groups) becomes a hard filter on the ANN search (authentication & authorization). The model never sees a chunk the user can't see, so there is nothing for it to leak.

Two dimensions:

  • Tenant isolation — hard boundary between customers/business units. Chunks from tenant A must never be candidates for tenant B's query. Enforce with a mandatory filter and, where possible, physical separation (namespace/collection per tenant).
  • ACL within a tenant — groups/roles that may read a document. Enforce with a filter on acl_groups intersected with the caller's groups.

How to implement

  1. Schema: required fields tenant_id, acl_groups: list[str], visibility: enum, owner, source_permission_ref (an opaque ID you can re-resolve). No defaults.
  2. Connector resolves permissions to groups, not to user lists. Use the source's group model (AD/Entra groups, Google groups, Confluence groups). Expand "everyone in org" to a well-known group tenant:<id>:all.
  3. Normalise group IDs to a canonical form (idp:<group_id>), never display names — names change, IDs don't.
  4. Propagate through the pipeline unchanged: parser passes it through, chunker copies it onto every chunk, embedder ignores it, indexer writes it as payload and creates a payload index.
  5. Validate at index write: reject chunks with empty tenant_id or empty acl_groups. Fail the document loudly.
  6. Query-time resolution: authn → caller identity → group memberships (from IdP, cached with short TTL) → filter tenant_id == caller.tenant AND acl_groups ∩ caller.groups ≠ ∅.
  7. ACL sync path: permission changes at the source often don't change content. Poll permission change feeds or re-fetch ACLs on schedule; update chunk payloads without re-embedding (change detection).

Small example

# Ingest side: connector → normalised ACL
def resolve_acl(sharepoint_item, graph_client, tenant_id) -> dict:
    perms = graph_client.get(f"/drives/{sharepoint_item.drive}/items/{sharepoint_item.id}/permissions")
    groups = set()
    for p in perms["value"]:
        if "grantedToV2" in p and "group" in p["grantedToV2"]:
            groups.add(f"entra:{p['grantedToV2']['group']['id']}")
        elif "grantedToV2" in p and "user" in p["grantedToV2"]:
            groups.add(f"user:{p['grantedToV2']['user']['id']}")        # rare; direct user grants
        elif p.get("link", {}).get("scope") == "organization":
            groups.add(f"tenant:{tenant_id}:all")
    if not groups:
        raise ValueError(f"no readable ACL for {sharepoint_item.id}")  # never default to public
    return {"tenant_id": tenant_id, "acl_groups": sorted(groups),
            "visibility": "internal", "owner": sharepoint_item.created_by}

# Query side: mandatory filter built from the caller, never from the prompt
def build_filter(caller):
    return {"must": [
        {"key": "tenant_id",  "match": {"value": caller.tenant_id}},
        {"key": "is_current", "match": {"value": True}},
        {"key": "acl_groups", "match": {"any": list(caller.groups | {f"user:{caller.id}", f"tenant:{caller.tenant_id}:all"})}},
    ]}

hits = vector_store.search(qvec, limit=30, query_filter=build_filter(caller))

Pros

  • Security is structural, not behavioural. No prompt can talk the model into reading a chunk that was never retrieved.
  • Filtering is cheap with payload indexes; ACL-aware retrieval adds milliseconds.
  • Auditable: every chunk states who may read it; every query states who asked.
  • Multi-tenant SaaS becomes possible on a shared index (with the right store).

Cons

  • Permission models are messy. Inheritance, deny rules, link-sharing, external guests, nested groups. Perfect fidelity is hard; err on the side of more restrictive.
  • ACL drift: source permissions change without content changes; you need a separate sync path.
  • Group explosion: a document shared with 3,000 individual users has a 3,000-element ACL list; payload indexes struggle. Need a group-ID indirection.
  • Retrofit pain is real: an index built without ACLs must be fully re-ingested; there's no shortcut.

When to use

Always, from the first document. Even a single-tenant internal tool has "HR only" and "everyone" documents.

When NOT to use

  • Only if the entire corpus is genuinely public to all callers (public docs site). Even then, keep tenant_id and set visibility=public; scope creeps.

Production at scale

Physical vs logical tenant isolation. Logical (shared collection + mandatory filter) is cheaper and simpler; a single missed filter is a cross-tenant leak. Physical (collection/namespace per tenant, or per-tenant DB) is safer and lets you scale/tune per tenant; thousands of small tenants make it operationally heavy. Common pattern: physical isolation for large/regulated tenants, logical for the long tail — and the mandatory filter in both (defence in depth).

Filter at the store, never post-hoc. Post-filtering top-100 by ACL can return zero results for a user who can see 1% of the corpus (filtered search). Use native filtered ANN, and set k high enough or use the store's guaranteed-k filtered search.

Group-ID indirection for large ACLs. If the source grants to many individuals, create a synthetic group doc-acl:<hash of member set> per unique member set, store that ID on chunks, and maintain the membership map in a side table. Query-time: expand caller → groups → synthetic groups (cached). Payload stays small.

Caller group resolution. Cache IdP group lookups per user with a short TTL (1–5 min). Invalidate on explicit revocation events where available. Log the group set used for every query (auditability) — it's your audit answer to "why did they see that".

Deny rules. Most vector stores only do allow-list matching. Model a deny as removing the group from the allow list at ingest (resolve allow − deny at the connector), not as a query-time negation.

ACL sync SLA. Define it: "a permission revocation at the source is reflected in retrieval within X minutes". Poll change feeds (Graph delta, Drive changes) for sensitive sources; otherwise scheduled ACL re-fetch. Alert when ACL sync lags.

Never default open. The failure mode is always the same: a new connector, a new field, a migration script — and acl_groups is null. Null must mean nobody, enforced by the validator, and a nightly scan must find any chunk with empty security fields.

Testing. Permission tests are part of the eval suite: for each tenant fixture, a set of (caller, query) pairs with expected forbidden documents. Run in CI on every retrieval or connector change (the testing pyramid).

Observability. Per tenant: chunks with empty ACL (must be 0), ACL sync lag, filtered-search zero-result rate (a signal that filters are too restrictive or k too small), cross-tenant results in canary queries (must be 0; run synthetic canaries continuously).

Prod checklist

  • Required tenant_id, acl_groups, visibility on every chunk; validator rejects empties
  • Connector resolves source permissions → canonical group IDs; deny resolved at ingest
  • Synthetic group indirection for large member sets
  • Mandatory filter built from caller identity in one shared retrieval function
  • Native filtered ANN; no post-filtering
  • Physical isolation for large/regulated tenants + logical filter everywhere
  • ACL sync path independent of content changes, with SLA and alert
  • Caller group cache with short TTL; group set logged per query
  • Permission regression tests in CI; continuous cross-tenant canaries
  • Nightly scan for empty security fields

Metrics to watch

MetricWhy
Chunks with empty tenant_id / acl_groupsMust be zero
ACL sync lag (source change → index)Revocation SLA
Cross-tenant canary hitsMust be zero
Filtered zero-result rateOver-restriction / k too small
ACL list size p95Need for indirection