Source ingestion → Connectors

RAG Data Connectors Explained: ACLs, Cursors, Deletions

What it is

A connector is the adapter between an external system and your ingestion pipeline. Its job: enumerate the documents that exist, fetch their bytes and metadata (owner, permissions, modified time, URL), and hand them to the parser — reliably, incrementally, and without getting your API keys banned.

Each source has its own quirks:

SourceAuthEnumerationGotchas
S3 / GCSIAM roleListObjectsV2 paginatedMillions of keys; use inventory reports for big buckets
SharePoint / OneDriveOAuth app (Graph API)Delta queriesThrottling (429) is aggressive; permissions are per item
ConfluenceAPI token / OAuthCQL search + paginationPage bodies are storage-format XHTML; attachments separate
Gmail / DriveOAuth per user + domain-wide delegationchanges.list with page tokensPer-user quotas; attachments; shared drives have different ACLs
DatabasesService accountIncremental on updated_at / CDCRow = doc? Joins to build a "document"
Web crawlNone / cookiesSitemap + link graphrobots.txt, dedupe, JS rendering, politeness

How to implement

  1. One interface, many implementations. list_changes(since_cursor) → iterator[DocRef], fetch(doc_ref) → RawDoc(bytes, mime, metadata, acl). Every connector implements the same two methods.
  2. Cursor-based incremental sync. Store a per-source cursor (delta token, updated_at watermark, change-feed offset). Full scans are the bootstrap path only.
  3. Capture ACL and tenant at fetch time (tenant / ACL capture at ingest). The connector is the only place that knows the source's permission model.
  4. Rate limiting + retry. Token bucket per source; exponential backoff with jitter on 429/5xx; honour Retry-After. Circuit-break when error rate spikes.
  5. Pagination that survives restarts. Persist the page token after each page; resume from it.
  6. Emit RawDoc to a queue, not directly into the parser. Decouples fetch throughput from parse throughput.
  7. Dead-letter queue for documents that fail fetch after N retries; alert, don't drop silently.

Small example

from dataclasses import dataclass
from typing import Iterator
import time, random, requests

@dataclass
class DocRef:
    source: str; external_id: str; modified_at: str; etag: str; url: str

@dataclass
class RawDoc:
    ref: DocRef; bytes: bytes; mime: str; acl_groups: list[str]; tenant_id: str

class Connector:
    def list_changes(self, cursor: str | None) -> tuple[Iterator[DocRef], str]: ...
    def fetch(self, ref: DocRef) -> RawDoc: ...

def with_backoff(fn, max_tries=6):
    for attempt in range(max_tries):
        try:
            return fn()
        except requests.HTTPError as e:
            if e.response.status_code in (429, 500, 502, 503, 504) and attempt < max_tries - 1:
                wait = float(e.response.headers.get("Retry-After", 0)) or (2 ** attempt + random.random())
                time.sleep(wait); continue
            raise

class ConfluenceConnector(Connector):
    def __init__(self, base, token, tenant_id):
        self.s = requests.Session(); self.s.headers["Authorization"] = f"Bearer {token}"
        self.base, self.tenant = base, tenant_id

    def list_changes(self, cursor):
        since = cursor or "1970-01-01"
        start, refs = 0, []
        while True:
            r = with_backoff(lambda: self.s.get(f"{self.base}/rest/api/content/search",
                params={"cql": f'lastmodified >= "{since}" order by lastmodified', "start": start, "limit": 100}))
            r.raise_for_status(); data = r.json()
            for p in data["results"]:
                refs.append(DocRef("confluence", p["id"], p["version"]["when"], str(p["version"]["number"]),
                                   self.base + p["_links"]["webui"]))
            if "next" not in data["_links"]: break
            start += 100
        new_cursor = max((r.modified_at for r in refs), default=since)
        return iter(refs), new_cursor

    def fetch(self, ref):
        r = with_backoff(lambda: self.s.get(f"{self.base}/rest/api/content/{ref.external_id}",
                                            params={"expand": "body.storage,restrictions.read"}))
        r.raise_for_status(); j = r.json()
        groups = [g["name"] for g in j["restrictions"]["read"]["restrictions"]["group"]["results"]] or ["all-employees"]
        return RawDoc(ref, j["body"]["storage"]["value"].encode(), "text/html", groups, self.tenant)

Pros

  • Uniform interface lets the rest of the pipeline ignore where documents came from.
  • Incremental cursors make daily sync cheap: only changed documents flow.
  • ACL capture at source is the only correct place to get permissions.
  • Queue decoupling lets fetch and parse scale independently.

Cons

  • Every source is a snowflake. Auth flows expire, APIs change, throttling rules differ. Connectors are perpetual maintenance.
  • Deletions are hard to detect incrementally on many sources (no "deleted since" API); you need periodic full reconciliation.
  • Permission models don't map cleanly (SharePoint inheritance, Drive shared-with-link, Confluence space vs page restrictions).
  • Rate limits cap ingestion speed regardless of how much compute you have.

When to use

Always — but choose buy vs build per source. Managed connector frameworks (Airbyte, Unstructured platform, LlamaCloud, Glean-style products, cloud-provider knowledge-base connectors) cover the common SaaS sources; build your own for internal databases, proprietary systems, and anything where you need custom ACL capture.

When NOT to use (build your own)

  • When a managed connector already covers the source and exposes ACLs and change cursors. Building a SharePoint connector from scratch is months of edge cases.
  • Don't build a generic web crawler for a handful of known URLs; a sitemap fetch is enough.

Production at scale

Bootstrap vs steady state are different jobs. The initial full load of 10M documents runs for days, needs parallelism and checkpointing (state & durability), and can saturate source APIs. Steady-state delta sync runs every few minutes on a few hundred changes. Design both paths; run the bootstrap with lower priority and a rate cap so it doesn't starve the delta sync.

Cursors are per source and per tenant. In multi-tenant systems each tenant's Confluence has its own token, quota, and cursor. Store (tenant, source, cursor, last_success_at) and alert when last_success_at ages beyond SLA.

Token/credential lifecycle. OAuth refresh tokens expire, get revoked, or the admin leaves. Monitor auth failures per tenant, notify the tenant admin automatically, and keep ingesting other tenants. One expired token must never stall the fleet.

Deletion reconciliation. Nightly (or weekly) list all IDs from the source, diff against the index, soft-delete the missing ones (document versioning). Without this, deleted documents stay searchable — a privacy incident.

Permission drift. ACL changes usually don't bump modified_at. Poll permission-change feeds where they exist (Graph delta includes permission changes; Drive has permissions in changes), otherwise re-fetch ACLs on a schedule for sensitive sources.

Throughput engineering. Async I/O (aiohttp/httpx) with a per-source concurrency semaphore; a token bucket calibrated to the documented quota minus margin. Track 429 rate; if it climbs, back off globally for that source.

Large objects. Stream to object storage; never hold a 2 GB PDF in memory. Pass a storage reference to the parser, not bytes. Set size limits and route oversize files to a special path.

Idempotency. Fetching the same document twice must produce the same RawDoc and the same downstream IDs. Combine with change detection (change detection) so duplicates are cheap.

Observability. Per (tenant, source): documents listed, fetched, failed, DLQ size, cursor age, 429 count, bytes fetched, p95 fetch latency. A cursor that hasn't advanced in 24h is an alert.

Security. Connector credentials in a secrets manager, scoped read-only, rotated. Connectors run in a network segment that can reach sources but not the LLM/serving tier.

Prod checklist

  • Common Connector interface; per-source implementation or managed connector
  • Incremental cursor persisted per (tenant, source); bootstrap path separate
  • ACL + tenant captured on every RawDoc
  • Backoff + jitter + Retry-After; per-source token bucket; circuit breaker
  • Queue between fetch and parse; DLQ with alerting
  • Deletion reconciliation job
  • Permission-change polling for sensitive sources
  • Credential rotation + auth-failure alerts per tenant
  • Cursor-age and 429-rate dashboards

Metrics to watch

MetricWhy
Cursor age per (tenant, source)Stalled sync
Fetch error rate, 429 rateQuota health
DLQ depthSilent data loss
Docs/hour during bootstrapCapacity planning
Reconciliation deletes per runDeletion coverage