Source ingestion → Connectors
What it is
A connector is the adapter between an external system and your ingestion pipeline. Its job: enumerate the documents that exist, fetch their bytes and metadata (owner, permissions, modified time, URL), and hand them to the parser — reliably, incrementally, and without getting your API keys banned.
Each source has its own quirks:
| Source | Auth | Enumeration | Gotchas |
|---|---|---|---|
| S3 / GCS | IAM role | ListObjectsV2 paginated | Millions of keys; use inventory reports for big buckets |
| SharePoint / OneDrive | OAuth app (Graph API) | Delta queries | Throttling (429) is aggressive; permissions are per item |
| Confluence | API token / OAuth | CQL search + pagination | Page bodies are storage-format XHTML; attachments separate |
| Gmail / Drive | OAuth per user + domain-wide delegation | changes.list with page tokens | Per-user quotas; attachments; shared drives have different ACLs |
| Databases | Service account | Incremental on updated_at / CDC | Row = doc? Joins to build a "document" |
| Web crawl | None / cookies | Sitemap + link graph | robots.txt, dedupe, JS rendering, politeness |
How to implement
- One interface, many implementations.
list_changes(since_cursor) → iterator[DocRef],fetch(doc_ref) → RawDoc(bytes, mime, metadata, acl). Every connector implements the same two methods. - Cursor-based incremental sync. Store a per-source cursor (delta token,
updated_atwatermark, change-feed offset). Full scans are the bootstrap path only. - Capture ACL and tenant at fetch time (tenant / ACL capture at ingest). The connector is the only place that knows the source's permission model.
- Rate limiting + retry. Token bucket per source; exponential backoff with jitter on 429/5xx; honour
Retry-After. Circuit-break when error rate spikes. - Pagination that survives restarts. Persist the page token after each page; resume from it.
- Emit
RawDocto a queue, not directly into the parser. Decouples fetch throughput from parse throughput. - Dead-letter queue for documents that fail fetch after N retries; alert, don't drop silently.
Small example
from dataclasses import dataclass
from typing import Iterator
import time, random, requests
@dataclass
class DocRef:
source: str; external_id: str; modified_at: str; etag: str; url: str
@dataclass
class RawDoc:
ref: DocRef; bytes: bytes; mime: str; acl_groups: list[str]; tenant_id: str
class Connector:
def list_changes(self, cursor: str | None) -> tuple[Iterator[DocRef], str]: ...
def fetch(self, ref: DocRef) -> RawDoc: ...
def with_backoff(fn, max_tries=6):
for attempt in range(max_tries):
try:
return fn()
except requests.HTTPError as e:
if e.response.status_code in (429, 500, 502, 503, 504) and attempt < max_tries - 1:
wait = float(e.response.headers.get("Retry-After", 0)) or (2 ** attempt + random.random())
time.sleep(wait); continue
raise
class ConfluenceConnector(Connector):
def __init__(self, base, token, tenant_id):
self.s = requests.Session(); self.s.headers["Authorization"] = f"Bearer {token}"
self.base, self.tenant = base, tenant_id
def list_changes(self, cursor):
since = cursor or "1970-01-01"
start, refs = 0, []
while True:
r = with_backoff(lambda: self.s.get(f"{self.base}/rest/api/content/search",
params={"cql": f'lastmodified >= "{since}" order by lastmodified', "start": start, "limit": 100}))
r.raise_for_status(); data = r.json()
for p in data["results"]:
refs.append(DocRef("confluence", p["id"], p["version"]["when"], str(p["version"]["number"]),
self.base + p["_links"]["webui"]))
if "next" not in data["_links"]: break
start += 100
new_cursor = max((r.modified_at for r in refs), default=since)
return iter(refs), new_cursor
def fetch(self, ref):
r = with_backoff(lambda: self.s.get(f"{self.base}/rest/api/content/{ref.external_id}",
params={"expand": "body.storage,restrictions.read"}))
r.raise_for_status(); j = r.json()
groups = [g["name"] for g in j["restrictions"]["read"]["restrictions"]["group"]["results"]] or ["all-employees"]
return RawDoc(ref, j["body"]["storage"]["value"].encode(), "text/html", groups, self.tenant)
Pros
- Uniform interface lets the rest of the pipeline ignore where documents came from.
- Incremental cursors make daily sync cheap: only changed documents flow.
- ACL capture at source is the only correct place to get permissions.
- Queue decoupling lets fetch and parse scale independently.
Cons
- Every source is a snowflake. Auth flows expire, APIs change, throttling rules differ. Connectors are perpetual maintenance.
- Deletions are hard to detect incrementally on many sources (no "deleted since" API); you need periodic full reconciliation.
- Permission models don't map cleanly (SharePoint inheritance, Drive shared-with-link, Confluence space vs page restrictions).
- Rate limits cap ingestion speed regardless of how much compute you have.
When to use
Always — but choose buy vs build per source. Managed connector frameworks (Airbyte, Unstructured platform, LlamaCloud, Glean-style products, cloud-provider knowledge-base connectors) cover the common SaaS sources; build your own for internal databases, proprietary systems, and anything where you need custom ACL capture.
When NOT to use (build your own)
- When a managed connector already covers the source and exposes ACLs and change cursors. Building a SharePoint connector from scratch is months of edge cases.
- Don't build a generic web crawler for a handful of known URLs; a sitemap fetch is enough.
Production at scale
Bootstrap vs steady state are different jobs. The initial full load of 10M documents runs for days, needs parallelism and checkpointing (state & durability), and can saturate source APIs. Steady-state delta sync runs every few minutes on a few hundred changes. Design both paths; run the bootstrap with lower priority and a rate cap so it doesn't starve the delta sync.
Cursors are per source and per tenant. In multi-tenant systems each tenant's Confluence has its own token, quota, and cursor. Store (tenant, source, cursor, last_success_at) and alert when last_success_at ages beyond SLA.
Token/credential lifecycle. OAuth refresh tokens expire, get revoked, or the admin leaves. Monitor auth failures per tenant, notify the tenant admin automatically, and keep ingesting other tenants. One expired token must never stall the fleet.
Deletion reconciliation. Nightly (or weekly) list all IDs from the source, diff against the index, soft-delete the missing ones (document versioning). Without this, deleted documents stay searchable — a privacy incident.
Permission drift. ACL changes usually don't bump modified_at. Poll permission-change feeds where they exist (Graph delta includes permission changes; Drive has permissions in changes), otherwise re-fetch ACLs on a schedule for sensitive sources.
Throughput engineering. Async I/O (aiohttp/httpx) with a per-source concurrency semaphore; a token bucket calibrated to the documented quota minus margin. Track 429 rate; if it climbs, back off globally for that source.
Large objects. Stream to object storage; never hold a 2 GB PDF in memory. Pass a storage reference to the parser, not bytes. Set size limits and route oversize files to a special path.
Idempotency. Fetching the same document twice must produce the same RawDoc and the same downstream IDs. Combine with change detection (change detection) so duplicates are cheap.
Observability. Per (tenant, source): documents listed, fetched, failed, DLQ size, cursor age, 429 count, bytes fetched, p95 fetch latency. A cursor that hasn't advanced in 24h is an alert.
Security. Connector credentials in a secrets manager, scoped read-only, rotated. Connectors run in a network segment that can reach sources but not the LLM/serving tier.
Prod checklist
- Common
Connectorinterface; per-source implementation or managed connector - Incremental cursor persisted per (tenant, source); bootstrap path separate
- ACL + tenant captured on every
RawDoc - Backoff + jitter +
Retry-After; per-source token bucket; circuit breaker - Queue between fetch and parse; DLQ with alerting
- Deletion reconciliation job
- Permission-change polling for sensitive sources
- Credential rotation + auth-failure alerts per tenant
- Cursor-age and 429-rate dashboards
Metrics to watch
| Metric | Why |
|---|---|
| Cursor age per (tenant, source) | Stalled sync |
| Fetch error rate, 429 rate | Quota health |
| DLQ depth | Silent data loss |
| Docs/hour during bootstrap | Capacity planning |
| Reconciliation deletes per run | Deletion coverage |
Related
- Change detection — what to do with fetched docs
- Document versioning — deletions and updates
- Tenant/ACL capture at ingest
- State & durability — checkpointed bootstrap jobs
- Failure handling — backoff, breakers