Source ingestion → Change detection (the size/mtime → SHA-256 ladder)
What it is
Re-processing a document costs parse time (seconds to minutes for OCR), embedding tokens (money), and index writes. Most sync runs touch documents that haven't actually changed — someone re-saved a file, a backup restore reset every mtime, a migration copied a folder. Change detection is a ladder of increasingly expensive checks that stops at the first proof of "unchanged":
walk gives you: path, size, mtime, ACL ← metadata only, no file read
(size, mtime) unchanged? ──► SKIP entirely. Never open the file.
│ changed
▼
read bytes + compute SHA-256 ← one pass over bytes
│
hash == stored hash? ──► content identical (resave / touch /
│ robocopy / restore). Update mtime, skip
│ differs parse + embed.
▼
re-parse → re-chunk → re-embed
Three tiers, three costs: free (listing metadata), cheap (one read + hash), expensive (parse + embed).
How to implement
- Store per document:
external_id, size, mtime/etag, content_hash (SHA-256), parser_version, chunking_config_version, embedding_model_version, last_processed_at. - Tier 1 — metadata compare. From the listing, compare
(size, mtime)oretagwith stored. Equal → skip, don't fetch. - Tier 2 — hash compare. Fetch bytes, stream them through SHA-256 (don't load into memory), compare with stored hash. Equal → update
mtime, skip. - Tier 3 — process. Hash differs → parse → chunk → embed → upsert with a new
version(document versioning); store the new hash. - Config-driven reprocessing. If
parser_version/chunking_config_version/embedding_model_versionstored ≠ current, the document needs reprocessing even if the hash matches. Make that a separate, explicit, rate-limited job. - Chunk-level hashing (optional, big win for large docs). Hash each chunk's text; re-embed only chunks whose hash changed. A one-paragraph edit in a 200-page manual re-embeds one chunk.
Small example
import hashlib, sqlite3
db = sqlite3.connect("ingest_state.db")
db.execute("""CREATE TABLE IF NOT EXISTS docs(
ext_id TEXT PRIMARY KEY, size INT, mtime TEXT, sha256 TEXT,
parser_v TEXT, chunk_v TEXT, emb_v TEXT)""")
CURRENT = {"parser_v": "docling-2.14", "chunk_v": "rec-400-sent", "emb_v": "bge-m3-1"}
def sha256_stream(fp, block=1 << 20):
h = hashlib.sha256()
for b in iter(lambda: fp.read(block), b""):
h.update(b)
return h.hexdigest()
def decide(ref, open_bytes):
row = db.execute("SELECT size, mtime, sha256, parser_v, chunk_v, emb_v FROM docs WHERE ext_id=?",
(ref.external_id,)).fetchone()
if row and (row[0], row[1]) == (ref.size, ref.modified_at) and row[3:] == tuple(CURRENT.values()):
return "skip:metadata" # Tier 1: never opened the file
with open_bytes(ref) as fp:
digest = sha256_stream(fp)
if row and row[2] == digest and row[3:] == tuple(CURRENT.values()):
db.execute("UPDATE docs SET size=?, mtime=? WHERE ext_id=?", (ref.size, ref.modified_at, ref.external_id))
return "skip:hash" # Tier 2: touched but identical
# Tier 3: process, then record
# ... parse / chunk / embed / upsert (new version) ...
db.execute("REPLACE INTO docs VALUES (?,?,?,?,?,?,?)",
(ref.external_id, ref.size, ref.modified_at, digest, *CURRENT.values()))
return "processed"
Pros
- Eliminates the dominant waste in ingestion: re-embedding unchanged content.
- Cheap tiers first — most documents are decided without a byte being read.
- Robust to false "modified" signals (touch, restore, resave) via the hash tier.
- Makes incremental sync fast enough to run every few minutes.
- Version-aware reprocessing turns migrations into explicit, controllable jobs.
Cons
- State to maintain: a document table that must stay consistent with the index. Lose it and you re-process everything.
- Hash tier still reads every candidate — on sources with unreliable mtime (some SharePoint/Drive exports) you end up hashing a lot.
- Semantic changes without byte changes aren't detected — e.g. ACL changed but content didn't (needs a separate ACL-sync path, connectors).
- Chunk-level hashing adds complexity: chunk boundaries can shift after an edit, so "unchanged chunk" detection needs content-addressed chunk IDs, not positional ones.
When to use
Always, from day one. Retrofitting change detection onto a pipeline that re-indexes everything nightly is painful because the pipeline has come to rely on the full rebuild to fix consistency bugs.
When NOT to use
- Truly append-only sources (immutable logs, event streams): there's nothing to detect; just track the offset.
- Tiny corpora (hundreds of documents) where a full rebuild is seconds — but still store hashes; you'll grow.
Production at scale
Where the state lives. Postgres (or the same DB as your document table) with ext_id primary key. Keep it outside the vector store; vector DBs are bad at "give me all doc IDs and their hashes" scans. At 100M documents, partition by source/tenant.
Metadata tier depends on source honesty. S3 etag is an MD5 (for non-multipart) — reliable. SharePoint mtime changes on permission edits — noisy. Drive modifiedTime is reliable but version is better. Learn each source's signal and encode it in the connector.
Streaming hashes. Never read() a whole file; stream in 1 MB blocks. For object storage, many services return a content hash in the listing (S3 etag, GCS md5Hash/crc32c) — use that as Tier 1.5 and skip the download entirely when it matches your stored value.
Reprocessing waves. A parser or embedding-model upgrade means every document needs Tier 3 despite unchanged hashes. Do not let that happen implicitly on the next sync. Run it as a controlled migration: a job that walks the document table filtered on emb_v != current, rate-limited, checkpointed, with a dual-index cutover (versioning & migration). Track progress as a percentage.
Chunk-level incrementalism. Use content-addressed chunk IDs (sha256(doc_id + chunk_text)) alongside positional index. After re-chunking a changed doc: chunks with an existing content hash → reuse the stored vector; new ones → embed; missing ones → delete. For long, frequently edited documents this cuts embedding spend by >90%.
Consistency on crash. Write the state row after the index upsert succeeds, in that order. If the process dies between, the next run re-processes the doc (idempotent upsert), which is safe. The reverse order can leave a document marked processed with nothing in the index.
Don't skip ACL updates. A hash match must still update ACL fields if the connector reports an ACL change (tenant / ACL capture at ingest). Tie ACL sync to the metadata tier: (size, mtime, acl_hash).
Metrics tell you where the money goes. Log the decision per document: skip:metadata, skip:hash, processed, processed:config. The ratio processed / listed on a steady-state sync should be single-digit percent. If it's high, either your source mtime is noisy (add the hash tier) or you have a config-version mismatch bleeding into normal syncs.
Prod checklist
- Document state table with size/mtime/etag, SHA-256, and all pipeline version fields
- Tier 1 metadata skip; use listing-provided hashes where available
- Streaming SHA-256; no whole-file reads
- Tier 2 hash skip updates mtime
- Version-mismatch reprocessing is an explicit, rate-limited migration job
- Content-addressed chunk IDs for chunk-level incrementalism (large docs)
- State row written after successful index upsert
- ACL changes flow even when content hash matches
- Decision-ratio dashboard per source
Metrics to watch
| Metric | Why |
|---|---|
processed / listed per sync | Waste detection |
skip:hash / skip:metadata ratio | Source mtime reliability |
| Bytes hashed per sync | Tier-2 cost |
| Reprocessing-wave progress % | Migration tracking |
| Embedding tokens per sync | The bill |
Related
- Connectors — supply size/mtime/etag
- Deduplication — same hash, different documents
- Document versioning — what "processed" creates
- Embedding versioning — the config-driven wave