Source ingestion → Change detection (the size/mtime → SHA-256 ladder)

Incremental indexing — detect changed documents so unchanged ones aren't re-embedded

What it is

Re-processing a document costs parse time (seconds to minutes for OCR), embedding tokens (money), and index writes. Most sync runs touch documents that haven't actually changed — someone re-saved a file, a backup restore reset every mtime, a migration copied a folder. Change detection is a ladder of increasingly expensive checks that stops at the first proof of "unchanged":

walk gives you: path, size, mtime, ACL          ← metadata only, no file read

(size, mtime) unchanged?  ──► SKIP entirely. Never open the file.
        │ changed
        ▼
read bytes + compute SHA-256                     ← one pass over bytes
        │
hash == stored hash?      ──► content identical (resave / touch /
        │                     robocopy / restore). Update mtime, skip
        │ differs             parse + embed.
        ▼
re-parse → re-chunk → re-embed

Three tiers, three costs: free (listing metadata), cheap (one read + hash), expensive (parse + embed).

How to implement

  1. Store per document: external_id, size, mtime/etag, content_hash (SHA-256), parser_version, chunking_config_version, embedding_model_version, last_processed_at.
  2. Tier 1 — metadata compare. From the listing, compare (size, mtime) or etag with stored. Equal → skip, don't fetch.
  3. Tier 2 — hash compare. Fetch bytes, stream them through SHA-256 (don't load into memory), compare with stored hash. Equal → update mtime, skip.
  4. Tier 3 — process. Hash differs → parse → chunk → embed → upsert with a new version (document versioning); store the new hash.
  5. Config-driven reprocessing. If parser_version / chunking_config_version / embedding_model_version stored ≠ current, the document needs reprocessing even if the hash matches. Make that a separate, explicit, rate-limited job.
  6. Chunk-level hashing (optional, big win for large docs). Hash each chunk's text; re-embed only chunks whose hash changed. A one-paragraph edit in a 200-page manual re-embeds one chunk.

Small example

import hashlib, sqlite3

db = sqlite3.connect("ingest_state.db")
db.execute("""CREATE TABLE IF NOT EXISTS docs(
    ext_id TEXT PRIMARY KEY, size INT, mtime TEXT, sha256 TEXT,
    parser_v TEXT, chunk_v TEXT, emb_v TEXT)""")

CURRENT = {"parser_v": "docling-2.14", "chunk_v": "rec-400-sent", "emb_v": "bge-m3-1"}

def sha256_stream(fp, block=1 << 20):
    h = hashlib.sha256()
    for b in iter(lambda: fp.read(block), b""):
        h.update(b)
    return h.hexdigest()

def decide(ref, open_bytes):
    row = db.execute("SELECT size, mtime, sha256, parser_v, chunk_v, emb_v FROM docs WHERE ext_id=?",
                     (ref.external_id,)).fetchone()
    if row and (row[0], row[1]) == (ref.size, ref.modified_at) and row[3:] == tuple(CURRENT.values()):
        return "skip:metadata"                                  # Tier 1: never opened the file
    with open_bytes(ref) as fp:
        digest = sha256_stream(fp)
    if row and row[2] == digest and row[3:] == tuple(CURRENT.values()):
        db.execute("UPDATE docs SET size=?, mtime=? WHERE ext_id=?", (ref.size, ref.modified_at, ref.external_id))
        return "skip:hash"                                      # Tier 2: touched but identical
    # Tier 3: process, then record
    # ... parse / chunk / embed / upsert (new version) ...
    db.execute("REPLACE INTO docs VALUES (?,?,?,?,?,?,?)",
               (ref.external_id, ref.size, ref.modified_at, digest, *CURRENT.values()))
    return "processed"

Pros

  • Eliminates the dominant waste in ingestion: re-embedding unchanged content.
  • Cheap tiers first — most documents are decided without a byte being read.
  • Robust to false "modified" signals (touch, restore, resave) via the hash tier.
  • Makes incremental sync fast enough to run every few minutes.
  • Version-aware reprocessing turns migrations into explicit, controllable jobs.

Cons

  • State to maintain: a document table that must stay consistent with the index. Lose it and you re-process everything.
  • Hash tier still reads every candidate — on sources with unreliable mtime (some SharePoint/Drive exports) you end up hashing a lot.
  • Semantic changes without byte changes aren't detected — e.g. ACL changed but content didn't (needs a separate ACL-sync path, connectors).
  • Chunk-level hashing adds complexity: chunk boundaries can shift after an edit, so "unchanged chunk" detection needs content-addressed chunk IDs, not positional ones.

When to use

Always, from day one. Retrofitting change detection onto a pipeline that re-indexes everything nightly is painful because the pipeline has come to rely on the full rebuild to fix consistency bugs.

When NOT to use

  • Truly append-only sources (immutable logs, event streams): there's nothing to detect; just track the offset.
  • Tiny corpora (hundreds of documents) where a full rebuild is seconds — but still store hashes; you'll grow.

Production at scale

Where the state lives. Postgres (or the same DB as your document table) with ext_id primary key. Keep it outside the vector store; vector DBs are bad at "give me all doc IDs and their hashes" scans. At 100M documents, partition by source/tenant.

Metadata tier depends on source honesty. S3 etag is an MD5 (for non-multipart) — reliable. SharePoint mtime changes on permission edits — noisy. Drive modifiedTime is reliable but version is better. Learn each source's signal and encode it in the connector.

Streaming hashes. Never read() a whole file; stream in 1 MB blocks. For object storage, many services return a content hash in the listing (S3 etag, GCS md5Hash/crc32c) — use that as Tier 1.5 and skip the download entirely when it matches your stored value.

Reprocessing waves. A parser or embedding-model upgrade means every document needs Tier 3 despite unchanged hashes. Do not let that happen implicitly on the next sync. Run it as a controlled migration: a job that walks the document table filtered on emb_v != current, rate-limited, checkpointed, with a dual-index cutover (versioning & migration). Track progress as a percentage.

Chunk-level incrementalism. Use content-addressed chunk IDs (sha256(doc_id + chunk_text)) alongside positional index. After re-chunking a changed doc: chunks with an existing content hash → reuse the stored vector; new ones → embed; missing ones → delete. For long, frequently edited documents this cuts embedding spend by >90%.

Consistency on crash. Write the state row after the index upsert succeeds, in that order. If the process dies between, the next run re-processes the doc (idempotent upsert), which is safe. The reverse order can leave a document marked processed with nothing in the index.

Don't skip ACL updates. A hash match must still update ACL fields if the connector reports an ACL change (tenant / ACL capture at ingest). Tie ACL sync to the metadata tier: (size, mtime, acl_hash).

Metrics tell you where the money goes. Log the decision per document: skip:metadata, skip:hash, processed, processed:config. The ratio processed / listed on a steady-state sync should be single-digit percent. If it's high, either your source mtime is noisy (add the hash tier) or you have a config-version mismatch bleeding into normal syncs.

Prod checklist

  • Document state table with size/mtime/etag, SHA-256, and all pipeline version fields
  • Tier 1 metadata skip; use listing-provided hashes where available
  • Streaming SHA-256; no whole-file reads
  • Tier 2 hash skip updates mtime
  • Version-mismatch reprocessing is an explicit, rate-limited migration job
  • Content-addressed chunk IDs for chunk-level incrementalism (large docs)
  • State row written after successful index upsert
  • ACL changes flow even when content hash matches
  • Decision-ratio dashboard per source

Metrics to watch

MetricWhy
processed / listed per syncWaste detection
skip:hash / skip:metadata ratioSource mtime reliability
Bytes hashed per syncTier-2 cost
Reprocessing-wave progress %Migration tracking
Embedding tokens per syncThe bill