Structure extraction → Headers / footers / boilerplate stripping
What it is
Boilerplate is content that repeats across pages or documents and carries no answer: "CONFIDENTIAL — Acme Corp — Page 12 of 90", "This email and any attachments…", "© 2024 All rights reserved", navigation menus, cookie notices, "Table of Contents", running chapter titles, letterhead, fax cover text, forwarded-message headers.
Why it matters:
- Embeddings: a 200-token chunk with a 60-token legal footer is 30% footer; the vector drifts toward "confidentiality notice", and every chunk in the corpus looks alike.
- BM25: boilerplate terms get high document frequency and pollute scoring.
- Cost: tokens embedded, stored, and later stuffed into the LLM context for nothing.
- Chunking: a footer in the middle of a paragraph (from page breaks) splits sentences.
- Answers: the model quotes the disclaimer.
How to implement
- Parser labels first: Docling / layout models tag
page_header,page_footer; drop them from text, keep in metadata if they carry the doc ID or page label. - Position + repetition within a document: text blocks in the top/bottom ~8% of the page that recur (exact or near-exact after digit normalisation) on ≥ 30% of pages are running headers/footers.
- Cross-document frequency: hash normalised lines/blocks; a block appearing in > N documents (per source/tenant) is boilerplate (email disclaimers, site navigation). Feed from chunk-hash frequency (deduplication).
- Source-specific rules: HTML — use readability-style extraction (main content), drop
nav,footer,aside, cookie banners; email — strip quoted replies and signatures (--, "On … wrote:", talon/mailparser); Office — headers/footers are separate parts, don't merge them. - Keep an allow-list of patterns that look like boilerplate but matter (e.g. "Effective date:", "Version 3.1", "Approved by") — extract as metadata instead of discarding.
- Record what was stripped (
stripped: [{type, text_hash, positions}]) for debugging and for the allow-list review. - Validate: stripped fraction per document should be small (< 15%); higher → review the rules for that source (production per-page heuristics).
Small example
import re
from collections import Counter
DIGITS = re.compile(r"\d+")
def norm(s): return DIGITS.sub("#", " ".join(s.lower().split()))
def find_running_blocks(pages, band=0.08, min_ratio=0.3):
"""pages: [[{bbox, text, page_size}]]. Returns set of normalised texts to strip."""
cnt = Counter()
for blocks in pages:
seen = set()
for b in blocks:
H = b["page_size"][1]; y0, y1 = b["bbox"][1], b["bbox"][3]
if y0 < band * H or y1 > (1 - band) * H:
seen.add(norm(b["text"]))
cnt.update(seen)
return {t for t, c in cnt.items() if c >= min_ratio * len(pages) and len(t) < 200}
GLOBAL_BOILERPLATE = { # populated from cross-document frequency job
norm("This email and any attachments are confidential and intended solely for the addressee."),
}
ALLOW = [re.compile(r"effective date", re.I), re.compile(r"\bversion\s+\d", re.I)]
def strip(pages):
running = find_running_blocks(pages)
out, stripped = [], []
for pno, blocks in enumerate(pages, 1):
for b in blocks:
n = norm(b["text"])
if (n in running or n in GLOBAL_BOILERPLATE) and not any(p.search(b["text"]) for p in ALLOW):
stripped.append({"page": pno, "text": b["text"][:80]})
continue
out.append(b | {"page": pno})
return out, stripped
Pros
- Cleaner embeddings and BM25 statistics → better retrieval.
- Lower cost at embed, storage, and generation.
- Fewer split sentences from page-break footers.
- Model stops quoting disclaimers.
Cons
- Over-stripping removes real content: a policy whose every page repeats "Applies to all employees in the EU" is repeating a fact. Allow-lists and human review needed.
- Rules per source are maintenance; templates change.
- Cross-document frequency needs a corpus-level job (not per-document) and must be tenant-scoped.
- Near-duplicate detection on footers with changing page numbers/dates requires normalisation; easy to get wrong.
When to use
- Any PDF corpus (running headers/footers are universal).
- Email, web, and Office sources (disclaimers, navigation, signatures).
- Before computing near-dup signatures (deduplication) — boilerplate inflates similarity.
When NOT to use (aggressively)
- Legal documents where headers/footers carry document identity, exhibit numbers, or execution details — extract to metadata, don't discard.
- Very short documents where the "footer" is a meaningful part.
Production at scale
Two jobs, two scopes. Per-document stripping runs inline in the parser (position + within-doc repetition). Cross-document boilerplate discovery runs as a periodic batch job per (tenant, source): hash normalised blocks, count documents, publish a boilerplate set that the inline stage consults. Tenant-scoped — one tenant's disclaimer is another tenant's contract clause.
Metadata, not deletion. Page labels, document IDs, classification markings ("CONFIDENTIAL", "DRAFT"), version strings, and effective dates that appear in headers/footers are valuable: parse them into metadata fields (page_label, classification, doc_version_label, effective_date) and then remove them from text. classification in particular can drive ACL (tenant / ACL capture at ingest).
Email is its own world. Quoted history (the previous 14 messages) in every reply is the dominant duplication in email corpora. Strip quoted blocks and signatures, index each message once, and keep thread structure as metadata. Libraries: talon, email-reply-parser, mailparser.
Web pages. Main-content extraction (trafilatura, readability) before anything else; drop nav/footer/aside/cookie/consent blocks; dedupe site-wide blocks by cross-document frequency.
Feedback loop with dedupe. The chunk-hash frequency table from deduplication is the discovery mechanism for global boilerplate: chunks appearing in > N docs get reviewed (human, sampled) and either added to the boilerplate set or to the allow-list.
Monitoring. Stripped fraction per document and per source; alert on outliers (> 15%) and on sudden changes (template change or rule bug). Track "boilerplate in top-k" in retrieval evals — a chunk of pure disclaimer showing up in results is the visible symptom.
Idempotence and versioning. Boilerplate sets and rules are versioned config; a rule change re-parses affected sources (change detection config wave). Store boilerplate_rules_version with parsed documents.
Never strip inside tables or code. Restrict the position heuristic to page bands and the frequency rule to text elements.
Prod checklist
- Parser header/footer labels used; position + in-document repetition heuristic
- Tenant-scoped cross-document frequency job publishing boilerplate sets
- Allow-list for meaningful repeated content; metadata extraction for page labels, classification, versions, dates
- Email quoted-reply and signature stripping; web main-content extraction
- Stripped content recorded per document; stripped-fraction monitoring
- Rules/sets versioned; change triggers re-parse for affected sources
- Tables/code excluded from stripping
- "Boilerplate in top-k" tracked in retrieval evals
Metrics to watch
| Metric | Why |
|---|---|
| Stripped fraction per doc/source | Over/under-stripping |
| Boilerplate chunks in top-10 (eval) | Retrieval pollution |
| Tokens embedded before/after stripping | Cost saved |
| Allow-list hits | Rules catching real content |
Related
- Deduplication — chunk-hash frequency feeds discovery
- Headings — running headers must not enter the stack
- Page labels — extracted from footers
- ACL — classification markings