Indexing → Filtered search (pre-filter vs post-filter)
What it is
Three ways to combine "nearest vectors" with "must satisfy filter":
| Strategy | How | Fails when |
|---|---|---|
| Post-filter | ANN top-K (say 100) → drop those failing the filter → return survivors | Filter is selective (tenant = 1% of corpus): most of the 100 are dropped; you get 0–3 results, or none |
| Pre-filter (brute force) | Evaluate the filter first → exact search over the matching subset | Subset is large (filter matches 50% of 50M): a brute-force scan is slow |
| Filter-aware ANN (integrated) | Graph traversal skips non-matching nodes, or partitions by filter value, or per-filter sub-indexes | Extremely selective filters can strand the graph (disconnected matching nodes); needs the store's implementation to handle it (Qdrant, Weaviate, Milvus, Vespa, Pinecone, pgvector with recent versions do variations) |
Good stores switch strategy by estimated selectivity: brute-force over small filtered sets, filter-aware graph search otherwise. The common production filters — tenant_id, is_current, acl_groups ∩ caller_groups, doc_type, date ranges — are often very selective per query (one tenant of thousands), which is exactly where naive post-filtering breaks.
How to implement
- Use a store with native filtered ANN (vector store choice) and create payload/metadata indexes on every filter field (keyword for IDs/enums, numeric/int for dates as epoch).
- Order filters by selectivity in your query builder (tenant first) — some engines benefit; all benefit from clarity.
- Set
kandeffor filtered queries: with filters, raiseef_search(ornprobe) and request more candidates than you need; measure recall vs a filtered flat search (index types: Flat, HNSW, IVF-PQ). - Partition when selectivity is extreme and stable: per-tenant collections/namespaces or the store's tenant sharding, so the "filter" becomes "choose the index".
- Never post-filter security fields in application code as the only mechanism — it both loses results and risks bugs; if you must post-filter (legacy store), over-fetch heavily (K = 10× k) and still treat it as temporary.
- Test with realistic callers: a user with narrow ACL groups, a small tenant, a date range that matches 0.1% — check result counts and recall.
- Monitor zero-result rate and recall on filtered canaries.
Small example
# Demonstrate the post-filter failure and the fix (Qdrant-style; same idea elsewhere)
def bad_post_filter(store, qvec, caller, k=10):
hits = store.search("chunks", qvec, limit=100) # no filter
mine = [h for h in hits if h.payload["tenant_id"] == caller.tenant_id and h.payload["is_current"]]
return mine[:k] # often empty for small tenants
def good_filtered_ann(store, qvec, caller, k=10):
flt = {"must": [{"key": "tenant_id", "match": {"value": caller.tenant_id}},
{"key": "is_current", "match": {"value": True}},
{"key": "acl_groups", "match": {"any": list(caller.groups)}}]}
return store.search("chunks", qvec, query_filter=flt, limit=k * 3,
search_params={"hnsw_ef": 160, "exact": False})[:k] # filter during traversal; higher ef for selective filters
# selectivity-aware exact fallback (when the filtered subset is tiny, brute force is both exact and fast)
def adaptive(store, qvec, caller, k=10):
n_match = store.count("chunks", query_filter=build_filter(caller)) # cheap with payload indexes
exact = n_match < 20_000
return store.search("chunks", qvec, query_filter=build_filter(caller), limit=k, search_params={"exact": exact})
# pgvector: make sure the planner uses the HNSW index with the WHERE clause; check EXPLAIN ANALYZE
# SET hnsw.ef_search = 200; -- and consider partial indexes per tenant for very large tenants
Pros (of native filtered ANN)
- Correct result counts for selective filters; no empty pages for small tenants.
- Security filters are enforced inside the engine, not in app code that can be bypassed.
- Predictable latency across selectivity ranges when the engine switches strategies.
Cons
- Engine-dependent behaviour: recall under selective filters varies widely; must be measured per store and version.
- Higher
ef/candidate counts raise latency for filtered queries. - Payload indexes cost memory and slow writes.
- High-cardinality filters (ACL lists with thousands of groups) strain index structures; need group indirection (tenant / ACL capture at ingest).
- pgvector's planner can still choose a sequential scan or post-filter path depending on statistics — needs
EXPLAINdiscipline.
When to use
Every multi-tenant or permission-aware system — which is nearly every enterprise system. Also any system with date/type filters that users rely on ("only current policies", "only Confluence").
When NOT to use (post-filter)
- Never as the security mechanism. Acceptable only for non-security cosmetic filters over broad result sets (e.g. "hide already-shown results"), with over-fetch.
Production at scale
Selectivity profile per filter. Measure how selective each filter is in practice (tenant sizes follow a power law: a few huge, thousands tiny). Tiny tenants → exact search on their subset is fastest; huge tenants → filter-aware HNSW with tuned ef; the engine may do this automatically, but verify with EXPLAIN-equivalents or timing.
Partition the big ones. Physical collections/shards for the largest tenants remove the filter cost entirely and isolate their load; keep the tenant filter anyway (defence in depth).
ACL cardinality. acl_groups any-of caller_groups with callers in 200 groups and docs with 50 groups is a wide set intersection; use synthetic ACL-set IDs (tenant / ACL capture at ingest) to reduce cardinality, and index them as keywords.
is_current and version filters on every query (document versioning); the payload index on a boolean is cheap; some engines prefer you to delete old versions rather than filter — do both (filter now, GC later).
Recall canary with filters. Daily: golden queries executed as realistic callers (small tenant, restricted groups); recall vs filtered flat search; alert on drops after upgrades/compaction.
Zero-result handling. If a filtered query returns nothing, decide: widen non-security filters (dates), fall back to BM25 (hybrid search), or abstain (score thresholding & abstention). Never widen the security filter.
Latency budget. Filtered queries are slower; include them in the p95 SLO measurement rather than benchmarking unfiltered.
Cross-index consistency. BM25/sparse indexes must apply the same filters (hybrid search); fusing an ACL-filtered dense list with an unfiltered BM25 list leaks.
Prod checklist
- Store with native filtered ANN; payload indexes on all filter fields
- Security filters (tenant, ACL, current) built by one function, inside the engine query
-
ef/candidate counts tuned for filtered recall; adaptive exact search for tiny subsets - Physical partitioning for largest tenants; filter kept anyway
- ACL cardinality controlled via synthetic set IDs
- Filtered recall canary with realistic callers; zero-result rate monitored
- Same filters applied to BM25/sparse index before fusion
- pgvector: query plans verified with EXPLAIN after every schema/index change
Metrics to watch
| Metric | Why |
|---|---|
| Filtered recall vs filtered flat | Engine behaviour under selectivity |
| Zero-result rate by tenant size bucket | Post-filter symptoms |
| Filtered p95 vs unfiltered p95 | Cost of filtering |
| Payload index memory; write latency | Index cost |
Related
- Index types —
ef/nprobetuning - Store choice — native filtered ANN support
- ACL capture, AuthZ — security filters
- Hybrid search — consistent filtering across indexes
- Self-query — user-driven filters