Query understanding → Self-query (natural language → metadata filters)

Self-Query Retriever Explained: Natural Language to Metadata Filters

What it is

Embeddings are bad at constraints. Dates, numeric ranges, enumerated categories, and negations ("not the draft version") are structural facts that vector similarity approximates poorly — a 2019 contract and a 2024 contract embed almost identically. But your chunks already carry that metadata (metadata on every chunk). Self-query uses an LLM to parse the user's question into:

{
  "semantic_query": "vendor termination clauses",
  "filters": {"signed_date": {"gte": "2023-01-01"}, "region": {"eq": "EMEA"}, "doc_type": {"eq": "contract"}}
}

The semantic part goes to the retriever; the filters go to the vector store's filter clause (filtered search), combined with the mandatory security filters. The result: precision that dense retrieval cannot achieve, and result counts the user can trust ("47 contracts match").

How to implement

  1. Publish the filterable schema to the LLM: field names, types, allowed enum values, and date semantics. This is a closed vocabulary — the model must not invent fields or values.
  2. Extraction prompt (fast tier, temperature 0) with the schema and 3–4 examples, including one with no filters and one with an ambiguous phrase that should not become a filter.
  3. Validate hard: unknown field → drop; enum value not in the allowed list → drop (or fuzzy-map to the nearest allowed value with a confidence threshold); malformed date → drop. Never pass an unvalidated filter to the store.
  4. Resolve relative dates ("last quarter", "since the merger") in code against a reference date passed into the prompt, not by the model's arithmetic.
  5. Merge with security filters — self-query filters are ANDed on top of tenant/ACL/current-version filters, never replacing them (tenant / ACL capture at ingest).
  6. Handle empty results: if filters yield zero, retry without the non-security filters and tell the user which constraint was relaxed. Never relax security filters.
  7. Evaluate: filter extraction precision/recall against a labelled set of queries, plus end-to-end retrieval quality.

Small example

from datetime import date
from pydantic import BaseModel, field_validator

FILTER_SCHEMA = {
    "doc_type":    {"type": "enum", "values": ["contract", "policy", "invoice", "report", "email"]},
    "region":      {"type": "enum", "values": ["EMEA", "AMER", "APAC"]},
    "signed_date": {"type": "date"},
    "vendor":      {"type": "string"},
    "status":      {"type": "enum", "values": ["draft", "executed", "expired"]},
}

SELFQ_SYS = f"""Extract search filters from the user's question. Today is {{today}}.
Available fields (use ONLY these, with ONLY these values):
{json.dumps(FILTER_SCHEMA, indent=2)}
Operators: eq, in, gte, lte, ne. Resolve relative dates to absolute ISO dates using today's date.
If a phrase does not clearly map to a field, leave it in semantic_query instead of inventing a filter.
Output JSON: {{"semantic_query": string, "filters": {{field: {{op: value}}}}}}

<example>
<q>termination clauses in contracts signed after 2023 with EMEA vendors</q>
<out>{{"semantic_query": "termination clauses", "filters": {{"doc_type": {{"eq": "contract"}}, "signed_date": {{"gte": "2023-01-01"}}, "region": {{"eq": "EMEA"}}}}}}</out>
</example>
<example>
<q>what is our parental leave policy</q>
<out>{{"semantic_query": "parental leave policy", "filters": {{}}}}</out>
</example>"""

ALLOWED_OPS = {"eq", "in", "gte", "lte", "ne"}

def validate_filters(raw: dict) -> tuple[dict, list[str]]:
    clean, dropped = {}, []
    for field, cond in (raw or {}).items():
        spec = FILTER_SCHEMA.get(field)
        if not spec: dropped.append(f"unknown field {field}"); continue
        for op, val in cond.items():
            if op not in ALLOWED_OPS: dropped.append(f"{field}: bad op {op}"); continue
            if spec["type"] == "enum" and val not in spec["values"]:
                dropped.append(f"{field}: value {val!r} not allowed"); continue
            if spec["type"] == "date" and not re.fullmatch(r"\d{4}-\d{2}-\d{2}", str(val)):
                dropped.append(f"{field}: bad date {val!r}"); continue
            clean.setdefault(field, {})[op] = val
    return clean, dropped

def self_query(question: str, caller):
    out = llm("self_query", SELFQ_SYS.format(today=date.today().isoformat()),
              [{"role": "user", "content": question}], max_tokens=250)
    parsed = json.loads(out.text)
    filters, dropped = validate_filters(parsed.get("filters"))
    log_self_query(question, filters, dropped)
    combined = merge_filters(build_acl_filter(caller), filters)      # security filters always win and are never removable
    hits = vec.search(embed_queries([parsed["semantic_query"]])[0], k=30, filter=combined)
    if not hits and filters:                                          # relax NON-security filters only
        hits = vec.search(embed_queries([parsed["semantic_query"]])[0], k=30, filter=build_acl_filter(caller))
        return hits, {"relaxed": list(filters)}
    return hits, {"applied": filters, "dropped": dropped}

Pros

  • Precision on constraints embeddings can't express: dates, ranges, categories, negation.
  • Trustworthy counts and scoping — "search only executed contracts" actually does that.
  • Smaller, cleaner candidate sets → reranking and generation both improve.
  • Enables faceted UX (show the parsed filters as removable chips — users love this and it makes the system's reasoning visible).

Cons

  • Hallucinated filters silently exclude the right documents — the worst failure here, because the user sees a confident "no results" or a wrong subset.
  • Schema coupling: the prompt must stay in sync with the actual metadata fields and enum values.
  • Over-filtering on ambiguous phrases ("recent", "big") that shouldn't have become filters.
  • Selectivity interaction: adding filters to an ANN query changes recall behaviour (filtered search).
  • Another LLM call on the critical path (fold into the rewrite call).

When to use

  • Corpora with rich, reliable metadata and users who ask scoped questions (legal, finance, compliance, procurement, ticketing).
  • Any UI where users expect filters but type sentences.
  • When retrieval evals show failures caused by returning the wrong version, date, or type of an otherwise-relevant document.

When NOT to use

  • Sparse or unreliable metadata — filters on fields that are often null exclude good documents. Fix ingestion first (metadata on every chunk).
  • Purely semantic corpora (wiki prose with no structured attributes).
  • When the user's phrasing rarely contains constraints — measure the firing rate before building it.

Production at scale

Show the filters, let users remove them. Rendering parsed filters as chips is the single best mitigation for hallucinated filters: the user sees "region: EMEA" and clicks it off. It converts a silent failure into a visible, correctable one.

Enum grounding from the index, not the prompt. Generate the allowed-values list from the actual distinct values in the corpus (per tenant, refreshed periodically) rather than hardcoding it. A prompt claiming region ∈ {EMEA, AMER, APAC} while the data says emea/EU produces filters that match nothing.

Fuzzy value mapping with a threshold: "Europe" → EMEA only if a similarity/alias table says so; otherwise leave it in the semantic query. Maintain the alias table per tenant.

Never relax security filters on empty results — only user-intent filters, and say which one was relaxed. Log every relaxation.

Zero-result loop: filters → 0 results → relax → still 0 → abstain (score thresholding & abstention). Track the zero-result rate by whether filters were applied; a high rate with filters means extraction is too aggressive or metadata coverage is poor.

Fold into one call. Rewrite (query rewriting) + decomposition detection (decomposition) + filter extraction in a single fast-tier structured output saves 2 round trips (structured outputs & tool use).

Metadata coverage dashboard. Percentage of chunks with each filterable field populated, per source. Self-query on a field that's 40% null is a trap.

Evaluation set of (query → expected filters) pairs, scored on precision and recall per field. Filter precision matters more than recall: a wrong filter excludes the answer; a missing filter just means a broader search that reranking can still fix.

Prod checklist

  • Filterable schema generated from actual index values, per tenant, refreshed
  • Extraction validated against the closed schema; unknown fields/values dropped and logged
  • Relative dates resolved in code against a passed reference date
  • Security filters merged in and never relaxable
  • Parsed filters surfaced in the UI as removable chips
  • Zero-result relaxation path (non-security only) with user-visible explanation
  • Folded into the rewrite call to save a round trip
  • Metadata coverage per field per source monitored
  • Labelled (query → filters) eval set; per-field precision tracked

Metrics to watch

MetricWhy
Filter extraction precision per fieldHallucinated filters exclude answers
Zero-result rate with vs without filtersOver-filtering
Filter relaxation rateExtraction aggressiveness
Metadata field coverage per sourceWhether filtering is viable at all