Query understanding → Routing (vector, SQL, graph, web, or no retrieval)

Query Routing for RAG Explained: SQL, Graph, API or Vector?

What it is

A RAG system that sends everything to the vector index answers "how many contracts did we sign last quarter" by retrieving five contracts and letting the model guess a count. The question needed SELECT COUNT(*). Routing classifies the information need and dispatches:

RouteQuestion shapeBackend
No retrievalChit-chat, formatting requests, "summarise what you just said", general knowledgeAnswer directly (saves latency and cost)
Vector / hybrid"What does the policy say about X" — narrative, unstructuredThe RAG pipeline (retrieval & ranking)
SQL / text2SQLCounts, sums, averages, rankings, "how many", "top 5 by revenue"Warehouse / OLAP
Graph"Which vendors are connected to this entity", multi-hop relationships, "what themes recur"Knowledge graph (graphRAG / knowledge graph)
Web / externalCurrent events, public facts, competitor infoWeb search tool
Direct API"What's the status of ticket 4471", live stateSystem of record
ClarifyGenuinely ambiguousAsk one question back

Routing is a classification problem (routing & escalation is the model-tier cousin; this is the backend cousin). It's usually the cheapest large quality win in a mature system, because it stops the pipeline from answering the wrong kind of question with the wrong kind of evidence.

How to implement

  1. Enumerate the routes you actually have. Don't build a router for backends that don't exist.
  2. Rules first: regex/keyword signals cover a surprising share ("how many", "total", "average", "top N" → SQL; ticket-ID pattern → API; "hi"/"thanks" → no retrieval).
  3. LLM classifier for the rest: fast tier, closed label set, one example per route including the tricky boundaries. Fold into the rewrite call (query rewriting) as an extra output field.
  4. Confidence and fallback: low confidence → default to hybrid retrieval (the safest general route), or to a multi-route fan-out that queries two backends and lets reranking/synthesis decide.
  5. Per-route context assembly (context engineering): SQL results become a table block, API results a structured block, retrieved chunks passage blocks — each with its own budget.
  6. Per-route evaluation: route accuracy against a labelled set, and end-to-end quality per route (a correct route with a broken backend is still a failure).
  7. Log the route on every request; it's the first field you look at when debugging a bad answer.

Small example

ROUTES = ["no_retrieval", "vector", "sql", "graph", "web", "api", "clarify"]

AGG = re.compile(r"\b(how many|count|total|sum|average|avg|median|top \d+|most|least|trend|per (month|quarter|year))\b", re.I)
TICKET = re.compile(r"\b(TICKET|INC|CHG)-\d{3,}\b", re.I)

ROUTER_SYS = """Classify the user's question into exactly one route. Output JSON: {"route": ..., "confidence": 0-1, "reason": "..."}
Routes:
- no_retrieval: greetings, formatting, meta-questions about the conversation, general knowledge needing no company data
- vector: questions answered by narrative text in internal documents (policies, contracts, wikis)
- sql: aggregations, counts, rankings, time series over structured business data
- graph: relationships between entities, multi-hop connections, cross-document themes
- web: public/current information not in internal systems
- api: live status of a specific record identified by an id
- clarify: the question is too ambiguous to route
<example><q>How many vendors did we onboard in Q3?</q><out>{"route":"sql","confidence":0.95,"reason":"aggregation over structured vendor data"}</out></example>
<example><q>What does the MSA say about termination?</q><out>{"route":"vector","confidence":0.93,"reason":"narrative clause lookup"}</out></example>
<example><q>Thanks, can you shorten that?</q><out>{"route":"no_retrieval","confidence":0.98,"reason":"formatting request about prior answer"}</out></example>"""

def route(query: str, history) -> dict:
    if TICKET.search(query):                     return {"route": "api", "confidence": 0.9, "reason": "ticket id"}
    if AGG.search(query) and has_warehouse():    return {"route": "sql", "confidence": 0.8, "reason": "aggregation keywords"}
    r = json.loads(llm("router", ROUTER_SYS, [{"role": "user", "content": query}], max_tokens=80).text)
    if r["confidence"] < 0.6:
        r = {"route": "vector", "confidence": r["confidence"], "reason": "low confidence → safe default"}
    log_route(query, r)
    return r

HANDLERS = {"no_retrieval": answer_directly, "vector": rag_answer, "sql": text2sql_answer,
            "graph": graph_answer, "web": web_answer, "api": api_answer, "clarify": ask_clarifying_question}

def handle(query, caller, history):
    r = route(query, history)
    return HANDLERS[r["route"]](query, caller, history), r

Pros

  • Right evidence for the question type — the fix for a whole class of "RAG is bad at numbers" complaints.
  • Cost and latency savings: no-retrieval routing skips embedding + search + large context entirely, and it's often 15–30% of conversational traffic.
  • Aggregations become correct instead of hallucinated arithmetic over five sampled rows.
  • Clear debugging: route is the first branch in the trace.

Cons

  • Misroutes are expensive: an SQL question sent to vector search returns plausible-looking nonsense; a vector question sent to SQL returns an error or an empty table.
  • Each route is a backend to build, secure, and evaluate — text2SQL in particular has its own hard failure modes (wrong joins, wrong filters, no ACLs).
  • Router drift as the product adds features and question types.
  • Another classification call (fold it in).

When to use

  • Any system with more than one evidence backend.
  • Any assistant where users ask both "what does the policy say" and "how many did we do last month".
  • Even single-backend systems benefit from the no_retrieval and clarify routes alone.

When NOT to use

  • Single-backend, single-question-type systems (a pure document Q&A over one corpus) — add only the no-retrieval branch.
  • Don't build routes for backends you haven't built; route to "I can't answer that kind of question yet" instead, which is a better user experience than a hallucinated number.

Production at scale

Start with two routes and grow. no_retrieval vs vector is worth building on day one. Add sql when you have a warehouse and the aggregation-question rate justifies it (measure it in logs first).

Fan-out for ambiguity instead of guessing: for medium-confidence cases, query two backends in parallel and let the synthesis step use whichever produced usable evidence. Costs more, fails less. Cap it.

text2SQL needs its own hardening: a read-only role, a schema allow-list, row-level security matching the caller (authentication & authorization), query timeouts, LIMIT injection, and a validation pass before execution. A router that sends user text to a database is an injection surface (prompt-injection defence).

Route-level SLOs and dashboards. Latency, cost, error rate, and satisfaction per route. An overall p95 hides that the SQL route takes 8 seconds.

Misroute detection. Sample answers per route through an LLM judge asking "was this the right kind of evidence for this question?" (LLM-as-judge). Feed misroutes back as router examples (data flywheel).

Security per route. Each backend enforces its own ACLs; the router must never be the security boundary. The caller's identity is passed to every handler and enforced there (authentication & authorization).

Graceful "no route" answers. When the question needs a backend you don't have, say so specifically ("I can't query live billing data") rather than retrieving something adjacent. This is the abstention discipline applied at routing level (refusal / no-answer path).

Cost attribution by route feeds the $/task number (cost levers); routes have wildly different unit costs.

Prod checklist

  • Route set matches actual backends; unbuilt routes answer with an explicit limitation
  • Rules first, LLM classifier for the rest, folded into the rewrite call
  • Low-confidence default to hybrid retrieval or bounded fan-out
  • Per-route context assembly with its own budget
  • text2SQL hardened: read-only role, schema allow-list, row-level security, timeouts, LIMIT, validation
  • ACLs enforced inside every handler, never by the router
  • Route logged on every request; per-route latency/cost/quality dashboards
  • Misroute sampling via judge; misroutes become router examples

Metrics to watch

MetricWhy
Route distribution and drift over timeProduct changes, router drift
Route accuracy on labelled setClassifier health
Per-route p95 latency, $/query, error rateSLOs and unit economics
Misroute rate from judge samplingSilent failures