Query understanding → Routing (vector, SQL, graph, web, or no retrieval)
What it is
A RAG system that sends everything to the vector index answers "how many contracts did we sign last quarter" by retrieving five contracts and letting the model guess a count. The question needed SELECT COUNT(*). Routing classifies the information need and dispatches:
| Route | Question shape | Backend |
|---|---|---|
| No retrieval | Chit-chat, formatting requests, "summarise what you just said", general knowledge | Answer directly (saves latency and cost) |
| Vector / hybrid | "What does the policy say about X" — narrative, unstructured | The RAG pipeline (retrieval & ranking) |
| SQL / text2SQL | Counts, sums, averages, rankings, "how many", "top 5 by revenue" | Warehouse / OLAP |
| Graph | "Which vendors are connected to this entity", multi-hop relationships, "what themes recur" | Knowledge graph (graphRAG / knowledge graph) |
| Web / external | Current events, public facts, competitor info | Web search tool |
| Direct API | "What's the status of ticket 4471", live state | System of record |
| Clarify | Genuinely ambiguous | Ask one question back |
Routing is a classification problem (routing & escalation is the model-tier cousin; this is the backend cousin). It's usually the cheapest large quality win in a mature system, because it stops the pipeline from answering the wrong kind of question with the wrong kind of evidence.
How to implement
- Enumerate the routes you actually have. Don't build a router for backends that don't exist.
- Rules first: regex/keyword signals cover a surprising share ("how many", "total", "average", "top N" → SQL; ticket-ID pattern → API; "hi"/"thanks" → no retrieval).
- LLM classifier for the rest: fast tier, closed label set, one example per route including the tricky boundaries. Fold into the rewrite call (query rewriting) as an extra output field.
- Confidence and fallback: low confidence → default to hybrid retrieval (the safest general route), or to a multi-route fan-out that queries two backends and lets reranking/synthesis decide.
- Per-route context assembly (context engineering): SQL results become a table block, API results a structured block, retrieved chunks passage blocks — each with its own budget.
- Per-route evaluation: route accuracy against a labelled set, and end-to-end quality per route (a correct route with a broken backend is still a failure).
- Log the route on every request; it's the first field you look at when debugging a bad answer.
Small example
ROUTES = ["no_retrieval", "vector", "sql", "graph", "web", "api", "clarify"]
AGG = re.compile(r"\b(how many|count|total|sum|average|avg|median|top \d+|most|least|trend|per (month|quarter|year))\b", re.I)
TICKET = re.compile(r"\b(TICKET|INC|CHG)-\d{3,}\b", re.I)
ROUTER_SYS = """Classify the user's question into exactly one route. Output JSON: {"route": ..., "confidence": 0-1, "reason": "..."}
Routes:
- no_retrieval: greetings, formatting, meta-questions about the conversation, general knowledge needing no company data
- vector: questions answered by narrative text in internal documents (policies, contracts, wikis)
- sql: aggregations, counts, rankings, time series over structured business data
- graph: relationships between entities, multi-hop connections, cross-document themes
- web: public/current information not in internal systems
- api: live status of a specific record identified by an id
- clarify: the question is too ambiguous to route
<example><q>How many vendors did we onboard in Q3?</q><out>{"route":"sql","confidence":0.95,"reason":"aggregation over structured vendor data"}</out></example>
<example><q>What does the MSA say about termination?</q><out>{"route":"vector","confidence":0.93,"reason":"narrative clause lookup"}</out></example>
<example><q>Thanks, can you shorten that?</q><out>{"route":"no_retrieval","confidence":0.98,"reason":"formatting request about prior answer"}</out></example>"""
def route(query: str, history) -> dict:
if TICKET.search(query): return {"route": "api", "confidence": 0.9, "reason": "ticket id"}
if AGG.search(query) and has_warehouse(): return {"route": "sql", "confidence": 0.8, "reason": "aggregation keywords"}
r = json.loads(llm("router", ROUTER_SYS, [{"role": "user", "content": query}], max_tokens=80).text)
if r["confidence"] < 0.6:
r = {"route": "vector", "confidence": r["confidence"], "reason": "low confidence → safe default"}
log_route(query, r)
return r
HANDLERS = {"no_retrieval": answer_directly, "vector": rag_answer, "sql": text2sql_answer,
"graph": graph_answer, "web": web_answer, "api": api_answer, "clarify": ask_clarifying_question}
def handle(query, caller, history):
r = route(query, history)
return HANDLERS[r["route"]](query, caller, history), r
Pros
- Right evidence for the question type — the fix for a whole class of "RAG is bad at numbers" complaints.
- Cost and latency savings: no-retrieval routing skips embedding + search + large context entirely, and it's often 15–30% of conversational traffic.
- Aggregations become correct instead of hallucinated arithmetic over five sampled rows.
- Clear debugging: route is the first branch in the trace.
Cons
- Misroutes are expensive: an SQL question sent to vector search returns plausible-looking nonsense; a vector question sent to SQL returns an error or an empty table.
- Each route is a backend to build, secure, and evaluate — text2SQL in particular has its own hard failure modes (wrong joins, wrong filters, no ACLs).
- Router drift as the product adds features and question types.
- Another classification call (fold it in).
When to use
- Any system with more than one evidence backend.
- Any assistant where users ask both "what does the policy say" and "how many did we do last month".
- Even single-backend systems benefit from the
no_retrievalandclarifyroutes alone.
When NOT to use
- Single-backend, single-question-type systems (a pure document Q&A over one corpus) — add only the no-retrieval branch.
- Don't build routes for backends you haven't built; route to "I can't answer that kind of question yet" instead, which is a better user experience than a hallucinated number.
Production at scale
Start with two routes and grow. no_retrieval vs vector is worth building on day one. Add sql when you have a warehouse and the aggregation-question rate justifies it (measure it in logs first).
Fan-out for ambiguity instead of guessing: for medium-confidence cases, query two backends in parallel and let the synthesis step use whichever produced usable evidence. Costs more, fails less. Cap it.
text2SQL needs its own hardening: a read-only role, a schema allow-list, row-level security matching the caller (authentication & authorization), query timeouts, LIMIT injection, and a validation pass before execution. A router that sends user text to a database is an injection surface (prompt-injection defence).
Route-level SLOs and dashboards. Latency, cost, error rate, and satisfaction per route. An overall p95 hides that the SQL route takes 8 seconds.
Misroute detection. Sample answers per route through an LLM judge asking "was this the right kind of evidence for this question?" (LLM-as-judge). Feed misroutes back as router examples (data flywheel).
Security per route. Each backend enforces its own ACLs; the router must never be the security boundary. The caller's identity is passed to every handler and enforced there (authentication & authorization).
Graceful "no route" answers. When the question needs a backend you don't have, say so specifically ("I can't query live billing data") rather than retrieving something adjacent. This is the abstention discipline applied at routing level (refusal / no-answer path).
Cost attribution by route feeds the $/task number (cost levers); routes have wildly different unit costs.
Prod checklist
- Route set matches actual backends; unbuilt routes answer with an explicit limitation
- Rules first, LLM classifier for the rest, folded into the rewrite call
- Low-confidence default to hybrid retrieval or bounded fan-out
- Per-route context assembly with its own budget
- text2SQL hardened: read-only role, schema allow-list, row-level security, timeouts, LIMIT, validation
- ACLs enforced inside every handler, never by the router
- Route logged on every request; per-route latency/cost/quality dashboards
- Misroute sampling via judge; misroutes become router examples
Metrics to watch
| Metric | Why |
|---|---|
| Route distribution and drift over time | Product changes, router drift |
| Route accuracy on labelled set | Classifier health |
| Per-route p95 latency, $/query, error rate | SLOs and unit economics |
| Misroute rate from judge sampling | Silent failures |
Related
- Structured + unstructured hybrid — the SQL route in depth
- GraphRAG — the graph route
- Routing/escalation — model tiers, same pattern
- Refusal path — "I don't have that backend"
- Injection — text2SQL as an attack surface