Advanced RAG → GraphRAG / knowledge graph

GraphRAG Explained: Knowledge Graph RAG for Global Questions

What it is

Vector RAG retrieves the k passages most similar to a query. That structurally cannot answer two classes of question:

  • Global/aggregate questions: "What are the recurring themes across our 4,000 incident reports?" No single passage contains the answer; it's a property of the whole corpus.
  • Multi-hop relational questions: "Which vendors supply components to products affected by the Q3 recall?" The chain vendor → component → product → recall is a path through relationships, not a similarity neighbourhood.

GraphRAG builds a knowledge graph at ingest: an LLM extracts (entity, relation, entity) triples plus entity descriptions from each chunk; entities are resolved and merged across the corpus; the graph is partitioned into communities (Leiden clustering); and an LLM writes a summary for each community at several levels of hierarchy.

Query time has two modes:

  • Local search: find the entities in the query, pull their neighbourhood (related entities, relationships, and the source chunks), answer from that.
  • Global search: map-reduce over community summaries — ask each relevant community summary to contribute, then reduce into an answer. This is what makes "what themes recur" answerable.

How to implement

  1. Decide it's worth it first (see the cost warning below). Confirm you have real global/multi-hop questions failing on vector RAG.
  2. Define the schema: entity types (Person, Vendor, Product, Policy, Incident…) and relation types. A constrained schema massively improves extraction quality over open-ended triple extraction.
  3. Extract per chunk with an LLM: entities with types and descriptions, relationships with descriptions and a strength/weight. Structured output with schema enforcement (structured outputs & tool use).
  4. Resolve entities: normalise names, merge duplicates (embedding similarity + string similarity + LLM adjudication for ambiguous pairs). This is the hardest part and where quality is won or lost.
  5. Build the graph in a graph store (Neo4j, Memgraph, ArangoDB) or as tables + an in-memory graph library; keep chunk_ids on every node and edge for provenance.
  6. Community detection (Leiden) at multiple resolutions; summarise each community with an LLM, hierarchically.
  7. Query: classify local vs global (routing); local → entity lookup + neighbourhood expansion + source chunks; global → map over community summaries, reduce to an answer.
  8. Always keep vector retrieval alongside — most questions are still ordinary lookups; the graph augments, it doesn't replace.

Small example

EXTRACT_SYS = """Extract entities and relationships from the text.
Entity types: {types}. Relation types: {rels}.
Output JSON: {{"entities": [{{"name","type","description"}}],
               "relationships": [{{"source","target","type","description","strength":1-10}}]}}
Use exact names as written. Do not invent entities not present in the text."""

def extract_graph(chunk, types, rels):
    out = json.loads(llm("graph_extract", EXTRACT_SYS.format(types=types, rels=rels),
                         [{"role": "user", "content": chunk["text"]}], max_tokens=1200).text)
    for e in out["entities"]: e["chunk_ids"] = [chunk["chunk_id"]]; e["tenant_id"] = chunk["tenant_id"]
    for r in out["relationships"]: r["chunk_ids"] = [chunk["chunk_id"]]; r["tenant_id"] = chunk["tenant_id"]
    return out

def resolve_entities(entities, threshold=0.92):
    """Merge duplicates: exact-normalised match, then embedding similarity, then LLM adjudication."""
    by_key, merged = {}, []
    for e in entities:
        key = normalise(e["name"])
        if key in by_key:
            by_key[key]["chunk_ids"] += e["chunk_ids"]; by_key[key]["descriptions"].append(e["description"]); continue
        e["descriptions"] = [e.pop("description")]; by_key[key] = e; merged.append(e)
    vecs = embed_documents([m["name"] + ": " + m["descriptions"][0] for m in merged])
    for i, j in candidate_pairs(vecs, threshold):                          # ANN over entity embeddings
        if merged[i]["type"] == merged[j]["type"] and llm_says_same(merged[i], merged[j]):
            union(merged[i], merged[j])
    return dedupe(merged)

GLOBAL_MAP_SYS = """Given this community summary, list points relevant to the question with an importance score 0-100.
Output JSON: {"points": [{"point": string, "score": int, "entities": [string]}]}. Return [] if nothing is relevant."""

def global_search(question, communities, level=1, top_communities=20):
    relevant = rank_communities(question, [c for c in communities if c["level"] == level])[:top_communities]
    partials = parallel_map(lambda c: json.loads(llm("graph_map", GLOBAL_MAP_SYS,
                            [{"role": "user", "content": f"<summary>{c['summary']}</summary>\n<question>{question}</question>"}]).text),
                            relevant)
    points = sorted((p for r in partials for p in r["points"]), key=lambda p: -p["score"])[:40]
    return llm("graph_reduce", GLOBAL_REDUCE_SYS, [{"role": "user",
               "content": f"<question>{question}</question>\n<points>{json.dumps(points)}</points>"}])

def local_search(question, graph, caller, hops=2):
    seeds = link_entities(question, graph, tenant=caller.tenant_id)          # entity linking from the query
    nodes, edges = graph.neighbourhood(seeds, hops=hops, tenant=caller.tenant_id)
    chunk_ids = {cid for n in nodes for cid in n["chunk_ids"]} | {cid for e in edges for cid in e["chunk_ids"]}
    chunks = [c for c in chunk_store.get_many(chunk_ids) if caller_can_read(caller, c)]   # ACL re-check, always
    return generate(question, rerank_hosted(question, chunks, top_n=8), graph_context=(nodes, edges))

Pros

  • Answers questions vector RAG structurally cannot: corpus-level themes, patterns, "what's common across", multi-hop relationship chains.
  • Explainable paths: the answer comes with a traversable chain of entities and relations, each linked to source chunks.
  • Entity-centric views: "everything we know about Vendor X" assembled across thousands of documents.
  • Community summaries are themselves a valuable artifact (executive overviews of a corpus).

Cons

  • Expensive to build. An LLM call per chunk for extraction, plus entity resolution, plus community summarisation — commonly 10–100× the cost of plain embedding for the same corpus. Microsoft's own guidance flags indexing cost as the primary adoption barrier.
  • Extraction quality is the ceiling: missed or wrong triples produce a confidently wrong graph, and errors are hard to spot.
  • Entity resolution is genuinely hard — "Acme Corp", "Acme Corporation", "ACME" and the other Acme in a different industry.
  • Incremental updates are awkward: a changed document can invalidate entities, edges, and the community structure; naive re-clustering churns summaries.
  • ACLs on a graph are hard: a path may traverse a node the caller can't see, and the existence of an edge can leak information.
  • Much more infrastructure: graph store, extraction pipeline, community pipeline, two query paths.

When to use

  • Corpora where relationships are the domain: incident/investigation corpora, supply chains, org and entity networks, research literature, legal entity structures, fraud analysis.
  • When users demonstrably ask global "what patterns/themes" questions that vector RAG returns nonsense for.
  • Corpora stable enough that rebuild cadence is manageable.

When NOT to use

  • Ordinary document Q&A. If your questions are "what does the policy say about X", GraphRAG is a large cost for no benefit. This is the most common mistake.
  • Rapidly changing corpora where the graph is stale by the time it's built.
  • Before hybrid + reranking + parent–child are in place and measured.
  • Tight ingestion budgets — model the extraction cost against corpus size before prototyping.

Production at scale

Cost model before code. chunks × (extraction tokens in + out) × price plus community summarisation plus re-runs. For 1M chunks this is a serious number. Prototype on a stratified 1–2% sample, measure the answer improvement on the global-question subset of your golden set, then extrapolate cost. Many teams stop here — correctly.

Constrained schema beats open extraction. Fixed entity and relation types cut extraction noise dramatically and make resolution tractable. Start narrow (5–10 entity types) and expand only when evals demand it.

Hybrid architecture is the norm: vector RAG handles local lookups; the graph handles global and multi-hop. Route per query (routing). Keep chunk IDs on every node and edge so the graph always resolves back to citable source text (citations).

Incremental maintenance: extract only for changed chunks (change detection); merge into the graph; re-run community detection on a schedule (nightly/weekly) rather than per document; version community summaries so answers are reproducible. Treat a re-clustering like a re-index (versioning & migration).

ACLs: propagate tenant_id and ACL sets onto nodes and edges (a node inherits the union of its source chunks' ACLs — which means a node can be visible while some of its edges are not). Filter traversal by the caller's permissions, re-check ACLs on the source chunks before generation, and treat edge existence as sensitive. This is the least-solved part of GraphRAG in practice; if you can't do it correctly, restrict the graph to a non-sensitive corpus.

Cheaper alternatives to evaluate first: hierarchical summarisation (RAPTOR-style summary trees over clusters of chunks) gives much of the global-question capability at a fraction of the cost; metadata aggregation plus SQL (structured + unstructured hybrid) answers many "how many / which vendors" questions outright.

Evaluate the graph itself: sample triples for precision against human judgement; measure entity-resolution accuracy on a labelled set; then measure end-to-end answer quality on global vs local question sets separately.

Prod checklist

  • Global/multi-hop question classes documented and failing on vector RAG
  • Extraction cost modelled on a sample and extrapolated before committing
  • Constrained entity/relation schema; structured extraction with validation
  • Entity resolution pipeline with measured accuracy
  • chunk_ids on every node and edge; citations resolve to source text
  • Tenant/ACL propagated to nodes and edges; traversal filtered; source-chunk ACL re-checked
  • Incremental extraction for changed chunks; scheduled re-clustering; versioned community summaries
  • Query routing between vector (local) and graph (global) paths
  • Separate evals for local and global question sets; triple precision sampled
  • Cheaper alternatives (summary trees, SQL aggregation) evaluated first

Metrics to watch

MetricWhy
Extraction $ per 1k chunks; total build costThe adoption barrier
Triple precision; entity-resolution accuracyGraph quality ceiling
Global-question answer quality vs vector baselineJustification
Graph staleness (time since last rebuild)Freshness
Traversal ACL violations (must be 0)Security