Knowledge-Graph RAG: What Building One Actually Taught Me

Everyone building RAG systems eventually hits the same wall: vector search is great at "find me the paragraph that sounds like this question" and terrible at "which suppliers appeared in more than one tender last quarter?" The second question isn't a similarity problem. It's a structure problem — and structure is what graphs are for.

I hit that wall building KRAG, a knowledge-graph RAG system over B2B tender documents. This post is what I'd tell myself before starting.

When a graph earns its keep

Pure vector retrieval fails predictably on three question shapes:

• Multi-hop: "Which projects used the same contractor as project X?" — the answer lives in no single chunk. It lives in the path between chunks.
• Aggregation: "How many tenders mention penalty clauses above 5%?" — similarity search returns examples, not counts.
• Negation and absence: "Which bids did NOT include a warranty section?" — embeddings cannot represent what isn't there. A graph with a known schema can.

If your users only ask "what does the document say about X", skip the graph. Vector search plus a reranker will beat you to production by a month. The graph earns its cost only when the questions are relational.

The part nobody warns you about: extraction

The graph itself is the easy 20%. The hard 80% is entity and relation extraction — turning "the Contractor shall indemnify the Employer" into (Contractor)-[INDEMNIFIES]->(Employer) reliably, across thousands of documents that never agreed on a template.

Three lessons from doing this on real tender documents:

1. Fix the schema first, extract second. An open-ended "extract all entities and relations" prompt gives you a different ontology per document. Decide your node types (Party, Clause, Deadline, Amount, Deliverable) and your edge types up front, and force the extractor to fit them. A smaller, boring schema you can trust beats a rich one you can't.
2. Extraction is a batch job with a QA gate, not a pipeline step. LLM extractors fail quietly — a mangled date here, a swapped subject/object there. Sample and score every batch. The graph amplifies errors: one wrong edge corrupts every multi-hop answer that walks through it.
3. Keep provenance on every edge. Each relation should carry the document ID and span it came from. When an answer looks wrong — and one will, in front of someone important — you need to trace it back to the sentence in seconds, not re-run extraction to find out.

Hybrid retrieval, not graph-instead-of-vectors

The setup that actually worked was boring: vector search finds the neighborhood, the graph answers the question. Embed the question, retrieve candidate entities, then expand 1–2 hops from those nodes and hand the resulting subgraph — not raw chunks — to the model as context. The model gets structure it can reason over, and the graph never has to be complete, only correct where it's dense.

The inverse also matters: every graph answer should be able to fall back to plain retrieval when the subgraph is thin. A graph that refuses to answer outside its schema is a feature at write time and a bug at read time.

The numbers, two years in

Since I built KRAG the field has produced enough measurement to sanity-check the design calls above. On complex summarization-style questions, graph-augmented retrieval reports fifty to seventy percent better comprehensiveness than vector RAG; on multi-hop QA benchmarks, exact-match gains on the order of 0.39 to 0.47 over KG-RAG baselines. The number that vindicates the unglamorous work: one production report puts complex multi-hop accuracy going from around forty percent to over ninety — attributed not to a better model but to proper entity resolution and hierarchical community detection. The graph literature has also settled into recognizable families — entity-relation graphs, community-summary hierarchies, and text-centric graph-guided retrieval — and every family shares the same failure economics: extraction inherits every inconsistency in the source data, so duplicate nodes and ambiguous entities remain the tax everyone pays first.

What I'd do differently

• Start with 50 documents, fully verified, before scaling to thousands. Graph quality compounds; so does graph rot.
• Budget more for entity resolution than extraction. "ACME Ltd.", "Acme Limited" and "the Contractor" being three nodes instead of one broke more answers than any extraction error.
• Treat the graph as a cache of verified structure, not a source of truth. Documents change; edges must expire.

"Graph engineering" gets thrown around as if the graph database were the work. It isn't. The work is the unglamorous loop of schema discipline, extraction QA, and provenance — the same loop as any data engineering, just with edges. Get that loop right and the graph is the easiest part of the system.