GraphRAG evaluation: test relationships, summaries, and cost

Define which GraphRAG system you are evaluating

Graph-based retrieval is a broad design family. This guide uses Microsoft GraphRAG as a concrete documented example, while the evaluation worksheet can be adapted to other implementations. Microsoft documents local search, global search, DRIFT search, and a basic vector-search mode; these paths should not be treated as interchangeable configurations.

Its local-search documentation describes combining extracted entities and relationships with document text. Global search uses community reports to assemble a dataset-wide response. DRIFT incorporates community information and follow-up local searches. These are architecture descriptions from the project documentation checked on October 16, 2026, not a claim that any mode wins on your data.

Evidence: GraphRAG: Query Engine [1]GraphRAG: Local Search [2]GraphRAG: Global Search [3]GraphRAG: DRIFT Search [4]

Build a test set that can expose graph-specific errors

Rubrex recommends labeling question families before reviewing answers. A local fact question tests a different capability from a request for organization-wide themes. Give each family its own denominator so that many easy fact questions cannot conceal poor relationship reasoning. Include ambiguous entity names, changing relationships, and questions whose answer is absent.

For a small pilot, a reviewer can author questions from a controlled document collection and record the exact supporting passages. Hold back a separate set for the final comparison. Do not generate all questions from the same community summaries being evaluated: an omitted source topic could then disappear from both the index and the test set.

Audit extracted relationships before scoring the final answer

Select a sample of entity records, relationship records, and summary claims. For each, retain the extraction output, source identifier, relevant passage, and a reviewer decision. Mark unsupported, reversed, outdated, or unresolved relationships separately. The number of graph edges alone measures size, not correctness.

Calculate reviewed-edge support as supported reviewed edges divided by all reviewed edges. Label this a sample-based estimate and state the sampling method. To assess missing relationships, create a separate reference set of required relationships from source documents and count how many are represented correctly. That denominator is the labeled reference set, not every relationship that might exist in the corpus.

Apply the same discipline to community summaries. Break an important summary into checkable claims and connect each claim to original passages. A summary citing another generated summary is an intermediate link; continue to source evidence for the sampled audit. Record ambiguous claims instead of forcing them into a positive label.

Keep answer support, coverage, and retrieval diagnoses separate

Use ordinary RAG answer checks alongside graph-specific diagnostics. A supported-claim rate asks whether checkable answer claims follow from the supplied evidence. Required-point coverage asks whether the answer includes the points the reference task required. Both can be high or low independently; a short answer can be fully supported while omitting the central relationship.

For global synthesis, define the themes and acceptable alternative formulations before comparing systems. Review counterexamples and minority patterns as well as common themes. Avoid asking an unconstrained judge which answer feels more comprehensive: length and confident phrasing can obscure unsupported generalization.

An illustrative relationship error that answer fluency misses

Consider three invented passages: D1 says Cedar depends on Harbor; D2 says Platform owns Harbor; D3 says Cedar changed owners last month. The test asks which team owns the component Cedar depends on. The reference answer is Platform, supported by D1 and D2. D3 is a distractor about Cedar itself.

Suppose an authored candidate answer says the current Cedar owner also owns Harbor. Even if that answer includes all three citations, it fails the required relationship. The review record should mark the unsupported ownership claim, identify the missing or misused D2 relationship, and retain the source passages. This is an illustrative diagnosis, not output obtained from a model or a GraphRAG run.

A useful follow-up isolates the stage: inspect whether the correct edge existed in the index, whether retrieval selected it, and whether generation used it correctly. Rebuilding the whole graph before locating the failure makes the next comparison harder to interpret.

Compare like-for-like snapshots and include document changes

Freeze document contents, permissions, task labels, model configuration, and evaluation rules for the baseline and candidate. Run both on the same questions. Retain task-level outcomes and investigate improvements and regressions rather than comparing a single aggregate score. Repeat variable paths enough to expose unstable results, within a stated budget.

Then run a separate update trial. Replace an ownership statement, delete a synthetic record, and add a new document with a unique harmless marker. Check whether derived entities, relationships, summaries, and answers reflect the intended snapshot. Track the time and work required to make the update visible. A fresh passage retrieved beside an outdated summary is still a mixed-version evidence problem.

Access rules also need a separate check: an answer built from a summary must not reveal restricted source content to a caller who cannot access it. Use isolated synthetic roles and verify the permission design of your implementation. This is a proposed acceptance condition, not a claim that Microsoft GraphRAG provides a complete authorization layer.

Make the decision with indexing costs and a retained failure list

Record extraction and summarization work during indexing separately from query-time work. Add reindexing after changes, evaluator costs, retries, and incomplete queries. For an illustrative planning calculation, total period cost equals initial indexing plus update work plus all query and evaluation work; divide by accepted tasks only when defining cost per successful task. If no tasks pass, report zero accepted tasks and the total spend; cost per successful task is undefined. Do not substitute an unverified vendor rate.

Keep GraphRAG when the measured gains on required relationship or synthesis tasks justify its operating burden and no critical boundary fails. Keep the vector baseline when it meets the actual task contract with less work. If gains appear only in one question family, consider a bounded route for that family and evaluate the routing decision separately. Record the configuration, unresolved failures, owner, and the next review trigger.

Common questions