Test GraphRAG community summaries for support and coverage

Define the summary’s evidence boundary

GraphRAG documents generated community reports as derived artifacts used for reading and downstream search. Treat them as generated evidence transformations, not authoritative source documents. Identify which entities, relationships, and text units were available to each report. Record report revision and source lineage. A summary may compress many graph records, so its evaluation needs a clear required-fact inventory rather than an expectation that every detail must appear in the same wording.

Evidence: GraphRAG indexing overview [1]GraphRAG default dataflow [2]

Build a minority-exception fixture

Author a graph community in which most fictional branches offer same-day support while one regulated branch requires prior approval. Ask for the general service pattern and any important exception. A summary that says all branches offer immediate support fails even if it captures the dominant theme. Another summary that cautiously says most branches do so but omits the approval exception may be supported yet incomplete. Keep those failure types separate in the fixture’s expected judgments.

Trace claims through intermediate artifacts

For each consequential summary statement, inspect the relationship description and original supporting text, not just the already summarized entity description. Errors can accumulate across extraction and summarization. Record whether an unsupported claim first appeared in the graph or in the community report. If a source conflict exists, check whether the report preserves uncertainty instead of selecting the more common statement. A polished paragraph should not hide that its apparent consensus comes from repeated copies of one source.

Measure coverage with explicit priorities

List critical facts, optional background, and known exclusions for the report’s intended query use. Score coverage of those requirements separately from factual support. A summary can avoid unsupported claims by saying almost nothing, so support alone is not enough. Conversely, a long report can cover many facts while inventing connections. Inspect entity confusion, time qualifiers, numerical aggregation, and missing exceptions as distinct failure groups. Keep the reviewed fact inventory stable when comparing report-generation prompts.

Test the report in downstream answers

Ask held-out questions that depend on both the dominant theme and the exception. Verify that retrieval exposes enough provenance to inspect the summary’s claims and, where needed, recover original evidence. If a report cannot support a narrow factual answer, the application may need a more specific retrieval path. After changing a source or graph relation, confirm that affected reports are refreshed or made ineligible. A stale derived summary can reintroduce a corrected claim even when the underlying graph is current. Include repeated copies of a minority exception and a single authoritative general rule. Frequency is not authority, so report evaluation should follow the declared source policy rather than automatically treating repeated text as stronger support.

Common questions