Audit GraphRAG relationships back to source passages

Define what an edge claims

GraphRAG’s dataflow links extracted knowledge to text units and derives relationship descriptions. An edge should be treated as a claim with scope, not merely a line connecting names. Record source and target entity IDs, relation meaning, source text unit, revision, and any temporal qualifier. A summary description can broaden a narrow source statement. Your evaluation should inspect whether the generated description still follows from the original passage rather than trusting the existence of a provenance ID alone.

Evidence: GraphRAG default dataflow [1]

Author direction and negation traps

Create fictional passages stating that Company A supplies Company B during a pilot and that Company C does not own Company B. Ask who supplies B and who owns it. The expected graph preserves the supply direction and pilot limitation and must not turn the negated ownership statement into a positive edge. Include another passage about a similarly named company. The acceptance rule requires the correct entities and exact relationship scope, not just a path between plausible-looking node labels.

Review provenance as evidence, not decoration

Open the referenced text unit and confirm it supports the relationship description. A valid identifier may point to a passage that mentions both entities without asserting the relationship. Record unsupported edges, wrong-direction edges, missing qualifiers, and invalid source links separately. If several sources jointly support a relationship, preserve that structure. A relationship inferred from multiple premises should be labeled as an inference with inspectable premises rather than presented as a direct quotation from one source.

Test paths before generating conclusions

For a multi-edge answer, verify every required relationship and the rule for composing them. Supply chains, ownership, and membership are not interchangeable relations, and many are not safely transitive. A valid A-to-B edge and B-to-C edge do not automatically support the statement that A owns C. Keep the path, relation types, and original passages with the answer judgment. This makes it possible to distinguish unsupported path reasoning from an incorrect individual edge.

Use provenance failures to guide repair

A wrong endpoint suggests entity resolution or extraction work; a missing qualifier suggests extraction or summarization work; an unsupported path suggests query planning or generation work. Retain representative cases for each failure class. Recheck derived summaries after a relationship correction so the old unsupported claim does not remain available through another retrieval path. The acceptance evidence should show that consequential graph claims can be independently inspected, not merely that the index produces connected results. Check whether an edge supported only by a withdrawn source remains eligible. Its historical provenance can remain recorded while its current evidentiary status changes.

Common questions