Test entity resolution in a GraphRAG index

Define identity independently of extracted labels

GraphRAG’s documented dataflow extracts entities and merges some graph information by entity labels and types. Your evaluation needs a task-specific identity rule beyond trusting those generated labels. Define the stable entity behind each authored mention and which aliases are acceptable. Two organizations can share a short name; one organization can appear under several historical names. Keep evidence for the identity relationship so an evaluator can distinguish an intentional alias from an accidental text match.

Evidence: GraphRAG default dataflow [1]

Author a merge-and-split fixture

Create fictional records for Northstar Logistics and Northstar Analytics, plus an abbreviated mention of each. Add a documented rename of the logistics organization. The expected graph keeps the two companies separate while connecting the renamed and abbreviated logistics mentions to the same entity when their context supports it. Include an unresolved short-name mention with insufficient context. A cautious unresolved result can be preferable to a forced merge that transfers a contract or risk relationship to the wrong company.

Measure the two error directions

Record false merges, where distinct entities become one, and false splits, where one entity becomes several. Keep the judged mention pairs and source text visible. Aggregate pair scores can be distorted by many easy negative pairs, so report the difficult alias and shared-name groups separately. Inspect whether a model or normalization change fixes one class while worsening the other. Do not treat a smaller graph as automatically better; it may simply have collapsed important distinctions.

Trace identity errors into answers

Ask an authored question about which company holds a particular contract. Compare the expected source-backed relationship with the graph path and final answer. A false merge can create a connection never asserted by any source, while a false split can hide a valid cross-document connection. Preserve entity IDs, mention provenance, relationship endpoints, and source revisions in the failure record. This separates an extraction or resolution defect from a later query or generation error.

Gate resolution changes on critical distinctions

Choose acceptance requirements around consequential entity groups, not only an overall graph statistic. Retain an explicit unresolved state when identity cannot be established. Recheck community membership and generated descriptions after a merge-policy change because derived artifacts may still encode the old identity decision. Keep a small set of known aliases and known same-name nonmatches as regression controls. The release evidence should explain which distinctions are preserved and which ambiguous mentions still require review. Test a source that explicitly distinguishes two same-name entities. If the index still merges them, preserve that contrastive passage with the failure so the repair can target identity evidence rather than merely adding more aliases.

Common questions