On this page
The short answer
Evaluate entity extraction by separating span detection, type assignment, value normalization, and complete-record acceptance. A partially overlapping name may be a useful diagnostic hit but an unusable output for the application. State the matching rule before scoring and keep missing, extra, and ambiguous entities distinct.
What to take away
- Separate spans, types, normalization, and identity.
- Name strict and relaxed matching distinctly.
- Check complete-record usability where required.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Choose the entity representation first
SpaCy’s Scorer documents labeled span precision, recall, and per-type results. An LLM extraction task may instead return normalized fields or linked identifiers, so choose the representation the application actually consumes before borrowing a span metric.
Define character offsets, token offsets, or source-value references consistently. Specify whether nested and overlapping entities are allowed, whether titles belong inside a person’s name, and how repeated mentions are represented. A reviewer cannot label boundary accuracy reliably when these decisions remain implicit.
Evidence: spaCy: Scorer [1]
Score stages without hiding downstream errors
Measure detection separately from entity type and canonical-value correctness. Preserve the original source text and the normalized output. A correct surface name linked to the wrong customer ID can be a more consequential failure than a harmless boundary mismatch.
Use strict matching for the production contract and optional relaxed matching for diagnosis. Report both with clear names. Do not present an overlap-based diagnostic score as exact extraction accuracy, and avoid counting two predicted fragments as two successful entities when they refer to one required source mention.
Build a fixture with ambiguous and repeated mentions
Consider an invented note mentioning Alex Morgan at Cedar Labs and another Alex Morgan at Harbor Group. The task requires the person tied to Cedar’s contract. A name-only match cannot establish the correct entity identity. Preserve organization context or the permitted reference identifier needed to disambiguate.
Include a missing value, a repeated mention, a nested organization phrase, and a misleading capitalization pattern. Define the expected output for each. Review full-record acceptance as well as field scores when downstream work requires all relevant entities to be correct together.
| Case or record | Expected check | Decision |
|---|---|---|
| Two people share a name | Use required organization context | Reject wrong identity despite name match |
| Partial overlapping name | Does not meet strict boundary rule | Diagnostic partial hit only |
| Repeated source mention | Defined deduplication policy applies | Do not inflate correct entity count |
Inspect errors by type and document condition
Keep counts of required entities, correct predictions, extras, misses, type errors, and normalization errors. Report per-type support so a frequent easy entity cannot obscure a rare difficult one. Preserve unresolved reference cases separately from candidate failures.
Use error categories to choose the repair: parsing, prompt examples, schema design, linking data, or reviewer guidelines. Compare the same source snapshot across candidates. A high extraction score on clean short notes does not establish performance on scanned documents or permission-restricted records that were never included in the test.
Limits of the evidence
The names and organizations are fictional examples. No extractor was tested. Span scoring, normalized-value matching, and entity linking answer different questions, and a reference set must define their boundaries before results are interpretable.
Common questions
Can partial overlap count as correct?
It can be a separate diagnostic measure. Use the application’s required boundary and value rules for acceptance.
Is field accuracy enough?
Not when the downstream task needs a complete correct record. Report record-level acceptance alongside field and entity-type results.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- spaCy: Scorer Explosion · Living reference · Maintained documentationReviewed: span scoring, entity precision/recall/F-score and per-type results. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.