Measure citation precision and citation recall separately

Define two different denominators

The ALCE paper evaluates generated text with citations; its research motivates checking citation behavior rather than treating references as decoration. For a local review, define citation precision as supported citation links divided by evaluated citation links, and citation recall as adequately cited evidence-requiring claims divided by those claims. These are declared local definitions, not a claim that every tool uses identical formulas. State how a citation attached to several claims is expanded into links before counting.

Evidence: Enabling Large Language Models to Generate Text with Citations [1]

Work an inspectable support example

An authored answer says a fictional plan costs forty units monthly and includes weekend support, citing passage A. Passage A establishes the monthly price but says nothing about support. Split the sentence into two factual claims. If both are treated as linked to A, one of two links is supported, giving local citation precision of 0.5. If both require evidence and only the price is adequately cited, citation recall is also 0.5. This is a teaching fixture, not a model result.

Handle alternatives and shared support

Some claims require two sources together, while others can be supported by either of several equivalent passages. Store those relationships explicitly instead of insisting on one exact citation ID. An answer can cite a different valid source and still pass. Conversely, citing a relevant document does not prove support for the claim. Check the actual passage, its effective date, and whether it applies to the described account or product. Keep support judgments separate from whether the source itself is authoritative.

Review implicit and compound claims

Claims can be introduced by headings, comparisons, arithmetic, and connective words, not just standalone sentences. “Therefore all customers qualify” may add an unsupported conclusion even when the preceding facts are cited correctly. Decide which common-knowledge or purely procedural statements require citations in your application. Keep disagreements visible and adjudicate unclear cases before automating the score. Otherwise differences in claim segmentation can look like changes in citation quality when the underlying answer stayed the same.

Use the failure type to choose a repair

Low link support suggests wrong citation attachment, unsupported generation, or misleading context assembly. Low coverage with otherwise sound links suggests that the answer leaves some factual claims uncited. Evaluate both on the same held-out cases after a fix. Preserve answer text, claim inventory, source excerpts, citation links, and judgment reasons. That record lets an engineer reproduce a failure without relying on a screenshot of a citation badge or an unexplained aggregate percentage. Document whether unsupported citation links attached to nonfactual wording are excluded, because inconsistent inclusion changes the precision denominator.

Common questions