Evaluate RAG groundedness at the claim level

Define the evidence boundary

Ragas documents a faithfulness measure based on claims and retrieved context. Your local evaluation should state which evidence the answer was allowed to use: retrieved passages, permitted structured records, or both. A model’s plausible background knowledge does not automatically count as supplied support. Conversely, a claim can be supported by the supplied context while that context is stale or wrong. Keep groundedness distinct from external truth and document eligibility so the score has an interpretable scope.

Evidence: Faithfulness metric [1]

Create supported and contradicted claims

Use an authored passage stating that a trial lasts fourteen days and applies only to individual accounts. An answer says the trial lasts fourteen days, covers team accounts, and includes telephone support. The first claim is supported, the second contradicts the eligibility condition, and the third lacks evidence. Preserve those three categories instead of merging every failure into one generic hallucination label. The acceptance criterion for this fixture rejects either the contradicted eligibility or the invented support channel.

Account for deductions and calculations

An answer may combine facts rather than quote them. For a numerical claim, retain the source values, unit conversion, and arithmetic. For a comparison, identify which sources establish each side. Decide whether the inference is licensed by the task and evidence, not merely whether its wording resembles a passage. A judge should be able to explain which premise is missing when it rejects a claim. If the reasoning requires unavailable business rules, label the outcome unresolved rather than accepting a plausible conclusion.

Calibrate automated judgments on disagreements

Have reviewers label a small set containing paraphrases, negations, numerical claims, and partially supported compound sentences. Compare the automated judge’s reasons with that record before scoring a larger set. A fluent explanation from the judge is not a substitute for checking the cited evidence. Keep claim-splitting rules stable between versions. If a prompt change causes fewer, broader claims to be extracted, the apparent groundedness gain may come from a changed denominator rather than better answers.

Use groundedness with completeness checks

A short answer that repeats one safe fact can be fully supported while failing the user’s question. Pair claim support with a list of required answer elements and appropriate abstention behavior. When a failure occurs, inspect whether the unsupported claim originated in the source selection, context compression, or generation step. Save the exact supplied context and answer so the team can repair the first broken stage. Recheck both supported and rejected examples after the intervention. Keep contradictory source revisions visible rather than letting the judge choose whichever one makes the answer appear supported.

Common questions