Check whether a RAG answer covers the required sources

Start from the question’s obligations

RAGBench presents an explainable benchmark framing for RAG evaluation. For a product-specific task, create your own required-evidence map rather than borrowing dataset labels without checking fit. A comparative question may require a rule from two vendors; an eligibility question may require a base policy and a local exception. Define those obligations before looking at the answer. Otherwise the evaluator can mistake a long, well-cited explanation of one source for a complete response to a multi-source request.

Evidence: RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems [1]

Build a two-policy comparison

Author two fictional service policies. Provider A permits monthly cancellation; provider B permits it only after a minimum term. Ask for the difference and any qualification. An answer describing A accurately with three citations to A still misses half the task. The fixture passes only if it identifies both policies and B’s qualification. Treat equivalent authoritative versions as valid alternatives if the question’s date and scope permit them, rather than requiring one arbitrary document ID.

Score retrieval coverage and answer coverage

For each evidence group, record whether an applicable supporting passage reached the final context and whether the answer used it correctly. This creates four useful states: present and used, present but omitted, missing and appropriately acknowledged, or missing and invented. A single percentage cannot explain those differences. Include the group labels and source relationships in the evaluation record so a retrieval team can distinguish an absent document from a generator that ignored available evidence.

Handle conflicting evidence explicitly

Sometimes the required coverage includes acknowledging a conflict rather than selecting a convenient winner. Store the rule for determining authority or time applicability when one exists. If no resolution is available, the expected answer should state the disagreement and the missing information needed to decide. Do not reward an answer for mentioning every source while blending incompatible claims into one apparent policy. Coverage means using the evidence according to the task, not mechanically listing document titles.

Use omissions to focus the next test

Group failures by missing evidence type: exception, comparator, date, numerical input, or second reasoning step. If the group is absent from retrieval, repair corpus eligibility or search. If it is available but omitted, test answer planning or context assembly with the same passages. Preserve a concise coverage matrix with question, required group, eligible source alternatives, context presence, and answer treatment. That artifact makes review more useful than an aggregate citation count. Add a case whose optional background is absent but all mandatory evidence is present. It should pass, preventing the coverage rule from becoming an indiscriminate demand to mention every available source.

Common questions