Write RAG reference answers that preserve acceptable alternatives

Separate a reference from the scoring rule

LangChain’s evaluation concepts describe reference outputs as part of datasets. A reference answer is useful only when its role is explicit. It may illustrate one acceptable response, provide expected facts, or drive a reference-dependent metric. Do not assume those roles are interchangeable. Store the required answer elements and evidence separately from one polished example so an evaluator can accept a shorter paraphrase or a different valid source without losing critical constraints.

Evidence: Evaluation concepts [1]

Author an alternative-answer fixture

Create a fictional policy allowing cancellation by email or through an account page. A reference that describes only email is incomplete as an acceptance specification if either method is valid. Record both allowed options, the required notice period, and the source passages. An answer mentioning one permitted method with the correct notice period can pass if the question asks for any valid method. If the question asks for all methods, the coverage requirement changes. The task wording determines the reference obligations.

Handle unsupported and partial answers

Include a case where the corpus supports the cancellation method but not the refund amount. The reference should identify the supported portion and expected qualification rather than invent a complete answer for convenience. Mark claims that must not appear, such as a guaranteed refund. Keep absence labels tied to the corpus snapshot. A reference-dependent evaluator can reward an unsupported expected answer unless the reference itself has been reviewed against applicable evidence.

Review source applicability and disagreement

For every required fact, retain source revision, effective date, relevant passage, and any authority rule. When reviewers disagree, inspect whether the question or reference is underspecified before treating the disagreement as evaluator noise. A reference can become stale after a legitimate policy update. Version the answer requirements and rerun both compared systems on the revised labels when necessary. Keep old references linked to old corpus snapshots for historical analysis rather than silently overwriting the ground truth.

Test the evaluator against valid variations

Create authored paraphrases, reordered answers, numerical equivalents with declared units, and alternate valid citations. Also create a near-match answer missing one critical condition. The evaluator should accept the valid variations and reject the consequential omission. Preserve those checks when changing judge prompts or similarity thresholds. The reference set should make the scoring decision more inspectable, not narrow the product’s acceptable behavior to the exact wording a content author happened to write first. Include a concise acceptable answer alongside a longer one. Both should preserve the required facts, preventing the reference from unintentionally rewarding verbosity or unnecessary background detail.

Common questions