Build unanswerable questions that meaningfully test RAG

Define absence relative to a corpus

LangChain’s evaluation concepts distinguish dataset inputs, reference outputs, and evaluation logic. An unanswerable label needs an additional corpus boundary: which snapshot and eligibility rules were checked? A question can be answerable on the public web but unsupported inside the application’s approved documents. Record that distinction. Also separate missing evidence from ambiguity, prohibited access, and a temporarily failed search service. Those conditions can require different user responses and should not share one unexplained refusal label.

Evidence: Evaluation concepts [1]

Author a near-match challenge

Create a fictional help center describing password reset and account recovery but no account-deletion retention period. Ask how long deleted records remain. The correct response identifies that the approved material does not establish the period, without borrowing a number from the reset policy. Add a related answerable question about the reset link’s lifetime and a partially answerable question combining reset and retention. These controls test whether the assistant can answer supported portions without inventing the missing policy.

Verify the negative label carefully

Search the eligible corpus using synonyms, likely headings, and relevant metadata before declaring absence. Keep the audit notes and snapshot ID with the case. Ask a second reviewer to inspect disputed negatives when feasible. A retriever miss should not be relabeled as an unanswerable question merely because the initial top results were empty. Conversely, a source in another tenant or an unpublished draft does not make the question answerable for the current user.

Score behavior beyond a refusal phrase

Check whether the answer avoids unsupported specifics, explains the bounded gap, and offers a relevant next step without pretending to perform it. An assistant that says “I do not know” and then supplies a guessed period still fails. An assistant that refuses the answerable control also fails the usefulness check. Track unsupported-answer rate and unnecessary-abstention rate separately. Retain the actual returned passages to determine whether a confident error was prompted by a misleading near match.

Refresh negatives when knowledge changes

A valid negative can become answerable after a policy publication or source update. Revalidate the label whenever the relevant corpus changes, preserving the old snapshot for historical comparisons. Treat changes in answerability as dataset updates, not spontaneous model improvements. Maintain a small set of stable synthetic negatives alongside production-derived cases so a corpus expansion does not erase all refusal tests. Record who owns reviewing each negative label and what evidence would cause it to change. Include one negative whose wording strongly resembles an answerable question except for a changed date or product version. That isolates whether the assistant respects the unsupported condition instead of answering the familiar neighboring task.

Common questions