Evaluate corrective RAG retrieval decisions against a fixed baseline

Keep the original method distinct

The 2024 CRAG paper describes a retrieval evaluator that can trigger different knowledge-retrieval actions, including web search and evidence refinement. A generic retry loop is not automatically a reproduction of that method. State whether your implementation follows the paper or merely borrows the idea of corrective branching. Record its evaluator, decision thresholds, available actions, and source policy. The paper’s reported experiments do not establish an improvement for your own corpus or current model configuration.

Evidence: Corrective Retrieval Augmented Generation [1]

Author three decision conditions

Create a fictional product-support fixture with complete applicable evidence, irrelevant evidence, and partly useful evidence missing a regional condition. Label the appropriate branch for each under your policy: answer, seek approved additional evidence, or disclose an unresolved gap. Add a control where the original evidence is complete so correction should not replace it unnecessarily. The criterion checks both branch choice and the final supported answer, preventing extra searches from being treated as success by themselves.

Compare against a fixed retrieval baseline

Use the same original query set, corpus snapshot, answer model, and context allowance for the baseline and corrective system. Preserve initial evidence, evaluator output, chosen branch, additional sources, and final context. Count cases repaired, cases unnecessarily changed, and cases made worse. Measure extra calls, delay, and failure handling separately. If the corrective path receives a larger context budget, state that difference or run an equal-budget comparison before attributing a gain to the decision method.

Test external-source authority and safety

When fallback search reaches beyond the approved corpus, define eligible domains, authority requirements, date applicability, and data that may be sent in the query. Use a synthetic test query rather than disclosing private source content. Add a plausible but inapplicable web passage and an instruction-bearing distractor. The correction policy should not promote an external source merely because the initial retrieval was weak. Preserve source provenance and test how uncertainty is communicated when no approved evidence resolves the gap.

Calibrate the branch evaluator on failure costs

A false insufficient judgment wastes work and can replace good evidence; a false sufficient judgment can produce an unsupported answer. Evaluate those error types separately using held-out cases. Inspect whether confidence changes correspond to useful evidence distinctions rather than answer fluency. Keep an explicit stopping rule and a maximum work budget. The release record should show when correction is justified, when it stops, and what the application does when the correction attempt fails to improve evidence. Retain a case where correction retrieves newer but inapplicable evidence, so the branch is judged on applicability rather than recency alone.

Common questions