Evaluate self-reflective RAG decisions without trusting self-scores

Describe the method accurately

The 2023 Self-RAG paper trains models to retrieve and reflect using special reflection tokens. Prompting an arbitrary assistant to rate its own answer is a different system. State which design you are testing, including model training, available signals, and inference policy. Do not transfer the original paper’s reported performance to a new model or workflow. A documentation-based evaluation plan can borrow the separation of retrieval and reflection decisions without claiming that it reproduced the learned method.

Evidence: Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection [1]

Build decisions with independent labels

Author cases where retrieval is necessary, unnecessary, and insufficient even after a search. For a fictional product question, one case requires a current compatibility rule while another asks to rephrase user-supplied text without external facts. Label expected behavior from the task and evidence, not from the system’s own critique. Add a confident but unsupported answer snapshot. The fixture should fail that answer even if the system assigns itself a favorable support or usefulness signal.

Compare policies under controlled inputs

Use a fixed baseline, such as always retrieving under the same evidence budget, and a declared self-reflective policy. Preserve retrieval decisions, reflection outputs, generated text, context, and final judgments. Score answer acceptance and unnecessary retrieval separately. A policy that saves calls by skipping required evidence fails even if its average latency improves. Conversely, additional retrieval is not a quality gain unless it adds applicable evidence or improves a required outcome under the agreed constraints.

Test self-critique as a fallible evaluator

Compare reflection signals with independent labels on paraphrases, negations, stale sources, and partially supported answers. Inspect false confidence and unnecessary rejection separately. If the same model generates and judges an answer, shared blind spots can produce agreement without correctness. Keep human-owned or deterministic checks for critical requirements. A fluent critique that explains a wrong answer convincingly is a failure case to retain, not evidence that the model has verified its reasoning.

Bound the inference policy and reporting

Declare stopping rules, retrieval limits, and how the application handles conflicting reflection signals. Measure calls and failures across the complete workflow rather than only successful final answers. Keep the evaluation dataset separate from policy tuning and preserve case-level traces for disputes. Report which decisions were tested and which remain unknown. The useful result is evidence about a specific retrieval-and-reflection policy, not a broad claim that self-reflection makes an assistant reliable across tasks. Include an answer whose self-critique notices a real defect but the final revision repeats it. Evaluate the revised output independently; acknowledging an error in a trace is not equivalent to repairing the user-facing result.

Common questions