On this page
The short answer
Provide a RAG judge with the evidence and rules needed for its specific task. Keep the application’s actual context distinct from ideal reference evidence, and prevent gold labels or later information from silently changing what the judge evaluates.
What to take away
- Declare each judge input’s role.
- Keep actual and ideal context separate.
- Test leakage with adversarial fixtures.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Define the judge’s information contract
Ragas documents context-based faithfulness, while DeepEval documents reference-dependent contextual recall. Those tasks need different inputs. A support judge should assess whether the answer follows from the supplied application context; a retrieval-recall judge may compare that context with reference evidence. Name the task and every input field. If ideal passages are passed to a support judge as though they were retrieved, the evaluation can accept an answer that the actual application had no evidence to produce.
Author an unavailable-evidence trap
Create a fictional question whose answer requires a regional exception. The application receives only the general policy, while the evaluation dataset retains the exception as reference evidence. An authored answer confidently states the exception. For a groundedness check, it fails because the actual context does not support it. For a task-correctness check against the reference, it may be factually correct. Both judgments can be useful, but merging the contexts or labels would hide the distinction.
Remove answer cues from inappropriate fields
Inspect case names, metadata, labels, and filenames for clues such as “bad answer” or the expected result embedded in an ID. Use opaque case identifiers where appropriate and keep acceptance labels outside model-visible inputs unless the metric explicitly requires reference content. Add paired fixtures differing only in those hidden cues to detect leakage. A judge that changes its support decision because a metadata field says “pass” is responding to the evaluation harness rather than the evidence task.
Preserve source boundaries and hostile text
Separate evaluator instructions, user question, answer, retrieved passages, and reference information using a documented serialization. Include an authored passage that asks the evaluator to ignore its rubric. The expected judge behavior follows the evaluation task, not instructions embedded in untrusted source text. Do not assume delimiters alone make the evaluator immune; inspect the result and retain failure reasons. Keep the exact serialized judge request for approved synthetic fixtures so another engineer can reproduce the issue.
Validate input construction before scoring scale
Review a small sample of judge requests against original application traces. Confirm that no passage was added, removed, or replaced during adapter construction and that dates and source IDs remain accurate. Pin the judge model and rubric for comparisons. If the adapter changes, rerun the leakage and unavailable-evidence fixtures before interpreting score trends. A well-written rubric cannot compensate for an evaluation pipeline that supplies the wrong evidence or reveals the desired answer through unrelated fields. Keep a checksum or exact snapshot of the approved synthetic judge input so adapter repairs can be compared against a stable evidence contract.
Limits of the evidence
The fixtures are authored and no judge-security guarantee is claimed. Different metrics legitimately require different reference inputs; verify the exact documented task and configuration.
Common questions
Can a judge see the reference answer?
When the declared metric requires it, yes; do not confuse that with a support judgment based only on actual context.
Why inspect serialized requests?
They reveal evidence substitutions, missing passages, and hidden label cues that are invisible in an aggregate score.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- Faithfulness metric Ragas · Living reference · Maintained documentationReviewed: claim-to-context faithfulness definition and implementation requirements. Accessed October 16, 2026.
- Contextual Recall Confident AI · Living reference · Maintained documentationReviewed: documented retrieval-context and expected-output dependence. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.