On this page
The short answer
Validate the adapter that turns application traces into metric inputs before interpreting scores. Preserve the distinction between actual answers, retrieved passages, and reference evidence, then use negative controls to catch missing context, swapped fields, and broken source identities.
What to take away
- Declare each metric’s actual input contract.
- Keep missing context distinct from empty retrieval.
- Reject corrupted adapters before comparing scores.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Map fields by meaning before mapping API names
Current Ragas documentation separates faithfulness inputs from reference-dependent context precision; DeepEval lists different required fields for contextual relevancy and contextual recall. Those documented requirements are API facts, not interchangeable names for one universal RAG record. Pin the exact metric and version, then write an adapter contract from your application’s trace fields to the selected metric. An actual answer must remain the observed output, while a reference remains evaluation evidence. Passing both through a generic text field can change what the score means without producing a schema error.
Evidence: Faithfulness metric [1]Context Precision [2]Contextual Relevancy [3]Contextual Recall [4]
| Field | Meaning to preserve | Adapter check |
|---|---|---|
| question | Actual user task at the evaluated turn | Do not substitute later clarification. |
| actual_answer | Observed application output | Do not fill failures with a gold answer. |
| retrieved_context | Passages actually supplied to generation | Preserve order, content, source IDs, and revision. |
| reference_answer | Reviewed expected facts or acceptable answer | Supply only to metrics whose task uses it. |
| reference_context | Ideal supporting evidence retained for evaluation | Do not merge into actual retrieved evidence. |
| source_identity | Document, revision, and passage identity | Check linkage after normalization and deduplication. |
Keep unknown context separate from an empty result
Use an explicit application-side state for context unavailable because logging failed, distinct from a successful retrieval returning an empty list. These are proposed harness rules, not a claim that every library assigns the same score to null or empty inputs. A missing trace should produce an unscorable diagnostic or another declared error state, rather than a favorable default. An empty retrieval can still support an abstention test, while a metric requiring contextual evidence may need a separately defined applicability rule. Record that decision instead of coercing both states to an empty string.
Run three deliberate adapter corruptions
First, replace actual retrieved passages with ideal reference passages in an authored missing-exception case. The adapter audit should detect the mismatch against the preserved application trace before a judge is called. Second, swap the actual and reference answers in a case containing an unsupported claim. The field-lineage check should reject the swap rather than treating a favorable resulting score as application improvement. Third, attach the correct passage text to the wrong document revision. The provenance check should fail even if text-based scoring remains favorable. These are proposed negative controls with expected diagnostics, not executed framework results.
Declare when the metric is applicable
For each metric, list required inputs, allowed missingness, reference dependence, and the question it can answer. A context-support judgment does not establish source freshness; a reference-dependent recall judgment does not prove the reference is correct. Test one valid control alongside each corruption so the harness cannot pass by rejecting every record. Preserve normalization decisions such as whitespace changes, list ordering, and duplicate removal. If an adapter intentionally transforms evidence, retain both representations and explain which one the metric receives so comparisons do not silently mix different evidence conditions.
Choose tooling only after the contract survives
Require all three corruption controls to produce the intended adapter diagnostics and the valid control to reach scoring with its identity intact. Then compare candidate metrics on the same validated record, keeping metric-specific field requirements explicit. Save the adapter revision, field mapping, applicability decision, and trace comparison with each evaluation run. If the contract changes, replay the controls before interpreting score trends. A framework comparison cannot establish useful measurement when one adapter supplies the real retrieved context and another supplies an idealized substitute.
Limits of the evidence
This is a proposed adapter-validation method based on checked input documentation, not an executed Ragas or DeepEval trial. Null handling, required arguments, and API compatibility must be verified in pinned versions.
Common questions
Should missing context become an empty list?
Only when it genuinely means successful retrieval with no passages. Missing instrumentation should remain a distinct diagnostic state.
Can reference evidence be passed to every metric?
Supply it only according to the declared metric task; never silently replace actual retrieved evidence with ideal passages.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- Faithfulness metric Ragas · Living reference · Maintained documentationReviewed: claim-to-context faithfulness definition and implementation requirements. Accessed October 16, 2026.
- Context Precision Ragas · Living reference · Maintained documentationReviewed: context precision variants and reference-dependent inputs. Accessed October 16, 2026.
- Contextual Relevancy Confident AI · Living reference · Maintained documentationReviewed: documented input and score behavior, not a hands-on tool ranking. Accessed October 16, 2026.
- Contextual Recall Confident AI · Living reference · Maintained documentationReviewed: documented retrieval-context and expected-output dependence. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.