RAG evaluation metrics: measure retrieval and answers separately

What current metric frameworks cover

Ragas documents separate metrics for context precision, context recall, faithfulness, answer relevance, and noise sensitivity. EnterpriseRAG, an August 2026 preprint, additionally examines noisy evidence, knowledge gaps, conflicting facts, and multiple instructions. The latter highlights conditions that a clean question-answer benchmark may omit.

Evidence: List of available metrics [1]EnterpriseRAG: Benchmarking LLM Instruction Adherence and Robustness under Non-Ideal Enterprise Retrieval [2]

Choose metrics from the failure you need to detect

Context precision asks how much retrieved material is useful under the chosen relevance rule. Context recall asks whether the required evidence was recovered. Faithfulness concerns support for the generated claims in the supplied context. Answer correctness needs a task-specific reference or another defensible judgment process. These are not interchangeable questions.

Define what counts as a relevant passage before scoring retrieval. A passage may mention the right topic without containing the exception required to answer the question. Similarly, an answer can be supported by outdated context and still be wrong for the user’s current request. Keep source freshness and authority visible.

Build a diagnostic evaluation matrix

Rubrex recommends freezing a corpus snapshot and creating cases for sufficient evidence, missing evidence, contradictory evidence, and irrelevant additions. Run the same question across these conditions where the transformation remains realistic. Record retrieval configuration and the final context supplied to the generator.

Evaluate the answer separately from the retriever. An experiment that changes both components can be useful for the product decision but is difficult to diagnose. First measure the end-to-end effect; then isolate components to understand where the change came from.

Illustrative example: recall rises, usefulness falls

A retriever starts returning more passages. It now includes the correct policy more often, but also supplies old versions with similar wording. Retrieval coverage improves while the answer increasingly applies outdated rules. A dashboard that reports only recall would miss the regression.

Inspect document versions and authority, then test a filtering or reranking change. Keep the question set fixed and compare both retrieval and final-answer behavior. Do not assume that reducing the number of passages is always the remedy; it may remove necessary evidence for multi-document questions.

Make the report explain tradeoffs

Present per-slice results, representative failures, the configuration used, and the uncertainty in judged scores. Include critical constraint violations beside averages. A useful readout explains which change to test next and which result would support or reject the hypothesis. It does not stop at a color-coded metric panel.

Run a RAG evaluation from question to evidence

For a first RAG eval, create a record containing the question, corpus version, supporting passages, retrieved passages, final answer, and expected behavior. Mark questions that should be answerable and those where the system should ask for clarification or abstain. This lets a reviewer distinguish a missing source from an answer that ignores a source already present.

Rubrex recommends running RAG evals with a frozen question set before changing retrieval depth, chunking, or the answer prompt. Compare the baseline and candidate record by record. For example, if a candidate retrieves the right policy but cites an obsolete exception, retain both the retrieval success and the answer failure. Agree which failures block the decision before aggregating the scores.

A four-case RAG example you can inspect

The downloadable worksheet contains four short, invented passages and four questions. Both the baseline and candidate answers are authored snapshots; no LLM generated them. They illustrate how to diagnose failures, not an observed improvement in a production system. The acceptance contract is explicit: use current evidence, cite it, and abstain when the corpus cannot answer.

The corpus deliberately contains conflicting refund policies. refund-current allows refunds on unused subscriptions within 14 days; refund-old says 30 days and is marked archived. shipping-current states 3 to 5 business days. cancellation-current says to cancel in Settings > Billing and retain access until the paid period ends. No passage contains a telephone support number. A passage can support an answer while still being ineligible under the current-policy contract.

• Download the passages, questions, answers, and labels (JSON)

Calculate separate scores before choosing a fix

Evidence coverage counts answerable cases retrieving every required current passage: baseline 2/3 and candidate 3/3. Exclude the unanswerable phone case from this denominator because it has no required passage. This is a case-level coverage check, not chunk-level context recall. It says nothing about irrelevant extra chunks or retrieval precision.

Context support counts factual answers whose claims are all supported by their retrieved passages: baseline 2/4 and candidate 3/3. The candidate abstention is excluded from factual-answer scoring and judged under task acceptance instead. The stale refund answer counts as supported in the baseline, which exposes why support alone is insufficient. These are explicit human-authored binary labels, not Ragas scores or automated semantic judgments.

Task acceptance counts complete answers meeting the contract, including correct abstention: baseline 1/4 and candidate 4/4. Do not compare the support percentages without their changing denominators. The worksheet provides every label so a second reviewer can disagree with an individual decision rather than debate an unexplained average.

The proposed fixes are version filtering for refunds, tighter claim-to-passage checking for cancellation, and an evidence-sufficiency rule for missing information. Test each change independently on a larger held-out set before combining them. Four deliberately constructed cases cannot estimate a real error rate, establish statistical significance, or justify a release. Keep difficult counterexamples, record reviewer disagreements, and rerun the unchanged shipping control after each intervention.

Common questions