Use mean reciprocal rank for RAG with explicit relevance labels

State the first-hit assumption

Sentence Transformers exposes MRR at explicit cutoffs in its retrieval evaluator. The measure rewards a relevant result near the beginning of a ranking; it does not count how many additional relevant documents follow. Define whether relevance applies to a whole document or to a passage that actually supports the question. Use stable source IDs and retain the judged evidence. A document about the right product can still be irrelevant to a question about its specific cancellation exception.

Evidence: Sentence Transformers evaluation reference [1]

Calculate a transparent toy example

Suppose three authored queries have their first relevant passage at ranks one, two, and nowhere within the first five results. Their reciprocal-rank contributions at five are 1, 0.5, and 0. The mean is 0.5. This is arithmetic on an illustrative ranking, not a measured retriever result. Keep the third query in the denominator. Dropping it would change the question from overall first-hit quality to quality conditional on already finding a relevant passage.

Inspect labels and cutoff behavior

Save the ranked IDs, judged relevant IDs, and cutoff with each result. When two passages duplicate the same evidence, the first relevant position still determines reciprocal rank, but their redundancy can harm the final context. Report duplicate evidence separately. Distinguish unjudged documents from judged nonrelevant ones according to a documented policy. If the label pool is incomplete, a low score may reflect missing judgments rather than a true retrieval miss, so inspect disputed results before interpreting small differences.

Choose the measure from the task

MRR is useful for a lookup where one decisive passage is enough. Pair it with evidence coverage or recall when an answer requires a policy and a separate exception. A system can improve the first hit while pushing the second required source outside the context window. Compare paired queries across versions and inspect regressions in critical groups before selecting a retriever. Publish the formula, cutoff, and labeling unit with the score so another engineer can reproduce its meaning.

Use paired ranks to explain version changes

For each query, compare the old and new first relevant ranks before calculating the mean difference. A movement from rank one to two loses more reciprocal-rank value than a movement from nine to ten, which reflects the measure’s emphasis on early access. Inspect whether that emphasis matches the user workflow. If the application always reads five passages and requires their combined evidence, a first-hit gain can overstate practical benefit. Keep the per-query rank changes with answer outcomes so reviewers can see when a metric change corresponds to a meaningful product difference.

Common questions