Evaluate a RAG reranker without hiding candidate-retrieval failures

Define the reranker boundary

Elastic describes reranking as a refinement of candidates produced by an earlier retrieval stage. Define that boundary explicitly in your evaluation record. Store candidate IDs, texts, initial ranks, and any metadata the reranker receives. Decide whether the target is finding one decisive passage or ordering several complementary passages. A reranker that promotes one relevant paragraph may still harm a question that requires two independent sources in the final context.

Evidence: Semantic reranking [1]

Create a candidate-pool counterexample

Author a fictional returns question requiring a general policy and a regional exception. In pool A, include both passages plus a distractor. In pool B, omit the exception entirely. A useful reranker should place both necessary passages within the allowed context for pool A. Pool B remains an upstream recall failure even if its best available paragraph receives a high score. Do not remove pool B from end-to-end reporting merely because it cannot be solved by reranking.

Run conditional and full-pipeline checks

For the conditional test, feed each reranker exactly the same pool and compare the rank of labeled evidence under the same output limit. For the pipeline test, replay original queries through candidate generation and reranking, then evaluate the generated answers. Report both results, including candidate-retrieval misses. Record reranking latency, input truncation, and token usage where applicable. Inspect whether long passages lose their qualifying sentences before the reranker scores them, because that transformation can mimic a ranking defect.

Make promotion requirements explicit

A proposed acceptance rule is that the reranker preserves every critical evidence pair already present in the candidate pool and improves the targeted ordering failures without exceeding the agreed budget. Confirm that on held-out cases rather than choosing a threshold from the demonstration fixture. Keep the original candidate ranking available for rollback. If the answer remains wrong despite correct evidence ordering, investigate context assembly and generation before adding another reranking stage.

Keep scoring errors and timeouts in the run record. A reranker that fails on unusually long evidence should not appear superior because its difficult requests disappeared from the evaluated set.

Check the text that the scorer actually sees

Long candidate passages may be truncated or serialized with repeated headings before reaching the reranker. Save the actual scorer input for the trial, with sensitive data handled under the agreed policy. Place a qualifying clause near the end of an authored passage and verify that it survives. If it disappears, shorten or restructure the candidate before concluding that the ranking model cannot understand the exception. Also compare identical evidence presented with and without repeated page boilerplate. The result can identify an assembly defect that would otherwise be mistaken for a model limitation.

Common questions