On this page
The short answer
Use nDCG when some retrieved evidence is more useful than other evidence. Declare the relevance scale, gain function, rank discount, and cutoff before comparing runs, then inspect disagreements in the underlying labels.
What to take away
- Define graded labels with examples.
- Keep the ideal ranking reproducible.
- Pair ranking scores with answer checks.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Choose a meaningful relevance scale
Sentence Transformers supports nDCG evaluation, while the interpretation depends on your judgments and calculation convention. Define grades in terms of the reader task: zero for irrelevant, one for useful background, and two for direct support. A grade should not simply encode which retriever produced the document. Keep the evidence supporting each judgment. If your chosen evaluator accepts only binary relevance, do not present its result as a test of a richer graded scheme that you never supplied.
Work through an explicit ranking
For an illustrative question, suppose three returned passages have grades 2, 0, and 1. With gain 2 raised to the grade minus 1 and discount log2(rank plus 1), DCG is 3 plus 0 plus 0.5, or 3.5. The ideal order 2, 1, 0 has DCG about 3.631. nDCG is therefore about 0.964. These are hand calculations on authored labels, not benchmark results. A different gain convention can yield a different value and must be named.
Keep zero-relevance queries visible
A query with no judged relevant document has an ideal gain of zero. Document how your evaluation handles that case rather than dividing by zero or silently removing it. An unanswerable query may need a separate abstention evaluation instead of a conventional relevance ranking. Also distinguish a truly empty relevant set from incomplete judgments. Ask a reviewer to inspect newly retrieved high-ranked documents before interpreting a score drop as evidence that the retriever became worse.
Inspect what a good score can hide
A high nDCG can coexist with a missing required exception if several background documents earn positive grades. Review the grade definitions against the actual answer contract. For a question requiring two independent documents, add an explicit evidence-completeness check rather than assuming a smooth ranking measure guarantees both. Record permission eligibility separately. An ineligible but highly informative document must never become a ranking success just because its content would help answer the question.
Compare versions using stable judgments
Freeze the query set, relevance unit, gains, cutoff, and corpus snapshot for a paired comparison. Save ranked IDs and per-query changes, then inspect critical regressions before choosing a winner from the aggregate. If labels change, rerun both versions against the revised judgment set and record that change. Use nDCG to prioritize ranking investigations; require accepted answers and adequate evidence coverage before treating the ranking improvement as a useful application improvement. Retain the ideal ranked grade sequence with the run. That small artifact makes normalization errors and disagreement about the best possible ordering much easier to inspect without reconstructing the entire relevance-label pipeline.
Limits of the evidence
This guide explains a declared calculation on synthetic grades. It does not establish a universal gain scheme or reproduce a benchmark evaluator’s implementation.
Common questions
Can binary labels produce nDCG?
Yes, but they measure binary relevance ordering rather than distinctions your labels do not contain.
Is a higher score enough to release?
No. Check missing required evidence, critical failures, permissions, and final answers separately.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- Sentence Transformers evaluation reference Sentence Transformers · Living reference · Maintained documentationReviewed: retrieval evaluator inputs and supported metrics; formulas explained independently. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.