Choose and interpret precision at k for retrieval

Define the result being judged

Sentence Transformers exposes precision and recall at specified cutoffs. Decide whether your result unit is a document, chunk, or distinct evidence span. A broad document can contain the right subject while the returned chunk omits the answer. Write a relevance rule that matches what generation actually receives. Also decide how to handle duplicate chunks and ineligible records. These choices belong in the evaluation contract because changing them can move the reported score without changing the underlying retrieval behavior.

Evidence: Sentence Transformers evaluation reference [1]

Calculate a fixed-cutoff example

Suppose five authored retrieved passages contain three relevant items and two distractors. Precision at five is 3 divided by 5, or 0.6. If the system returns only three relevant results, a fixed-k convention can still divide by five and yield 0.6; precision over returned results would instead be 1.0. Neither number should be published without its denominator. Pick the convention appropriate to your retrieval interface and keep the shorter-result behavior visible alongside the score.

Add a missing-exception counterexample

A fictional eligibility question requires a policy and an exception. A result list containing five relevant paragraphs about the general policy may score well while missing the exception needed to answer correctly. Label required evidence groups separately from passage relevance. Report whether all mandatory groups appear in the final context. This distinguishes useful topical precision from completeness. A higher precision score achieved by returning fewer but incomplete facts is not a success for the user’s decision.

Control the context assembly step

Compare retrieval results before and after deduplication, filtering, and truncation. The top five candidates may be precise, yet the final prompt may contain only their first lines or may drop an essential passage to fit the token budget. Store both candidate IDs and final context IDs. When testing a new cutoff, measure the resulting generation context and answer outcome rather than assuming that the ranked list is identical to what the model can use.

Use precision to diagnose noise

Inspect low-precision cases by cause: wrong entity, obsolete document, duplicate evidence, or genuinely irrelevant passage. A metadata correction may fix one group while a reranker helps another. Compare precision and recall together when choosing k, and retain a small held-out set to avoid selecting a cutoff that merely fits the development examples. State which critical answer requirements must remain satisfied even when the average noise level improves. That makes the metric an actionable diagnosis rather than a decorative summary. Keep a query with no relevant results as a control when changing k, so altered empty-list handling cannot manufacture an improvement.

Common questions