Benchmark contamination: why a private test set is not enough

A recent taxonomy broadens the question

An August 2026 preprint organizes contamination around the mitigations it can defeat, including exposure acquired during evaluation. It also reports limitations in the reliability of parts of its disclosure instrument. Its contribution is a way to ask sharper questions, not a detector that proves any benchmark is uncontaminated.

Evidence: Benchmark Contamination: A Taxonomy Organized by Defeated Mitigation [1]

Trace how evaluation information can reach the system

List the places where test material can appear: prompts, retrieved documents, few-shot examples, developer notes, fine-tuning data, shared conversations, and tools. A case can remain outside model training yet still be incorporated into the application’s prompt after repeated debugging. That changes what success on the case means.

For web-enabled systems, also consider whether a test question leads directly to its published answer. That may be legitimate for a retrieval task and invalid for a closed-book capability claim. The same behavior can be acceptable or contaminating depending on the evaluation contract.

Use controls that match the intended claim

Rubrex recommends maintaining separate development, regression, and decision sets. A known failure belongs in regression testing even after engineers have seen it. A held-out set serves a different purpose: measuring transfer beyond those fixes. Do not erase regression value merely because a case is no longer secret.

Limit access to the decision set, record when it was exposed, and freeze scoring rules before running the final comparison. If tools are allowed, record their access boundaries. New questions should be reviewed for similarity to previous examples and for accidental clues in identifiers or reference documents.

Illustrative example: the memorized escalation

A team fixes an assistant after a support escalation and adds the exact interaction to its prompt as an example. The next evaluation passes that interaction. This is a useful regression result, but weak evidence that the assistant handles unfamiliar cases of the same kind.

Build separate cases that exercise the same underlying rule with different evidence, phrasing, and boundary conditions. Keep the original escalation, label its exposure, and report generalization cases separately. A transparent distinction is more useful than one larger success percentage.

Include an exposure record beside the score

A compact record should describe dataset origin, who saw the cases, how the application was tuned, whether tools could retrieve related material, and what remains unknown. Attach it to the run, because exposure can change between executions. A benchmark label alone does not describe the conditions under which a score was obtained.

• Name the capability or workflow claim the test is intended to support.
• Record known training or tuning exposure where available.
• Explain retrieval and network permissions during evaluation.
• Keep the final decision set separate from routine debugging feedback.

Common questions