On this page
The short answer
When many prompts are compared on the same cases, the best observed result can partly reflect chance and repeated selection. Keep exploratory tuning separate from a protected confirmation set, record every candidate considered, and choose an analysis that matches the comparison family and dependencies. One unadjusted significant result is not enough to establish a reliable winner.
What to take away
- Keep prompt search and confirmation separate.
- Define the tested family and error objective.
- Report all candidates that influenced selection.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Record the search that produced the winner
SciPy documents false-discovery-rate procedures and their assumptions. Such adjustments address a defined family of tests; they do not repair an undocumented prompt search in which losing candidates or inconvenient metrics disappear from the record.
List the candidate prompts, model settings, primary outcome, task population, and stopping rule before the confirmatory stage. During exploration, retain the full search history. Trying another prompt after every look at the same final cases consumes their value as independent confirmation even if each individual comparison seems reasonable.
Evidence: SciPy: false_discovery_control [1]
Define the comparison family and goal
Decide whether the goal is one release decision, exploratory discovery, or a set of separately actionable improvements. Family-wise error and false discovery rate answer different questions. Do not choose a procedure solely because it yields a preferred decision.
The tests may be dependent because prompts share cases and outputs. Preserve paired task structure and obtain statistical review when the analysis is complex. Correcting a collection of invalid p-values does not make them valid; the underlying statistic, sampling unit, and hypothesis still need to fit the evaluation.
Use a numerical selection illustration
Under an illustrative assumption of twenty independent null comparisons, each tested at 0.05, the probability of at least one false positive is 1 minus 0.95 to the twentieth power, approximately 64.2%. Actual prompt comparisons are often dependent, so that number is an explanation of the mechanism rather than a forecast for your experiment.
A simple final workflow can choose a candidate using development data, freeze its prompt and analysis, then compare it with the baseline on untouched cases. Report the development search separately. A failure on confirmation should remain a failure; repeatedly rewriting the candidate until the same confirmation set passes restarts exploration.
| Case or record | Expected check | Decision |
|---|---|---|
| Twenty independent null tests at 0.05 | 1−0.95^20≈64.2% chance of any false positive | Illustration only; dependency matters |
| Winner chosen on development cases | Freeze prompt before confirmation | Evaluate on protected cases |
| Repeated looks at final failures | Final set becomes development material | Use new confirmation evidence |
Report the selected result with its context
Include the number of candidates and outcomes examined, the selection rule, the confirmatory sample, and the adjustment or validation approach. Show the practical effect and its uncertainty, not only whether a threshold was crossed. Preserve failed runs and planned exclusions.
When evidence is exploratory, label it that way and use it to design the next experiment. Do not advertise the highest score as a universal prompt improvement. A reproducible candidate that wins on a protected, relevant comparison is more useful than a long leaderboard of selectively reported development results.
Limits of the evidence
The 64.2% figure is computed from an explicitly hypothetical independence assumption. This guide does not prescribe a universal testing correction. Dependent prompts, adaptive selection, and clustered tasks require an analysis appropriate to the actual experiment.
Common questions
Must every exploratory result be adjusted?
Choose an analysis suited to the claim. Exploratory results can guide work, but clearly separate them from confirmatory release evidence.
Does a correction solve benchmark overfitting?
No. Adaptive prompt selection, leakage, invalid labels, and unrepresentative cases require design controls beyond a multiple-testing adjustment.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- SciPy: false_discovery_control SciPy · Living reference · Maintained documentationReviewed: false discovery rate, Benjamini-Hochberg and dependence assumptions. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.