On this page
The short answer
Judge an improvement by its size, uncertainty, and consequences for the actual workflow. Define the smallest useful change before comparing candidates. A statistically detectable difference can be operationally trivial, while a modest aggregate gain can hide a consequential regression in a supported task or permission boundary.
What to take away
- Use raw counts and absolute effect sizes.
- Predefine the smallest useful improvement.
- Inspect paired regressions behind a net gain.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Choose an effect in meaningful units
SciPy’s bootstrap interface supports uncertainty estimates for a statistic, including paired samples. The choice of statistic remains a product decision: an answer-quality point, an accepted-task percentage point, or reviewer time saved can imply very different value.
Write the effect in the units the release owner understands. For binary task acceptance, an absolute percentage-point difference is often clearer than a relative percentage alone. For latency, preserve seconds and the relevant percentile. Keep critical failures separate rather than assuming their consequences are captured by the average.
Evidence: SciPy: bootstrap [1]
Set a practical target before reading results
Ask what change would justify integration effort, new operating costs, and migration risk. Record that target with the comparison plan. It may differ by workflow: improving a frequent ordinary task and eliminating a severe boundary defect are not interchangeable objectives.
Avoid defining the threshold after seeing the observed gain. A candidate that improves by a tiny amount should not become meaningful merely because the team can describe the same decimal as a large relative change against a small baseline error rate. Show both denominators and the raw counts.
Translate an illustrative improvement into outcomes
Suppose two invented systems answer the same 200 tasks. The baseline passes 160 and the candidate passes 170. The absolute improvement is 10/200, or five percentage points; relative pass-rate improvement is 5/80, or 6.25%. These are different descriptions of the same hypothetical totals.
Now inspect paired transitions. Fifteen baseline failures becoming passes and five baseline passes becoming failures produce the same net ten-task gain. Review those five regressions and their consequences before accepting the aggregate. The totals alone cannot tell whether the candidate broke an important workflow.
| Case or record | Expected check | Decision |
|---|---|---|
| 160/200 to 170/200 passes | Absolute gain 5 percentage points | Do not call it a 10% gain |
| 15 fixes and 5 regressions | Net 10 extra accepted tasks | Review regression severity |
| Uncertain gain below practical target | Benefit not established | Keep baseline or gather targeted evidence |
Combine magnitude with uncertainty and cost
Analyze the paired differences with a method appropriate to task dependence and the chosen metric. Compare the uncertainty range with the predeclared useful-effect target. If the evidence cannot distinguish a worthwhile gain from a negligible change, report that uncertainty rather than a confident business conclusion.
Add measured implementation and operating costs where available, clearly separating estimates from observations. Keep the baseline if the benefit does not justify the change, or run a targeted follow-up on the unresolved slice. A release decision can be rational without a universal winner; preserve the actual trade-off and who accepted it.
Limits of the evidence
All counts and percentages are hypothetical arithmetic. No practical improvement threshold is universal, and aggregate effects can change with case mix. Uncertainty analysis must reflect paired observations, clustering, and label quality.
Common questions
Is statistical significance enough to ship?
No. Also assess practical magnitude, critical failures, operating cost, and whether the tested cases represent the intended workflow.
Which effect size should I use?
Choose one connected to the decision and retain its units. Absolute task-success differences are often interpretable, but other workflows may need different outcomes.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- SciPy: bootstrap SciPy · Living reference · Maintained documentationReviewed: resampling confidence intervals and paired shared-index behavior. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.