On this page
The short answer
Plan statistical power by defining the improvement worth detecting, the analysis, and the data-generating assumptions before collecting the final sample. Sample size alone does not determine power. Preserve paired or clustered structure and separate planning estimates from evidence obtained after the experiment.
What to take away
- Define the smallest useful effect before sampling.
- Model paired and clustered outcomes explicitly.
- Label precision examples separately from power estimates.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Start with the effect and decision rule
SciPy documents simulation-based power estimation under an explicitly specified alternative hypothesis and test. The simulation is only as relevant as those assumptions; invoking a power function does not establish that an LLM task sample follows the chosen model.
Define the primary outcome, smallest useful improvement, significance rule if used, and acceptable uncertainty. Identify whether the observations are independent tasks, customer clusters, or repeated attempts. A comparison of two systems on the same cases usually contains pairing information that a generic independent-samples calculation would discard.
Evidence: SciPy: power [1]
Use pilot evidence to describe plausible variation
Estimate baseline outcome rates, task heterogeneity, missingness, and candidate-baseline disagreement from a pilot that is separate from final confirmation. Treat those estimates as uncertain and examine several plausible scenarios rather than one convenient point assumption.
For a paired binary comparison, the number and direction of discordant cases matter. Two systems can have the same marginal pass rates under different joint outcome patterns. Preserve the pilot’s case-level transition table so the planning model reflects how often candidates fix and regress the same tasks.
Separate precision arithmetic from a power result
For an illustrative independent binary rate of 0.8, the plug-in standard-error calculation is square root of 0.8 times 0.2 divided by n. It gives 0.04 at n equals 100 and 0.02 at n equals 400. That demonstrates how precision scales under the assumed model; it is not a computed power guarantee.
Likewise, invented paired totals of 80 and 85 passes out of 100 can arise from five fixes and no regressions, or twenty fixes and fifteen regressions. The net gain is identical, but the comparison structure differs. A planning analysis should not assume those situations provide the same evidence.
| Case or record | Expected check | Decision |
|---|---|---|
| p=0.8; n=100 versus 400 | Plug-in standard error 0.04 versus 0.02 | Illustrative precision, not power |
| Five fixes; zero regressions | Net gain five tasks | Retain paired transition counts |
| Twenty fixes; fifteen regressions | Same net gain; different discordance | Use the appropriate planning structure |
Freeze the plan and state uncertainty
Document the selected scenarios, analysis code or method, planned sample, stopping rule, and handling of missing outcomes. If resources cannot support a decisive study for the useful effect, report that limitation and consider a narrower task population or a more informative paired design.
Do not reinterpret an underpowered result as proof of no meaningful difference. After execution, report the observed effect and uncertainty rather than using post hoc power as a substitute. Revisit the plan when the task distribution or outcome definition changes, and preserve the distinction between a planning simulation and measured model results.
Limits of the evidence
The numeric examples are hypothetical calculations, not a power analysis for a model. No simulation was run. Incorrect distribution, dependence, or label assumptions can make planning estimates misleading even when the arithmetic is correct.
Common questions
Is there one sample size for 80% power?
No. It depends on the effect, outcome variation, test, dependence, significance rule, and missingness assumptions.
Does a nonsignificant result show equivalence?
No. Equivalence requires a suitable design and practical margin; an inconclusive result may simply leave substantial uncertainty.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- SciPy: power SciPy · Living reference · Maintained documentationReviewed: simulation-based power estimation under an explicitly specified alternative and test. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.