On this page
The short answer
A paired bootstrap comparison resamples the same task identities for baseline and candidate, then recomputes their difference. This preserves the observed pairing instead of treating the systems’ outputs as unrelated samples. Define the statistic, resampling unit, and interval method explicitly, and check whether task clustering requires a different unit.
What to take away
- Resample paired identities together.
- Choose task or cluster units deliberately.
- Report statistic, interval method, and missing-data policy.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Choose the resampling unit from the experiment
SciPy’s paired bootstrap option uses shared resampled indices across corresponding arrays. That matches a simple paired-case design, but an application with several cases per customer or conversation may require resampling whole clusters rather than individual rows.
Start with one aligned record per intended analysis unit, containing both systems’ outcomes. Resolve duplicate attempts through the declared policy. Keep missing candidate or baseline outcomes visible and explain the paired-complete subset if the statistic excludes them. A bootstrap cannot restore information removed by selective filtering.
Evidence: SciPy: bootstrap [1]
Define the statistic before resampling
For task acceptance, a useful statistic can be the candidate pass fraction minus the baseline pass fraction on the same resampled tasks. For other outcomes, specify whether the comparison concerns means, medians, cost ratios, or another quantity. Retain units and the direction that favors the candidate.
Choose the number of resamples, random generator settings, interval method, and confidence level in the analysis record. Different interval methods have different behavior, especially for small or degenerate samples. Do not quietly switch methods after seeing which interval excludes zero.
Work through a tiny authored resample
Suppose five invented task-level candidate-minus-baseline differences are minus one, zero, zero, one, and one. Their observed mean is 0.2. A resample using indices five, five, four, two, and three produces differences one, one, one, zero, zero and mean 0.6. These are illustrative calculations, not an uncertainty interval or model result.
The same indices must select both systems’ original outcomes before calculating each difference. Independently resampling baseline and candidate would break the case pairing. Repeat the declared process to form the bootstrap distribution, then compute the selected interval rather than treating the single illustrative resample as evidence.
| Case or record | Expected check | Decision |
|---|---|---|
| Differences[−1,0,0,1,1] | Observed mean 0.2 | Authored arithmetic example |
| Indices[5,5,4,2,3] | Resampled differences[1,1,1,0,0] | Mean 0.6; not a confidence interval |
| Several cases per conversation | Rows may be dependent | Consider conversation-level resampling |
Inspect assumptions before interpreting the interval
Check alignment, cluster membership, sample size, unusual weights, and the prevalence of identical or boundary outcomes. When every sampled task behaves the same way, some methods can become degenerate; report the condition and obtain an appropriate analysis rather than hiding a warning.
Present the observed difference, interval, resampling unit, case count, and missing-data policy together. Compare the interval with the practical decision margin, not only zero. A narrow interval can still concern an unrepresentative workload, and resampling the observed cases cannot discover failure mechanisms absent from the original sample.
Limits of the evidence
The five-task example is intentionally tiny and illustrates indexing only. No bootstrap interval or model experiment was computed. Valid inference depends on the resampling unit, statistic, sample, and assumptions; clustered or sparse data may need specialized analysis.
Common questions
Why not resample the two systems independently?
When both answered the same tasks, independent resampling discards their pairing and can change the uncertainty estimate.
Does more resampling create more evidence?
It reduces simulation noise for the chosen bootstrap procedure. It does not create new independent tasks or repair an unrepresentative original sample.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- SciPy: bootstrap SciPy · Living reference · Maintained documentationReviewed: resampling confidence intervals and paired shared-index behavior. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.