On this page
The short answer
Use confidence intervals to show how much sampling uncertainty remains around a task pass rate. State the unit of observation, sample selection, and interval method. An interval does not repair biased cases or dependent observations, and a narrow interval on the wrong task population can still mislead a release decision.
What to take away
- State successes and denominator beside every rate.
- Match the interval to the sampling unit.
- Allow an inconclusive release decision.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Define the rate and the sampling unit
SciPy documents exact and Wilson confidence intervals for a binomial proportion. Those methods address a particular statistical model; they do not establish that repeated agent attempts or copied customer questions behave like independent draws from the intended population.
Define a pass before examining candidate results and decide whether the observation is a task, customer, conversation, or repeated attempt. Keep the number of successes and total eligible observations with the rate. If many observations come from one incident, do not present them as evidence from equally many independent incidents.
Evidence: SciPy: BinomTestResult.proportion_ci [1]
Choose the interval before comparing releases
Record the confidence level, method, and treatment of missing outcomes in the evaluation plan. Use an implementation whose assumptions and edge behavior you understand. Avoid switching methods after seeing which one makes a preferred candidate appear acceptable.
For an illustrative report, retain a row containing passes, attempted tasks, evaluable tasks, and the computed interval. Keep unknown outcomes visible beside it. If the rate excludes infrastructure failures, explain that conditional population and publish a separate completion measure so the interval cannot be mistaken for end-to-end reliability.
For a computed illustrative example, 80 passes in 100 independent binary tasks gives an observed rate of 80%. A two-sided 95% Wilson interval is approximately 71.1% to 86.7%, calculated with z = 1.9599639845. These are authored counts, not model results. The bounds describe the stated independent-task sampling model; copied cases or clustered conversations do not satisfy that assumption merely because the calculation runs.
Connect uncertainty to an action
Write a practical acceptance threshold before running the comparison. A product team might require a lower bound above an agreed minimum for ordinary task success while separately blocking any reproduced unauthorized action. The numerical threshold must come from the product decision, not a generic rule supplied by the interval function.
If the result straddles the decision boundary, label it inconclusive and choose the next evidence-gathering step. Additional independent representative cases may help. More repetitions of the same easy task can narrow a misleading calculation without resolving the uncertainty that actually matters.
| Case or record | Expected check | Decision |
|---|---|---|
| Independent binary task outcomes | Use a documented proportion interval | Report method and assumptions |
| Repeated copies of one task | Outcomes share a case family | Do not count as independent cases |
| Interval crosses acceptance boundary | Evidence does not resolve the threshold | Collect targeted additional evidence |
Describe what the interval cannot say
Do not interpret a frequentist interval as a probability that this already calculated interval contains the true rate. Explain it as uncertainty from a stated sampling procedure, and state the main threats to that interpretation in plain language.
Keep task-level failures available for inspection alongside the statistical summary. For a paired baseline comparison, an interval for one candidate’s pass rate is not automatically an interval for the improvement between candidates. Use an analysis that preserves the paired task structure when the release question concerns a difference.
Limits of the evidence
This is statistical evaluation guidance, not a computed assessment of a model. The proposed decision rules require appropriate sampling and product-specific thresholds. Dependence, label errors, selection bias, and changing workloads can dominate the sampling uncertainty represented by an interval.
Common questions
Does a 95% interval mean the system is 95% reliable?
No. Reliability is the quantity being estimated; the confidence level describes the interval procedure under its assumptions.
Can I compare two intervals by checking overlap?
That is not generally a substitute for analyzing the difference, especially when both systems answer the same cases. Preserve the paired observations.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- SciPy: BinomTestResult.proportion_ci SciPy · Living reference · Maintained documentationReviewed: exact and Wilson binomial proportion interval methods and returned bounds. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.