On this page
The short answer
Report results by slices that change the user task or failure consequence, such as workflow, language, permission role, or evidence condition. Preserve counts and sampling context for each slice. An overall improvement can coexist with a regression for one group, and a small slice can be too uncertain to support a ranking.
What to take away
- Define meaningful slices before comparing winners.
- Preserve overlapping membership and missing labels.
- Show counts and uncertainty beside rates.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Choose slices before inspecting winners
Scikit-learn documents per-label support and different averaging rules. Although application slices need not be classification labels, the same reporting discipline matters: retain the size of each group and explain how an aggregate weights it.
Start from product requirements and known failure mechanisms. A language slice can reveal parsing or terminology problems; a permission-role slice can expose a boundary the aggregate obscures. Avoid inventing dozens of slices only after noticing favorable results. Distinguish planned analysis from exploratory findings that need confirmation.
Keep slice definitions stable and inspectable
Assign slice labels using fields available independently of candidate performance. Define overlapping membership explicitly: one task can be both multilingual and permission-sensitive. Do not add overlapping slice counts together as though they form a partition of all tasks.
Record missing slice metadata and cases excluded by a definition. If a task’s difficulty is labeled after viewing one candidate’s answer, that label can favor the same candidate. Use a reference process or a separate task characteristic, and version changes to the labeling rules.
Show counts that explain the aggregate
In an illustrative two-workflow report, a candidate gains ten accepted tasks in a large FAQ slice but loses three in a small account-update slice. A positive total does not settle the release if the lost updates involve a required behavior. Keep both the raw transitions and the slice denominators.
Report per-slice pass, fail, missing, and unresolved counts before calculating rates. An empty slice is untested, not perfect. A tiny slice with a high observed rate should carry an uncertainty note rather than the same visual confidence as a much larger representative sample.
| Case or record | Expected check | Decision |
|---|---|---|
| FAQ gains 10 accepted tasks | Large ordinary-task slice | Retain the benefit |
| Account updates lose 3 tasks | Potentially consequential regression | Review before aggregate approval |
| No cases in a language | No measured evidence | Mark untested, not 100% |
Use findings to narrow the next decision
Compare baseline and candidate on the same cases within each slice. Investigate whether a regression comes from retrieval, input format, scorer behavior, or a changed requirement before creating a new model route. Routing around the problem adds its own selection and maintenance tests.
Keep the overall result, slice results, weighting policy, and limitations together. Confirm surprising exploratory differences on new cases where possible, especially after many slice comparisons. Use the report to decide what to fix or withhold, rather than turning every small numerical difference into a claim about an entire user population.
Limits of the evidence
The workflow counts are invented illustrations. Slice analysis can be misleading when labels are post hoc, samples are sparse, or many comparisons are explored. A group label does not by itself explain a causal performance difference.
Common questions
Should every slice have equal weight?
That is a decision policy, not a default truth. Show raw results and explain whether the aggregate reflects observed frequency, balanced coverage, or another objective.
Can small slices reveal important defects?
Yes. A reproduced boundary failure can matter even in a small slice, but the slice’s observed rate may still be too uncertain for a population estimate.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- scikit-learn: precision_recall_fscore_support scikit-learn · Living reference · Maintained documentationReviewed: per-label metrics, support, macro/micro/weighted averaging and undefined denominators. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.