On this page
The short answer
Account for every scheduled evaluation case, including those that never produced an answer or a valid score. Keep application failure, evaluator error, intentional exclusion, and missing evidence separate. Report quality on evaluated outputs alongside completion over the scheduled population, with a clear reconciliation between the two.
What to take away
- Start from the scheduled case manifest.
- Keep application and evaluator failures distinct.
- Publish conditional and end-to-end denominators.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Create statuses that preserve the failure stage
Pytest distinguishes skipped tests, expected failures, ordinary failures, and unexpected passes. LLM evaluation needs similarly explicit statuses, adapted to the stages of its own workflow rather than copied mechanically from a unit-test report.
Track whether a case was admitted, executed, completed, parsed, and scored. A provider timeout occurs before a usable answer; a judge timeout can occur after the application succeeded. Collapsing both into null loses the distinction needed to fix the system and makes the final denominator depend on an undocumented cleanup step.
Evidence: pytest: Skip and xfail [1]
Reconcile IDs before calculating metrics
Begin with the scheduled case manifest and left-join attempts and judgments to it. Find missing case IDs, duplicate attempts, orphan scores, and partially completed batches. Use a declared retry policy to choose which attempt contributes to each reported view.
Keep first-attempt and eventual-success results separate when retries matter. Selecting the best output from several attempts while reporting one attempt per case creates an advantage that the product may not offer users. Preserve invalid outputs and raw error records so a reviewer can locate the stage where evidence disappeared.
Report conditional quality and end-to-end completion
In an illustrative batch of 100 scheduled tasks, 90 produce scoreable outputs and 81 pass. Quality among scoreable outputs is 81/90, or 90%; accepted tasks over all scheduled tasks is 81/100, or 81%. The remaining ten tasks still need stage-specific statuses. These are authored arithmetic examples, not benchmark results.
If some tasks were intentionally out of scope, record that decision before execution or explain the post-run exclusion. Show the excluded count and reason beside the relevant denominator. Do not silently recategorize disappointing outputs as invalid simply because they lower the quality rate.
| Case or record | Expected check | Decision |
|---|---|---|
| 100 scheduled;90 scoreable;81 passes | 81/90 quality;81/100 accepted tasks | Show both rates |
| Judge timeout after valid output | Application output already exists | Rescore the saved output |
| Missing case row | No terminal status | Reconcile before publication |
Make recovery auditable
Retry recoverable infrastructure failures with bounded attempts and a stable case identity. Retain each attempt and the reason for recovery. For evaluator failures, rescore the saved application output rather than rerunning the application unless a new output is actually required.
Block publication when unexplained missing rows could change the release decision. A reconciled incomplete report can still be useful if it states what was measured and what remains unknown. Add a routine check that scheduled equals the sum of terminal categories, while ensuring those categories are mutually exclusive for the reported view.
Limits of the evidence
The batch counts are hypothetical calculations. Status categories and retry selection must match the product contract. Reconciliation prevents silent omissions but does not establish that scored outputs or reference labels are correct.
Common questions
Should timeouts count as wrong answers?
Keep their stage and status explicit. They can fail end-to-end task acceptance without being a factual-quality judgment about an answer that never existed.
Can I exclude malformed outputs?
You may define a conditional quality view, but retain malformed outputs in the full task outcome and explain the exclusion.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- pytest: Skip and xfail pytest · Living reference · Maintained documentationReviewed: distinct skip, expected failure, failure and unexpected-pass reporting. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.