On this page
The short answer
An LLM judge should abstain when the available evidence cannot support the requested judgment. Define abstention reasons and a review path before deploying it. Track abstentions separately from passes, failures, and technical errors so uncertainty remains visible instead of silently improving the reported score.
What to take away
- Name unknown outcomes separately from errors.
- Tie abstention to required evidence.
- Report decisive coverage beside accuracy.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Separate a score from a decision
Scikit-learn’s threshold guide distinguishes a predictive score from the action taken from it. A judge’s numeric output likewise does not determine what the application should do when source evidence is missing or the rubric does not apply.
Rubrex recommends naming allowed outcomes explicitly: pass, fail, insufficient evidence, ambiguous criterion, and evaluator error. A timeout is not an evidence-based abstention. A statement that the answer is probably correct is not a pass unless the task’s acceptance process permits that uncertainty. These distinctions make escalation and measurement possible.
Write evidence-based abstention triggers
List the materials required for each criterion. A faithfulness judgment needs the answer and the relevant source passages. A side-effect judgment needs the operation result or final state. A judge should decline that criterion when the required evidence is absent rather than infer what probably happened.
Provide examples of legitimate ambiguity, including two valid interpretations of an underspecified question. Also provide a clearly supported pass and a clearly supported failure. Without those controls, a prompt encouraging caution can produce a judge that declines everything, reducing useful coverage without improving its decisions.
Route the unknown result to an owner
For an illustrative ticket evaluation, a trace records a proposed update but omits the final record state. The judge can assess whether the proposed arguments match the request but cannot establish that the update completed. Preserve those as separate criterion outcomes instead of returning one universal unknown label.
Send unresolved consequential criteria to a reviewer who can retrieve the permitted missing evidence. Record the reason, required artifact, owner, and deadline. Do not repeatedly ask the same judge to guess until it returns a decisive label; that replaces a controlled abstention policy with selective acceptance of convenient answers.
| Case or record | Expected check | Decision |
|---|---|---|
| Final state missing | Cannot establish completed side effect | Abstain on completion criterion |
| Source directly contradicts answer | Evidence sufficient for a judgment | Fail rather than abstain |
| Complete supported answer | No missing required artifact | Pass rather than defer |
Measure coverage alongside conditional accuracy
Report decisive judgments divided by attempted judgments as coverage, and evaluate correctness on independently reviewed decisive cases. Retain a separate audit of abstentions: some may be justified, while others conceal a usable source passage the evaluator overlooked.
Tune abstention rules on development examples and validate on held-out outputs. Compare failure detection and review workload as well as coverage. Approve the policy only when unknown outcomes reach the intended queue and remain visible in release summaries. A higher pass rate conditional on fewer decided cases is not evidence that the product itself improved.
Limits of the evidence
This is a proposed evaluator policy, not a measured selective-classification system. Abstention thresholds and escalation capacity depend on the workflow. A documented unknown state improves transparency but does not establish that decisive judgments are correct.
Common questions
Should abstentions count as failures?
For product completion they may block acceptance; for evaluator accuracy they are a distinct outcome. Publish both views and their denominators.
Can the judge’s confidence choose abstention?
Only after validating that confidence against reference judgments. Evidence-availability rules remain useful even when confidence scores are poorly calibrated.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- scikit-learn: Tuning the decision threshold scikit-learn · Living reference · Maintained documentationReviewed: distinguishes predicted scores from decision thresholds and warns against tuning on training data. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.