On this page
The short answer
Triage evaluator disagreements by identifying what each reviewer saw, which rule each applied, and whether the underlying reference is sound. Do not average incompatible labels before diagnosing the cause. A disagreement ledger should lead to a changed rule, repaired evidence, corrected reference, or explicitly unresolved decision.
What to take away
- Keep original and adjudicated labels.
- Classify the cause before resolving the score.
- Match the repair to evidence, rubric, or policy.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Preserve the original labels and reasons
W3C’s PROV overview describes provenance through the entities, activities, and people involved in producing data. For a disputed evaluation label, that suggests preserving the original judgments and their inputs before recording any adjudicated result.
Keep reviewer identifiers or permitted pseudonyms, guideline versions, evidence references, and timestamps. Record whether the judgment came from a person or an automated evaluator. If those conditions differ, disagreement may reflect different tasks rather than inconsistent reasoning. Do not overwrite an initial label with the consensus result and lose the path that explains it.
Classify the cause before choosing a winner
Compare the evidence each reviewer had and the criterion they interpreted. Use separate categories for missing evidence, ambiguous wording, reference error, task-policy disagreement, and execution mistake. Allow multiple causes when a bad interface and an unclear rule interact.
For a supported-answer dispute, inspect the claim and cited passage directly. For a tool-action dispute, inspect the final state and authorization context. A confident explanation from an evaluator is not enough to settle either case. The purpose is to locate the source of disagreement, not to reward the most fluent rationale.
Connect each category to a repair
Consider an invented cancellation-policy answer. Reviewer A accepts the general deadline; reviewer B rejects the omitted exception. If the task requires that exception, clarify completeness anchors and relabel related cases. If the source never establishes which policy applies, preserve an insufficient-evidence outcome rather than manufacturing certainty.
A reference defect needs a corrected reference and affected-run review. A missing document needs repaired evidence delivery. A valid difference in product priorities needs an owner’s policy decision. Sending all three problems back to the same judge with a stronger instruction is unlikely to produce an interpretable repair.
| Case or record | Expected check | Decision |
|---|---|---|
| Different source versions | Reviewers evaluated different evidence | Repair snapshot delivery |
| Omitted required exception | Completeness rule unclear | Revise anchors and relabel |
| Unsettled product policy | No authoritative acceptance rule | Escalate to the policy owner |
Retest the corrected decision boundary
Create a small follow-up set containing the disputed case, a clear positive control, a clear negative control, and a new borderline case. Have reviewers apply the revised rule without seeing the desired outcome. Preserve any remaining disagreement and its reason.
Report how many cases changed after adjudication and whether the change affects prior release conclusions. Keep unresolved critical cases visible as unresolved. Agreement is useful when it reflects shared evidence and a coherent rule; it is not a substitute for correctness, and forcing consensus can conceal a product-policy decision that still needs an accountable owner.
Limits of the evidence
The triage categories are Rubrex proposals, and the cancellation example is invented. No adjudication study was conducted. Provenance improves inspectability but cannot establish that an adjudicator, reference, or product policy is correct.
Common questions
Should majority vote settle every dispute?
No. A majority can share the same missing evidence or mistaken assumption. Inspect the source and the rule before deciding how votes should contribute.
Is disagreement always an annotation defect?
No. It can reveal genuinely ambiguous tasks or unsettled product priorities. Preserve those cases rather than forcing a false ground truth.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- W3C PROV-Overview (2013 Working Group Note) W3C · 2013 · Working Group NoteReviewed: abstract and introduction; provenance describes entities, activities and people; historical non-normative overview. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.