On this page
The short answer
Adjudication should explain why reviewers disagreed and record an evidence-backed resolution without overwriting their original judgments. Separate an incorrect label from an ambiguous rubric, missing evidence, or a genuinely unresolved question. The goal is a usable decision and a better evaluation process, not agreement produced by hiding disagreement.
What to take away
- Preserve initial judgments before discussion.
- Classify the reason for disagreement before changing a score.
- Version rubric changes and reassess affected cases consistently.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Keep independent judgments independent long enough to compare them
LangSmith documents reviewer settings and rubric feedback within annotation queues. Whatever tool is used, Rubrex recommends preserving the initial judgment, rationale, evidence snapshot, and rubric revision before reviewers discuss a disputed case. Otherwise the final labels cannot show whether the instructions were independently understandable.
Use a fictional product-support answer that gives the right general policy but omits a regional exception. One reviewer marks it correct because the main fact is true; another marks it incomplete because the exception changes the user’s action. The disagreement concerns the acceptance definition, not simply which reviewer has more experience or confidence.
Evidence: LangSmith: Use annotation queues [1]
Classify the cause before choosing a resolution
Inspect the disputed evidence together with the task the answer was supposed to accomplish. Ask whether the reviewers saw the same document versions, interpreted the same scope, and applied the same rule. Separate factual mistakes, missing context, differing severity interpretations, and instructions that permit more than one reasonable judgment.
In the support example, check whether the question identified the region and whether the rubric required applicable exceptions. If neither specifies that requirement, a retrospective verdict alone leaves the underlying ambiguity unresolved. Record the gap, clarify the rule, and identify other cases that may have been labeled under the same competing interpretations.
Write a resolution that another reviewer could reproduce
The adjudication record should identify the original judgments, decisive evidence, adopted interpretation, final state, and adjudicator. Use an unresolved state when the evidence cannot decide the matter. Do not convert uncertainty into a passing answer merely because the release process expects a binary score.
Distinguish a corrected label from a new rubric. A label correction applies an existing clear rule that was missed. A rubric change alters what counts as acceptable and can require rescoring relevant baseline and candidate outputs. Preserve both versions and explain which one supports the reported comparison, rather than changing only the candidate’s difficult cases.
| Case | Evidence to keep | Decision |
|---|---|---|
| Missed explicit exception | Same clear rubric and decisive passage | Correct the label with a reason |
| Unspecified exception requirement | Two defensible interpretations | Version the rubric and reassess affected cases |
| Missing regional evidence | Neither reviewer can establish applicability | Retain an unresolved state |
Use the disagreement to improve the next independent trial
Add a clear positive example, a clear negative example, and a boundary example to the annotation guidance. Then test the revised guidance on fresh cases without coaching each reviewer toward the desired label. A discussion that produces consensus on one case does not establish that future independent judgments will be dependable.
Monitor which disagreement causes recur. If evidence access is the problem, repair the packet. If the task itself is underspecified, improve the product contract. Keep adjudicated training examples separate from a final held-out set so teaching the exact resolutions does not create a misleading impression that reviewer reliability improved on unseen work.
Limits of the evidence
The example and adjudication record are proposed, not observations from a Rubrex annotation study. Agreement does not establish correctness, and some disputes remain unresolved until the task owner supplies evidence or a clearer acceptance definition.
Common questions
Should the majority label always win?
A majority can share the same mistaken interpretation. Resolve the task against evidence and the applicable rubric, preserving uncertainty when necessary.
Can adjudicated cases train reviewers?
Yes, when identified as training examples. Keep a separate unseen set for checking independent application of the revised guidance.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- LangSmith: Use annotation queues LangChain · Living reference · Maintained documentationReviewed: current run/thread distinctions, single and pairwise queues, configured rubrics and reviewer settings; no product comparison. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.