On this page
The short answer
Measure position bias by judging each answer pair in both orders and mapping every decision back to the original answer identity. A judge that prefers the first position after a swap can create a false model advantage. Preserve ties and inconsistencies rather than counting each presentation as an independent task.
What to take away
- Map display labels back to answer identities.
- Treat swapped presentations as related observations.
- Predefine the handling of inconsistent pairs.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Define identities independently of display order
The MT-Bench paper discusses position bias as a limitation of LLM-based judging. Use that as motivation for a controlled local test, not as evidence that a particular current judge has a known error rate.
Give answers stable hidden identifiers and present temporary labels such as A and B to the judge. Store the mapping for every presentation. An analysis that counts all A selections as wins for the same model after swapping the displayed answers has a bookkeeping defect even if the judge itself is perfectly consistent.
Evidence: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena [1]
Hold the content and rubric fixed
Select pairs from representative tasks, including clear wins and legitimate ties. Present the same two answer texts in both orders with identical task evidence and scoring instructions. Remove provider names, response metadata, and formatting differences that are unrelated to the intended comparison.
Separate the swap test from a prompt rewrite or evaluator-model change. If several variables change together, a disagreement cannot be attributed to order. Retain invalid outputs and abstentions with the presentation record so they do not disappear when the analysis joins the two orders.
Use a decision table for swapped outcomes
An illustrative pair contains answer X and answer Y. If X wins whether it is shown first or second, the preference is order-consistent. If the first displayed answer wins both presentations, the pair is position-sensitive. A tie in one presentation and a win in the other is another form of instability worth retaining.
Report the count of original tasks and the number of presentations separately. The two orders are related observations of one pair, not two independent customer tasks. Keep repeated judgments nested within that pair when summarizing uncertainty or deciding which examples need human review.
| Case or record | Expected check | Decision |
|---|---|---|
| X wins in both orders | Identity preference survives swap | Order-consistent win |
| First position wins twice | Identity preference reverses | Position-sensitive; review |
| Tie then X wins | Decision changes with order | Retain as instability |
Choose how inconsistent pairs affect the release
Write the policy before viewing aggregate wins. You might send position-sensitive pairs to review, treat them as unresolved, or require order-consistent preference for a narrow comparison. None of those policies should silently convert uncertainty into a win for the preferred candidate.
After revising the judge prompt, repeat the swap test on held-out pairs and confirm that clear reference wins remain detectable. Report the remaining inconsistent pairs and their task types. An apparently balanced overall win rate can conceal a strong position preference if the answer ordering was balanced but individual decisions remained unreliable.
Limits of the evidence
The procedure uses authored or sampled answer pairs and does not measure a named current model. Order effects can interact with content and rubric design. An order-consistent judgment can still be wrong, so this check complements reference-quality validation rather than replacing it.
Common questions
Does random answer order solve position bias?
It can reduce systematic advantage in an aggregate, but it does not make an individual judgment dependable. Swap checks reveal which pairs remain unstable.
Should inconsistent pairs be discarded?
Keep them visible. Exclusion changes the comparison population; state the policy and report how many pairs it removes from decisive results.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena Zheng et al. · 2023 · PreprintReviewed: abstract and bibliographic record; position, verbosity and self-enhancement biases; no benchmark replication. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.