On this page
The short answer
Test verbosity bias by changing answer length while keeping correctness, required coverage, and task suitability stable. Compare paired judgments and inspect the reasons. A longer answer can be better when it supplies required information, so length preference becomes a problem only when it rewards irrelevant or unsupported expansion.
What to take away
- Match required facts before changing length.
- Include unsupported and genuinely useful expansions.
- Retain ties and order-sensitive results.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Distinguish length from useful coverage
The MT-Bench paper’s abstract identifies verbosity bias among limitations of LLM judging. That finding motivates a local check; it does not establish the size of the effect in your evaluator, prompt, model, or task.
Define the product’s required facts and acceptable response style before constructing pairs. A concise confirmation can be ideal for an operational task, while a troubleshooting explanation may legitimately need several steps. Without that contract, a reviewer cannot tell whether a judge favored verbosity or correctly noticed missing information in the shorter answer.
Evidence: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena [1]
Author controlled answer pairs
Write a short answer that fully satisfies the case, then expand it with harmless restatements while preserving the same claims. Have an independent reviewer verify equivalent correctness and coverage. Avoid adding examples that change usefulness unless that is the separate variable you intend to test.
Create a second pair where added length introduces an unsupported claim. The judge should not reward the expansion merely because it sounds thorough. Include a third pair where the longer answer adds a genuinely required exception; otherwise the test may train the team to equate brevity with quality regardless of user need.
Balance presentation and retain all outcomes
Randomize pair order and remove model names or other identity cues. Run the same rubric on the same pairs and preserve ties, invalid responses, and abstentions. Repeat enough to observe unstable judgments within your budget rather than treating one preference as a reliable effect.
For an illustrative policy question, both answers may state the same deadline, but one repeats it in four paragraphs. An equivalent-quality label expects a tie. A longer answer inventing an extension should lose on factual support; a longer answer explaining a required exception can win on completeness.
| Case or record | Expected check | Decision |
|---|---|---|
| Same facts; repeated wording | No additional task value | Expect equal quality |
| Longer answer invents exception | Unsupported material claim | Prefer the supported answer |
| Longer answer includes required exception | Improves required coverage | Permit a justified preference |
Repair the rubric without rewarding omissions
Add explicit instructions to evaluate required content, factual support, and task-appropriate concision separately. Test the revised rubric on held-out pairs and retain its decisions before and after the change. A successful repair reduces unjustified length preference while still recognizing useful explanation.
Report the number of equivalent-content pairs, the direction of preferences, order sensitivity, and reviewer disagreements. Avoid claiming a universal judge-bias rate from authored examples. If the effect persists in a critical workflow, use direct checks or human review for that criterion instead of trusting a pooled quality score.
Limits of the evidence
This is an authored bias-test design, not a reproduction of MT-Bench or a measurement of a current judge. Equivalent-content labels can be disputed, and length can interact with readability, completeness, and task type. Interpret results only within the tested cases and rubric.
Common questions
Should the shortest answer always win?
No. Compare answers against the user’s task. Brevity is useful when it preserves the required facts, qualifications, and actionable steps.
Is a single long-answer preference a bias finding?
It is an observation to investigate. Controlled equivalent-content pairs, order balancing, repeated judgments, and reference review make the conclusion more defensible.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena Zheng et al. · 2023 · PreprintReviewed: abstract and bibliographic record; position, verbosity and self-enhancement biases; no benchmark replication. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.