On this page
The short answer
Evaluate a translation by comparing its meaning with the source, checking task-specific terminology and locale, and recording errors by consequence. Fluent target-language prose can still reverse a condition or omit a required exception. Use qualified bilingual review for the tested language pair and distinguish material errors from stylistic preferences.
What to take away
- Prioritize source meaning over target fluency.
- Record category, severity, and evidence spans.
- Evaluate each language pair and locale deliberately.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Define a source-to-target acceptance contract
Google’s WMT MQM evaluation repository documents error categories, severity, and attention to document context. Its historical evaluation protocol is a useful reference, but this guide does not reuse its rankings or prescribe its weights for every product.
Specify the language pair, audience, domain, locale, and intended action. Preserve terminology that must remain stable, such as product names and account identifiers. Identify whether the task is faithful translation, localized adaptation, or a translated summary. A summary should not be judged as if every omitted sentence were a translation error unless full fidelity was required.
Evidence: WMT MQM Human Evaluation [1]
Review meaning before polish
Ask reviewers to identify the source span, target span, error category, severity, and correction rationale. Separate mistranslation, omission, added unsupported content, terminology, and fluency. Define when an issue changes the user’s action and therefore deserves a material severity label.
Keep the surrounding document where pronouns, references, or formality depend on context. An isolated sentence can be ambiguous even when its document-level meaning is clear. Do not use back-translation as the sole ground truth; it can introduce or conceal errors of its own.
Use a task-specific contrast fixture
In an illustrative software instruction, the source says do not delete the backup until verification completes. A fluent translation that drops the negation changes the required action and should fail materially. A less elegant translation that preserves the constraint may still satisfy the functional contract.
Include dates, decimal separators, named entities, ambiguous pronouns, and required glossary terms. Review locale transformations under an explicit policy. A day-month date rendered in an ambiguous numeric format can create a usability failure even when every word is grammatically correct.
| Case or record | Expected check | Decision |
|---|---|---|
| Negation removed from backup instruction | Required action reverses | Material failure |
| Awkward but faithful wording | Meaning and constraint preserved | Style issue; assess task tolerance |
| Ambiguous numeric date | Locale policy requires clarity | Fail or revise under the contract |
Report error distributions and reviewer limits
Report material-error counts and accepted segments or documents with their denominators. If weighted severity is used, show the raw categories and weights so a low total cannot conceal a critical meaning reversal. Preserve non-translation or unreviewable outcomes separately.
Compare candidates on the same source material and use independent review before adjudication. State the reviewers’ actual role without inventing credentials. Expand the test with domain-specific cases when generic benchmark text does not represent the application; successful performance on one language pair does not establish performance on another.
Limits of the evidence
The backup example is invented and no translation system was assessed. MQM-related source material provides an error-analysis reference, not proof of current model performance. Bilingual competence, source ambiguity, and domain context limit the review.
Common questions
Can an English-only reviewer evaluate translation quality?
They can inspect limited output properties, but source-to-target meaning usually needs competent review in both languages or a validated alternative process.
Should all errors have the same weight?
No universal weighting fits every task. Define consequence-based severity and keep raw error categories visible beside any aggregate.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- WMT MQM Human Evaluation Google Research · Living reference · Maintained documentationReviewed: README annotation categories, severity and document-context instructions; no historical ranking figures reused. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.