On this page
The short answer
Build multilingual evaluations using both comparable cross-language tasks and work authored for the actual language community and product context. Track language, script, locale, and task coverage separately. Translating an English benchmark can support one comparison, but it does not establish that the resulting cases represent local user needs.
What to take away
- Track language variety, script, and task scope.
- Combine aligned tests with locally authored work.
- Validate evaluators in the languages they judge.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Define the language scope precisely
The current FLORES+ dataset card describes multilingual translation evaluation and language-variety identifiers. That is a narrower scope than every multilingual assistant task. The public card was checked today; this guide does not claim to have downloaded or reviewed its gated test sentences.
List the product’s supported languages, scripts, regional conventions, and workflows. A language name alone can conceal a script distinction or code-switching pattern that changes input handling. Record unsupported combinations rather than implying that a general model capability automatically establishes product support.
Evidence: FLORES+ Dataset Card [1]
Use aligned and locally authored cases
Create aligned cases when the intended question is whether the same requirement survives a language change. Review each translation for equivalent difficulty and meaning. Keep shared case-family identifiers so aligned versions are not treated as unrelated independent tasks.
Add locally authored cases for terminology, formats, user expectations, and workflows that were absent from the English starting set. Include code-switching or transliterated input only where it reflects the intended audience. A mechanical expansion into many languages can multiply case count without improving relevance.
Test evidence and evaluator competence too
For an illustrative support task, the user asks in Urdu while the policy evidence is in English and the expected answer must preserve a specific exception. Evaluate retrieval, understanding, answer language, and exception coverage separately. A correct English answer may still violate the product’s response-language contract.
Give reviewers the necessary source evidence and a language-appropriate rubric. If an automated judge is used, validate it for each relevant language and error type. Agreement with English judgments on translated labels does not establish that it recognizes local phrasing or culturally specific ambiguity.
| Case or record | Expected check | Decision |
|---|---|---|
| Urdu question; English policy | Required exception retained in Urdu answer | Check task and language separately |
| Aligned translated cases | Same underlying requirement | Group as related observations |
| Unsupported script variant | No representative test coverage | Disclose the support gap |
Report coverage before making cross-language claims
Show case counts, task mix, invalid inputs, incomplete outputs, and accepted tasks by language and script. Keep missing-language cells visible. Do not combine unequal task populations into a language ranking without explaining the differences.
Investigate a low result by stage: encoding, parsing, retrieval coverage, generation, or evaluator quality. Confirm improvements on new locally relevant examples. State where the evidence is sparse and where qualified review was unavailable. A multilingual release decision should identify the exact supported scope rather than treating one aggregate as proof of universal language parity.
Limits of the evidence
The multilingual support scenario is authored. No language or model comparison was run, and only the public FLORES+ card was reviewed. Language varieties, local workflows, evaluator competence, and uneven sampling can limit cross-language conclusions.
Common questions
Is translating an English test set useful?
Yes, for controlled comparisons when semantic equivalence is reviewed. Supplement it with locally authored tasks rather than assuming it covers every user need.
Can I use one judge for every language?
Only within a validated scope. Check its errors and uncertainty in the relevant languages, scripts, and task types.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- FLORES+ Dataset Card Open Language Data Initiative · Living reference · Dataset cardReviewed: public dataset card: translation evaluation scope, language/script/variety fields and aligned sentence IDs; gated data files not accessed. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.