Evaluate multilingual retrieval across query and document languages

Define the language conditions

Azure documents language-aware analysis and separate indexing and search analyzers. Multilingual evaluation should record the configured analysis path as well as the embedding model. A translated query, a multilingual embedding, and a language-specific lexical analyzer are different interventions. Define whether users may receive evidence in another language and which answer language is expected. A correct passage in an unreadable language may not satisfy the application’s user experience, even when semantic relevance is high.

Evidence: Analyzers for linguistic and text processing [1]

Build a paired-language fixture

Author a short fictional travel policy in English and Urdu, with equivalent approval rules but distinct document IDs. Ask the same intent in both languages and include a transliterated product name. Have a qualified reviewer verify the meaning before treating the pairs as ground truth. Add an intentionally outdated translation to test applicability. The acceptance rule follows the current approved rule, not whichever language variant is easiest for the retriever to match.

Measure retrieval and translation effects

For each language pair, retain question text, reviewed intent, relevant evidence IDs, and final answer support. Compare original-language queries with translated queries under the same eligibility and context budgets. Inspect whether translation loses negation, dates, or entity distinctions. Record empty results and processing failures in their original groups. A strong English-English result should not compensate for a systematic Urdu-English failure when both groups are within the supported product scope.

Inspect lexical and numerical failures

Named entities, model codes, punctuation, and numerals can behave differently across scripts and normalization rules. Add questions containing an unchanged identifier, a localized spelling, and a numerical unit. Check tokenization and normalization before assuming that every mismatch requires a new embedding model. Preserve the original text in the evidence record so a reviewer can identify whether meaning changed during preprocessing. Avoid comparing multilingual scores whose label quality differs substantially without stating that limitation.

Set support boundaries from evidence

Declare supported language pairs and question types based on the tested coverage, not merely on a model’s advertised language list. If evidence is available only in another language, test whether the answer preserves qualifiers when translating it. Keep an escalation or clarification path for unresolved entity names and unreviewed language pairs. Recheck the paired fixture after analyzer, translation, or embedding changes. Report the weakest required groups alongside the average so the release decision reflects the actual audience. Retain a terminology glossary only when it reflects approved meanings for the domain. Test a word with multiple valid translations so the system cannot pass by mechanically choosing a fluent but task-incompatible interpretation.

Common questions