Test retrieval of acronyms and ambiguous entity names

Define entities independently of wording

Azure’s analyzer documentation distinguishes linguistic processing from the original text. Evaluation labels should identify the intended entity, not only an exact string. Two documents can share an acronym while describing unrelated organizations or products. Store stable entity IDs, accepted aliases, and scope where available. Keep the user’s authorized context separate from model-generated expansions. A query rewrite that expands an acronym to the wrong organization can produce a highly relevant result for the wrong task.

Evidence: Analyzers for linguistic and text processing [1]

Author a shared-acronym example

Create fictional documents for ARC Analytics and ARC Archive, each with a different data-export policy. Ask “Does ARC support bulk export?” once with an earlier analytics context and once with no resolving context. The first case requires the analytics policy; the second requires clarification or an explicitly qualified response. Include a product code ARC-17 to test exact matching. A result mentioning ARC-71 should not pass merely because its text is semantically similar.

Inspect the whole name-resolution path

Record the original query, resolved entity ID, rewritten query if any, retrieved source entities, and final answer attribution. Compare exact-code queries, alias queries, and misspellings separately. If the correct entity is resolved but the wrong source ranks highly, investigate retrieval or metadata filters. If the resolution itself is wrong, reranking will not repair the mistaken interpretation. This staged record prevents every ambiguous-name failure from being treated as a generic semantic-search problem.

Test normalization without erasing distinctions

Case folding, punctuation removal, transliteration, and abbreviation expansion can help some queries while collapsing different identifiers. Include examples where a hyphen or suffix matters to the product version. Review analyzer output on those strings when available. Keep an identity-preserving control and an intentionally ambiguous case. Do not accept a preprocessing change solely because it increases matches; require that it preserves the distinctions needed for correct evidence selection and answer applicability.

Use uncertainty as a valid outcome

Define when the application should ask a clarifying question rather than forcing a single match. A short clarification can be the correct result when two equally plausible entities have incompatible policies. Score unnecessary clarification separately on clearly identified queries so the system cannot pass by asking every user to restate their request. Preserve the alias map revision and disputed cases with the evaluation. Entity vocabularies change, and a previously unique name can become ambiguous after a corpus update. Add a case where the user explicitly corrects an expanded acronym. The corrected entity should replace the prior assumption in retrieval, and the failure record should preserve both forms rather than silently overwriting the evidence of the mistake.

Common questions