Test query rewriting for preserved intent and useful recall

Make preserved meaning inspectable

LangChain documents retrieval as a combination of modular stages. Query transformation is one stage whose output should be inspected separately. For each original question, identify entities, requested action, time scope, negation, and any comparison being requested. Keep authorization scope outside model-generated rewrite text. A rewrite may make wording clearer, but it should not broaden eligible documents or silently replace an unresolved entity with a guess that produces a more convenient search result.

Evidence: Retrieval [1]

Author a constraint-sensitive example

Suppose a user asks whether an annual plan bought before June can be canceled without a fee, excluding business accounts. A valid rewrite may expand the product abbreviation while preserving purchase date and the business-account exclusion. An invalid rewrite asks for the current standard cancellation policy. Provide both outputs as authored cases. The latter may retrieve a clean, relevant-looking policy, yet it answers a different question and must fail the preservation check before retrieval scores are considered.

Measure conditional retrieval improvements

Run original and rewritten queries against the same index snapshot, filters, and budgets. Score the rewrite’s retained constraints first, then compare the labeled evidence found by each query. Count queries that improved recall only after losing a constraint as failures, not wins. Record empty rewrites, multiple competing rewrites, and requests that bypass transformation. Inspect whether entity expansion helps one abbreviation while harming an unrelated entity with the same initials. Those differences inform a narrower rewriting policy.

Limit rewriting to justified cases

Adopt a rewrite policy only for the query groups where it preserves meaning and improves useful evidence under the agreed budget. Keep an identity-rewrite control and a path to search the original query when the transformation is ambiguous. A clarifying question may be more appropriate than a forced rewrite when two products share a name. Preserve original question, rewritten text, evidence IDs, and failure reasons together so an engineer can repair intent drift without changing the answer generator.

Treat unresolved references as explicit uncertainty

A question such as “Does that rule apply to mine?” cannot safely be rewritten without knowing which prior rule and account the user means. Add cases with two plausible antecedents and label the expected behavior as clarification rather than a guessed expansion. Store the minimum conversation context needed to resolve each reference. Evaluate whether the rewrite introduces a product, date, or account type that was never established. This catches a common failure hidden by retrieval metrics: the system can find excellent evidence for a confident interpretation that the user did not authorize or intend. Keep the identity rewrite as a scored baseline even after deployment so transformation remains justified by observed evidence.

Common questions