Evaluate multi-query retrieval against its extra cost

Specify what extra queries should add

Qdrant documents staged queries that combine candidate sets. An evaluation of generated query variants needs an additional intent-preservation check. State which missing evidence each variant is expected to recover: a synonym, a second document, or a different description of the same event. More searches can increase work without broadening useful coverage when they all retrieve the same popular paragraph. Keep a contribution record rather than treating every returned result as independent evidence.

Evidence: Hybrid and multi-stage queries [1]

Construct a two-source fixture

Use an authored support question requiring a setup guide and a regional restriction. The baseline query retrieves only setup instructions. One variant names the region and can recover the restriction; another merely paraphrases setup and produces duplicates. Add a drifting variant about an unrelated region. The expected acceptance rule requires both applicable sources and rejects the unrelated restriction. This fixture separates genuine evidence expansion from repeated retrieval and from breadth achieved by changing what the user asked.

Evaluate merging before final answers

Record every query, its candidate IDs, rank, latency, and any generation cost. Deduplicate merged results using stable source identities, then apply the same final context allowance as the baseline. Score new required evidence, displaced evidence, and unsupported regions separately. Compare a fixed maximum query count with a policy that stops after the required evidence is found. Account for canceled and failed calls, because their work and delay still affect the experience even when they contribute no context.

Require marginal value for expansion

A useful policy adds queries only when the expected evidence gap justifies their cost and permitted delay. Test that policy on held-out questions, including easy ones where a single search is already sufficient. Keep a single-query fallback for failures in query generation or merging. Review cases where extra queries improve recall but worsen the answer by introducing contradictory sources. The decision should explain when expansion is allowed, when it stops, and how the application presents unresolved evidence gaps.

Distinguish a failed call from a useful empty search

An expanded query may return no results because its hypothesis is wrong, because no source exists, or because the search service failed. Preserve those states separately. A useful empty result can help the policy stop pursuing an unsupported branch, while a timeout provides no evidence about the corpus. Test a fixture where one optional branch times out but the required sources are already present. Define whether the application may answer, retry, or disclose a limitation. That policy determines acceptable behavior more clearly than demanding that every generated query return a passage. Record whether parallel queries share a deadline, since one optional slow branch can otherwise delay an already answerable task.

Common questions