Compare hybrid search with lexical and vector baselines

Identify complementary retrieval needs

Elastic documents reciprocal rank fusion for combining ranked result lists. That mechanism does not establish that hybrid retrieval improves your application. Build query groups for exact identifiers, paraphrases, and questions requiring both a named entity and its meaning. Specify what counts as relevant evidence for each group. A hybrid average can improve because common paraphrases improve while a smaller but important product-code group loses the exact matching document.

Evidence: Reciprocal rank fusion [1]

Build an identifier-versus-meaning fixture

Use a synthetic equipment catalog containing part AX-17, part AX-71, and a prose troubleshooting guide that describes the same symptom without the part number. Ask one question by exact code and another by a symptom paraphrase. The expected evidence differs across the two questions. Keep a distractor with the right symptom but incompatible model. This fixture can reveal whether fusion promotes a broad semantic match ahead of the document containing the precise compatibility restriction.

Compare the stages under fixed budgets

Run lexical-only, vector-only, and hybrid retrieval against the same corpus snapshot and filters. Record component candidate depth, fusion settings, final context size, and the IDs contributed by each component. Score evidence recall and ordering before generating answers. Then test whether the answer retains the correct part and restriction. Record latency and retrieval work separately. Increasing both component pools and the final context can produce a different result without proving that the fusion method itself is responsible.

Choose a policy from the failure slices

Prefer the configuration that meets the stated identifier and paraphrase requirements without unacceptable cost or critical regressions. If no single fusion setting serves both, test a query-routing rule on held-out questions rather than tuning it on the same examples. Preserve cases where one component found the answer but fusion displaced it. Those are actionable fusion failures, unlike corpus gaps where neither component had an eligible supporting document to retrieve.

Before interpreting an improvement, inspect whether the fusion implementation changes duplicate handling or source truncation. Those differences belong in the comparison record because they can explain a win even when the underlying ranked lists are unchanged.

Inspect disagreement between component lists

For every changed answer, write down which lexical and vector ranks contributed to the final placement of its supporting passage. If lexical search finds an exact part code at rank one but fusion displaces it, the next experiment concerns merging, not tokenization. If neither component finds the correct code, inspect indexing and query analysis first. Keep a small disagreement table with question, relevant ID, component ranks, merged rank, and answer outcome. That evidence lets a reviewer judge whether an aggregate improvement reflects useful complementarity or simply a larger candidate budget.

Common questions