Test RAG fallback search when the first retrieval fails

Define the failure that triggers recovery

LangChain’s retrieval documentation separates retrieval architecture choices and stages. A fallback policy should name the condition it addresses: no candidates, missing required evidence, a service error, or low confidence from a tested evaluator. Those are not equivalent states. An empty result can be correct when no eligible source exists, while a timeout says nothing about corpus content. Define the permitted next action for each state before evaluating final answers.

Evidence: Retrieval [1]

Build recovery and no-recovery controls

Author a fictional support question whose answer is absent from the primary collection but present in an approved secondary manual. Add a second question absent from both sources and a third already answered by the primary collection. The policy should recover the first, disclose the gap in the second, and avoid unnecessary expansion in the third. Include a secondary source about a different product so the fallback cannot pass by accepting the first plausible result.

Preserve scope across the branch

Record the original query, resolved entity, authorization context, fallback query, source set, and final evidence. A fallback that drops a region or product version may improve apparent recall while answering a different question. Keep user-specific or sensitive details out of external queries unless the application’s data policy permits them. Test that secondary retrieval cannot bypass the eligibility rules applied to the primary collection. A change of backend is not permission to widen access or use unsupported sources.

Measure recovery outcomes and extra work

Compare the fallback system with a fixed primary-only baseline on the same questions. Report genuine repairs, unnecessary fallbacks, worsened answers, unresolved gaps, and failed calls separately. Count retries and elapsed time even when no useful evidence is added. Inspect the final context for contradictions introduced by recovery. A successful secondary search is an intermediate event, not the outcome: the evidence must be applicable and used correctly in the final response.

Require an explicit stopping decision

Set a maximum work budget and a condition for concluding that the approved search paths have not established an answer. Test that the application can stop without manufacturing certainty. Offer a relevant next step, such as identifying the missing source or asking a clarifying question, only when it fits the actual gap. Preserve traces for failed recoveries and replay them after changing triggers or source policies. The goal is useful, bounded recovery, not a loop that searches until something sounds convincing. Include a recovery source that corroborates existing evidence but adds no missing fact. The policy should not count that result as a repaired question, even when the secondary call itself completed successfully.

Common questions