On this page
The short answer
Evaluate fallback against an explicit eligibility matrix, not just a list of reachable models. For each task, define permitted destinations, required capabilities, data conditions, and failure behavior. A route that restores availability can still violate the task contract if it changes processing scope or returns an incompatible result.
What to take away
- Approve fallback eligibility per task class.
- Separate policy refusals from availability failures.
- Test a reachable but prohibited backup.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Build an eligibility matrix the gateway can enforce
LiteLLM documents fallback groups after failed attempts. Rubrex recommends binding that mechanism to an application-owned route matrix. Include task class, eligible deployment, required output features, permitted data treatment, and the evidence used to approve the route.
Use a synthetic internal-document task and a public-summary task. Their allowed destinations may differ. Record provider or deployment identifiers explicitly rather than relying on a nickname such as safe backup. Verify changing commercial or data conditions from the actual applicable terms; this guide does not assert any provider’s current policy.
Make the easy backup ineligible
Configure a controlled failure on the primary and leave one fast reachable backup outside the approved set. The correct result is to exclude that backup, even when doing so leaves the task incomplete. Inspect the actual upstream destination and payload, not only the selected route name.
Add a backup lacking a required structured-output or tool capability. A successful HTTP response should still fail the task check if the result cannot satisfy the contract. Keep refusal handling as a separate case: availability recovery should not silently reroute a request whose policy rejection requires a deliberate product decision.
Check inherited and dynamically changed routes
Exercise default routes, per-request overrides, queued tasks, and retries after a policy update. A matrix enforced only in the first request path can be bypassed by a later fallback or a stale worker. Retain the policy version used at admission and the actual destination used for each attempt.
Include a permitted backup and a no-backup case. Measure unauthorized destination admissions, incompatible outputs, controlled failures, and valid recoveries separately. Do not average a boundary violation into a general reliability score where many successful low-sensitivity tasks can conceal it.
| Case | Evidence to retain | Acceptance condition |
|---|---|---|
| Reachable prohibited backup | Actual destination ledger | No request sent there |
| Missing output capability | Task validation result | No false success |
| Approved backup | Policy version and receipt | Bounded compatible recovery |
Accept constrained recovery with clear ownership
Release the internal-document path when only eligible deployments receive its content and incompatible backups cannot create a false success. Assign ownership for eligibility evidence and review triggers, such as a new deployment, capability change, or revised processing requirement.
Retain the route matrix, configuration, destination receipts, and acceptance outputs. Re-run when adding a provider or changing global defaults. Fallback should be a controlled continuation of the same authorized task; it should not become an opportunity for the generated plan to choose a broader data path than the user or organization intended.
Limits of the evidence
The route classes and destinations are illustrative. No provider terms, regions, or legal requirements are asserted here. This proposed test verifies a chosen matrix only when implemented against the actual gateway and applicable agreements.
Common questions
Is API compatibility enough for fallback?
No. Evaluate required behavior, data conditions, permissions, and task quality.
Should a refusal automatically trigger another provider?
Define that explicitly in product policy; it is different from recovering an availability error.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- LiteLLM: Fallbacks (Provider Failover) LiteLLM · Living reference · Maintained documentationReviewed: documented fallback behavior and Proxy mock-testing deprecation beginning with version 1.85.0. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.