On this page
The short answer
An LLM gateway mediates application requests to model providers and can centralize routing and operational policies. Evaluate the exact gateway configuration by checking approved destinations, bounded retries, identity and budget enforcement, cache scope, streaming failures, and trace completeness. A successful fallback response is useful only if it still satisfies the application’s task and data requirements.
What to take away
- Distinguish a retry within a route from a fallback to a different route.
- Approve fallback destinations for capabilities and data handling before enabling them.
- Test errors through the deployed gateway path, not only an isolated routing function.
- Count every attempt and verify cache equivalence for authenticated users.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Define which responsibilities belong to the gateway
A gateway can provide a common request entry point, while a router selects an eligible destination. LiteLLM’s architecture documentation distinguishes retries inside a model group from fallbacks to another group. Its virtual-key documentation describes model access and usage controls. These documented mechanisms are configuration surfaces to test, not a substitute for the application’s own authorization and task checks.
Rubrex recommends drawing the actual request path: authenticated user, application, gateway, provider, and any application tool that may perform a side effect. Name the owner of each decision. A gateway API key identifying a backend service may not identify an individual customer or tenant; decide where that mapping is verified and which limits apply.
Evidence: LiteLLM: Life of a Request [1]LiteLLM: Virtual Keys [3]
Write an approved route matrix before enabling failover
For each task class, list eligible providers and deployments, required input and output features, acceptable data locations and handling terms, maximum latency, and fallback policy. Verify those requirements against the actual provider agreement and deployment. A compatible request shape does not prove equivalent tool behavior, structured output, context handling, or privacy terms.
Keep refusal or content-policy handling separate from availability recovery. Decide explicitly whether any policy rejection may be retried or rerouted under your product rules. A failure should not silently broaden the allowed destination set. Test a request whose only configured backup is prohibited and verify a controlled failure before content reaches that backup.
| Requirement | Record before rollout | Acceptance check |
|---|---|---|
| Allowed destination | Named deployment and permitted task classes | No unlisted fallback receives the request |
| Required capability | Tool schema, response format, input modalities | The real task still satisfies its output contract |
| Data handling | Approved location, retention, and agreement reference | Primary and backup are both within the approved scope |
| Work budget | Deadline and maximum total attempts | Retry and fallback paths remain bounded |
| Failure response | What the caller sees when no eligible route works | No false success or unsupported substitute answer |
Inject controlled failures through the path you will deploy
Use a non-production gateway and instrumented test upstreams. Make one return a rate-limit error, another delay beyond a configured timeout, and another return an invalid response. Record upstream calls and the final client response. Then restore a normal upstream as a control. This is a proposed fixture; no gateway experiment was run for this guide.
LiteLLM’s current fallback guide says Proxy requests strip the mock_testing fallback flags beginning with version 1.85.0; direct Router tests still support them. Its documentation advises exercising Proxy fallbacks with a provider error in a non-production environment. Check your installed version and test the actual boundary rather than assuming a historical sample flag still triggers failover.
Add an all-upstreams-unavailable case. Verify the deadline, the number of admitted attempts, the error surfaced to the caller, and whether late work continues. Keep a distinction between transport completion and task success: receiving a response body does not prove that the backup followed the intended output or evidence contract.
Budget retries across the client, gateway, and provider layers
Retries can multiply when the application and gateway each retry independently. For an illustrative upper-bound calculation, suppose a client permits two attempts and each gateway invocation permits three provider attempts in total. Up to six provider attempts may then occur for one user task. If the gateway limit instead applies separately to each fallback group, the bound changes; inspect and test that exact semantics.
Keep a stable task identifier and a distinct identifier for each attempt. Record requested route, actual destination, start and completion status, usage provenance, and retry reason. Include failed and canceled work in cost accounting where it consumed resources. Distinguish a budget admission decision from a later billing estimate; one does not demonstrate the other.
For workflows that execute application tools, preserve idempotency and reconciliation at the side-effect boundary. A repeated model request can produce another tool proposal. Do not treat a successful model fallback as permission to repeat an already completed business operation.
Test key restrictions and exhausted budgets as separate boundaries
LiteLLM documents model permissions and budget or rate-limit settings on virtual keys, with inheritance depending on owner and team configuration. Use that as a reason to review the effective policy rather than only the settings displayed on one key. This guide does not claim all controls apply identically across editions or deployment modes.
Create small synthetic allowances in a test environment. Try an allowed route, a forbidden route, an expired or revoked test key, and two concurrent requests competing for the final allowance. Inspect whether the denial occurs before new provider work is admitted. Also run a legitimate sufficient-budget control so a blanket outage cannot be mistaken for effective enforcement.
If several gateway replicas share a quota, repeat the concurrency case across those replicas and inspect the shared reservation behavior. A test against one process cannot validate distributed enforcement. Record which storage and failure modes were exercised, including the behavior when the quota store is unavailable.
Evidence: LiteLLM: Virtual Keys [3]
Check whether two requests are actually allowed to share a response
Cloudflare AI Gateway’s caching documentation says its default key includes provider, endpoint, model, provider authentication header, and the full request body. It also documents custom keys that make requests with the same custom key share a response. Those are product-specific behaviors checked on October 16, 2026; inspect your gateway’s own key construction.
Provider authentication can be shared by many application users, so it is not a substitute for customer authorization context. For an illustrative test, two synthetic users submit the same question through a shared backend credential but have different permitted documents. Warm the cache with one user, then request as the other. Verify that the response contains only permitted evidence.
Include document version, prompt version, output contract, and relevant authorization scope when deciding equivalence, or bypass caching for responses that cannot safely share. Test updates and revocation as well as cache hits. This is an acceptance-design recommendation, not a claim of a vulnerability in Cloudflare or any specific gateway.
Evidence: Cloudflare AI Gateway: Caching [4]
Inspect streaming failures and keep a rollback decision
Use a test upstream that emits a partial stream and then fails. Observe what the client has already displayed and whether a second response replaces, appends to, or duplicates it. Define a usable incomplete state before enabling automatic recovery. A fallback that works before the first byte may behave differently after partial output has reached the user.
Release the configuration only when the approved route matrix, task-quality checks, data boundaries, and failure fixture meet the written criteria. Keep a reversible rollout, an owner, the exact configuration version, and a way to disable a failing route. Evaluate model-routing economics in the separate routing guide; this checklist establishes whether the gateway enforces the intended behavior during normal and failed requests.
Limits of the evidence
The fixture, arithmetic example, and acceptance checks are proposed methods, not measured gateway results. Official documentation was checked on October 16, 2026; deployed versions, editions, client retries, provider terms, and custom policies may differ. This guide does not verify any production gateway’s security, distributed quotas, data handling, availability, or cost savings.
Common questions
What is the difference between an LLM gateway and an LLM router?
A gateway is an entry point for requests and shared operational controls. Routing is the decision about which eligible destination serves a request. One product can provide both, but test their responsibilities separately.
Does adding a fallback make an AI application reliable?
It can recover some availability failures. The backup still needs to pass task-quality, feature, permission, data-handling, budget, and latency checks.
Can cached answers be reused across tenants?
Only when the response is equivalent and permitted for every caller sharing that cache entry. Shared provider credentials or identical user text alone do not establish that condition.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- LiteLLM: Life of a Request LiteLLM · Living reference · Maintained documentationReviewed: documented gateway request lifecycle and retry-versus-fallback distinction. Accessed October 16, 2026.
- LiteLLM: Fallbacks (Provider Failover) LiteLLM · Living reference · Maintained documentationReviewed: documented fallback behavior and Proxy mock-testing deprecation beginning with version 1.85.0. Accessed October 16, 2026.
- LiteLLM: Virtual Keys LiteLLM · Living reference · Maintained documentationReviewed: documented model permissions, owner/team inheritance, and budget/rate-limit settings; no enforcement test. Accessed October 16, 2026.
- Cloudflare AI Gateway: Caching Cloudflare · Living reference · Maintained documentationReviewed: documented default cache-key inputs and response sharing through custom cache keys; no vulnerability claim. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.