On this page
The short answer
Evaluate streaming errors after the client has already received part of an answer. Check what remains visible, whether a retry replaces or duplicates content, and which task status is stored. A stream ending is not equivalent to a complete answer, and a successful fallback cannot erase effects already shown or executed.
What to take away
- Separate partial output from completed task state.
- Test recovery after the first visible content.
- Prevent incomplete streams from authorizing actions.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Define completion at the consumer
LiteLLM documents fallback handling across model groups. Rubrex recommends testing the client-visible stream separately because recovery before any output differs from recovery after output has appeared. The consumer needs explicit partial, completed, failed, and canceled states.
Use a synthetic report stream that emits a title, two facts, and a final completion event. Only a complete report may be marked ready for review. The interface may show partial text, but it should label it accurately and avoid presenting an incomplete final paragraph as a verified recommendation.
Interrupt at different visible boundaries
Make a mock upstream fail before the first event, after the title, in the middle of a structured record, and just before completion. Add a clean stream and a user cancellation. Capture raw events and the rendered result so parsing and presentation failures can be distinguished.
Enable the intended recovery policy in a separate branch. Observe whether a second answer replaces the partial draft, appends to it, or creates a new artifact. A repeated title or contradictory continuation can confuse the user even when the provider’s second attempt technically succeeds.
Inspect final state and downstream consumers
Check stored report status, displayed status, export availability, and any action triggered by completion. A timeout handler should not publish a partially accumulated object merely because some fields parse. For tool-using workflows, inspect whether a duplicated event can repeat an operation already admitted.
Record time to first content, time to usable completion, incomplete outcomes, recovery attempts, and user-visible duplication separately. Keep failed and canceled streams out of completed-answer latency summaries unless the report explicitly describes that censoring. Include an event with an unknown type to test controlled parsing without silently discarding the completion signal.
| Case | Evidence to retain | Acceptance condition |
|---|---|---|
| Mid-record failure | Raw events and stored report | Incomplete state preserved |
| Fallback after title | Rendered output comparison | No unexplained duplicate continuation |
| Missing completion event | Consumer state | No false ready artifact |
Accept truthful recovery rather than seamless appearance
Release the report stream when incomplete work is clearly identified, only completed artifacts enter the ready state, and recovery does not duplicate downstream effects. If automatic continuation cannot preserve meaning, offer an explicit restart or retained partial draft under the product contract.
Retain event fixtures and screenshots at the failure points. Re-run after changes to SDK event formats, rendering, proxies, or fallback policy. The goal is a trustworthy transition from partial information to a usable result, not an interface that looks successful while hiding an interrupted generation.
Limits of the evidence
The stream events and report are synthetic proposed fixtures. No provider stream was benchmarked. Event semantics, cancellation, and fallback behavior differ across SDKs and clients, so the acceptance record must name the tested versions and consumer.
Common questions
Does an HTTP success mean the answer completed?
No. Inspect the stream completion contract and the usable final artifact.
Should partial text be discarded?
That is a product choice; label and handle it explicitly rather than silently treating it as final.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- LiteLLM: Fallbacks (Provider Failover) LiteLLM · Living reference · Maintained documentationReviewed: documented fallback behavior and Proxy mock-testing deprecation beginning with version 1.85.0. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.