On this page
The short answer
Run a provider-outage drill against controlled upstreams to test the application’s full failure path. Observe detection, admission, retries, approved fallback, queued work, and recovery. A backup returning text is only one part of success; the resulting task must still meet its quality, permission, budget, and data-handling contract.
What to take away
- Exercise application behavior with isolated upstream failures.
- Verify backlog recovery after the provider returns.
- Keep quality and boundary checks on backup responses.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Choose a narrow failure and a declared role
Google SRE discusses canarying as a way to limit exposure while evaluating change. Rubrex recommends applying the same bounded mindset to a synthetic outage drill. Define the affected upstream, task classes, expected stop conditions, observation window, and person responsible for ending the exercise.
Use a test assistant with a primary mock provider and one approved backup. One task supports the backup’s output contract; another requires a capability the backup lacks. The expected outcome differs by task. A blanket rule that any response is better than failure can silently break the second task.
Evidence: The Site Reliability Workbook: Canarying Releases [1]
Simulate outage and degraded recovery
Return connection errors from the primary, then timeouts, then a slow recovery rather than switching immediately from total failure to perfect health. Keep the test traffic small and synthetic. Record which requests are admitted, retried, rerouted, queued, or rejected at each stage.
Add an all-upstreams-unavailable case and a task whose deadline expires in the queue. Verify that the user receives a truthful incomplete status and that expired work does not later produce an unexpected action. The application should have an explicit policy for a backlog after recovery, not simply release every delayed attempt together.
Inspect backup outcomes and operational evidence
Run the same acceptance checks on backup responses as on primary responses: required fields, supported facts, permitted destinations, and task completion. Preserve actual provider or mock destination in the trace. A dashboard showing available traffic can hide that a critical task class now receives unusable output.
Measure task outcomes, oldest queued age, attempt amplification, work admitted after deadlines, and recovery time under the defined fixture. Include a healthy-primary control and an intentionally unsupported-backup control. Keep results scoped to the test configuration rather than turning a successful drill into an availability percentage claim.
| Case | Evidence to retain | Acceptance condition |
|---|---|---|
| Primary unavailable | Route and final task result | Approved compatible backup only |
| No eligible backup | Failure response and call ledger | Controlled truthful stop |
| Recovery backlog | Queue age and admissions | Expired tasks not revived |
Keep the drill result actionable
Accept the supported task path when it recovers through the approved backup within its bounds, and the unsupported task stops without substituting an invalid answer. Assign fixes to routing, queue admission, adapters, or user messaging according to the observed failure. Preserve a clear way to disable the backup.
Retain configuration, failure timeline, task ledger, and unresolved findings. Repeat the drill after changing provider integrations or queue policy. The release owner should leave with an inspectable recovery contract and known limits, including which tasks deliberately remain unavailable when the primary fails.
Limits of the evidence
This article proposes an isolated mock-provider exercise and reports no measured outage result. It does not advise disrupting production services. Real provider failures, traffic patterns, and recovery guarantees can differ materially from the fixture.
Common questions
Should every task use a backup?
Only when that backup satisfies the task’s capability and boundary requirements.
Does a drill prove production uptime?
No. It exercises a defined failure path and configuration under synthetic conditions.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- The Site Reliability Workbook: Canarying Releases Google · 2018 · Engineering book chapterReviewed: foundational staged-release and comparison principles; no AI-specific threshold asserted. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.