On this page
The short answer
Evaluate an AI canary by comparing a deliberately limited exposed group with an appropriate control, while monitoring critical events that can stop the release immediately. Retain task denominators, configuration identities, missing outcomes, and rollback evidence. A quiet site-wide average can hide a substantial regression within a small candidate group.
What to take away
- Inspect the candidate group separately from the site-wide average.
- Use critical-event stops alongside rate comparisons.
- Do not expand before the chosen task population is represented.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Specify exposure and observation before launch
Google's SRE Workbook describes canarying as limited production exposure coupled with an evaluation process. Applying that foundation to an AI application requires additional outcome checks: a request can return successfully while its answer is unsupported or its tool action violates the task contract.
Define the assignment unit, eligible traffic, comparison window, intended outcome, and configuration fingerprint. For a synthetic support deployment, pin each conversation to a branch so follow-ups do not switch models unexpectedly. Keep the contemporaneous control visible, and record material workload differences such as language, customer tier, or tool access that could explain an apparent branch effect.
Evidence: The Site Reliability Workbook: Canarying Releases [1]
Keep the small-group denominator visible
Consider an illustrative window of 1,000 tasks: 50 reach the candidate and 950 reach the control. If four candidate tasks fail and nineteen control tasks fail, the branch failure rates are 8% and 2%, while the combined failure rate is 2.3%. These are authored arithmetic examples, not measured model results.
The site-wide number hides the size of the candidate regression. It also does not establish statistical certainty about a small group. Preserve task-level results and uncertainty, and investigate whether failures cluster around one incident or task family. More requests that repeat the same easy case may not supply the coverage needed to evaluate a critical workflow.
Write both rate gates and event gates
Separate ordinary quality-rate comparisons from events that demand immediate containment. A reproduced cross-tenant disclosure, unauthorized write, or broken approval boundary may warrant stopping the affected path even when its frequency estimate is uncertain. Define the event classes and responsible owner before exposure rather than negotiating them after an incident.
Include timeouts, canceled work, unscored results, and unavailable telemetry in the review. Decide which missing evidence pauses expansion. A candidate should not appear safer because its failing traces never reached the dashboard. Keep synthetic permission probes alongside observed user tasks so important boundaries remain inspectable even when ordinary traffic rarely exercises them.
| Case | Evidence to keep | Decision |
|---|---|---|
| Candidate failures | 4 of 50 tasks; 8% | Inspect the exposed group directly |
| Control failures | 19 of 950 tasks; 2% | Preserve the comparison population |
| Critical unauthorized action | One reproduced event with trace | Apply the predefined containment rule |
Use expansion and rollback as separate decisions
Before increasing exposure, verify the agreed task coverage, inspect unresolved critical cases, and run a controlled return-path check. Confirm which configuration new work receives after rollback and how active candidate tasks are stopped, pinned, or reconciled. Restoring a routing percentage does not reverse a business action already committed by a tool.
Record a decision for the observed window: expand, continue collecting, restrict a task slice, or roll back. Include denominators, uncertainties, evidence gaps, and the named decision owner. Keep the artifact after the release completes so a later regression can be compared with the actual conditions under which the candidate was accepted.
Limits of the evidence
The numerical example is illustrative and no canary was run for this article. Exposure assignment, statistical analysis, critical-event policy, and rollback behavior must fit the application; a canary cannot prove the absence of rare failures.
Common questions
Is five percent always a suitable canary?
Choose exposure from consequences, task coverage, traffic, and detection time. This guide does not prescribe a universal percentage.
Can overall availability decide an AI rollout?
Also examine answer and action outcomes by branch. An HTTP success can contain a failed user task.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- The Site Reliability Workbook: Canarying Releases Google SRE · Living reference · Foundational book referenceReviewed: foundational limited-exposure and evaluation concepts; not a newly published 2026 study; synthetic 50/950 arithmetic independently checked. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.