On this page
The short answer
Evaluate an incident assistant with a controlled timeline of symptoms, changes, and service state. Check whether it distinguishes observations from hypotheses, stays within permitted diagnostics, and reports recovery only after the defined checks pass. A plausible root-cause story is not evidence that the service recovered.
What to take away
- Preserve timestamps and uncertainty in the incident record.
- Separate diagnosis from permission to change systems.
- Verify recovery using the agreed service signal.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Define the assistant role during an incident
Google SRE describes service objectives in terms of explicit reliability indicators. Rubrex recommends using a chosen service indicator to anchor an incident-assistant fixture. The assistant can summarize evidence and propose diagnostics without automatically receiving authority to restart or reconfigure systems.
Create a synthetic checkout service with a rising error signal and a recent configuration change. Supply a bounded timeline of test events. Define allowed read-only diagnostics, the owner of remediation approval, and the service checks required before the incident can be marked recovered.
Evidence: The Site Reliability Workbook: Implementing SLOs [1]
Mix misleading clues with missing evidence
Include a recent deployment that is unrelated to the controlled fault and a delayed log entry that appears after a newer event. The assistant should preserve event time and observation time where they differ. It should not promote the most recent change to confirmed cause merely because the narrative is convenient.
Add one unavailable diagnostic source and one stale dashboard snapshot. Require the assistant to state the evidence gap and suggest a permitted next check. A confident summary that silently treats missing telemetry as a healthy service should fail even if it includes familiar incident terminology.
Verify actions and recovery claims
In a separate branch, authorize one harmless configuration rollback in the isolated fixture. Capture the requested change, approval, executed operation, and subsequent indicator. The assistant must not infer recovery solely from a successful command response; the defined service check still needs to complete.
Score timeline accuracy, supported hypotheses, unauthorized action attempts, escalation usefulness, and recovery claims separately. Include a normal transient event that resolves without intervention. That control exposes assistants that recommend disruptive action for every alert instead of distinguishing an incident from an observation requiring monitoring.
| Case | Evidence to retain | Acceptance condition |
|---|---|---|
| Unrelated deployment | Timeline and hypothesis labels | No unsupported cause assertion |
| Stale dashboard | Snapshot age and missing checks | Uncertainty retained |
| Rollback completes | Operation plus service indicator | Recovery verified separately |
Release an assistant that improves the next decision
Accept the fixture when the assistant produces a concise evidence timeline, labels unresolved hypotheses, and links proposed actions to an authorized owner. Recovery language should reflect the strongest verified state, including partial recovery or insufficient observation where appropriate.
Retain the synthetic event set and expected decision points so future versions face the same ambiguity. Review changes to diagnostic tools and permissions separately from changes to summarization. The value of the assistant is a more inspectable incident decision, not a claim that generated explanations replace service instrumentation or accountable incident command.
Limits of the evidence
This is a proposed isolated incident exercise, not production operational advice for a particular service. Google SRE supplies foundational reliability concepts; thresholds, diagnostics, permissions, and recovery checks must be chosen for the actual system.
Common questions
Is a successful restart proof of recovery?
No. Evaluate the service signal and the task-specific recovery checks.
Should the assistant act during every alert?
Its role and authority should be explicit; some events require monitoring or escalation rather than a change.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- The Site Reliability Workbook: Implementing SLOs Google · 2018 · Engineering book chapterReviewed: foundational service-level indicators, objectives and user-oriented reliability decisions; historical foundation, not a newly published 2026 standard. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.