Define AI service objectives around completed user tasks

Start with the user outcome

Google SRE’s SLO guidance uses explicit service indicators and objectives. Rubrex recommends adapting that discipline to a bounded AI task rather than adopting a universal model-quality percentage. Name the population, successful outcome, observation window, and the evidence required to label success.

For a synthetic policy assistant, a usable completion is an answer supported by the authorized current policy or a precise evidence-gap response when the corpus cannot answer. A fast hallucinated answer does not satisfy that definition. Distinguish it from the separate indicator that the API was reachable.

Evidence: The Site Reliability Workbook: Implementing SLOs [1]

Build the denominator from admitted tasks

Include attempted tasks that time out, cancel, fail validation, or remain unlabeled, according to an explicit reporting policy. Do not quietly remove difficult tasks from the denominator because they lack a clean final answer. Keep unknown quality states visible when evidence or review is missing.

Define the latency event carefully: first token, complete answer, or usable task result. The appropriate event depends on the product. A streaming assistant can look fast on first-token latency while taking too long to produce a usable answer, and an agent can finish its text while a consequential tool remains unresolved.

Keep quality and boundaries inspectable

Use the synthetic task ledger to compute transport availability, usable completion, completion latency, and boundary violations separately. Add a case with a correct answer that reads an unauthorized document and another with a polite unsupported answer. The first should not pass merely because the answer is correct.

Choose which failures require an immediate stop or investigation independently of the overall objective. A weighted average can conceal a severe permission failure among many routine successes. Include confidence and sample coverage when reporting reviewed quality; an objective is not stronger than the labeling process supporting it.

Use objectives to choose an operational response

Set targets from the product’s needs and observed baseline, with an owner and a response when the indicator moves. For the policy assistant, an index outage may call for a clear evidence-gap mode, while an authorization failure may require disabling the affected path. Those responses should be concrete before rollout.

Retain definitions, label provenance, time windows, and exclusions with the dashboard. Review them when the task or evaluator changes. The objective should help the team decide what to investigate or stop; it should not become a marketing accuracy number stripped of its task population and evidence limits.

Common questions