On this page
The short answer
Compare latency using the same workload, start and end definitions, and treatment of incomplete tasks. Separate time to first output from time to a usable final result. Report percentiles and threshold completion rates with counts; excluding timeouts can make an unreliable workflow look fast.
What to take away
- Distinguish first output from useful completion.
- Preserve timeouts outside successful-latency distributions.
- State workload, quantile method, and sample size.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Define the user-visible timing boundary
Google’s SRE workbook describes latency indicators as proportions of requests faster than defined thresholds. For an AI workflow, define whether the measured event is a provider response, first token, completed tool action, or accepted user task. Those times answer different questions.
Use a monotonic clock for durations within one process and preserve correlation across services when needed. Record queue wait, retrieval, model attempts, tools, and postprocessing without assuming their durations add when steps run concurrently. Keep the overall end-to-end duration as the primary task observation.
Hold workload and concurrency explicit
Compare baseline and candidate under the same task mix, input lengths, concurrency, cache policy, and provider conditions where possible. Interleave runs or use repeated windows when external load changes. A warm cached response should not be compared with an uncached full workflow without labeling the difference.
Record failures, cancellation, and deadline expiry. A task that stops at a timeout has no observed successful-completion latency. Preserve its status rather than inserting a convenient zero or dropping it from every metric. Report both successful-task latency and overall completion within the product’s deadline.
Use a small transparent percentile example
For invented completed durations of one, two, two, three, and twelve seconds, the mean is four seconds. Under a declared nearest-rank method, the median is two seconds and the 95th percentile is twelve seconds. These are illustrative calculations; other quantile conventions can differ for tiny samples.
Now add a sixth task that times out before producing a usable result. Do not quietly calculate every performance headline from only the five successes. Preserve the timeout and show the proportion of all scheduled tasks that completed within the stated threshold.
| Case or record | Expected check | Decision |
|---|---|---|
| Durations 1,2,2,3,12 seconds | Mean 4; median 2; nearest-rank p95=12 | Illustrative five-task calculation |
| Sixth task times out | No successful completion observed | Report incomplete outcome |
| Faster first token; slower final action | Different timing boundaries | Compare the user-relevant endpoint |
Connect tails to task outcomes
Inspect slow cases and their traces rather than optimizing a percentile without understanding the cause. Retries, long outputs, tool waits, and overloaded queues can require different changes. Check whether a speed improvement degrades answer quality, evidence coverage, or task completion.
Publish the sample window, counts, quantile method, workload slices, and incomplete outcomes alongside the timing results. Tail estimates need enough observations to be meaningful, and a small local fixture is not a production service objective. Choose rollout actions based on the complete task contract and measured operating conditions.
Limits of the evidence
The durations are invented and no latency benchmark was run. Percentiles depend on method and sample size, while provider load and task mix can change between runs. The guide does not prescribe a universal AI latency objective.
Common questions
Should I report p99 from a tiny test?
You can show the calculation, but explain its instability and sample size. A handful of tasks cannot characterize a production tail reliably.
Can timeouts be included as the timeout duration?
Use an explicit censored or deadline-based view if appropriate. Do not label that duration as an observed successful completion.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- Google SRE Workbook: Implementing SLOs Google · Living reference · Online book referenceReviewed: latency threshold proportions, request populations and histogram-based measurement limitations. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.