On this page
The short answer
Measure repeatability by repeating a defined evaluation under controlled conditions and identifying which layer changes the result. Rescore fixed outputs to test evaluator variability, then rerun the application separately. Preserve case-level distributions; similar overall averages can hide individual tasks that repeatedly switch between success and failure.
What to take away
- Rescore fixed outputs separately from new model runs.
- Report mixed-outcome cases, not only averages.
- Evaluate the retry policy users actually receive.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Define what is supposed to repeat
NumPy’s random compatibility policy explains that reproducible random streams depend on more than a seed, including generator, call sequence, build, and environment. An LLM workflow adds provider behavior, retrieval state, tools, and evaluator configuration to the conditions that may vary.
Write which layer is fixed in each trial. A saved output judged twice measures evaluator stability. A new model output judged by a deterministic rule measures application variability. A full workflow rerun with live tools measures a broader operational system. Mixing those experiments makes the source of instability difficult to locate.
Evidence: NumPy random: Compatibility policy [1]
Use a crossed trial design when practical
Select representative cases and preserve the input and evidence snapshot for each. Generate several candidate attempts within an agreed budget, then apply repeated evaluator judgments to selected fixed outputs if the evaluator itself is variable. Keep attempt and judgment identifiers distinct.
Record model settings, actual destination, prompt revision, retrieval snapshot, tool fixture state, and execution errors. Interleave configurations or control timing when external load may matter. Do not claim an exact-repeatability setting simply because a temperature parameter is low; verify the observed behavior of the complete tested path.
Report stability at the task level
For an illustrative task run five times, outcomes pass, pass, fail, pass, fail produce a pass fraction of 3/5. That is evidence about those five attempts on that one task, not five independent customer tasks. Another task that always fails deserves a different diagnosis from one that intermittently succeeds.
Show counts of always-passing, always-failing, and mixed-outcome cases alongside aggregate success. For fixed outputs, report judgment flips and the criteria involved. A stable average across batches can conceal different failures in every batch, so retain task identities when comparing runs.
| Case or record | Expected check | Decision |
|---|---|---|
| One task:3 passes in 5 attempts | Within-case pass fraction 3/5 | Do not call it five independent tasks |
| Same output receives different labels | Candidate text held fixed | Investigate evaluator variability |
| Different retrieved snapshot | Input conditions changed | Separate from a controlled repeat |
Turn variability into a release condition
Decide whether the product requires single-attempt success, bounded retries, or human fallback. Evaluate that exact policy rather than selecting the best observed attempt. For critical failures, retain every occurrence and the path that produced it, even when most repeats passed.
Investigate variability by layer: changing source results, tool timing, candidate sampling, evaluator reasoning, or environment drift. Change one suspected cause and repeat the relevant subset before rerunning the whole suite. Keep the residual instability and trial count in the decision record so a deterministic-looking headline does not overstate the evidence.
Limits of the evidence
The repeated outcomes are illustrative, not observations from a model. Trial counts may be too small for stable estimates, and dependencies between runs matter. A repeatable evaluator can still be systematically wrong, while an unstable result may originate outside the model.
Common questions
Does temperature zero guarantee repeatability?
Do not assume it across the complete application. Record and test provider, retrieval, tool, and evaluator behavior under the actual configuration.
Should I report best-of-five performance?
Only if the product implements and evaluates a defensible selection policy. Otherwise report the attempts and the user-facing success condition.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- NumPy random: Compatibility policy NumPy · Living reference · Maintained documentationReviewed: stream reproducibility conditions involving generator, seed, call sequence, build and environment. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.