On this page
The short answer
A reproducible benchmark record lets another reviewer reconstruct what was asked, what ran, and how the score was calculated. Pin datasets and task definitions, preserve raw outputs and terminal statuses, and distinguish an exact replay from a new stochastic run. A random seed alone is not a complete evidence record.
What to take away
- Pin data, prompts, evaluators, and runtime settings.
- Keep raw outputs before answer extraction.
- Distinguish rescoring saved outputs from rerunning models.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Preserve the complete experiment identity
The EleutherAI evaluation-harness README documents result saving and sample logging. Those capabilities support inspection, but a saved aggregate alone cannot reconstruct every application-specific condition that produced it.
Create a manifest containing the dataset revision, case list, task template, candidate configuration, evaluator revision, and relevant runtime environment. Include retrieval snapshots and tool fixtures when the benchmark uses them. Record values that were resolved dynamically, such as actual model destination or fallback route, rather than only the alias the request originally named.
Retain raw artifacts before aggregating
Store the exact candidate-visible input and output for each attempt under the permitted data policy. Preserve errors, cancellations, timeouts, and parse failures. Join them to stable case and attempt identifiers so a second reviewer can independently recompute the reported denominator.
Keep normalization and postprocessing code with its version. A seemingly minor answer-extraction change can alter a score without changing model behavior. Save the unprocessed output alongside the extracted answer, and record which stage rejected a record instead of replacing every failure with an empty string.
Check replayability with a small audit
For an illustrative audit, select a correct output, a scored failure, and an incomplete attempt. Recompute their judgments from the saved artifacts without calling the candidate again. The expected result is the same scoring decision under the same evaluator implementation, or a documented source of evaluator nondeterminism.
Then perform a separate rerun if needed. A fresh output can differ even when the request configuration matches, so label it as a new attempt rather than evidence that the original record was wrong. Keep caching and resume behavior explicit so previously computed samples are not mistaken for newly executed work.
| Case or record | Expected check | Decision |
|---|---|---|
| Saved pass | Recompute from the recorded output | Same rule yields same decision |
| Parse failure | Retain original output and extraction error | Do not silently drop the case |
| Resumed benchmark | Mark cached versus newly executed samples | Preserve execution provenance |
Publish limits a reviewer can act on
Record what cannot be reproduced: an unavailable provider snapshot, a deleted source document, nondeterministic evaluator behavior, or a dependency that was not pinned. Explain how each limit affects the conclusion. A transparent partial record is more useful than an unsupported promise of exact repeatability.
Before sharing, verify that the package contains enough permitted evidence to trace one headline metric down to its cases. Redact or reference sensitive content without silently changing the task. Approve the report only when the manifest and artifacts support the stated comparison and any remaining reconstruction gap is visible.
Limits of the evidence
The reproducibility package is a proposed workflow, not a verified reproduction of a published benchmark. Hosted models, changing dependencies, and inaccessible source snapshots can prevent exact reruns. Saving configuration does not establish task validity or label correctness.
Common questions
Does a seed guarantee identical API outputs?
Do not assume it. Record provider behavior and execution conditions, and treat fresh responses as new attempts unless exact replay is demonstrated.
Must every artifact be public?
No. Preserve access-controlled evidence or redacted references as appropriate. State which parts a recipient can inspect and which limits remain.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- EleutherAI Language Model Evaluation Harness EleutherAI · Living reference · Maintained documentationReviewed: repository README result saving, sample logging, configuration and cache/resumption notes. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.