Version a RAG corpus for reproducible evaluation

Define a manifest beyond filenames

The BEIR project represents retrieval using a corpus, queries, and relevance judgments. A changing product corpus needs additional lineage. Record stable source ID, revision, content hash, parser and splitter settings, eligibility metadata, and index revision. A filename is not a reliable version because its contents can change in place. Keep deleted or inaccessible content under the organization’s retention rules; a manifest can preserve identifiers and hashes without retaining unrestricted copies of every historical source.

Evidence: BEIR official benchmark repository [1]

Author a reproducibility mismatch

Imagine two evaluation runs using the same fictional refund questions and prompt. Between them, the policy changes from seven to fourteen days. A correct answer changes even if the model and retriever do not. The fixture should link each reference expectation to the corpus revision that established it. Replaying the old questions against the new corpus may be useful, but it is a new evaluation condition and should not be presented as a like-for-like quality improvement.

Track derived artifacts explicitly

Source text can remain unchanged while parsing, chunking, embedding, or graph extraction changes. Include those transformation versions in the manifest and map derived IDs back to source revisions. Store the final retrieved context for failures so a reviewer can inspect what the model actually received. If a hosted service does not expose every internal detail, record that limitation and preserve the observable configuration. Reproducibility should describe what can be rerun, not imply control over hidden provider internals.

Separate comparison questions

Use a fixed corpus snapshot to study a retrieval or model change. Use a fixed application configuration with a changed corpus to study knowledge updates. When both change, run an additional comparison or state that attribution is unresolved. Maintain a manifest difference report listing added, removed, modified, and eligibility-changed records. Inspect whether expected evidence labels need revision. A case that becomes unanswerable after a legitimate deletion should not remain an unquestioned ground-truth answer.

Make the record useful to another engineer

Package the manifest ID with query set, relevance labels, run configuration, retrieved IDs, and answer judgments. Verify that a colleague can locate one failed source revision and reproduce the intended retrieval condition. If they cannot, identify the missing artifact before relying on the evaluation for a release decision. Keep a compact synthetic corpus for routine regression checks and separately document the limits of restoring large or sensitive historical corpora. That balances traceability with operational and retention constraints. Keep the manifest generation procedure under version control too. A manifest that omits permission changes or normalizes away meaningful source differences can create the appearance of reproducibility while hiding the actual changed condition.

Common questions