On this page
The short answer
Label leakage occurs when the system under test receives information that should exist only in the scoring process. Audit the actual serialized model inputs, retrieval contents, and tool responses. A clean user prompt is insufficient if a reference answer survives in metadata, an example, a filename, or an evaluator feedback loop.
What to take away
- Inspect complete serialized requests.
- Separate scorer references from candidate context.
- Invalidate results affected by demonstrated leakage.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Draw a boundary between evidence and labels
Scikit-learn describes leakage as using information unavailable at prediction time. For an LLM evaluation, apply that distinction to the complete application input, not only a training table. A legitimate policy passage can be available evidence while an annotator’s expected decision remains a private label.
List which fields the model, retriever, evaluator, and human reviewer may see. Keep expected answers and acceptance notes in a separate scoring object. If a judge requires the reference, that does not authorize sending it to the candidate model or indexing it alongside customer documents.
Evidence: scikit-learn: Common pitfalls and recommended practices [1]
Inspect what is actually sent
Capture the fully assembled request after templates, examples, history, metadata transformations, and tool context are added. Compare it with the intended input contract. Review filenames and record identifiers as well as obvious text fields: names such as approved-answer-yes can reveal labels even when a reference column is absent.
For retrieval systems, audit which evaluation directories or exported review notes entered the index. For agent workflows, inspect prior tool responses and cached outputs. Sanitizing the first prompt will not help if a later tool returns a scored answer from an earlier evaluation run.
Use a synthetic label marker as a control
Place an invented marker only inside the private scoring reference for a controlled fixture. Make the user input and permitted source evidence incapable of establishing that marker. Inspect the candidate’s serialized input and retrieved context for it before asking whether an output reproduces it.
Then remove the private reference from the entire candidate path and rerun the fixture. A changed score is a diagnostic clue, not proof by itself: output variation can have other causes. The decisive evidence is a demonstrated path carrying a protected label into material available to the candidate.
| Case or record | Expected check | Decision |
|---|---|---|
| Private reference marker | Absent from all candidate-visible material | Block if present |
| Permitted policy passage | Still available after the fix | Retain as legitimate evidence |
| Review export in index | Contains expected-answer notes | Remove and rebuild the affected snapshot |
Repair the pipeline and preserve the audit
Fix the data contract where the label crossed the boundary. Separate stores or explicit field allowlists can make the distinction reviewable. Test the original leak path and nearby paths such as few-shot examples, exports, and cached tool results. Keep a legitimate-evidence control to ensure the repair does not remove the passages the task genuinely requires.
Invalidate affected comparisons when leaked labels could influence their outcomes. Record which runs are compromised, which were rerun, and whether earlier tuning decisions relied on them. Merely deleting the marker from the latest prompt does not restore the evidentiary value of an already contaminated release comparison.
Limits of the evidence
These are proposed leakage-audit controls using synthetic material. No pipeline was audited for this article. Failure to find a marker does not establish that every label path is isolated, and previously exposed test cases may remain unsuitable as held-out evidence.
Common questions
Is a reference answer always forbidden context?
It depends on the task. If the product is explicitly given a reference, evaluate that contract. Otherwise keep scoring-only labels out of candidate inputs.
Does repeating the marker prove a leak?
It is a clue. Inspect the actual data path and use controls; chance, prior exposure, or a flawed fixture can also affect an output.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- scikit-learn: Common pitfalls and recommended practices scikit-learn · Living reference · Maintained documentationReviewed: prediction-time information boundary and preprocessing leakage. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.