On this page
The short answer
Test an evaluator by taking an acceptable answer and introducing a single deliberate defect with a known expected judgment. Preserve the original and the mutation description. An evaluator should detect the material change while continuing to accept the valid control; otherwise a high score may reflect weak discrimination or blanket rejection.
What to take away
- Mutate one material property at a time.
- Pair defects with valid originals and harmless edits.
- Measure detection separately from model failure frequency.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Choose a defect the rubric claims to detect
Promptfoo documents deterministic, model-based, and custom assertions. Those mechanisms can implement an evaluator, but their presence does not establish that the assertion distinguishes a correct answer from a minimally altered incorrect one.
Start from a reference-reviewed acceptable output. Select one task-relevant mutation: change a date, reverse a relationship, omit an exception, attach an irrelevant citation, or alter a required unit. Do not combine several defects initially. A single change makes it easier to identify which criterion the evaluator missed.
Evidence: Promptfoo: Assertions and Metrics [1]
Verify mutation validity independently
Have a reviewer confirm that the original passes and the altered answer violates the intended requirement. Preserve the relevant source passage and the exact edit. A mutation that merely changes style can be useful for a style criterion but should not be called a factual error.
Include harmless edits as controls, such as permitted wording changes or reordered independent facts. This prevents a detector that flags any textual difference from appearing competent. Keep the mutation’s label private from the evaluator unless the intended task explicitly includes it.
Score paired controls without rerunning the candidate
For an illustrative policy answer, the accepted original says cancellation is allowed within fourteen days. A mutation changes fourteen to forty while keeping the citation unchanged. The evaluator should reject the unsupported deadline. A paraphrase retaining fourteen days should remain acceptable.
Run the fixed outputs through the same scoring path used for candidate responses. Report detected material mutations divided by valid material mutations, and separately report false rejections of harmless edits and original answers. This measures the evaluator on the authored control set, not the model’s rate of producing those defects.
| Case or record | Expected check | Decision |
|---|---|---|
| Fourteen changed to forty days | Contradicts the same policy passage | Reject the mutation |
| Equivalent wording keeps fourteen | Meaning remains valid | Accept the harmless edit |
| Citation points to unrelated passage | Claim lacks cited support | Flag attribution defect |
Use misses to improve the specific criterion
When a mutation survives, inspect evidence availability, rubric wording, answer parsing, and evaluator reasoning. A numeric date check may be preferable to an unconstrained overall-quality judgment where the contract is exact. More complicated model judging is not automatically the right repair.
Retest the revised rule on held-out mutations and legitimate answers. Preserve mutation families so many versions of one date edit do not dominate the report. Keep undetectable or disputed mutations visible, and avoid treating a perfect authored-control score as proof that all naturally occurring failures will be caught.
Limits of the evidence
All mutations described here are authored examples. No evaluator was tested, and mutation detection on a designed suite may differ from detection of natural errors. A valid original and a valid defect label are necessary for an interpretable result.
Common questions
Is this a benchmark of the application model?
No. Fixed answer mutations primarily test the evaluator’s ability to recognize defined defects. They do not measure how often a candidate generates them.
Should every mutation be automatically generated?
Automation can propose edits, but independently verify their labels and preserve exact differences before using them as controls.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- Promptfoo: Assertions and Metrics Promptfoo · Living reference · Maintained documentationReviewed: deterministic and model-based assertions, custom JavaScript/Python scoring, result meanings. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.