On this page
The short answer
A useful LLM test-case schema separates what the application receives, what the evaluator expects, and what a run actually produced. Preserve stable identifiers and evidence versions so a failed case can be reproduced. Validate the record before execution rather than discovering missing context after a score has been published.
What to take away
- Keep cases, attempts, and judgments separate.
- Validate cross-field evidence requirements.
- Version migrations without rewriting history.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Separate case definitions from run records
Hugging Face Datasets features describe column names, types, labels, and nested structures. A typed representation can prevent malformed records, but the application still needs semantic rules about what each field means and who may see it.
Rubrex recommends three linked records: the authored case, the immutable run attempt, and the evaluation result. A case can have several attempts and several evaluator revisions. Overwriting the answer field after a rerun destroys the history needed to explain which output was scored and why two reports differ.
Define the minimum authored case
Give each case a stable ID, task family, input, permitted context references, required behavior, prohibited behavior, source snapshot, and provenance. Keep private scoring notes outside the candidate-visible input. Represent unknown or intentionally absent evidence explicitly; an empty string should not ambiguously mean missing, redacted, or irrelevant.
Specify field types and cross-field constraints. A case requiring a supporting passage needs a resolvable evidence reference. A conversation case needs ordered turns and the expected state at the point being tested. A permission case needs the caller role and allowed resource scope, rather than an explanation buried in an informal title.
Attach results without erasing the case
For each attempt, retain the case version, system version, timing, terminal status, actual output reference, and visible tool trace where relevant. Store evaluator version, score, reason, and review status separately. This makes a rescoring operation distinguishable from rerunning the application.
In an illustrative record, case C7 expects a denied write. Attempt A1 reports a tool rejection and no state change; evaluator E2 marks a pass. If E3 later interprets the attempted call as a separate warning, both judgments can coexist without changing what A1 actually did.
| Case or record | Expected check | Decision |
|---|---|---|
| Missing source version | Evidence-based task needs a resolvable snapshot | Reject before execution |
| Second application attempt | New attempt ID; same case reference | Append rather than overwrite |
| New evaluator rubric | Preserve old judgment and raw output | Rescore with a new result record |
Validate meaning as well as shape
Before admission, check unique IDs, resolvable source references, compatible task types, and the separation of labels from visible inputs. After execution, reject result rows whose case or attempt cannot be found. Preserve invalid rows in an error ledger rather than silently removing them from the denominator.
Review the schema with the people who investigate failures. If they cannot answer which document version, permission context, and output produced a score, add that evidence deliberately. Keep a schema version and a migration record so older cases remain interpretable when new fields are introduced.
Limits of the evidence
The schema is a proposed evidence model rather than a prescribed industry format. It needs adaptation for privacy, storage, and runtime requirements. Typed fields cannot establish that references are correct or that a test represents real product work.
Common questions
Should expected answers live beside inputs?
They can share a controlled authoring store, but serialize only permitted input fields to the candidate. Make the boundary explicit and test it.
Is JSON Schema enough?
It can validate shape and some constraints. Evidence availability, permission scope, and task meaning still need application-specific validation.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- Hugging Face Datasets: Dataset features Hugging Face · Living reference · Maintained documentationReviewed: column types, ClassLabel, nested features and serialization. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.