On this page
The short answer
Evaluate generated code against the requested behavior with independent tests and a controlled execution environment. Passing examples supplied in the prompt is only a starting point. Include boundary cases, constraints, and task-level completion, and keep functional correctness distinct from execution safety, maintainability, and performance.
What to take away
- Use held-out tests that distinguish plausible bugs.
- Inspect test and configuration changes.
- Evaluate the actual repair-attempt budget.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Define the coding task beyond sample outputs
EvalPlus documents expanded tests for generated-code correctness and an execution workflow using Docker. Those resources motivate stronger testing, but a container example does not establish that any local environment is a complete security boundary.
Write the function or repository contract, supported inputs, output semantics, dependency limits, and allowed file changes. Specify whether the task permits modifying tests. Keep evaluation-only tests separate from the candidate’s visible examples so a solution cannot succeed merely by encoding the examples already shown.
Build discriminating functional checks
Include empty inputs, repeated values, boundary sizes, invalid inputs where supported, and cases that distinguish plausible wrong algorithms. Validate the reference tests independently. A test suite with an incorrect oracle can confidently reject valid alternatives or accept the same bug as the reference.
For repository tasks, inspect the final diff and run checks relevant to the requested behavior. Compilation alone is not functional acceptance. Keep unrelated modifications, skipped tests, and changed configuration visible so the candidate cannot improve the apparent pass rate by removing the checks it fails.
Use a small illustrative correctness fixture
Imagine a task that returns unique values while preserving first-seen order. A solution using a sorted set can pass an already sorted example yet fail input [3,1,3,2], whose required output is [3,1,2]. The fixture tests the actual ordering requirement rather than code style.
Include a case where the input is empty and a case with repeated values at several positions. Execute only in an isolated environment with explicit file, network, process, time, and resource policies. Treat generated code as untrusted until those boundaries are enforced and verified for the evaluation environment.
| Case or record | Expected check | Decision |
|---|---|---|
| Input[3,1,3,2] | Expected[3,1,2] preserving first appearance | Reject sorted-set output |
| Empty input | Contract returns empty result | Check boundary behavior |
| Tests removed to obtain green run | Acceptance evidence changed | Reject the apparent improvement |
Report attempts and constraints honestly
Record first-attempt acceptance, any permitted repair attempts, execution errors, timeouts, and incomplete tasks. If the product permits an agent to inspect test feedback and revise code, evaluate that complete budgeted workflow rather than comparing its best result with a single-shot baseline.
Report which tests were run and which important properties remain untested. Keep code quality and performance criteria separate unless a combined policy is explicit. A finite functional suite provides evidence about tested behavior; it does not prove general correctness or establish that the program is safe to deploy.
Limits of the evidence
The function and test cases are authored examples. No generated code was executed. The cited framework does not certify an arbitrary sandbox or establish performance on your repository, dependencies, and production conditions.
Common questions
Does passing unit tests prove generated code is correct?
No. It establishes behavior on those tests under their oracle and environment. Inspect coverage, constraints, and relevant integration behavior too.
Can I use pass-at-k?
Only with a clearly defined sampling and selection interpretation. It is not the same as the success rate of one attempt or a real repair workflow.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- EvalPlus: Code correctness evaluation EvalPlus maintainers · Living reference · Maintained documentationReviewed: README expanded code tests, correctness evaluation and Docker execution workflow; no model rankings reused. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.