On this page
The short answer
Evaluate an AI code reviewer by checking whether its findings identify real, actionable defects in the changed code with enough evidence to reproduce them. Include known-defect and clean-change controls. Count unique confirmed defects, missed defects, false alarms, and review effort rather than rewarding the number or confidence of comments.
What to take away
- Require a trigger and evidence, not a confident warning.
- Pair seeded defects with corrected controls.
- Deduplicate findings by underlying bug.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Define an actionable review finding
GitHub’s SARIF support documents rule identifiers, result locations, and fingerprints for matching alerts. Those are useful evidence fields for structured findings, although an AI code review can use another format and still need the same clarity.
Require a finding to identify the affected path and lines, triggering condition, expected versus observed behavior, and why the change introduces the problem. Separate correctness defects from optional style suggestions. A plausible concern about unchanged code may be useful context, but it should not count as a newly detected regression without a supported connection to the diff.
Build defect and clean-change controls
Create controlled repository snapshots with independently reviewed defects and corresponding fixed or harmless changes. Preserve realistic surrounding code so the reviewer must reason about the actual context. Do not leak a defect description through branch names, filenames, or expected-output metadata.
Include boundary-condition mistakes, authorization changes, wrong assumptions about data shape, and false-positive traps such as an already enforced validation. Keep the reference defect list protected from the reviewer and verify each seeded defect with an executable reproducer or an evidence-backed analysis before using it as ground truth.
Judge findings against observable behavior
For an illustrative pagination change, a loop uses less-than where the final page boundary requires less-than-or-equal. A useful finding names the boundary input and the omitted result. A vague statement that pagination may be broken is weaker evidence and should not receive the same acceptance label.
Run the same review on a corrected control. Repeating the original warning after the condition is fixed is a false alarm unless another supported defect remains. Merge duplicate comments about one underlying bug so verbosity does not inflate recall, and retain speculative findings as unconfirmed rather than calling them reproduced.
| Case or record | Expected check | Decision |
|---|---|---|
| Boundary page omitted | Finding names input and missing result | Confirm the actionable defect |
| Corrected boundary condition | Original warning still emitted | Count unsupported warning as false alarm |
| Three comments about one bug | Same root cause and trigger | Count one unique detected defect |
Report utility for the human review process
Calculate detection against confirmed reference defects and precision among review findings, preserving raw counts and unresolved cases. Inspect severity and task type separately. A system that detects many trivial issues but misses the consequential change may not satisfy the intended workflow.
Measure the work needed to verify findings and whether their locations remain accurate after small edits. Compare configurations on identical diffs without allowing one access to privileged test labels. Preserve missed defects and false alarms as different repair targets; forcing more comments can improve apparent coverage while making the review less useful.
Limits of the evidence
The pagination scenario is invented and no reviewer was benchmarked. Seeded defects can differ from natural changes, and the reference list can miss genuine issues. Structured alert fields improve inspectability without proving a finding is correct.
Common questions
Should style comments count as false positives?
Keep them in a separate category if style review is in scope. Do not mix optional suggestions with correctness-defect precision.
Can the known-defect list be the only ground truth?
Treat novel findings as candidates for independent verification. A reviewer can discover a real defect absent from the original list.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- GitHub: SARIF support for code scanning GitHub · Living reference · Maintained documentationReviewed: documented results, rule IDs, file locations and duplicate-alert fingerprints. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.