On this page
The short answer
Evaluate a file-editing agent by comparing the requested transformation with the complete before-and-after state. Check changed paths, preserved content, format validity, and unintended deletions. A fluent explanation of an edit is insufficient when the saved file differs from the request or unrelated user work has been overwritten.
What to take away
- Keep a baseline of every permitted target.
- Inspect changed paths and content separately.
- Verify the saved artifact opens in its intended format.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Define the allowed transformation
Git documents content comparisons through diff operations. Rubrex recommends combining such comparisons with a task-specific acceptance contract, because a diff shows what changed without establishing whether the change was wanted. Capture the original file, allowed paths, requested edits, and protected content.
Use a synthetic project note where the user asks to correct a date in one paragraph. Another paragraph contains a deliberate unusual spelling and a neighboring file holds unrelated draft work. The agent should make the date correction while preserving both the unusual spelling and the other file.
Evidence: Git: git-diff documentation [1]
Exercise context, format, and path traps
Repeat the date in a quoted historical example so the agent must select the intended occurrence. Add a filename with spaces and a second file with a similar basename. Verify the exact target before and after the edit rather than rewarding any file that contains the new date.
Include a structured document with a simple format constraint, such as valid JSON and preserved key order where that matters to the workflow. For a binary document, use the appropriate application inspection instead of pretending a text diff captures layout. The evaluator should match evidence to the artifact type.
Check preservation as a first-class outcome
Record changed paths, additions, deletions, and the smallest relevant content comparison. Open or parse the saved artifact using its intended reader. An edit can be textually correct yet break encoding, formulas, links, or structure that the user needed to preserve.
Run a conflict case in which the source file changes after the agent reads it. The workflow should detect the stale baseline or merge deliberately rather than overwrite the later user edit. Include a clean unchanged-baseline control so a system that refuses every edit does not satisfy the test merely by avoiding damage.
| Case | Evidence to retain | Acceptance condition |
|---|---|---|
| Repeated date | Focused diff | Only intended occurrence changes |
| Concurrent user edit | Baseline and final version | Later work preserved or conflict raised |
| Structured file | Parser result and artifact | Valid saved format |
Accept only reviewable edits
For the date correction, require the intended occurrence to change, protected content to remain, and the saved file to pass its format check. If the agent needs a broader edit, it should explain the reason through the task flow before replacing unrelated material. Keep the original artifact recoverable during review.
Retain the request, baseline identifier, final artifact, and validation result. Add regression cases after a failure involving paths, encodings, or document structure. The useful unit is a correctly preserved artifact, not the number of editing commands issued or the confidence of the agent’s final message.
Limits of the evidence
The files and edits described here are proposed fixtures. No document was modified for this article. Diff tools reveal changes but do not establish semantic correctness, recoverability, or preservation for every artifact format.
Common questions
Does a small diff prove correctness?
No. It must still match the requested transformation and preserve required structure.
How should binary documents be evaluated?
Inspect them in a suitable reader or renderer and compare relevant content and layout evidence.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- Git: git-diff documentation Git project · Living reference · Maintained documentationReviewed: compare tracked content changes and expose before/after evidence; not verification of file-edit correctness. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.