On this page
The short answer
Annotation guidelines should let a reviewer make an evidence-based decision on a new case without guessing the author’s intent. Define the unit being labeled, acceptance boundaries, examples, and uncertainty handling. Pilot the instructions independently and revise the rules that produce disagreement before expanding the labeling queue.
What to take away
- Define the unit and required evidence.
- Use boundary examples with reasons.
- Version guideline changes and affected labels.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Define the annotation unit and evidence
Label Studio’s labeling guide directs annotators to project instructions and explains task submission and skipping. The software can present the workflow, but the project must supply the substantive rules that make its labels meaningful.
State whether the reviewer labels a whole task, an answer claim, an extracted field, or a tool action. List the evidence they may use and what to do when it is missing. A reviewer checking citation support should not be expected to infer hidden source text or use general knowledge to rescue a claim unsupported by the supplied passage.
Evidence: Label Studio: Labeling guide [1]
Write boundaries before adding examples
For each label, describe the observable condition, exclusions, and severity rule. Distinguish a wrong fact from missing required information, an acceptable alternative wording, and a style preference. Use plain language that fits the actual tasks rather than a broad instruction to judge quality.
Include borderline examples with short explanations. An answer that gives the main policy but omits a required exception can be incomplete even when every included sentence is true. A concise answer can be fully correct if the task needs only one fact. Those contrasts help reviewers apply the same boundary to unseen outputs.
Pilot independent labeling and uncertainty
Choose a small diverse pilot set and have reviewers label it independently before discussing disagreements. Ask them to record the evidence span and the rule they applied. Preserve uncertain and skipped cases with reasons so the pilot does not appear easier after difficult examples disappear.
In an illustrative extraction pilot, one reviewer marks an absent invoice field as null while another marks it missing. The disagreement may reveal an unclear task rule rather than weak reviewers. Decide which representation the product requires, add the boundary example, and relabel affected pilot cases under the revised instruction version.
| Case or record | Expected check | Decision |
|---|---|---|
| Truthful but missing required exception | Completeness requirement applies | Label incomplete |
| Equivalent permitted wording | Meaning and required coverage preserved | Accept |
| Source passage unavailable | Cannot assess support | Skip or mark unknown with reason |
Release the guideline as a versioned artifact
Publish the label definitions, evidence requirements, examples, revision history, and escalation owner with the annotation task. Check that the interface displays the instructions and necessary source material at the moment of review. Training a reviewer once is insufficient if the queue later changes context.
Monitor disagreement categories and examples that repeatedly require clarification. Update the guideline through a documented review, and identify which earlier labels need reconsideration. Avoid silently changing a label’s meaning while keeping the same benchmark score series; the resulting numbers would appear comparable even though the acceptance boundary moved.
Limits of the evidence
This is a proposed annotation-design workflow, not evidence of reviewer agreement on a real dataset. Guidelines cannot eliminate ambiguous source material or establish domain expertise. The labeling interface and access controls must support the evidence and uncertainty rules the project defines.
Common questions
Should reviewers discuss cases before labeling?
Use independent pilot labels first to expose ambiguity, then discuss evidence and rules. A coordinated first pass can hide the disagreement you need to diagnose.
How many examples should guidelines contain?
Use enough to explain consequential boundaries and common ambiguities. Add targeted examples when pilot evidence shows a rule is unclear.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- Label Studio: Labeling guide HumanSignal · Living reference · Maintained documentationReviewed: project instructions, task skipping, submitting annotations and collaboration workflow. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.