Version benchmark questions, references, and scoring rules
Track case, reference, rubric, and scorer revisions separately, then decide which historical scores remain comparable after a benchmark update.
Research briefing · Sources includedResearch & field notes
Research library — page 2
Read a guide, follow its sources, and check the limits before applying a finding. Search the full library from the research desk, or browse by topic.
Account for missing and invalid evaluation results
Reconcile scheduled cases with attempts, scores, skips, errors, and unknowns so missing rows cannot silently inflate quality or completion rates.
Research briefing · Sources includedMeasure evaluation repeatability across repeated runs
Separate candidate variability, evaluator variability, and infrastructure changes with repeated trials, fixed artifacts, and case-level stability reports.
Research briefing · Sources includedWrite annotation guidelines reviewers can apply consistently
Define task labels with evidence requirements, boundary examples, skip reasons, and a pilot process that exposes ambiguous rules before large-scale review.
Research briefing · Sources includedTriage evaluator disagreements into fixable causes
Classify disagreements by missing evidence, rubric ambiguity, reference defects, and judgment errors, then assign a specific corrective action.
Research briefing · Sources includedMaintain a golden test set as a product changes
Review trusted evaluation cases for stale evidence, changed policies, exposed labels, and missing workflows while preserving historical comparability.
Research briefing · Sources includedUse negative controls to test an evaluation pipeline
Run deliberately uninformative outputs and known good controls through scoring to detect leaked labels, broken joins, and evaluators that accept everything.
Research briefing · Sources includedTest LLM behavior with meaning-preserving transformations
Create reviewed transformations of a base task, specify which behavior should stay invariant, and separate robustness failures from changed meaning.
Research briefing · Sources includedUse deliberate answer mutations to test evaluators
Alter one material fact in a correct answer, preserve the original as a control, and measure whether evaluators detect the specific defect.
Research briefing · Sources includedAvoid false wins when comparing many prompts
Separate exploratory prompt search from final confirmation, define comparison families, and report selection effects instead of only the winning score.
Research briefing · Sources includedDecide whether an LLM score improvement matters
Translate score changes into task outcomes, predefine a practical improvement target, and inspect regressions before calling a small difference useful.
Research briefing · Sources includedReport LLM results by meaningful user and task slices
Break evaluation outcomes into prespecified workflow, language, role, and difficulty slices while retaining counts, uncertainty, and the overall population.
Research briefing · Sources includedTrack who labeled an evaluation case and why
Preserve label authorship, evidence references, guideline versions, adjudication, and allowed reuse so a disputed benchmark label can be reconstructed.
Research briefing · Sources includedEvaluate summaries for factual support and useful coverage
Review summaries against source-backed claims, required decisions, exceptions, and audience needs rather than relying on word overlap or fluent prose.
Research briefing · Sources includedEvaluate text-to-SQL with executable acceptance checks
Compare generated SQL by permitted execution results across controlled database fixtures, including nulls, duplicate rows, joins, and empty results.
Research briefing · Sources includedEvaluate LLM classification beyond overall accuracy
Evaluate class-specific precision, recall, abstention, and routing consequences with an explicit taxonomy and an imbalanced test-set control.
Research briefing · Sources includedEvaluate entity extraction at field and record level
Define span boundaries, entity types, normalized values, and record-level acceptance so partial matches cannot hide unusable extraction outputs.
Research briefing · Sources includedEvaluate AI translation with task-specific error categories
Review translations for meaning, omissions, terminology, locale, and severity with source context and bilingual adjudication instead of fluency alone.
Research briefing · Sources includedBuild multilingual LLM evaluations without treating English as universal
Combine aligned questions with locally authored tasks, review language and script coverage, and report outcomes without treating translated English as universal.
Research briefing · Sources includedTest AI document extraction across messy source layouts
Test extraction on scanned, rotated, multi-column, revised, and incomplete documents while preserving page evidence and separating parsing from field errors.
Research briefing · Sources included