How many examples does an LLM evaluation need?
Choose an evaluation sample around coverage, uncertainty, and the decision at stake instead of relying on a universal minimum number of test cases.
Research briefing · Sources includedResearch & field notes
Research library — page 9
Read a guide, follow its sources, and check the limits before applying a finding. Search the full library from the research desk, or browse by topic.
Build an LLM evaluation dataset that reflects real work
Design evaluation datasets with task coverage, provenance, held-out examples, and explicit expected behavior. Separate synthetic coverage from real traffic.
Research briefing · Sources includedBenchmark contamination: why a private test set is not enough
Understand training leakage, repeated tuning, and evaluation-time exposure. Build a practical disclosure record for trustworthy LLM comparisons.
Research briefing · Sources includedBuild an AI failure taxonomy that leads to fixes
Classify AI failures by observable symptoms and causal hypotheses so evaluation findings translate into focused experiments and clear ownership.
Research briefing · Sources includedRAG evaluation metrics: measure retrieval and answers separately
Choose RAG metrics that distinguish missing evidence, irrelevant context, unsupported claims, and incomplete answers instead of hiding them in one score.
Research briefing · Sources includedHow to test retrieval recall without rewarding irrelevant context
Build evidence labels, multi-document cases, and controlled retrieval comparisons that reveal what your RAG retriever actually misses.
Research briefing · Sources includedRAG citations: a correct answer can still cite the wrong evidence
Evaluate citation support, coverage, and faithfulness separately. Learn what recent attribution research means for practical RAG quality checks.
Research briefing · Sources includedTest RAG systems with conflicting and outdated documents
Create RAG evaluations for version conflicts, missing authority, and ambiguous dates so assistants do not turn contradictory evidence into confident answers.
Research briefing · Sources includedWhen should a RAG assistant say it does not know?
Evaluate answerability, abstention, and clarification together so a RAG assistant avoids unsupported answers without refusing useful work.
Research briefing · Sources includedAI agent evaluation: test outcomes, traces, and repeated runs
Design agent evaluations around state changes, tool behavior, repeatability, and recovery rather than grading only the final chat response.
Research briefing · Sources includedTool-calling evaluation: arguments, authorization, and side effects
Test tool selection, argument validity, authorization boundaries, and state reconciliation to catch failures that a valid function call can hide.
Research briefing · Sources includedEvaluate multi-turn assistants across the whole conversation
Test corrections, changing requirements, retained constraints, and useful clarification so conversational quality is measured beyond isolated responses.
Research briefing · Sources includedPrompt-injection evaluation for retrieval and tool-using agents
Design bounded prompt-injection tests that separate untrusted content from authority and measure both inappropriate actions and excessive blocking.
Research briefing · Sources includedMeasure AI agent cost per successful task, not per response
Compare agent quality, latency, retries, and total resource use using a consistent task budget and an explicit definition of success.
Research briefing · Sources includedHow to calibrate an LLM judge before trusting its scores
Validate an LLM judge against task-specific human labels, inspect directional errors, and recheck calibration when the candidate distribution changes.
Research briefing · Sources includedAn LLM evaluation rubric template with scoring anchors
Create rubrics with observable criteria, clear pass boundaries, evidence requirements, and examples that help reviewers reach defensible judgments.
Research briefing · Sources includedHuman evaluation agreement: diagnose disagreement before averaging
Measure reviewer agreement with the label distribution and task ambiguity in view. Use adjudication to improve the rubric instead of hiding disagreement.
Research briefing · Sources includedPairwise LLM evaluation: compare answers without hiding ties
Design blinded paired comparisons with explicit criteria, randomized order, tie handling, and task-level analysis for model and prompt decisions.
Research briefing · Sources includedWhere should limited human evaluation time go?
Allocate human review across representative traffic, uncertain judgments, and consequential failure modes without confusing targeted sampling with prevalence.
Research briefing · Sources includedLLM regression testing: turn failures into release gates
Build a layered regression suite for prompts, models, retrieval, and tools. Separate deterministic failures from noisy judgments and preserve reproducible runs.
Research briefing · Sources included