Research & field notes
Find a guide for your next decision.
Browse all 200 articles by topic. Choose the problem you need to solve, then read a guide and follow its sources. For a quick search or the full set of article summaries, visit the research library.
Evaluation design
Start with the decision your test needs to support. These guides cover test data, scoring, and fair comparisons. Use them to check what a score means, find gaps in your test set, and keep a record of changes. A higher score is useful when it helps your team make a sound release decision.
- Choosing an LLM evaluation framework: four options and a trial plan
- Deduplicate an LLM evaluation dataset without losing rare failures
- Split LLM evaluation data by customer, time, and workflow
- Find label leakage in an LLM evaluation pipeline
- Design an LLM test-case schema that preserves evidence
- Build an evaluation coverage matrix for an AI product
- Use confidence intervals in an LLM release decision
- Choose metric weights without hiding critical failures
- Evaluate rare AI failures without overstating confidence
- Make an LLM benchmark run reproducible
- Version benchmark questions, references, and scoring rules
- Account for missing and invalid evaluation results
- Measure evaluation repeatability across repeated runs
- Maintain a golden test set as a product changes
- Use negative controls to test an evaluation pipeline
- Test LLM behavior with meaning-preserving transformations
- Avoid false wins when comparing many prompts
- Decide whether an LLM score improvement matters
- Report LLM results by meaningful user and task slices
- Evaluate summaries for factual support and useful coverage
- Evaluate text-to-SQL with executable acceptance checks
- Evaluate LLM classification beyond overall accuracy
- Evaluate entity extraction at field and record level
- Evaluate AI translation with task-specific error categories
- Build multilingual LLM evaluations without treating English as universal
- Test AI document extraction across messy source layouts
- Evaluate table extraction without losing row relationships
- Evaluate numerical reasoning with units and tolerances
- Evaluate generated code with tests and execution boundaries
- Evaluate AI code review for useful findings and false alarms
- Evaluate generated search queries by the evidence they retrieve
- Evaluate answer completeness against a required-facts checklist
- Measure answer relevance without rewarding unsupported detail
- Evaluate refusals for safety, precision, and useful alternatives
- Evaluate AI-generated content for accessibility tasks
- Test long-context admission, truncation, and separated evidence
- Plan evaluation power around a practical decision
- Use paired bootstrap comparisons for LLM experiments
- Use Elo-style LLM rankings with explicit limits
- Compare LLM latency with percentiles and incomplete runs
- Prepare an evaluation review packet a release owner can inspect
- Choose what evaluation evidence to retain and for how long
- How to evaluate an LLM application before release
- How many examples does an LLM evaluation need?
- Build an LLM evaluation dataset that reflects real work
- Benchmark contamination: why a private test set is not enough
- Build an AI failure taxonomy that leads to fixes
RAG systems
A system that answers from documents can fail at several steps. It may miss the right source, retrieve old text, or make a claim the source does not support. These guides help you test each step. Start with the failure you see, then check the evidence used to produce the answer.
- GraphRAG evaluation: test relationships, summaries, and cost
- Agentic RAG evaluation: test search decisions and stopping rules
- Choose RAG chunk sizes with an evidence-focused trial
- Test whether chunk overlap helps retrieval or adds noise
- Evaluate document parsing before tuning a RAG retriever
- Test RAG metadata filters with explicit eligible documents
- Compare hybrid search with lexical and vector baselines
- Evaluate a RAG reranker without hiding candidate-retrieval failures
- Evaluate an embedding-model migration before reindexing
- Test query rewriting for preserved intent and useful recall
- Evaluate multi-query retrieval against its extra cost
- Test parent-child retrieval for context and precise evidence
- Evaluate context compression for lost evidence
- Test whether context order changes a RAG answer
- Use mean reciprocal rank for RAG with explicit relevance labels
- Use nDCG to evaluate graded retrieval relevance
- Choose and interpret precision at k for retrieval
- Measure duplicate passages in a RAG context window
- Measure citation precision and citation recall separately
- Evaluate RAG groundedness at the claim level
- Check whether a RAG answer covers the required sources
- Build unanswerable questions that meaningfully test RAG
- Test RAG answers to questions whose facts change over time
- Verify deleted documents stop influencing RAG answers
- Test incremental indexing for missing and duplicate records
- Monitor RAG index freshness with inspectable probes
- Version a RAG corpus for reproducible evaluation
- Evaluate retrieval from tables with row and column context
- Test PDF retrieval across headings, footnotes, and reading order
- Test spreadsheet retrieval with units, formulas, and sheet identity
- Evaluate multilingual retrieval across query and document languages
- Test retrieval of acronyms and ambiguous entity names
- Retrieve numerical evidence with the correct unit and period
- Evaluate questions that require evidence from multiple documents
- Test follow-up questions against the correct conversation context
- Separate RAG conversation memory from durable knowledge
- Test RAG cache invalidation when source documents change
- Choose public RAG datasets without assuming they match your product
- Use BEIR as a retrieval diagnostic with a local task set
- Trial Ragas metrics against a small human-reviewed fixture
- Trial DeepEval for RAG with inspectable case-level evidence
- Validate RAG metric input contracts before choosing evaluation tooling
- Evaluate corrective RAG retrieval decisions against a fixed baseline
- Evaluate self-reflective RAG decisions without trusting self-scores
- Test entity resolution in a GraphRAG index
- Audit GraphRAG relationships back to source passages
- Test GraphRAG community summaries for support and coverage
- Test GraphRAG index updates after a relationship changes
- Test RAG fallback search when the first retrieval fails
- Design and test a useful zero-result RAG response
- Write RAG reference answers that preserve acceptable alternatives
- Give a RAG evaluator the right evidence without leaking labels
- Classify RAG questions before choosing retrieval metrics
- Turn retrieval errors into specific indexing and ranking fixes
- RAG evaluation metrics: measure retrieval and answers separately
- How to test retrieval recall without rewarding irrelevant context
- RAG citations: a correct answer can still cite the wrong evidence
- Test RAG systems with conflicting and outdated documents
- When should a RAG assistant say it does not know?
Production reliability
A model test is one part of running an AI product. Teams also need to check cost, speed, access, and failure recovery. Use these guides to plan tests at the service level. Keep a clear record of the conditions you tested, the result you saw, and the limits that still need attention.
- LLM observability tools: a practical selection and trial guide
- LLM gateway evaluation: test fallbacks, budgets, and cache boundaries
- Build indirect prompt-injection tests from retrieved content
- Test system-prompt leakage without collecting real secrets
- Use synthetic canaries to test AI secret-handling boundaries
- Validate tool outputs before an agent acts on them
- Test unsafe output handling at application boundaries
- Test retrieval access control before a model sees documents
- Test cross-tenant isolation across retrieval, tools, and caches
- Scope AI service accounts to the operations they need
- Test AI rate limits across users, retries, and concurrent requests
- Test LLM budget enforcement before expensive work starts
- Prevent retry storms across AI application layers
- Test streaming errors after an answer has started
- Run a controlled LLM-provider outage drill
- Test fallback policies for provider, region, and data boundaries
- Test AI API-key rotation without losing the recovery path
- Choose trace sampling without hiding rare AI failures
- Test redaction across prompts, metadata, and error messages
- Verify observability exports preserve useful task evidence
- Define AI service objectives around completed user tasks
- Attribute LLM cost to tasks, retries, and supporting services
- Roll out a prompt version with a reversible decision
- Use feature flags for bounded AI releases
- Design shadow evaluation without confusing it with live acceptance
- Evaluate an AI canary release with explicit stop conditions
- Turn user-reported AI errors into reproducible cases
- LLM regression testing: turn failures into release gates
- Valid JSON is not enough: evaluate structured LLM outputs
- Evaluate an LLM model migration before switching production
- Monitor LLM production drift without collecting everything
- What should an AI evaluation engagement deliver?
- LLM security and bot attacks: a practical test plan
Human judgment
Some answers need a person to judge their quality. A clear scoring guide helps reviewers apply the same standard. These articles cover review tasks, disagreement, and checks on automated judges. Start with a small set of examples that people can discuss before using the scores to compare models or approve a release.
- Measure false positives in automated LLM evaluation
- Find the failures your LLM evaluator misses
- Test whether an LLM judge rewards longer answers
- Measure answer-order bias in pairwise LLM judging
- Give an LLM judge a defensible abstention option
- Calibrate a quality score against task acceptance
- Write annotation guidelines reviewers can apply consistently
- Triage evaluator disagreements into fixable causes
- Use deliberate answer mutations to test evaluators
- Track who labeled an evaluation case and why
- Evaluate tone and style without confusing preference with correctness
- Design escalation rules for human review of AI work
- Design a human review queue that preserves context
- Adjudicate AI review disagreements with an evidence record
- How to calibrate an LLM judge before trusting its scores
- An LLM evaluation rubric template with scoring anchors
- Human evaluation agreement: diagnose disagreement before averaging
- Pairwise LLM evaluation: compare answers without hiding ties
- Where should limited human evaluation time go?
AI agents
An agent can take several steps and use tools to act on a task. Test the final result and the path it took. These guides cover tool calls, memory, access limits, and recovery from errors. Pair failure tests with ordinary tasks so your controls preserve the work users need to complete.
- Test agent checkpoints before relying on recovery
- Test agent idempotency when a tool call is repeated
- Test cancellation through an AI agent workflow
- Evaluate agent recovery after a tool timeout
- Test concurrent agent runs that touch shared state
- Test agent approval boundaries before consequential actions
- Scope agent tool permissions with task-specific access
- Classify tool errors so an agent can recover appropriately
- Detect agent loops with observable stopping rules
- Test state consistency across agent steps
- Test handoffs between agents with an explicit task contract
- Evaluate agent memory for usefulness and incorrect persistence
- Evaluate browser agents by visible task completion
- Evaluate email drafting agents before permitting sends
- Evaluate scheduling agents for constraints and confirmation
- Evaluate file-editing agents with before-and-after evidence
- Evaluate support agents across resolution and escalation
- Evaluate an incident assistant using controlled failure scenarios
- Test an agent sandbox with explicit allowed operations
- Design an agent audit trail that explains consequential actions
- AI agent evaluation: test outcomes, traces, and repeated runs
- Tool-calling evaluation: arguments, authorization, and side effects
- Evaluate multi-turn assistants across the whole conversation
- Prompt-injection evaluation for retrieval and tool-using agents
- Measure AI agent cost per successful task, not per response
General AI
Begin with the job your users need done. These guides explain common AI product choices in plain terms, from when to use an agent to when a person should take over. Use them to frame a first trial, choose a useful measure of success, and decide which questions need more evidence.
- Ask an AI vendor for evidence you can actually verify
- Scope an AI evaluation engagement with acceptance artifacts
- How to choose your first B2B AI use case
- Chatbot, workflow, or AI agent: which should you build?
- RAG vs. fine-tuning: diagnose the problem first
- AI pilot success metrics that survive a real rollout
- Tokens and context windows: a practical guide for teams
- Prompt caching: measure the savings, not just the hit rate
- Model routing: lower AI cost without hiding quality losses
- Human handoff in AI products: design and measure the last mile
- Does an AI SaaS use your data? Questions that need evidence
- Build or buy an AI product: an evidence-first checklist
Model briefings
Model names alone do not tell you which option fits your task. These briefings link to dated sources and explain the settings behind reported results. Check the source date, price, and test conditions before relying on a comparison. Then try the model on examples from the work your users actually do.