Research & field notes

Research library — page 1

Read a guide, follow its sources, and check the limits before applying a finding. Search the full library from the research desk, or browse by topic.

Evaluation design6 min read

Choosing an LLM evaluation framework: four options and a trial plan

Compare DeepEval, Giskard, Ragas, and Promptfoo by evaluation task, then use a shared fixture to check scoring, exports, data handling, and repeatability.

Research briefing · Sources included

RAG systems7 min read

GraphRAG evaluation: test relationships, summaries, and cost

Evaluate GraphRAG against a vector baseline with relationship checks, source-backed global summaries, update tests, and complete indexing and query costs.

Research briefing · Sources included

RAG systems7 min read

Agentic RAG evaluation: test search decisions and stopping rules

Test when an agent retrieves, rewrites a query, switches sources, and stops. Use a bounded fixture with answer, permission, latency, and cost checks.

Research briefing · Sources included

Production reliability6 min read

LLM observability tools: a practical selection and trial guide

Compare Langfuse, LangSmith, and Phoenix using verified documentation, then test trace coverage, masking, missing events, export, and investigation workflows.

Research briefing · Sources included

Production reliability7 min read

LLM gateway evaluation: test fallbacks, budgets, and cache boundaries

Evaluate an LLM gateway with controlled failures, approved fallback routes, key restrictions, streaming tests, cache isolation, and complete attempt accounting.

Research briefing · Sources included

Evaluation design3 min read

Deduplicate an LLM evaluation dataset without losing rare failures

Remove duplicate evaluation records while preserving rare failure types, source lineage, split boundaries, and explicit sampling weights.

Research briefing · Sources included

Evaluation design3 min read

Split LLM evaluation data by customer, time, and workflow

Choose customer, conversation, document, and time boundaries for LLM development and held-out tests, then audit split membership for related cases.

Research briefing · Sources included

Evaluation design3 min read

Find label leakage in an LLM evaluation pipeline

Audit prompts, retrieval, metadata, filenames, and graders for hidden reference answers with synthetic markers and controlled evidence removal.

Research briefing · Sources included

Evaluation design3 min read

Design an LLM test-case schema that preserves evidence

Define stable case IDs, visible inputs, private references, task contracts, source versions, and outcome records without mixing authoring and run evidence.

Research briefing · Sources included

Evaluation design3 min read

Build an evaluation coverage matrix for an AI product

Map user tasks, permission roles, evidence conditions, and failure consequences to test cases, then distinguish missing coverage from a tested failure.

Research briefing · Sources included

Evaluation design3 min read

Use confidence intervals in an LLM release decision

Report uncertainty around task pass rates with explicit sampling assumptions, interval methods, and a practical rule for inconclusive release evidence.

Research briefing · Sources included

Human judgment3 min read

Measure false positives in automated LLM evaluation

Review acceptable answers that an evaluator rejects, report false-alarm denominators, and test repairs without weakening genuine failure detection.

Research briefing · Sources included

Human judgment3 min read

Find the failures your LLM evaluator misses

Audit accepted outputs for hidden failures, measure detection recall on labeled challenge cases, and keep missed-failure estimates separate from prevalence.

Research briefing · Sources included

Human judgment3 min read

Test whether an LLM judge rewards longer answers

Compare concise and expanded answers with the same facts, then separate legitimate completeness gains from evaluator preferences for length or polish.

Research briefing · Sources included

Human judgment3 min read

Measure answer-order bias in pairwise LLM judging

Swap answer order in paired LLM judgments, map labels back to answer identities, and preserve ties and inconsistent decisions in the report.

Research briefing · Sources included

Human judgment3 min read

Give an LLM judge a defensible abstention option

Give evaluators explicit unknown outcomes, required evidence fields, and review routing so incomplete context does not become an invented pass or fail.

Research briefing · Sources included

Human judgment3 min read

Calibrate a quality score against task acceptance

Compare score bands with independently reviewed acceptance outcomes, then separate probability calibration, ordinal rubrics, and release thresholds.

Research briefing · Sources included

Evaluation design3 min read

Choose metric weights without hiding critical failures

Separate blocking task requirements from weighted preferences, expose trade-offs, and use sensitivity checks before trusting an aggregate quality score.

Research briefing · Sources included

Evaluation design3 min read

Evaluate rare AI failures without overstating confidence

Combine targeted boundary tests with representative sampling, preserve zero-failure uncertainty, and avoid treating enriched challenge cases as prevalence.

Research briefing · Sources included

Evaluation design3 min read

Make an LLM benchmark run reproducible

Preserve benchmark inputs, configuration, raw outputs, evaluator versions, and execution conditions so another reviewer can reconstruct a reported result.

Research briefing · Sources included