GPT-6 Astra, Sol, and Luna: an evidence-based model guide
Compare current OpenAI GPT-6 models with dated API prices, context limits, and independent benchmark results. A practical selection guide for B2B teams.
Research briefing · Sources includedResearch & field notes
Better questions.
More reliable AI.
Research, methods, and practical decisions for teams shipping AI. Read the evidence. Understand the limits. Put it to work.
40briefings
across 7 topics
across 7 topics
Start here / Evaluation design
How to evaluate an LLM application before release
A practical evaluation plan for LLM applications: define success, build a representative dataset, choose graders, and investigate failures before release.
Read the briefingA repeatable quality bar
01 / Define the decision
02 / Measure the baseline
03 / Investigate the failures
04 / Test the next change
A better score should lead to a better decision.
The library
Find your next question.
40 articles
Model briefings / 5 articles
Inside the current models.
OpenAI, Claude, Meta, Kimi, and Gemini. Dated prices, benchmark settings, and practical evaluation advice. These are research snapshots, not live rankings.
Claude Opus 5.5: pricing, metrics, and what to test
Review Claude Opus 5.5 with dated pricing and independent planning metrics. Learn how to test coding quality, reviewer effort, and application permissions.
Research briefing · Sources includedMeta Muse Spark 1.3: metrics, pricing, and data-use tiers
Review Meta Muse Spark 1.3 with dated math benchmarks, API pricing, context limits, audio caveats, and the difference between Standard and Contributor tiers.
Research briefing · Sources includedKimi K3: benchmark evidence and a practical evaluation plan
Examine Kimi K3 with independent science and math metrics, API prices, and context limits. Learn how to evaluate long tasks without confusing model and agent.
Research briefing · Sources includedGemini 3.8 Flash: metrics, pricing, and migration checks
Evaluate Gemini 3.8 Flash using dated science and math results, context limits, introductory pricing, and practical migration and multimodal checks.
Research briefing · Sources includedGeneral AI / 10 articles
AI decisions, explained.
Ten practical guides for product teams: choosing a use case, understanding costs, protecting customer data, and measuring whether the work improves.
How to choose your first B2B AI use case
Choose an AI use case with measurable value, available evidence, and manageable failure costs. A practical scorecard for B2B product teams.
Research briefing · Sources includedChatbot, workflow, or AI agent: which should you build?
Compare chatbots, fixed AI workflows, and agents by task completion, tool use, latency, and recovery. Choose the right level of autonomy for your product.
Research briefing · Sources includedRAG vs. fine-tuning: diagnose the problem first
Understand when retrieval, prompting, or fine-tuning addresses an AI failure. Compare evidence access, behavior consistency, update costs, and evaluation metrics.
Research briefing · Sources includedAI pilot success metrics that survive a real rollout
Measure an AI pilot with accepted outcomes, net handling time, repeat use, and operational costs. Learn why demo accuracy alone cannot justify a rollout.
Research briefing · Sources includedTokens and context windows: a practical guide for teams
Learn how tokens, context limits, output budgets, and conversation history affect AI cost and reliability. Includes a measurement plan for long-document tasks.
Research briefing · Sources includedPrompt caching: measure the savings, not just the hit rate
Understand prompt caching, cache-read metrics, and cost per accepted answer. Learn how to test real savings without confusing cached input with cached responses.
Research briefing · Sources includedModel routing: lower AI cost without hiding quality losses
Design a model-routing experiment with route accuracy, escalation, task quality, and full cost accounting. Learn when a smaller-model path earns its place.
Research briefing · Sources includedHuman handoff in AI products: design and measure the last mile
Build AI handoffs that preserve context and make the next action clear. Measure escalation recall, review effort, and resolution instead of deflection alone.
Research briefing · Sources includedDoes an AI SaaS use your data? Questions that need evidence
Inspect AI SaaS data flows, training-use settings, retention, access, and deletion. A practical evidence checklist with coverage metrics for product teams.
Research briefing · Sources includedBuild or buy an AI product: an evidence-first checklist
Compare building and buying AI software with task quality, integration effort, data controls, total cost, and exit readiness. Includes a practical scorecard.
Research briefing · Sources includedResearch methods / 25 articles
Build a repeatable quality bar.
Evaluation design, RAG, agents, human judgment, and production reliability. Follow the evidence from a useful question to a release decision.
How to evaluate an LLM application before release
A practical evaluation plan for LLM applications: define success, build a representative dataset, choose graders, and investigate failures before release.
Research briefing · Sources includedHow many examples does an LLM evaluation need?
Choose an evaluation sample around coverage, uncertainty, and the decision at stake instead of relying on a universal minimum number of test cases.
Research briefing · Sources includedBuild an LLM evaluation dataset that reflects real work
Design evaluation datasets with task coverage, provenance, held-out examples, and explicit expected behavior. Separate synthetic coverage from real traffic.
Research briefing · Sources includedBenchmark contamination: why a private test set is not enough
Understand training leakage, repeated tuning, and evaluation-time exposure. Build a practical disclosure record for trustworthy LLM comparisons.
Research briefing · Sources includedBuild an AI failure taxonomy that leads to fixes
Classify AI failures by observable symptoms and causal hypotheses so evaluation findings translate into focused experiments and clear ownership.
Research briefing · Sources includedRAG evaluation metrics: measure retrieval and answers separately
Choose RAG metrics that distinguish missing evidence, irrelevant context, unsupported claims, and incomplete answers instead of hiding them in one score.
Research briefing · Sources includedHow to test retrieval recall without rewarding irrelevant context
Build evidence labels, multi-document cases, and controlled retrieval comparisons that reveal what your RAG retriever actually misses.
Research briefing · Sources includedRAG citations: a correct answer can still cite the wrong evidence
Evaluate citation support, coverage, and faithfulness separately. Learn what recent attribution research means for practical RAG quality checks.
Research briefing · Sources includedTest RAG systems with conflicting and outdated documents
Create RAG evaluations for version conflicts, missing authority, and ambiguous dates so assistants do not turn contradictory evidence into confident answers.
Research briefing · Sources includedWhen should a RAG assistant say it does not know?
Evaluate answerability, abstention, and clarification together so a RAG assistant avoids unsupported answers without refusing useful work.
Research briefing · Sources includedAI agent evaluation: test outcomes, traces, and repeated runs
Design agent evaluations around state changes, tool behavior, repeatability, and recovery rather than grading only the final chat response.
Research briefing · Sources includedTool-calling evaluation: arguments, authorization, and side effects
Test tool selection, argument validity, authorization boundaries, and state reconciliation to catch failures that a valid function call can hide.
Research briefing · Sources includedEvaluate multi-turn assistants across the whole conversation
Test corrections, changing requirements, retained constraints, and useful clarification so conversational quality is measured beyond isolated responses.
Research briefing · Sources includedPrompt-injection evaluation for retrieval and tool-using agents
Design bounded prompt-injection tests that separate untrusted content from authority and measure both inappropriate actions and excessive blocking.
Research briefing · Sources includedMeasure AI agent cost per successful task, not per response
Compare agent quality, latency, retries, and total resource use using a consistent task budget and an explicit definition of success.
Research briefing · Sources includedHow to calibrate an LLM judge before trusting its scores
Validate an LLM judge against task-specific human labels, inspect directional errors, and recheck calibration when the candidate distribution changes.
Research briefing · Sources includedAn LLM evaluation rubric template with scoring anchors
Create rubrics with observable criteria, clear pass boundaries, evidence requirements, and examples that help reviewers reach defensible judgments.
Research briefing · Sources includedHuman evaluation agreement: diagnose disagreement before averaging
Measure reviewer agreement with the label distribution and task ambiguity in view. Use adjudication to improve the rubric instead of hiding disagreement.
Research briefing · Sources includedPairwise LLM evaluation: compare answers without hiding ties
Design blinded paired comparisons with explicit criteria, randomized order, tie handling, and task-level analysis for model and prompt decisions.
Research briefing · Sources includedWhere should limited human evaluation time go?
Allocate human review across representative traffic, uncertain judgments, and consequential failure modes without confusing targeted sampling with prevalence.
Research briefing · Sources includedLLM regression testing: turn failures into release gates
Build a layered regression suite for prompts, models, retrieval, and tools. Separate deterministic failures from noisy judgments and preserve reproducible runs.
Research briefing · Sources includedValid JSON is not enough: evaluate structured LLM outputs
Test syntax, schema, field meaning, and downstream behavior separately so machine-readable output is also correct and useful to the consuming system.
Research briefing · Sources includedEvaluate an LLM model migration before switching production
Compare a candidate model on your tasks, judge calibration, output contracts, and operational limits before relying on public benchmark improvements.
Research briefing · Sources includedMonitor LLM production drift without collecting everything
Separate changes in traffic, quality, and evaluator behavior. Design privacy-aware monitoring that feeds representative failures back into evaluation.
Research briefing · Sources includedWhat should an AI evaluation engagement deliver?
A buyer’s guide to evaluation scope, datasets, calibrated scoring, failure analysis, reproducibility, and a handoff your engineering team can use.
Research briefing · Sources includedThe model decision desk
Which model fits the task?
Compare independent benchmark results, exact model settings, and API costs in one place.
Explore the model comparisonHow to read these notes
Evidence, with its limits attached.
These are research briefings and practical recommendations, not original experimental results. Sources include recent preprints and maintained technical documentation. Illustrative examples are labeled, and each article records when its sources were checked.
Our editorial approachPut the method to work