On this page
The short answer
An evaluation coverage matrix connects intended product behavior to concrete cases. Start with user tasks and consequential boundaries, then record which combinations have tested evidence. A large dataset can still leave an important workflow untested; case count alone does not establish coverage.
What to take away
- Derive axes from expected behavior.
- Give untested cells explicit owners.
- Report coverage separately from pass rate.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Choose coverage axes from the product
NIST’s AI RMF Core connects context, measurement, and documented risk management; its actions are not a universal checklist. The matrix below is a Rubrex planning method for turning an application’s own task contract into inspectable test coverage.
Begin with a small set of axes that change the expected behavior: workflow, caller role, evidence state, output action, and consequence of failure. Do not generate every possible combination mechanically. A language difference might matter for extraction, while an approval state is central to an agent that can modify records.
Evidence: NIST AI RMF Core, version 1.0 [1]
Give every required cell an owner
Write the behavior expected in each priority cell and link the cases that exercise it. Mark cells as covered, partially covered, untested, or out of scope with a reason. A test that merely mentions a workflow should not count if it never checks the boundary the cell describes.
Assign an owner to uncovered cells and record why they matter. A read-only role paired with a write request can deserve priority even if it is rare. Separate frequency evidence from consequence judgments so a product owner can see why low-volume cases consume part of the test budget.
Use the matrix to find false reassurance
Consider an invented invoice assistant with many successful extraction cases. The suite may cover clean PDFs and ordinary users while omitting revised invoices, ambiguous currencies, and users who cannot view the account. A single overall accuracy score would conceal those absences.
For a proposed review, ask a tester to select one case for each high-priority cell and explain its expected assertion. If no case demonstrates currency handling or record authorization, mark the corresponding cell untested even if a model produced a plausible answer during an informal demo.
| Case or record | Expected check | Decision |
|---|---|---|
| Clean invoice; ordinary user | Required fields and units checked | Covered only for that cell |
| Revised invoice; stale copy | Version choice must be checked | Untested until a case exists |
| Read-only user; write request | No unauthorized change | High-priority boundary case |
Keep coverage and performance as separate reports
Coverage answers whether the evaluation exercises a required condition. Performance answers what happened on the cases that were run. A covered cell can fail; an untested cell has no measured pass rate. Preserve that distinction in dashboards and release notes rather than coloring all empty cells green.
Revisit the matrix after a new tool, user role, document source, or workflow changes the product contract. Add or retire cells with a written rationale. Approve a release only against the agreed scope and unresolved gaps; filling a spreadsheet is not evidence that unrepresented combinations are safe or reliable.
Limits of the evidence
This matrix is a proposed planning aid, not a completeness proof or NIST certification. Selected axes and priority judgments can miss interactions. The illustrative invoice cases were authored for explanation and do not describe client or production results.
Common questions
Do I need every combination?
Usually choose combinations based on behavior and consequence, and record exclusions. Exhaustive-looking Cartesian products can waste effort without testing meaningful interactions.
Can one case cover several cells?
Yes, if it actually asserts each required behavior. Keep the mapping inspectable so an omitted assertion does not become claimed coverage.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- NIST AI RMF Core, version 1.0 NIST · 2023 · Framework excerptReviewed: 2023 framework context, measurement, documented risks and responsibility; page says revision in progress. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.