On this page
The short answer
Choose metric weights only after separating non-negotiable requirements from preferences. A weighted average can summarize trade-offs among acceptable outputs, but it should not allow excellent tone or speed to cancel an unauthorized action. Preserve component scores and the cases that determine the decision.
What to take away
- Apply blocking requirements before preferences.
- Keep raw metrics and normalization rules.
- Report weight-sensitive rankings as trade-offs.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Name the decision behind the score
NIST’s AI RMF Core frames measurement in the context of risks and intended use. It does not prescribe a universal set of LLM metric weights. Rubrex recommends starting with what the release owner can actually trade off and what must block acceptance.
For a support assistant, factual support and authorization may be required conditions, while brevity and response time can be preferences within an acceptable range. A single score that mixes them without that distinction makes a strong style result appear to compensate for a material factual or permission failure.
Evidence: NIST AI RMF Core, version 1.0 [1]
Normalize deliberately and retain raw units
Document each metric’s range, direction, denominator, and missing-value policy before combining it. A latency in seconds cannot be added directly to a quality percentage without an explicit transformation. Keep the raw values in the report so reviewers can inspect what a normalized point means.
Avoid using the observed candidate minimum and maximum as the only normalization anchors. Adding a poor candidate could then change the apparent score of systems that did not change at all. Prefer product-defined ranges or show how sensitive the conclusion is to the scaling choice.
Work through a labeled trade-off
Consider invented candidates X and Y on a ten-point quality scale and an explicitly transformed ten-point speed scale. With weights of 0.8 quality and 0.2 speed, X scoring 9 and 4 yields 8.0; Y scoring 8 and 9 yields 8.2. With equal weights, their scores are 6.5 and 8.5. These are arithmetic illustrations, not model measurements.
Neither comparison can decide whether a critical boundary violation is acceptable. Evaluate that gate first. Then inspect whether the small 0.2 difference under the first weighting is meaningful given label and sampling uncertainty, rather than declaring a confident winner from decimal arithmetic alone.
| Case or record | Expected check | Decision |
|---|---|---|
| Unauthorized action | Independent contract gate | Block regardless of average |
| X: quality 9, speed 4 | 0.8×9 + 0.2×4 | Illustrative weighted score 8.0 |
| Y: quality 8, speed 9 | 0.8×8 + 0.2×9 | Illustrative weighted score 8.2 |
Test sensitivity and keep disagreement visible
Repeat the calculation over a defensible range of weights and scaling assumptions. If the preferred candidate changes easily, report a trade-off instead of a robust ranking. Show component-level results by important task slice so aggregate improvements cannot conceal a regression for a supported workflow.
Record who selected the weights, why they fit the product, and which unresolved values require a decision. Change the policy through a versioned review rather than after seeing results. The final artifact should make the judgment inspectable, including the reasons an otherwise high-scoring candidate was blocked.
Limits of the evidence
The numbers are hypothetical arithmetic examples. This guide supplies no universal utility function or acceptable risk threshold. Metric transformations, task mix, and uncertainty can change the ranking even when the arithmetic is correct.
Common questions
Are equal weights more objective?
They are a policy choice too. Equal numeric weights can imply very different practical trade-offs depending on metric scales.
Should every metric enter the aggregate?
No. Keep diagnostics and blocking requirements separate when averaging them would obscure their meaning.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- NIST AI RMF Core, version 1.0 NIST · 2023 · Framework excerptReviewed: 2023 framework context, measurement, documented risks and responsibility; page says revision in progress. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.