On this page
The short answer
Evaluate refusals by checking whether prohibited requests are declined while legitimate requests remain usable. Inspect the actual tool or data outcome, not only refusal wording. Separate correct refusal, unnecessary refusal, partial compliance, and safe continuation, and define the relevant policy before judging the response.
What to take away
- Pair prohibited requests with legitimate controls.
- Check final state as well as refusal wording.
- Report false refusals and violations separately.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Write the product’s permitted boundary
XSTest’s abstract describes testing exaggerated refusals using safe prompts and unsafe contrasts. That motivates paired evaluation of blocking and usefulness; it does not supply the complete policy for your application or establish current model behavior.
Define what the caller may ask, which information the system may disclose, and which actions require authorization or review. Keep policy refusal distinct from inability to answer because evidence is missing. Both may produce a decline, but their causes and useful next steps differ.
Evidence: XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models [1]
Build legitimate and prohibited contrasts
Create pairs that share vocabulary while differing in permission or intent. An authorized user requesting their own test record and a user requesting another tenant’s record can require different outcomes even when their sentences look similar. Use synthetic accounts and harmless markers.
Include legitimate requests mentioning sensitive terminology without asking for a prohibited action. A blanket keyword filter can look protective on restricted cases while making ordinary work impossible. Keep reviewer labels tied to the declared policy and record unresolved cases instead of treating the evaluator’s intuition as policy.
Inspect behavior after the refusal text
In an illustrative read-only support task, retrieved text asks the assistant to send a message. A response saying it cannot send while a tool trace shows a completed send is a boundary failure. A polite refusal does not reverse the side effect.
The paired legitimate control lets an authorized user request a permitted summary of the same material. It should succeed without unnecessary blocking. For each case, preserve attempted tool calls, enforced denials, final state, and the user-facing response so refusal quality and actual boundary enforcement can be reviewed independently.
| Case or record | Expected check | Decision |
|---|---|---|
| Refusal text but completed unauthorized send | Side effect crossed the boundary | Fail enforcement |
| Authorized summary request | No prohibited action required | Allow useful completion |
| Missing source evidence | Cannot establish requested fact | Use an evidence-gap response |
Measure blocking and usability together
Report prohibited-case violations, legitimate-case false refusals, incomplete tasks, and permitted continuations with their own denominators. Preserve refusal reasons and any helpful alternatives. An alternative should stay within the user’s authorized scope rather than perform a different consequential action.
After a policy or guardrail change, retest the original violation and nearby legitimate cases. Keep model-based refusal separate from downstream authorization controls in the evidence. A finite test set can show failures and improvements within its scope, but a clean refusal report does not establish universal security or policy compliance.
Limits of the evidence
All support scenarios are authored controls and no refusal system was tested. XSTest was reviewed at abstract level as a foundational contrast-testing source. The application’s policy and downstream authorization require their own validation.
Common questions
Should every refusal include an alternative?
Offer an alternative when it is useful and permitted. Do not invent another action or disclose restricted information merely to make the refusal sound helpful.
Is refusal rate a safety metric?
It is a descriptive count. Its meaning depends on which requests should be refused and whether prohibited behavior actually stopped.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models Röttger et al. · 2023 · PreprintReviewed: abstract and bibliographic record; safe-request refusals and unsafe contrast prompts; no historical model scores reused. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.