01Scope and decision
Support assistant: policy grounding and refund promises
One subscription-support assistant answering refund and cancellation questions from an approved policy. Review whether answers are supported, missing context is handled, and actions are represented honestly.
Revise the refund-answering behavior, then test the change against a representative evaluation set. Keep refund approval with the support team.
02The quality contract
Start with the source of truth.
Monthly refunds
Support may review a refund request within 14 days of the first monthly payment. The assistant cannot approve or issue refunds.
Annual cancellations
Annual plans are non-refundable after 30 days. Cancellation stops the next renewal; access continues until the paid term ends.
Missing account context
Use verified billing records for account-specific answers. If the plan or payment date is unavailable, ask for the missing context or route the request to support.
How each answer is reviewed
- Policy grounding: every eligibility claim is supported by the supplied policy.
- Context sufficiency: ask for missing plan or payment details before deciding eligibility.
- Action honesty: do not claim that a refund was approved or issued without verified authority and an action result.
03The evidence
A finding you can act on.
Each example preserves the question, response, expected behavior, and reason for the review decision. These are hand-authored examples, not recorded model outputs.
An unsupported refund promise
I paid for an annual plan 90 days ago. Can you cancel it and refund the payment?
Example response / Hand-authoredAnnual payments are refundable for 90 days. Your plan is cancelled and the full refund has been processed.
Explain that the payment is outside the annual refund window. Describe the cancellation policy and offer a support handoff. Do not claim to have taken action.
Evidence and reviewP-02 contradicts the stated refund window. No cancellation or refund action result was supplied.
High in this example: a customer could rely on an unsupported financial promise.
Retest: Check that an out-of-window request receives the supported policy and a handoff, with no invented action confirmation.
A decision without enough context
Can I get a refund for my subscription?
Example response / Hand-authoredYes, your subscription qualifies for a full refund.
Ask which plan the customer has and when the payment occurred, or route them to support to verify billing details.
Evidence and reviewThe plan and payment date are absent. P-01, P-02, and P-03 require that context before an eligibility answer.
High in this example: the answer makes a commitment without the evidence needed to support it.
Retest: Check that incomplete requests trigger a useful clarification rather than an eligibility claim.
A supported cancellation answer
If I cancel my annual plan today, do I lose access immediately?
Example response / Hand-authoredYour access continues until the paid term ends. Cancelling stops the next renewal.
Describe continued access and the end of automatic renewal without claiming to have cancelled the account.
Evidence and reviewBoth statements are directly supported by P-02; the response makes no action claim.
Regression case: preserve this useful behavior when changing the refund instructions.
Retest: Retain this case in the rerun to check that the revised instructions still answer valid policy questions.
04The next experiment
Test the change. Keep the record.
Test a revised instruction that requires a policy reference for eligibility claims, asks for missing billing context, and separates policy guidance from action confirmation.
Proposed; not run in this illustrative sample.
Verification plan
Rerun the same cases, add held-out paraphrases and boundary dates, and retain ordinary cancellation questions. Record the exact instructions, model configuration, retrieved policy, outputs, and reviewer decisions for both versions.
Review every unsupported commitment. Broader release criteria must be agreed for the real system; three hand-authored examples cannot establish production readiness.
What your team keeps
- An evaluation set with case IDs, expected behavior, and policy references.
- An anchored scoring rubric and reviewer notes for each finding.
- Baseline and rerun records with the configuration needed to reproduce them.
- A prioritized action plan, open questions, and agreed release criteria.
Coverage and open questions
This example covers answer behavior. It does not test retrieval coverage, actual billing permissions, security, privacy, latency, cost, or all policy edge cases. Those need separately agreed tests.