On this page
The short answer
Evaluate an LLM classifier using the label contract, per-class outcomes, and consequences of incorrect routing. Overall accuracy can hide failures in a rare but important class. Keep unknown labels, invalid outputs, and abstentions visible, and verify that the label taxonomy actually fits the incoming tasks.
What to take away
- Define each label’s operational meaning.
- Inspect consequential classes separately.
- Report abstention and invalid-output counts.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Define labels as operational decisions
Scikit-learn documents per-label precision, recall, support, and macro or weighted averaging. Before using those measures, define what each application label means and which action follows it. A topic label for analytics has a different acceptance contract from a queue label that changes who handles a customer request.
Specify mutually exclusive versus multi-label behavior, an unknown category, and any abstention path. Give reviewers borderline examples. A request containing both billing and technical issues needs an explicit policy; forcing it into one category without that rule creates inconsistent references and misleading model errors.
Keep the evaluation taxonomy independent
Build a reference set from the intended workload and label it under a fixed guideline. Include rare classes, overlapping descriptions, negation, and requests outside the supported taxonomy. Preserve class counts and whether examples were sampled representatively or deliberately balanced.
Inspect the actual serialized output. A semantically reasonable explanation with no valid label may still fail a downstream routing contract. Conversely, a correctly formatted class can be wrong for the request. Keep schema validity and classification correctness separate so the repair can target output handling or semantic discrimination.
Use an imbalanced numerical control
In an illustrative set of 100 requests, ninety are routine and ten are urgent. A constant routine classifier achieves 90% accuracy and zero urgent-case recall. Those numbers are hypothetical arithmetic, not a model evaluation. The example shows why the product owner must inspect consequential classes separately.
For each class, retain correct assignments, false assignments, missed assignments, and support. Explain macro averaging as equal treatment of class metrics and support-weighted averaging as weighting by class frequency. Handle undefined denominators explicitly; a class with no reference examples has not demonstrated perfect recognition.
| Case or record | Expected check | Decision |
|---|---|---|
| Always routine on 90/10 fixture | Accuracy 90%; urgent recall 0% | Reject accuracy-only conclusion |
| Valid label but wrong queue | Semantic routing error | Count as classification failure |
| Unsupported request category | Taxonomy has no defensible label | Use the defined unknown path |
Compare candidate routes and review burden
Compare systems on the same cases and record label transitions. Investigate whether apparent gains shift work into a catch-all queue or increase human review. An abstention policy can reduce confident mistakes while moving the operating burden elsewhere; report both effects.
Apply release gates for any class whose mistakes have unacceptable consequences, then compare ordinary routing quality and operating cost. Revisit the taxonomy when real requests repeatedly fall outside it. Improving a prompt cannot repair a label scheme that does not represent the decisions the product needs to make.
Limits of the evidence
The class counts are invented, and no classifier was run. Reference-taxonomy quality, sampling, and class overlap constrain interpretation. A classification metric does not by itself establish whether the downstream routing workflow is useful or safe.
Common questions
Is macro F1 always better than accuracy?
It highlights class-level performance differently. Choose measures that fit the decision and retain per-class counts rather than treating one aggregate as universally best.
Should the test set be balanced?
A balanced challenge set can test each class, while a representative set estimates workload behavior. Label those purposes and do not merge their rates without a policy.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- scikit-learn: precision_recall_fscore_support scikit-learn · Living reference · Maintained documentationReviewed: per-label metrics, support, macro/micro/weighted averaging and undefined denominators. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.