Evaluating the system that writes the rubric.

01The problem

A generated scorecard needs its own checks.

Rubrex’s internal generator turns prospect context into evaluation rubrics. A polished scorecard is only useful if its criteria reflect the product, distinguish different kinds of failure, and tell reviewers how to score consistently.

We wanted to find where that broke down before using the generated rubric in an evaluation. The practical question was: could a reviewer apply this scorecard and explain the resulting judgment?

02The method

Review the rubric, not just the response.

We ran the generator against six representative prospect profiles. A 12-part meta-rubric supplied a common review framework, and three critic lenses examined the generated scorecards from different perspectives. This made the criticism concrete enough to turn into prompt changes.

We separated structural validation from semantic review. Deterministic checks cover the schema, weights, and scoring-anchor structure. Human review asks whether the criteria actually fit the product and whether the anchors communicate meaningful differences in quality. Both are needed before release.

03Findings and changes

The failures were specific enough to fix.

The review surfaced four recurring problems that a well-formed response alone would not reveal:

• Generic templates. Some rubrics amounted to a company-name swap, with too little connection to the product’s actual use case.
• Overlapping criteria. Separate dimensions could count the same issue twice, distorting the final judgment.
• Vague scoring anchors. Descriptions left reviewers without a clear basis for choosing one score over another.
• Invented product details. When the available evidence was thin, the generator could fill gaps with unsupported assumptions.

These findings produced seven prompt fixes. They gave the revision work clear targets: use the supplied context, separate the criteria, make scoring decisions explicit, and expose missing information rather than inventing it.

04What this shows

Turn criticism into a testable next change.

Start with representative inputs, define what good output requires, and keep each criticism attached to a specific failure. That gives your team a concrete change to test, instead of another round of subjective prompt tweaking.

This is the kind of work we scope in a Reliability Sprint: an evaluation set, a baseline, a failure breakdown, and a targeted experiment. This internal study demonstrates the method; it does not establish a measured accuracy uplift or a result for an external client.