Classify RAG questions before choosing retrieval metrics

Define intent as an evidence obligation

BEIR’s heterogeneous task framing illustrates why one retrieval setting does not describe every search problem. For a local RAG taxonomy, define categories through the required evidence: one decisive passage, multiple complementary sources, a numerical operation, a date-dependent rule, or an answerability decision. Avoid labeling solely from surface phrases such as “what” or “compare.” The same wording can require very different evidence depending on the corpus and the user’s actual request.

Evidence: BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models [1]

Build a small classification example

In an authored equipment corpus, “What connector does D-4 use?” is a lookup. “Which adapter works with D-4?” may require a cross-document join. “What applied before September?” adds temporal applicability. “Which unreleased model will replace it?” may be unsupported. Record the evidence obligations for each rather than assigning labels by intuition alone. The taxonomy is useful when it changes the evaluation design: first-hit rank for the lookup, complete-chain evidence for the join, and correct uncertainty for the unsupported question.

Allow overlap without losing clarity

A numerical comparison can also be temporal and cross-document. Permit multiple labels or a primary task plus modifiers, then define how combined cases are scored. Keep category rules and examples versioned. Ask reviewers to classify a small shared set and inspect disagreements before using the taxonomy for reporting. Frequent confusion can reveal unclear definitions or ambiguous product requirements. Do not force a case into a convenient category merely to maintain equal group sizes.

Choose measures by failure consequences

For each category, specify retrieval, answer, and operational checks. A lookup may emphasize early evidence access; a comparison needs all required source groups; a temporal question needs applicable revisions; an unsupported query needs restrained behavior. Keep critical eligibility checks across every category. Report category counts and exclusions so a strong high-volume group cannot conceal a weak but consequential group. A taxonomy should explain why scores differ rather than create a new aggregate whose meaning nobody can reproduce.

Use classification to plan collection and repair

Compare the taxonomy with production question samples under approved data handling and identify missing or rare categories. Collect representative cases for those gaps instead of generating many near-identical lookup questions. When an error cluster appears, connect it to a specific intervention: entity resolution, temporal filtering, source coverage, or abstention policy. Preserve the category and reasoning in the failure record. Revisit the taxonomy when the product’s supported tasks change, while retaining earlier versions for historical comparisons. Track newly unclassifiable questions rather than discarding them from reports. They may reveal a supported product task that the current taxonomy and evaluation dataset do not yet represent.

Common questions