On this page
The short answer
Evaluate text-to-SQL by checking whether a permitted query returns the intended result across controlled database cases. String equality can reject valid alternatives, while one execution can accept a query that coincidentally matches the reference on one database. Keep semantic correctness, authorization, and execution cost as separate acceptance conditions.
What to take away
- Use multiple discriminating database fixtures.
- Define null, duplicate, and ordering semantics.
- Keep execution permissions separate from result correctness.
Sources checked . Research synthesis and Rubrex recommendations. Examples and local experiments are labeled; they are not client results.
Define the allowed database task
Zhong, Yu, and Klein propose test-suite accuracy using multiple database instances to approximate SQL semantic correctness. Their paper motivates checking behavior beyond a single reference string; this guide does not reproduce its benchmark results or claim formal query equivalence.
Write the user question, schema snapshot, permitted tables, expected result semantics, and allowed query class. Decide whether ordering, duplicates, null handling, and numeric tolerance matter. Run evaluation in isolated read-only fixtures with explicit resource limits rather than executing candidate SQL against production records.
Evidence: Semantic Evaluation for Text-to-SQL with Distilled Test Suites [1]
Build fixtures that distinguish plausible wrong queries
Include data configurations that reveal missing joins, incorrect filters, aggregate mistakes, and accidental dependence on row order. A database containing only one matching row can make an incorrect query look right. Preserve the fixture contents and reference result with the case version.
For a monthly revenue task, include canceled orders, multiple line items, missing dates, and a boundary timestamp. Define the currency and time-zone assumptions. A query that returns a plausible total can still double-count a join or include the wrong period; the fixture should make those mistakes observable.
Compare results under a documented policy
For an illustrative customer-count task, a candidate omits DISTINCT while the reference counts unique customers. A fixture where each customer has one order hides the error. Add a customer with two orders: the candidate result now differs and should fail. These are authored fixtures, not an executed SQL benchmark.
Compare result sets or multisets according to the task, preserving column meaning and order requirements. Keep execution errors, timeouts, and forbidden operations distinct from incorrect result values. A parser accepting the query does not establish that it follows the user’s request or the caller’s allowed scope.
| Case or record | Expected check | Decision |
|---|---|---|
| Unique-customer count; repeated orders | COUNT without DISTINCT overcounts | Reject on the distinguishing fixture |
| Valid alternative join syntax | Same intended result across fixtures | Permit semantic equivalence |
| Correct total from forbidden table | Permission contract violated | Block despite numeric match |
Report acceptance and diagnose the query defect
Use case-level outcomes with the number of fixture instances each query passed. Do not treat several instances of one question as unrelated user tasks. Review near misses by category so a schema-grounding problem can be distinguished from a date filter or aggregation defect.
Block queries that cross the declared permission or operation boundary even when their returned values happen to match. Compare performance on the same schema and fixture versions, and retain the generated SQL and execution evidence. Passing a finite suite supports the tested behavior without proving equivalence for every possible database.
Limits of the evidence
No candidate SQL was executed for this guide. The fixtures are proposed examples and the cited paper was reviewed at abstract and publication-record level. Controlled execution tests do not prove universal semantic equivalence or production authorization correctness.
Common questions
Is exact SQL matching useful?
It can diagnose formatting or template behavior, but equivalent queries can use different syntax. Use task-appropriate execution checks for semantic acceptance.
Does passing all fixtures prove equivalence?
No. A finite suite can miss database configurations that distinguish queries. Document tested conditions and remaining limitations.
Sources & further reading
Primary sources behind this briefing. A source’s findings apply to its own study conditions; publication on arXiv does not establish peer review.
- Semantic Evaluation for Text-to-SQL with Distilled Test Suites Zhong, Yu and Klein · 2020 · PreprintReviewed: abstract and bibliographic record; execution-based comparison over multiple database instances; no reported numerical findings reused. Accessed October 16, 2026.
Questions or corrections? Write to Rubrex. Read our editorial approach.