ESQ-Bench: A Multi-Tier Enterprise Oracle Benchmark for Evaluating NL2SQL Dialect Generalization and Silent Semantic Divergence

Natural Language to SQL (NL2SQL) models often report high execution accuracy on academic benchmarks such as Spider and BIRD. Those benchmarks use relatively small, synthetic schemas and SQL dialects that differ from what enterprises encounter in production systems. As a result, strong scores on those tests do not guarantee that a model will perform reliably when faced with real‑world database complexity.

What You Need to Know

ESQ‑Bench addresses this gap by providing an Oracle‑first benchmark that spans three tiers of enterprise schema complexity. The benchmark consists of six fully populated schemas that contain a total of 465 tables and 164,682 rows, with no empty tables. The same seed data was loaded into Oracle, PostgreSQL, MySQL, and SQL Server, allowing direct comparison of a model’s behavior across different SQL dialects.

Each schema tier reflects increasing levels of realism: the first tier mimics simple reporting databases, the second adds normalized transactional tables with foreign‑key constraints, and the third introduces partitioned tables, materialized views, and complex data types. For every tier, the benchmark supplies 550 gold‑validated natural‑language question and SQL query pairs, curated by domain experts to ensure semantic correctness.

Evaluation goes beyond traditional exact‑match metrics. ESQ‑Bench uses a four‑metric harness: Exact Match (EM), Execution Accuracy (EX), Semantic Recovery (SR), and Silent Divergence (SD). SR measures whether a query returns the correct result set even if the SQL text differs, while SD captures cases where a query runs without error but produces a subtly wrong result—a failure mode that is easy to miss in standard testing.

Why It Matters

By exposing silent semantic divergence, ESQ‑Bench reveals a class of errors that existing benchmarks overlook. A model may achieve high EM and EX scores yet still generate queries that return incorrect aggregates or miss rows due to dialect‑specific handling of NULLs, date functions, or implicit casts. Enterprises that rely on NL2SQL for self‑service analytics need to know whether such subtle faults exist before deploying a model.

The multi‑tier design also helps pinpoint where a model’s generalization breaks down. If performance drops sharply from tier 1 to tier 3, developers can infer that the model struggles with advanced features such as window functions or hierarchical queries. This insight guides targeted improvements, whether through better training data, richer synthetic augmentation, or dialect‑specific post‑processing.

Key Details

• Six populated schemas, 465 tables, 164,682 total rows, zero empty tables.
• Identical seed data loaded into Oracle, PostgreSQL, MySQL, and SQL Server.
• Three enterprise schema complexity tiers (simple reporting, normalized transactional, advanced partitioning/views).
• 550 expert‑validated natural‑language question and SQL query pairs.
• Four‑metric evaluation harness: EM, EX, SR, SD.
• Focus on measuring silent semantic divergence across SQL dialects.

What’s Next

The authors plan to extend ESQ‑Bench to additional cloud‑native dialects such as Amazon Redshift and Google BigQuery, and to incorporate more complex workloads like recursive CTEs and temporal tables. They also invite the community to contribute new question‑query pairs and to benchmark emerging NL2SQL approaches, with the goal of building a shared enterprise‑grade evaluation suite that better reflects production realities.

📌 Source: Arxiv Ai