Redefining text-to-SQL metrics by incorporating semantic and structural similarity.

ABSTRACT: The rapid advancements in text-to-SQL systems have driven the scientific community to create increasingly complex benchmarks for this task. However, evaluation metrics often rely on simplistic or binary approaches that fail to capture the similarities and differences between equivalent SQL queries. Current metrics overlook critical aspects such as partial correctness, structural differences, and semantic equivalence. To address these limitations, we propose a novel metric for SQL query comparison, designed to offer a more precise assessment of the similarity between SQL queries at both the semantic (string) and execution result (resultant table) levels. This new metric allows for a granular evaluation of SQL query similarity, supporting a more accurate assessment and ranking of text-to-SQL

SUBMITTER: Pinna G 

PROVIDER: S-EPMC12216993 | biostudies-literature | 2025 Jul

REPOSITORIES: biostudies-literature