Yuchen Zhang, Yuze Gao, Bin Chen, Wenfeng Li, Shuo Sun, and Jian Su. 2025. High-Quality Complex Text-to-SQL Data Generation through Chain-of-Verification. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pages 2368–2379, Mumbai, India. The Asian Federation of Natural Language Processing and The Association for Computational Linguistics.
Abstract:
Can today’s Text-to-SQL (T2S) benchmarks
still stretch modern LLMs? We argue no. Spi-
der1.0 and BIRD, painstakingly hand-built, re-
main small, costly, and skewed toward middle
complex SQL. Meanwhile, LLM-generated cor-
pora are inexpensive but often superficial and
fragile suffering from shallow nesting, seman-
tic drift, template fatigue, and insufficient qual-
ity check. We address this gap with a Chain-
of-Verifications (CoVe) framework that turns
a handful of expert-labelled seeds into a large,
reliably checked dataset at a fraction of the
usual cost. The resulting corpus, AIGT2S, de-
livers: (1) 18k Question–SQL pairs across 113
databases, 41–77% larger than current English
sets; (2) 55% queries in the Ultra band of our
four-level difficulty taxonomy; (3) 87.5% inter-
annotator agreement; (4) ≥80% labour and
≥98% monetary savings versus earlier efforts.
Baselines including GPT-4o, Llama3, RESD-
SQL, and MAC-SQL, achieve at most 56% ex-
ecution accuracy, indicating substantial room
for improvement.
License type:
Publisher Copyright
Funding Info:
This research / project is supported by the Centre National de la Recherche Scientifique @ Campus for Research Excellence and Technological Enterprise (CNRS@CREATE) - Intelligent Modelling for Decision-making in Critical Urban Systems
Grant Reference no. : EC-2022-041
Description:
Publisher: Association for Computational Linguistics