We propose the following principles to guide the development of this taxonomy:
- Comprehensiveness: The taxonomy should encompass all potential errors that could occur during the NL2SQL conversion process.
- Mutual Exclusivity: Each error type should be clearly distinct with no overlap, to avoid ambiguity in error classification.
- Extensibility: The taxonomy should be adaptable to incorporate new error types as NL2SQL technologies and methodologies evolve.
- Practicality: The taxonomy should be practical and applicable in real-world settings, aiding developers in diagnosing and correcting errors effectively.
Following these principles, we attempted to design a taxonomy containing two levels:
- Error Localization: This level focuses on identifying the specific parts of the SQL where errors occur, such as in the
SELECTclause. It is vital for precisely locating where misunderstandings or misinterpretations arise, thereby facilitating targeted corrections. - Cause of Error: This level focuses on understanding why the model is wrong when generating SQL. For example, value errors in the
WHEREclause may indicate the model's insufficient ability to understand and retrieve database content. On the other hand, conditional errors in theWHEREclause typically reveal flaws in semantic understanding, where the model fails to grasp the logical requirements of the query.
We collected the errors generated by DIN-SQL on the Spider dataset and manually classified them according to the taxonomy we designed.
