Building Natural Language Interfaces to Databases (NLIDBs) has been identified as one of the most significant semantic parsing tasks for decades[3, 9, 18, 68, 73, 98]. By automatically converting text into the Structured Query Language (SQL), NLIDBs allow users to communicate with relational data in natural languages (NL) without any programming effort. These NL questions often cannot be directly answered by search engines. For example, in response to the question ‘What are the total population and average area of countries in the continent of North America whose area is bigger than 3000?’, an NLIDB would return 480753000 and 1344763 for the total population and average area respectively; while a search engine would present a number of tables and leave the computation to the user. Such interfaces (also known as Text-to-SQL models within the NLP community) relieve users who are not proficient in query languages from the burden of learning techniques for querying databases by allowing them to pose NL questions.

Within recent years, the emergence of complex, large, and human-annotated datasets consisting of NL questions and their corresponding SQL queries has significantly developed the field. Traditionally these have included in-domain datasets such as WikiSQL[98], ATIS[15, 39], and Advising[24], more recently the family of Spider cross-domain datasets, including Spider[92], SParC[93], and CoSQL[91] challenge the generalizability of models to unseen databases. Although recent studies have demonstrated the high accuracy (above 70%) of state-of-the-art Text-to-SQL models trained and evaluated on the Spider dataset, the performance of these models on complex queries is rather low, as many struggle to predict complex SQL queries, Complex SQL Generation. Parsing a question into a SQL query with nested queries, multiple SELECT clauses, GROUP BY, ORDER BY, UNION, INTERSECT, and EXCEPT requires a model to capture the semantic dependency between the NL question, database schema, and SQL syntax. According to the Spider criteria, SQL queries are classified into four difficulty levels – easy, medium, hard, and extra hard. The difficulty level is determined based on the number of SQL components, selections, conditions, nested sub-queries, column selections, aggregators, etc. Further, a question is complex when the corresponding SQL query is hard or extra hard.

Evaluating the accuracy of the top-five state-of-the-art Text-to-SQL models only on complex questions within the development set of Spider as the preliminary step, we found that their performance is below 50%. On questions with corresponding SQL queries of easy and medium difficulty levels, however, such models perform with an accuracy of over 80%. Therefore, we explore to what extent the decomposition of complex questions, as a novel stage within the Text-to-SQL pipeline, can bring us further in the area of Text-to-SQL. This is guided by our intuition that by decomposing complex questions into multiple easy and medium questions, Text-to-SQL models can convert them into correct SQL queries with a higher accuracy, thereby circumventing the challenge of complex SQL generation, illustrated in Figure 1.

Figure 1:(a) An example of a complex question in the Spider dataset. In addition to the complex question, the corresponding SQL query, the answer, and tables are shown. (b) The decomposition of the question in (a) is illustrated; Instead of feeding the complex question in (a) into Text-to-SQL models, we manually decompose the question into the three sub-questions. These sub-questions are classified as simpler than the original question. Executing sub-questions sequentially on the database, we can observe that answer to the complex question is the same as sub-question 3 in (a).

Note that the proposed decomposition stage is different from standard text simplification in NLP[59], a task in which text is rewritten to make it easier to process for a given audience. The complexity of questions in the Spider dataset originates from the underlying SQL query and the dependency between the text and database schema as opposed to the linguistic complexity of NL questions. To verify this, we analyzed whether metrics that are popularly used in text simplification tasks such as Flesch-Kincaid readability score, Flesch's reading ease score, Type-Token Ration, and Lexical variation are effective in distinguishing levels of difficulty in complex user questions. We found that easy and medium questions have the same lexical complexity and lexical richness as hard and extra hard questions, confirming that the existing text simplification methods are ill-suited for decomposing complex user questions. In order to assess the feasibility of decomposition, we thereby raise the following research questions:

To what extent can we leverage the decomposition of complex user questions as a means to circumvent the challenge of complex SQL generation facing existing Text-to-SQL pipelines?

To what extent can non-expert crowd workers aid in the decomposition of complex user questions in Text-to-SQL tasks in comparison to SQL experts?

To assess the potential benefit of decomposing complex questions, we first manually decomposed the questions and corresponding queries within the development set of the Spider dataset serving as an oracle decomposition. We then compared the accuracy gained by Text-to-SQL models with the new pipeline in which the oracle decomposition was augmented, realizing an increase in accuracy by over 163% (i.e., from 30% to 79%). Despite the promise of decomposition, to develop ML models that can (semi) automate the decomposition of complex user questions in a generalizable fashion, we would require a substantial amount of training data. Since hiring groups of experts is a costly endeavour[60], the viability of decomposing complex user questions at a beneficial scale hinges on its cost-effectiveness. Crowdsourcing has proved to be a reliable, effective, and efficient approach in many tasks[31, 53, 62] and across different domains[63], including within the NLP field[36, 80, 97]. Thus, we explored whether non-expert crowd workers (recruited from the Prolific crowdsourcing platform) can power such a cost-effective alternative. In comparison to the accuracy boost of 153% as a result of the decomposition carried out by a small group of SQL experts (N = 5), decomposition by non-expert crowd workers (N = 83) led to an accuracy boost of over 96%. Our findings show that crowd workers can effectively decompose complex user questions and thereby aid in circumventing the challenge of complex SQL generation in Text-to-SQL pipelines.

Our experiments pave the way towards extending crowd-powered decomposition on available Text-to-SQL datasets to gather a substantial amount of training data. This is a crucial prerequisite for building ML-based automatic decomposition models integrated into the existing Text-to-SQL pipeline to circumvent the challenge of complex SQL generation.

The remainder of the paper is structured as follows. In Section 2, we describe background and related work. We then explain the gold standard created to evaluate crowd-workers performance in Section 3 followed by illustrating the steps in crowd-powered decomposition, Section 4. We answer the research questions in Section 5. We discuss the implication, outlook, and threats to validity in Section 6. Finally, in Section 7, we pinpoint conclusion.

Context Independent Text-to-SQL Parsing: Generating SQL queries from natural language questions has been an active field of study for a long period in both database and NLP communities[1, 33, 46, 48, 57, 81, 84, 94, 95]. Previous Text-to-SQL parsers employed either expert-designed rules[72, 75, 85] or statistical techniques[42, 74, 94]. Over the past few years, driven by the development of a large in-domain context-independent dataset, WikiSQL[98], many deep learning models proposed by researchers have shown promising results for this task[30, 32, 69, 98]. All of these studies focus on mapping a single query to the corresponding SQL query which is known as context-independent parsing. Deep learning models generally adapt an encoder-decoder framework to solve the Text-to-SQL problem as a sequence-to-sequence problem[18, 20, 41, 69, 98]. To show and test the limitations of the Text-to-SQL models on generalizability on various domains and databases, Yu et al. [92] proposed a complex cross-domain dataset called Spider. In addition to the sequence-to-sequence paradigm, namely the generation-based methods, state-of-the-art neural models leverage more strategies such as sketched-based techniques (generates a SQL skeleton first and then fills the skeleton with database schema tokens)[13, 19, 32, 38, 51, 52, 86, 89, 98], data augmentation[78, 84], various attentional architectures for question/schema encoding such intermediate representation for decoding[27, 29, 34, 90], graph representation of databases in schema encoding[5, 6, 10, 12], schema linking (correctly identify column and value mentions in an natural language questions and link them to the given database schema)[6, 7, 17, 18, 21, 29, 45, 49, 50, 65, 77, 89].

While there are some attempts to tackle the complex SQL generation issue, it is still a significant challenge for Text-to-SQL models[24, 45, 92]. For instance, schema linking methods by capturing the alignment between text and table indirectly address this challenge. On the other hand, intermediate representation approaches are designed to bridge the gap between text and SQL. Furthermore, some studies have examined decomposing complex SQL queries within the decoder to generate multiple clauses and sub-queries[43, 90]. Unlike these studies, in this work, we investigate the potential performance gain by adding a decomposition stage in the Text-to-SQL pipeline to decompose complex natural language questions before submitting them to Text-to-SQL models.

Context Dependent Text-to-SQL Parsing: Recently, context-dependent Text-to-SQL parsing has drawn a lot of attention. Compared to benchmarks with single-turn questions, ATIS, a simple in-domain context-dependent benchmark, was proposed first. The models evaluated on ATIS leveraged the sequence-to-sequence framework[8]. Later, to overcome the lack of generalizability of models, two large-scale context-dependent datasets were introduced for the Text-to-SQL task, SParC[93] and CoSQL[91] modelling conversational dependencies between questions. The Text-to-SQL models, also known as conversational Text-to-SQL models, require understanding the context of sequentially related questions compared to single-turn models. Several studies were conducted on these two benchmarks that proposed EditSQL[96], IGSQL[8], IST-SQL[79], R 2 SQL[37], RAT-SQL-TC[47] models. In addition to employing strategies in the previous section to tackle the problem of translating Text to SQL, these models track dialogue states to generate SQL queries according to the context. Li et al. [47] conducted an exploratory study within context-dependent parsing to determine how far we are from effective context modelling. In this work, we employed R 2 SQL as the baseline to assess the accuracy gain of decomposing complex questions in the Text-to-SQL pipeline. It was the first open-source context-dependent model in the SParC leaderboard 1 at the time of carrying out the experiments in this paper.

Text-to-SQL Datasets: The growing interest in Text-to-SQL applications has led to various datasets including in-domain datasets ATIS[15, 39], GeoQuery[57, 94], Restaurants[57, 71], Scholar[39], Advising[24], Academic[46], Yelp[87], IMDB[87] which have been studied for decades. WikiSQL is among the first large-scale datasets with relatively simple questions and single tables extracted from Wikipedia. Although WikiSQL contains 80654 questions and SQL pairs for 24241 databases, it is generated from a limited set of templates and only covers the single SELECT column, aggregation, and WHERE clause. Furthermore, keywords like JOIN, GROUP BY, and ORDER BY are not included. The family of Spider datasets, Spider[92], SparC[93], and CoSQL[91] contain the most difficult questions having nested queries, covering many SQL syntaxes, and multiple table joins. These datasets evaluate the Text-to-SQL models to generalize not only new SQL queries and database schemas but also new domains. Spider contains 10,181 questions and 5,693 unique complex SQL queries on 200 databases with multiple tables, covering 138 domains. It also supports a wide range of SQL syntax. Due to these reasons, we examined our proposed solution on the development set of the Spider dataset.

Data Annotation & Crowdsourcing: Natural Language Processing research has been spurred on by the growing number of annotated corpora[36, 80, 97]. Such corpora are leveraged to train, evaluate, and compare NLP algorithms. However, annotating data is an expensive and time-consuming process[60]. The emergence of crowdsourcing [23] platforms such as MTurk 2 has led to a widespread adoption of crowd-powered workflows to create annotated corpora[4, 11, 14, 28, 44, 53, 54, 55, 56, 61, 66]. Crowdsourcing has been shown to be a cheaper and faster alternative compared to expert annotation[25, 76]. In addition to data labelling, crowdsourcing proved to be a reliable approach in many tasks[31, 53, 62] and domains[63]. Although much research is conducted to quality control and quality assurance in crowdsourcing[16, 26, 35], several studies have also shown the benefit of employing experts to provide higher quality labels[2, 83]. Prior works have proposed augmenting crowd worker labels with those from experts to optimize the cost and quality of data labelling[40, 58, 64, 88]. We employ both domain experts and crowd workers for data annotation in this work. Our findings suggest the potential benefit of leveraging crowd workers to create training data and then build ML-based decomposition model in the future.

This section introduces the steps for developing a gold standard for the decomposition, creating SpiderDec serving as the oracle decomposition. We then employ a Text-to-SQL model to assess the potential accuracy boost by decomposition. Note that the accuracy is measured based on comparing the execution result of each SQL query with the corresponding gold query.

SpiderDec, Extension of the Spider Dataset: In the Spider dataset, data is split into training, development, and a hidden test set. We manually decomposed the questions, and corresponding queries within the development set of the Spider dataset on questions with hard and extra hard SQL queries, thereby creating SpiderDec3. State-of-the-art Text-to-SQL models have over 80% execution accuracy for SQL queries with easy and medium hardness levels, while the performance is less than 50% for hard and extra hard SQL queries. So, decomposing hard and extra hard questions into multiple easy and medium questions can lead to a higher accuracy of Text-to-SQL pipeline. For simplicity, we refer to the hard and extra hard questions of Spider dataset as complex questions. We limited our approach to the development set first to explore the potential benefit of the decomposition task; we leave the annotation of the training set as future work in case of accuracy boost in the Text-to-SQL pipeline. Our rationale behind considering the Spider dataset as a lens to circumvent the problem of complex user questions is governed by the scale and diversity of the Spider dataset compared to others. Furthermore, the cross-domain setup of Spider allows Text-to-SQL models to use different databases for training and testing.

Within the Spider dataset, in total, there are 332 complex training examples over 20 databases. Each example consists of a natural language question and its corresponding SQL gold query. In the remainder of the paper, we refer to each instance in the dataset as a pair of the NL question and the SQL query. We annotated Spider development set in two stages: sub-SQL annotation and sub-question annotation. As the complexity of the Text-to-SQL task derives from the underlying SQL queries, we created SpiderDec from a SQL-centered perspective, first annotating SQL queries and then questions.

Sub-SQL Annotation: In the SpiderDec decomposition, we first broke down each complex SQL query into multiple subsequent easy or medium SQL sub-queries. Based on our rubric inspired by prior work[92], each sub-SQL meets one of the conditions in Table 1 to be considered as easy or medium. Among 332 pairs of the NL question and the SQL query, 26.8% of SQL queries contain these keywords: EXCEPT, UNION, and NOT IN on which decomposition to medium or easy is not applicable. Therefore, we only decomposed their nested sub-queries into simpler ones and kept the keyword without the necessity of having all the SQL sub-queries with the easy or medium level of difficulty.

Table 1:Criteria to identify whether a SQL query is easy or medium used as a guideline for decomposition.

Sub-Question Annotation: Given decomposed SQL sub-queries per SQL query from the previous stage, we assigned a natural language sub-question to each of the annotated sub-queries. In order to determine whether sub-questions are semantically equivalent to their associated complex questions, two experts manually evaluated them and resolved any conflicts with each other.

Assessing Accuracy of SpiderDec: To investigate the potential accuracy boost achievable by adding the decomposition stage to the Text-to-SQL pipeline, we are required to measure the performance of Text-to-SQL models on the newly generated dataset. Instead of complex questions, we gave decomposed sub-questions to pre-trained models as input data. We then calculated the execution accuracy gained on the entire development set and separately per hardness category. To this end, we leveraged R 2 SQL[37], a context-dependent BERT-based Text-to-SQL model trained on SParC [93] dataset. We then assessed the execution results of predicted sub-SQLs by R 2 SQL and compared them with the result obtained from the original development set of Spider (existing Text-to-SQL pipeline). R 2 SQL can effectively model contextual questions and database schemas. SParC dataset is built on top of the Spider, providing rich contextual phenomena and thematic relations between the questions. Because the sub-questions are thematically dependent on each other acting as contextual utterances, we adapted the context-dependent Text-to-SQL model, which maps the entire sub-questions to the corresponding SQL queries. Furthermore, the R 2 SQL is the first open-source model on the leaderboard at the time of experimenting 4.

We now describe our crowd-powered study in more detail. We go over the annotation tool, the task, participants, the workflow. We then explain our measurement to evaluate participants’ decomposition.

Annotation Tool. We developed an annotation tool on top of the R 2 SQL pre-trained model for crowd workers to decompose complex questions. We first created a Text-to-SQL API from R 2 SQL, translating contextual natural language questions into SQL queries. We leveraged Vue.js JavaScript framework 5 for the frontend and Flask 6 for the backend. Within the annotation process, the question to be decomposed and its associated database are presented to participants. They can easily interact with tables, search an item, sort rows, and scroll them. They can also execute the predicted SQL corresponding to their sub-questions.

Task. In creating SpiderDec, we decomposed SQL queries. We then assigned NL questions to the queries (SQL-centered decomposition), while the crowd workers only access the NL questions and decompose them (question-centered decomposition). In the real-world scenarios, we do not necessarily have the gold SQL queries and labeled data, so we designated our crowd-powered study to investigate the feasibility of decomposing natural language questions and explore to what extent the question-centered decomposition result in accuracy boost compared to SpiderDec.

Participants. In our study, participants included SQL experts and non-expert crowd workers. We hired five computer science students with at least two years of experience with SQL as experts. Due to the high cost of hiring experts, we limited the number of experts to five students. Since the number of students was below sufficient samples to carry out statistical comparisons, we assigned them the entire set to gain more insight into their decomposition performance and quality. Each student spent between 12-30 hours for the whole corpus in the development set of Spider. According to the institutional regulations, the participants were paid between 22-30 €per hour based on their course credits. In addition to experts, 83 non-experts were employed through the Prolific Academic Platform.7 With the Prolific platform, we required the participants to (i) have at least 100 accepted Prolific task submissions, (ii) to be native English speakers, and (iii) and have a minimum approval rate of 90%. The study took approximately 50 minutes to decompose six complex questions. We also paid our participants 7.5 £(9 €) per hour for the experiment. For simplicity, we refer to Prolific participants as non-experts in the remainder of the paper.

SQL Knowledge. We measured the SQL knowledge of participants in a post-test conducted right after the decomposition task to avoid cognitive biases[22]. To this end, we manually designed our survey as no standard SQL assessment test is available in the literature. The survey took 10 minutes to complete and consists of 10 questions. First, participants were asked one question to self-report on their SQL proficiency, followed by seven questions regarding the key concepts of SQL 8. Inspired by prior work[67, 70, 82], we employed the modified VKS test to measure participants’ knowledge across four levels. Our questions are related to key concepts of SQL, which are used in our dataset, including relational databases, primary key, foreign key, SELECT statement, WHERE clause, JOIN tables, and Aggregate functions. Participants were asked to write their concept definitions for levels (3) and (4). Finally, participants were given a simple question, Write a SQL query that returns the name of the 3 youngest winners across all matches found in the table matches., with a schema of the database to write down a SQL for. This question helps us to investigate their knowledge in practice.

I don't remember having seen this term/phrase before.

I have seen this term/phrase before, but I don't think I know what it means.

I have seen this term/phrase before, and I think it means .

I know this term/phrase. It means .

English Proficiency. In addition to SQL proficiency, we hypothesized that the participants’ proficiency in reading and writing could affect their performance. Decomposition a NL question first requires understanding the questions -associated with participants’ reading skill- and then paraphrasing them in multiple sub-questions- connected with participants’ writing skill. Therefore, we leveraged the self-assessment grid of CEFR scales, Common European Framework of Reference for Languages: Learning, Teaching, Assessment 9. CEFR is an international standard describing reading, listening, speaking, and writing skills on a six-point scale, starting from A1 as a beginner to C2 as a master. Our task was only dependent on participants’ reading and writing skills; we, therefore, included the self-assessment questions of these skills. As we included native English speakers in the study, we assumed their knowledge level is above A2 and excluded A1 and A2 from the options. In total, the participants answered three questions related to their reading and writing skill in English.

Figure 2:Overview of the flow of the user study and SpiderDec creation.

Workflow. When participants entered the study, a 15-minute tutorial video provided information about their task and how to interact with different annotation tool components to decompose questions. All explained concepts were simplified, avoiding any technical burden for participants. Furthermore, the study was also elaborated on two examples within two stages: 1) how to decompose a complex question into multiple sub-questions and 2) how to work with the annotation tool. Subsequently, the participants moved on to the training phase, where they were given those two examples again to work with the annotation tool and learn the task in practice. Participants could stay in this stage as long as they wish to. Participants were then redirected to the actual decomposition task by clicking the respective button in the training stage— all 332 complex questions from the development set of Spider were randomly assigned to experts, while non-experts were given six questions. They could also skip a question if they were not certain about how to decompose it.

The experiment ended with a post-test where the SQL knowledge survey and English proficiency self-assessment were given to participants. We set these surveys as the post-test to avoid cognitive biases[22] such as Anchoring Effect-where the participants may overlay focus on answering the survey question rather than the actual task-, Overconfidence or Optimism Bias- where the participants overestimate their ability to perform the task when they can answer all the questions in the survey-, and Loss Aversion Bias- when the participants suspect that the answers to the questions may affect their payment. Lastly, we included five questions regarding the annotation tool and tutorial,10 their experience working with our annotation tool, and the perceived performance. The workflow corresponding to data annotation by participants is illustrated in Figure 2.

In total, experts provided us with 1515 sub-questions. These sub-questions are associated with 623 decomposed questions, indicating that each question on average contains 2.43 sub-questions. Non-experts created 1082 sub-questions in total for 453 decomposed questions, showing on average 2.37 sub-questions per question.

Table 2:Example evaluation of a participant's sub-questions.

Assessment of Sub-questions. We manually evaluated whether sub-questions were semantically equivalent to questions or not. Among 1515 sub-questions provided by experts, we randomly sampled 312 questions with the confidence interval of 95% from the population size while we evaluated all sub-questions generated by non-experts. As we assured the quality of data generated by experts, we randomly sampled experts’ sub-questions rather than checking all of them. We labeled the entire block of sub-questions as either correct, partially correct, or incorrect. Examples of sub-questions and the labels we assigned to them are provided in Table 2. We employed the following criteria to judge the equivalency of sub-questions to the original complex question.

Correct. If a participant's sub-questions include all concepts that appear in a question, it indicates that the question is semantically equivalent to the sub-questions. In that case, the entire block of sub-questions is assigned the highest score of 2.

Partially Correct. If the participant's decomposition misses one concept from the original question, a score of 1 is given to that. For example, if the question asks about the name and birth date and the participant only included name, we labelled the decomposition as 1.

Incorrect. Sub-questions that are either entirely incorrect, incomplete, or missed more than one concept from the original question.

Assessment of Sub-SQLs. To examine whether participants’ decomposition leads to the Text-to-SQL pipeline performance boost, we employed our baseline, R 2 SQL, which is also in line with our approach in Section 3. First, the baseline predicted the sub-SQLs. We then compared the execution result of the block of sub-questions with the execution result of the gold SQL query. We labeled the sub-SQL's either correct or incorrect. The block of the sub-SQL's is correct if its execution result is equivalent to the execution result of the gold SQL; otherwise, it is incorrect.

5.1 Performance of the baseline on SpiderDec (RQ1)

5.2 Performance of Crowd Workers on Decomposition Task (RQ2)

This study shows that decomposition leads to the accuracy boost for the Text-to-SQL pipeline on complex questions. This can begin to shed light on the influence of decomposition as a promising approach in improving the accuracy of Text-to-SQL tasks. As the follow-up study, we can build a fully automatic ML-based decomposition model integrated into the existing Text-to-SQL pipeline. For training such a model, it is crucial to collect a substantial amount of labeled data, such as decomposition of the family of Spider dataset. Our findings support the evidence of employing crowd workers for this task as a scalable method. According to our error analysis in 5.2, a fully automatic decomposition model might face several challenges. Among five error groups discussed in 5.2, the major challenges for automatic decomposition can be error types of Complex Sub-Questions and Missed One Keyword. Circumventing the challenge of Complex Sub-Questions is difficult since verifying the sufficient level a question should be broken down is difficult for even a human. There is a trade-off between the granularity of sub-questions and their complexity. The fine-grained the sub-questions, the less complex sub-questions we have. However, we might oversimplify questions that are not required at all. In terms of Missed One Keyword challenge, we demand an attention model to identify the keywords within the questions and evaluate whether those keywords existed in the sub-questions. Many of these keywords are dependent on the schema of tables. However, the attention model in training would not be enough.

Caveats and Limitations. We had an imbalance number of SQL experts, restricting us from gaining insight into their performance individually and employing any statistical tests. We also did not consider workflows to optimize decomposition such as aggregation of crowd-workers’ answers and double-checking their decomposition answers by experts, which means that it would be possible to achieve higher accuracy than what we observed in our work when optimized. Furthermore, we only leveraged one Text-to-SQL model in our study. Although our decomposition approach is independently defined of any Text-to-SQL models, a comprehensive analysis of state-of-the-arts Text-to-SQL models can give us a better insight into the impact of decomposition on different models.

This paper explores the feasibility of decomposing complex user questions within the Text-to-SQL pipeline as a means to circumvent one of the significant shortcomings of Text-to-SQL models in complex SQL generation (RQ1). We first adapted the decomposition on the development set of the Spider dataset, breaking complex questions down into simpler sub-questions in a way that Text-to-SQL models can convert them correctly to corresponding SQL queries. We then investigated the feasibility of leveraging crowd workers to produce sufficient training data for building a ML-based model decomposing complex questions automatically (RQ2).

We defined the decomposition task for complex questions in which a complex question is split into multiple subsequent sub-questions. Having assessed the decomposition approach on complex questions in Spider dev. set (SpiderDec), we found that the accuracy raised remarkably from 30% to 79%. Our results support the evidence of decomposition as a promising approach to boost the performance of existing Text-to-SQL pipelines.

We then examined the performance of 88 crowd workers on decomposing the natural language questions within the development set of Spider. Compared to the accuracy boost of 153% (30% to 76%) as a result of the decomposition carried out by a small group of SQL experts (N = 5), decomposition by non-expert crowd workers (N = 83) led to an accuracy boost of over 96% (30% to 59%). Our findings show that crowd workers can effectively decompose complex user questions and thereby aid in creating training data at a beneficial scale for generalization of decomposition in Text-to-SQL pipelines.

This research has been supported by ICAI AI for Fintech Research.

1https://yale-lily.github.io/sparc

2https://www.mturk.com

3https://github.com/sarasal/decomposition

4https://yale-lily.github.io/spider

5https://vuejs.org

6https://flask.palletsprojects.com/en/2.0.x/

7https://www.prolific.co

8https://www.interviewbit.com/sql-interview-questions/#sql

9http://ebcl.eu.com/wp-content/uploads/2011/11/CEFR-all-scales-and-all-skills.pdf

10https://www.ueq-online.org