Salehi et al. [53] proposed a workflow for complicated writing tasks where workers first post their questions regarding the task to the requester. After discussing the questions with the requester, workers write a draft. Then, the requester rates the drafts and discusses them with the workers. The workers, in turn, edit the paragraphs based on the ratings and discussion and submit the final paragraph. This workflow is expensive in terms of time and money. Moreover, it highly relies on the workers’ communication and relation to the requester as well as on a well-structured feedback mechanism which seems to be challenging to achieve. Similarly, Manam et al. introduced TaskMate, a workflow that depends on the workers to improve the clarity of task descriptions. TaskMate offers a collaborative setting in which workers identify the ambiguities of a task description in the form of questions and suggest several answers for each question, other workers vote for the answers that most probably clarify the ambiguities, and a clearer task description is then created to be done by the workers [47]. This approach reduces the effort for requesters to create clear instructions. It puts all the responsibility on workers, assuming that they collaborate well together. Its impact thus depends on how reliable the workers are in terms of collaboration and general work quality. Gaikwad et al. [28] defines a workflow called Daemo which offers requesters to post several instances of their task, receive feedback from workers, on which basis they can optimize their task description. Experiments indicate that this method is effective in

Note 1 The Data and experiment code are available here: https://osf.io/m8njv/

principle. Despite the cost in terms of time and money for this pilot step, however, the approach may not be suitable for large crowds with diverse backgrounds and skills, due to its reliance on the subjective judgements of a limited number of workers submitting their feedback. In contrast, the computational approach we present in this paper proposes an automated tool that offers a faster and cheaper workflow on the requester’s side, since it fully removes the workers from the process. The intended workflow is independent from various challenges discussed in previous work [50], ranging from low-quality results submitted by workers, complications of poor requester-worker relationship, and the lack of a proper feedback system in the process. If helpful, it may still be followed by workflows or interventions that include the workers’ opinions.

A number of researchers have proposed models of what task clarity means as well as methods to analyze task clarity. Among these, Gadiraju et al. [27] proposed a computational model, which stresses predictive features such as role clarity and goal clarity as the major aspects of task clarity in crowdsourcing. Additionally, the authors presented a means to quantify clarity. Wu and Quinn [58] measured the effect of guidelines on workers’ perception of task quality and, consequently, on the outcome of the task including accuracy, throughput, trust, and worker satisfaction. Papoutsaki et al. [51] studied inexperienced requesters in a data collection task; how they design a task, what dimensions they consider, and similar. Their analysis contains helpful lessons for novice requesters. Khanna et al. [38] showed that the user interface, task instructions, and the workers’ cultural background prevent workers with low digital abilities from accepting and doing tasks on Mturk. They recommend to simplify the user interface and task instructions, along with language localization to harness the usability of workers. Finnerty et al. [22] provided evidence that clear instructions and a simpler task design which encourages the workers’ focus and awareness leads to higher quality results. To build computational methods, we extended the clarity aspects discussed by Gadiraju et al. [27]. Accordingly, we annotated the dataset they used in their work in order to investigate the assessibility of descriptions’ clarity flaws by only processing the plain text of the descriptions. We believe that the trained classifiers we present below will advance the development of assistance tools that are not only helpful for inexperienced requesters to improve the clarity of their task descriptions, but also for workers to better understand tasks and, hence, to deliver more accurate results.

Manam and Quinn [48] introduced WingIt, a system that relies on the workers’ assumptions and intuitions about the task requirements as well as the requesters’ desired result in the case of ambiguous task instructions. WingIt enables workers to either ask the requester for clarifications in the form of questions (“Q&A”) with the best answer that clarifies the ambiguity, or directly revise the instructions (“Edit”). The worker either waits for requester’s response within three minutes (synchronous) or submits the result

assuming the requester confirms the answer (asynchronous). Another approach is implemented in the SPROUT system [11] which organizes confusing questions and utilizes suggestions from crowd workers to edit ambiguous parts of task instructions. It provides requesters with the questions and allows them to prioritize the set of questions. These tools can potentially involve inexperienced workers and require additional time from workers and costs from requesters. Yet, the risks of misunderstandings and wrong perceptions of workers remain, which may consequently lead to rejection and a bad reputation. Kulkarni et al. [44] introduced a system called Turkomatic which works based on a price-divide-solve algorithm. Via step-by-step guidance, it utilizes the crowd to decompose and solve complex tasks provided by requesters. Turkomatic requires knowledgeable workers, supervision from the requesters, and a close feedback mechanism to be successful. With a similar idea, Chang et al. [14] proposed a collaborative system called Revolt which focuses on image-labeling tasks with vague or incomplete instructions. In Revolt, multiple workers are allowed to label the task with the given steps and have access to the description written by other workers. In case of a conflict, workers relabel the image according to other workers’ descriptions. Also, the system Fantasic [30] tests a task design to help novice requesters. It collects task requirements from requesters to then create and show a task description before posting on the platform, but this is limited to a narrow set of task types. Other systems include Soylent, a plug-in for Microsoft Word through which with the help of workers documents are edited, shortened, and proofread by hiding the complexity of task specification [10], as well as CrowdForge [42], and Crowd4u [34]. The two latter help to decompose complex tasks written in natural language into small tasks for crowdsourcing platforms, but they lack generalizability to all types of task specifications in crowdsourcing. TurKit, finally, helps requesters with iterative task deployment on MTurk [46]. Its architecture avoids receiving redundant submissions by saving intermediate results. TurKit makes the assumption that the way tasks are broken down will be determined by requesters in all cases. In contrast, we develop natural language processing methods to automatically detect clarity flaws in task descriptions independent of workers’ involvement and the necessity of their communication with requesters. In future, these automatic methods may be deployed to develop a tool by which requesters can identify and improve their task description’s clarity weaknesses before its publication on the platform. By avoiding worker interaction in the process, it is not only more efficient in terms of time and cost, but it may also increase effectiveness, because it is not affected by the various challenges arising out of workers’ and platforms’ involvement along with the complications that workers face with requesters in the process [50]. This may particularly help inexperienced requesters to learn what information is necessary for creating complete and unambiguous task descriptions.

With task clarity we refer to a twofold property of crowdsourcing task descriptions, which not only influences the level of comprehensibility and completeness of the instructions written in natural language, but also determines the extent to which information required for delivering a high-quality result to the task is provided by its requester. In addition to determining participation criteria and laying down eligibility constraints for crowdsourcing tasks (in terms of reputation, experience, demographic variables, language proficiency, and similar), task clarity is primarily shaped by the task design of requesters in crowdsourcing marketplaces. In particular, problems with task clarity may occur due to the inexperience of task requesters, who may lack an understanding of the diversity among target workers in terms of their background knowledge, skills, demographics, and culture. Similarly, they may lack sufficient awareness of the importance of a thorough task design and its direct influence on the quality of the submissions from workers. Unclear task descriptions can lead to inaccurate or incorrect responses from workers, which can in turn lead to task rejection and distrust between requesters and workers. Different researchers have addressed challenges pertaining to task clarity, and studied dimensions that lead to instructions being perceived as vague or incomplete. Among these, Gadiraju et al. [27] discussed goal clarity and role clarity as main dimensions of microtask descriptions. These terms refer to what is expected to be delivered from workers and how the work is planned to be done, respectively. Moreover, Wu and Quinn [58] introduced the notion of descriptive metrics and prospective metrics of task descriptions. For identifying the task clarity flaws, descriptive metrics are of particular interest. They include (a) the vocabulary or language used to describe the task, (b) the specification of the data that is expected to be delivered by crowd workers, and (c) the order of the steps that should be carried out in a task and the solution to be submitted. Prospective metrics, on the other hand, refer to the task properties which are more subjective relating to the workers’ personal feelings. Such metrics play a role when it comes to workers’ trust, confidence, and prediction of outcomes rather than their comprehension of the task, influenced directly by task descriptions in general. 2

Finally, other information adhering to best practices for a complete task instruction includes the interface on which the work should be performed, the expected format of the solution, and the specification of acceptance conditions [58].

Based on the prior studies (Section 3.1) on the characteristics of incomplete and unclear tasks, we formed a set of clarity flaws that

Note 2 Occasional exceptions may occur for tasks where the concrete object of investigation is specifically known to a worker. However, we are interested in the general form of a description rather than its concrete object of investigation.

define the basis for annotation guidelines in our dataset creation in Section 5, We here propose to model clarity by assessing the following clarity flaws. The first one, description unclear, can be understood as an overall unclarity assessment, while the remaining reflect sub-dimensions of unclear task descriptions:

(1) Description unclear. The task is unclear, i.e., it is not fully understandable how one can complete the task successfully and/or what the desired solution is. This refers to overall unclarity. (2) Difficult wording. The words and the grammatical constructions used in the task descriptions are not fully comprehensible. (3) Important terms undefined. Some terms that are potentially important to properly understand the tasks are not defined sufficiently. This refers to the vocabulary used for the description [29]. (4) Desired solution unspecified. The solution that is actually desired to be submitted in response to a task is not explained in sufficient detail. This refers to the goal clarity property. (5) Solution format unspecified. The format in which the solution should be submitted is not specified sufficiently. This refers to the required detailed information regarding the goal clarity. (6) Steps unspecified. The steps that need to be carried out one after another to complete a task are not defined sufficiently. This refers to the role clarity property. (7) Resources unspecified. The resources that are required to be used to complete the task are not sufficiently specified. Resources may include data, tools, links, websites, etc. This refers to the necessary data and links to perform the task. (8) Acceptance criteria unspecified. The acceptance criteria on which basis a requester decides about the acceptance of a solution submitted to a task are not sufficiently specified. This refers to the information which can help decide how much time to spend for performing the task and how much the work is rewarded [43].

In this paper, we employ two approaches to compare their effectiveness for clarity assessment in order to address Research Question RQ1 from Section 1: (a) we rely on transformer-based neural models which have been shown to be superior in a variety of natural language processing tasks, (b) we use linear SVM classifiers based

on six feature types, since they are found to be effective if data is limited. For neural approach, we rely on the widely used BERT model [18]. We explore two common variations of pre-trained BERT namely Bert-base-uncased, a case-insensitive model trained on lower-cased English text, and Bert-base-cased, a case-sensitive model trained on English text in its original format. Both variations have 12 layers, 768 hidden nodes, 12 heads, and almost 110 million parameters [3]. For feature-based approach, we collected six specific feature types based on which the classifiers assess the task clarity. In the following, we introduce the feature types in details.

Feature-based classification can provide detailed insights into the textual properties that are helpful in assessing each clarity flaw, allowing us to address RQ2. Concretely, our linear SVM classifiers are based on the following six feature types. These types cover both, standard features that have often been used in natural language processing and clarity flaw-specific features that we engineered based on the well-known aspects of task descriptions [27]:

(1) Content. Content is important in many text classification tasks. Consistent with common practice, we examine the effect of content-related properties via term frequency–inverse document frequency (TF-IDF) where we consider all lowercased token 1- to 3-grams including stop words as terms. (2) Length. To test whether clarity correlates with length, we include 26 features that reflect the extent of a task description. They cover the numbers of all characters, letters, digits, punctuation marks, whitespaces, non-whitespaces, unique words, words, fully upper-cased tokens, fully lower-cased tokens, capitalized words, phrases, and sentences. Additionally, we computed the mean of all counts per sentence (except for sentences). (3) Style. Clarity may be considered a property of style. We hence model style, namely via part-of-speech 1- to 3-grams and phrase 1- to 3-grams (created using the NLTK library [1]) as well as characters 3-grams and the functional words. For the latter, we consider the top-100 most frequent lower-cased words in the whole corpus. (4) Subjectivity. Subjective phrasing of task instructions have been shown to affect perceived task clarity [27]. We capture subjectivity using the Textblob library [8] which computes a subjectivity score, a polarity score, a negativity score, a positivity score, and an objectivity score for a given text. (5) Readability. As mentioned in Section 2, readability metrics have been used for clarity assessment. We consider Flesch- Kincaid Grade Level, ARI, Coleman-Liau, Flesch Reading- Ease, Gunning-Fog Index, LIX, SMOG Index, RIX, and Dale- Chall Index. All readability metrics are computed via the Pypi library [4]. (6) Flaw-specific. In line with the clarity flaws from Section 3, we hypothesize that the clarity of task descriptions is reflected in the completeness, in terms of resources and acceptance criteria, as well as the complexity of words and terms.

To study completeness and complexity, we introduce the following eight task-specific features. The first four are binary features capturing whether a description matches the given regular expression. a. Website. Regular expression for various token 1- or 2-gram which may refer to a web resource (e.g., “web page(s)”, “webpage(s)”, “site(s)”, “web site(s)”, etc.). b. Link. Regular expression for URls and placeholder words such as “link”. c. Given time. Regular expression for token 1-grams delivering information regarding the estimated time to complete, such as “5 minutes”, “1 minute 14 seconds”, “2 min”. d. Reward. Regular expression for token 1-grams delivering information regarding the specified reward (or bonus) for a task, such as “up to $0.57 + 50% bonus = $0.85 max”, “5 cents”, “avg rwrd+bns: $2.02” e. Entity. All token 𝑛-grams detected by Spacy [6] as locations, organizations, ordinal entities, products, or similar. f. POS categories. Frequencies of conceptually similar partof-speech tags found with by the Stanford Tagger [7], such as verbs, nouns, open and close part-of-speech tags, and similar. g. Discrete words. The 10 most frequent discrete lower-cased 1-gram tokens (excluding stop words) which appear either only in clear task descriptions or only in unclear descriptions in all dimensions. h. Complex words. Two different scores for the complexity of token 1-grams computed by Pypi [4] .

For our data compilation, we built on the previously published dataset of Gadiraju et al. [27] and Difallah et al. [19], which consists of a total of 7007 records of real task descriptions published on Amazon Mechanical Turk (mTurk) from October 2013 to September 2014.3 For each task, the title, body, date of publication, and some other metadata are given. For our study, the title text, a dot (as separator), and the body of the tasks compose the task descriptions in the dataset. The task descriptions are grouped into six different task types, namely Surveys (SU), Content Access (CA), Interpretation and Analysis (IA), Information Finding (IF), Content Creation (CC), as well as Verification and Validation (VV) [26]. During inspection of the original dataset, we observed that the 7007 task descriptions contains a lot of cases that are near-copies of others in terms of being multiple instances of the same task only with specific information replaced. Since we did not expect any clarity-specific differences in these, we filtered out all near-copies in a semi-automatic process, ending up in 2663 clearly distinct task descriptions. Due to the limited budget, we decided to select a 50% sample for manual annotation (i.e., 1332 records). To preserve the full diversity of task descriptions covered, we chose the sample representative with respect to the task types in the filtered set. Table 1 shows the distribution of the task types in our dataset in comparison to the source data. The 1332 task descriptions span 31,027 tokens (23.3 tokens per description on average), and cover 25,891 unique tokens.

Note 3 We are aware that the age of the data may impact what we observe. However, we decided to favor comparability to previous work over timeliness, also because we do not see a fundamental change in task descriptions after 2014. Besides, note that obtaining task descriptions is all but straightforward.

We decided to collect annotations for clarity flaws in the task descriptions directly from crowd workers, since they are eventually the ones to benefit from improved task descriptions, making their judgment decisive. In accordance with the source of the given task descriptions, we deployed the annotation tasks on Mturk, so that our annotators match potential workers for the described tasks in principle.

Task Design. Being task requesters ourselves in this setting, we took care to avoid the clarity flaws identified in Section 3 in our task design. In our annotation task, the workers had to assess a given task description from our dataset for the clarity flaws via a survey-like form. In the beginning, general instructions were given on how to fill the form (along with a privacy guarantee), asking the workers to put themselves into the role of the worker who takes on the task corresponding to the description. Then, each flaw was described following the definitions from Section 3. To determine a suitable setting in terms of the annotation scheme to use and the number of annotators to employ, we designed and deployed the annotation tasks in two phases: first, a pilot annotation study where we explored initial design decisions on Mturk to test whether our tasks lead to the desired results with sufficient quality; and second, the main annotation study where we collected the annotations of all 1332 task descriptions after improving the task design based on the findings from the pilot study.

Pilot Annotation Study. We compared two different annotation schemes in terms of which one leads to a higher inter-annotator agreement: (a) binary classification where workers either agreed or disagreed with statements covering each clarity flaw; and (b) 5point Likert scoring where workers reported the extent to which they agreed with the statements from “1: strongly disagree” to “5: strongly agree”. We then carried out a pilot study with two batches of 12 annotation tasks (one for binary, one for Likert). Each annotation task included four task descriptions, meaning 48 descriptions in total. In addition to the flaw assessments, the workers were asked to give a summary of the task description, which we used as a quality check to see whether they carefully read the task descriptions while deciding on the labels. We estimated 8 minutes to complete each task and paid USD 1.32 per task to each worker. Each task was to be annotated by three workers with more than 1000 approved tasks (HITs) on mTurk, as suggested by mTurk to ensure the quality of annotators based on their reputation. Our analysis of the results of the pilot study revealed that the 5-point Likert scale gave workers more freedom to make more accurate judgement, while leading to higher inter-annotator agreement; for the annotated flaws, we observed full agreement ranging from 40% to 63% for the binary scheme, and from 50% to 75% for 5-point likert scoring after discarding unreliable workers. Moreover, the

summaries written by the workers illustrated the necessity to select workers more restrictively to increase reliability.

Main Annotation Study. We decided to acquire only so called master workers with an approval rate higher than 95% who are English speakers from the US, Canada, England, Ireland, Australia, New Zealand, or India.4 Each task was tackled by five annotators, spending about 5 minutes on average with an hourly wage of around 10 USD per task. We refined the quality check by replacing the summary text field with two text fields for other problems that the workers potentially find with the description (an optional field) and a brief suggestion for improving the task descriptions (a mandatory field).5 Finally, we deployed all 1332 task descriptions in 333 annotation tasks, each of which included four task descriptions to be annotated. Although, there was no limits on the number of tasks an individual worker could complete, the annotation tasks were done after 10 days and 33 unique workers participated in the study.

Note 4 The title master worker is given to workers by a blackbox algorithms of the mTurk platform based on the workers’ performance.

Note 5 For the study at hand, we used these texts only evaluate the reliability of workers, but an analysis of the problems and suggestions given by the workers may be interesting in future work.

The distribution of the collected annotations for the clarity flaw Description unclear turned out to be unexpectedly skewed toward “strongly disagree”. We found three workers who had participated in annotating more than 150 task descriptions and had selected strongly disagree for all the statements of more than 95% of the task descriptions. To improve data quality, we discarded all assignments of these workers, still remaining with at least three annotations for all task descriptions. To obtain a single final annotation from the remaining annotations of each tasks, we relied on the multi-annotator competence estimation (MACE) [32]. MACE was designed particularly for crowdsourcing settings, where standard inter-annotator reliability measures such as Fleiss’ 𝜅and Krippendorff’s 𝛼do not apply due to varying annotator sets. It grades the reliability of workers based on their agreement with others and allows deriving

Figure 2: Pearson’s r correlation coefficient for each pair of the annotated clarity flaws in our dataset. The medium correlations suggest that all specific flaws add to some extent to overall unclarity (Description unclear), but none of them decides it alone.

Note 4 The title master worker is given to workers by a blackbox algorithms of the mTurk platform based on the workers’ performance. 5For the study at hand, we used these texts only evaluate the reliability of workers, but an analysis of the problems and suggestions given by the workers may be interesting in future work.one aggregate annotation for each instance on this basis.6 The competence value of the annotators ranged from 0.01 to 0.97. While the average was only 0.13, the top five all had a confidence above 0.32. In addition to the resulting MACE Likert scores from 1 to 5, we also established binary class for each task description, where we consider “strongly agree” (5) and “rather agree” (4) as Positive, and “partly agree/partly disagree” (3), and lower as Negative. Table 2 shows the distribution of scores in the consolidated dataset for each clarity flaw from Section 3. We observe that the distribution is slightly skewed towards lower scores in general, but that the whole scale is covered reasonably in most cases. Particularly, the binary classes are reasonably balanced. For development and evaluation, we split the dataset based on the tasks’ publication date on mTurk, in order to simulate the idea of unseen future tasks in testing. The training set contains 666 task descriptions (50%) with 15,128 tokens, the validation set contains 333 descriptions (25%) with 7,821 tokens, and the test set also contains 333 descriptions (25%) here with 8,078 tokens.

Given the final Likert scores, we carried out a correlation analysis for all clarity flaws, in order to roughly assess whether they can be distinguished and how well they predict that a description is unclear (overall unclarity). Figure 2 shows the Pearson’s 𝑟correlation coefficient for each pair of flaws. Only few flaws correlate strongly with each other, the highest coefficient being observed for Difficult wording and Important terms undefined (0.75), which makes sense. Also, it seems intuitive that the flaw Solution format unspecified often goes hand in hand with Steps unspecified and Acceptance criteria unspecified (both 0.70). The majority of correlations is rather medium, roughly between 0.5 and 0.6. An important observation is that none of the seven specific clarity flaws is highly correlated with Description unclear, suggesting that all of them may somewhat add to overall unclarity. An uneasy wording seems to have

Note 6 For comparison, we also explored the use of majority voting instead, using the rounded mean Likert scores, in case no majority exists. However, we decided for MACE, since it not only accounts for annotator reliability, but also the distribution of scores turned out to be notably more balanced.

a rather high impact (0.59), whereas a missing specification of the desired solution seems a little less important (0.49)—as far as single correlation coefficients allows such an inference.

We evaluated the following configurations of the given approaches and two baselines in our experiments:

Transformer-based Models. We study how effectively the task clarity flaws can be assessed using eight BERT-based binary classifiers, one for overall unclarity and seven for the other clarity flaws. In each case, we used the pre-trained Bert-base-cased and Bertbase-uncased models, and we employed the PyTorch library [5] to conduct the experiments on the BERT-based classification. We first preprocessed the texts of the task descriptions using BertTokenizer for both pre-trained models. Then, we converted the preprocessed texts to the required data type for each model, respectively. We adapted BertForSequenceClassification to the binary-label setup and used the BertAdam optimizer to fine-tune the parameters with learning rate of 2−5 and a warmup of 0.1. Using the training set, we tuned the models for four epochs and computed the test set accuracy for the Bert-base-cased and Bert-base-uncased classifiers separately.Feature-based Models. Specifically to examine what textual properties of task descriptions are helpful for the computational assessment of task clarity, we used binary SVMs with the six feature types from Section 4.2. We used the scikit-learn library [2] to conduct the experiment on feature-based classification. To account for the imbalances in the data distribution, we randomly resampled the training dataset for each classifier independently. Then, we trained eight linear SVM classifiers, again one for overall clarity assessment and seven for the other flaws. We did this once for each of the six feature types alone, for each feature ablation (all but one feature type), and for all features together. We optimized the cost hyperparameter of each classifier on the validation set (tested range: 2𝑖for −10 < 𝑖< 10). Finally, we computed the test set accuracy score of each best validation set configuration on the test set.

Baselines. We compare the approaches simply to a majority baseline, which always predicts the majority training class. Thereby, we make visible where actual learning success is achieved. For some test sets, the majority class is different from training. To make visible where we can definitely distinguish classes, we also show the “minority” baseline (predicting the minority training class) below, but we point out that this is not a reasonable baseline in practice.

Significance Tests. We performed a one-tailed independent 𝑡test to study whether (a) Bert-base-cased, (b) Bert-base-uncased, and (c) the SVM with all features can assess the task clarity flaws significantly better than the majority baseline at 𝑝< .05 (marked ) and 𝑝< .01 ().

Table 3 shows all test set accuracy results for overall unclarity (Description unclear) and the seven clarity flaws in task descriptions, both for the BERT models and for the SVM classifiers with single feature types (𝐴𝑖), feature ablation (𝐴𝑖), and all features (𝐴1−6). We first analyze the overall results in light of the first research question, RQ1. We evaluate the test set accuracy of the BERT models against the majority baseline to find out whether the considered task descriptions’ clarity flaws seem possible to be identified computationally. We see that Bert-base-cased and Bert-base-uncased succeed in assessing six flaws: unspecified important terms (0.69/0.67 vs. 0.31), desired solutions (0.60/0.62 vs. 0.38), solution format (0.60/0.61 vs. 0.36), steps to perform (0.56/0.58 vs. 0.43), resources (0.56/0.60 vs. 0.41), and acceptance criteria (0.57/0.55 vs. 0.44), while they fail to improve over the majority baselines overall unclarity (0.69/0.71 vs. 0.72) and difficult wording (0.71/0.71 vs. 0.75). Moreover, the hypothetical approach of predicting the minority class would lead competitive results for some flaws. Comparing the two BERT variants, we observe that the case-insensitive variant (Bert-base-uncased) performs slightly better, showing higher results in five cases. This suggests that our choice to lower-case all words for the features was appropriate. Given that the training data is not huge, the feature-based classifiers apparently benefit from their focused analysis; they reach higher test set accuracy compared to both BERT models for severalcases: unspecified desired solution (𝐴5 and 𝐴6 with 0.63 vs. Bertbase-uncased with0.62), solution format (𝐴2, 𝐴4, and 𝐴5 0.62 vs. 0.61), steps (𝐴2 0.62 vs. 0.58), resources (𝐴1 and 𝐴2 0.63 vs. 0.60), and acceptance criteria (𝐴3 0.63 vs. 0.57). For overall unclarity, the SVMs 𝐴2 and 𝐴5 perform best with 0.74 and show learning success over the majority baseline in classifying the task descriptions. Particularly the classifier without the length feature (𝐴2) seems strong in general (more on the features below). Finally, the results suggest that the clarity of having a difficult wording seems hard to assess; none of our approaches managed to beat the majority baseline (0.75). A reason may lie in the diversity of potentially difficulty words, which makes it hard to learn such words. However, the majority baseline result also shows that this clarify flaw shows a rather high distribution imbalance.

The individual feature type results in the upper part of Table 3 (𝐴𝑖) suggest that many of the considered textual properties are relevant to at least some of the clarity flaws. The content of task descriptions (measured in the form of TF-IDF) seems particularly important, achieving notably higher results than the other feature types for a number of clarity flaws, including for overall unclarity (0.72). The style of the descriptions performs best on unspecified acceptance criteria (0.63), possibly because of specific part-of-speech tags, and it is also an important indicator for an unspecified solution format in descriptions (0.60). Likewise, the readability of task descriptions helps best to identify unspecified desired solutions (0.63), whereas the flaw-specific features and the subjectivity of descriptions play an important role mostly in an ablation setting in the center part of Table 3 (𝐴𝑖). The insightful exception among the feature types is the Length (in terms of number of words, characters, digits), which achieves comparably low accuracy for all consider clarity flaws of task descriptions. This is also underlined by the feature ablation, showing that the best results overall are achieved when the length feature is left out. This finding suggests that the clarity of crowdsourcing task descriptions does not depend on their length—which is in contrast to related assessment tasks, such as predicting Wikipedia article quality [45] or argument quality [56].