1. Introduction

1The institution of distance learning paradigms, which has become crucial during the Covid-19 pandemic, showed the need to provide schools and universities with Natural Language Processing (NLP)-based tools to assist students, teachers and professors. Nowadays, language technologies are more and more exploited to develop educational applications, such as Intelligent Computer-Assisted Language Learning (ICALL) systems (Granger 2003) and tools for automated essay scoring (Attali and Burstein 2006) or automatic error detection and correction (Ng et al. 2013). A fundamental requirement for developing this kind of applications is the availability of electronically accessible corpora of learners’ productions. Corpora created so far differ in many respects. For instance, considering the types of examined learners, they can gather productions written by L2 students or by native speakers: the former have been built for many languages (e.g. English, Arabic, German, Hungarian, Basque, Czech, Italian), while the latter are mainly available for English. In both cases, a peculiarity of existing corpora is that they are cross-sectional rather than longitudinal. A notable exception in the context of Italian as L1 – which is the focus of our contribution – is represented by CItA (Corpus Italiano di Apprendenti L1), which was jointly developed by the Institute for Computational Linguistics of the Italian National Research Council (CNR) of Pisa and the Department of Social and Developmental Psychology at Sapienza University of Rome (Barbagli et al. 2016): it is the first digitalized collection of essays written by the same group of Italian L1 learners in the first two years of the lower secondary school2.

2The diachronic and longitudinal nature of CItA makes it particularly suitable to study the evolution of L1 writing competence over the two years, assuming that many remarkable changes in writing skills occur in this period. For instance, in their recent work, Miaschi, Brunato, and Dell’Orletta (2021) showed that it is possible to automatically learn the writing development curve of students: they extracted a wide set of linguistic features from the essays and used them to train a binary classification algorithm able to predict the chronological order of two productions written by the same pupil at different times.

3The present study ranks among research based on CItA, but chooses a different approach from the one just mentioned: instead of tracking the development of students’ writing competence, we focused on the perception of writing quality by Italian L1 speakers with the aim of understanding whether it is possible to find the linguistic features that are crucially involved in the distinction between ‘better’ and ‘worse’ essays according to our target reader.

Contributions

4To the best of our knowledge, this is the first paper that (i) introduces a dataset of evaluated essays in terms of perceived writing quality by means of a crowdsourcing task, (ii) deals with the correlation between linguistic features and perceived quality of writing and (iii) assesses the impact of students errors on quality perception.

2. Corpus Collection

5As previously mentioned, the starting point of our study was the CItA corpus. It comprises 1,352 essays, written by 156 pupils of seven lower secondary schools in Rome (three in the historical center and four in the suburbs) during the school years 2012-2013 and 2013-2014. The productions respond to 124 writing prompts that pertain to five textual typologies: reflexive, narrative, descriptive, expository and argumentative. An additional ‘common prompt’ was presented at the end of each school year, in which students were asked to write a letter to advise a younger friend how to compose better essays. The common prompts were aimed at understanding how learners internalize the different writing instructions given by teachers.

6Each essay contained in CItA is also provided by a set of metadata tracking students’ biographical, sociocultural and sociolinguistic information. Beyond the longitudinal nature, the most significant novelty introduced by CItA regards error annotation, which was manually performed by a middle school teacher according to a new three-level schema including: the macro-class of error (i.e. grammatical, orthographic and lexical); the class of error (i.e. verbs, prepositions, monosyllabes); and the corresponding type of modification required to correct it. More details about the CItA collection are reported in Barbagli et al. (2016).

2.1 Essay Selection

7For the purpose of our investigation, we selected 200 essays from CItA to be submitted to human evaluation. The essays ranged from a minimum of 141 tokens to a maximum of 1153 tokens and their average length was 359.4 tokens. Then, to gather judgments on writing quality, we created ten questionnaires, each one consisting of ten pairs of essays of the same grade, and distribute them to native speakers of all ages and cultural background.

8Table 1 reports the criteria we adopted to select the pairs of essays. As it can be seen, Survey 1 allows the comparison between essays responding to the common prompts written by students attending the first or the second grades. In surveys 2-8, we chosen essays pertaining to the same textual typology – assuming that their similarity with regard to the content could let the annotator focus on stylistic issue to orient their judgment – and paired them according to the school year in which they were written. Instead, essays in questionnaires 9 and 10 were paired according to their number of errors: for each year, we divided the range between the minimum amount of errors (0) and the maximum one (49 for the first year, 43 for the second one) into ten error bins and designed the two surveys choosing a couple of productions for each bin. Surveys comparing essays with a similar amount of errors were meant to understand which categories of errors have a greater impact on human judgment.

2.2 Human Evaluation

9After designing the surveys, we moved on to their implementation using the QuestBase platform3. We defined a three-section structure including the filling-in instructions, the personal data entry form and the essays evaluation pages.

Figure 1: Comparison of a pair of essays extracted from one of the ten surveys

10Filling-in instructions. The first section reported the following submission guidelines:

Ciao! Il presente sondaggio è rivolto a partecipanti di madrelingua italiana. La sua compilazione richiede circa 20 minuti. Pima di proseguire, dando il consenso alla partecipazione, ti spieghiamo in cosa consiste. Nelle pagine che seguono leggerai dieci coppie di temi scritti da studenti del primo e del secondo anno di scuola media. I testi possono contenere un certo numero di errori. Per ciascuna coppia ti chiediamo di indicare quale dei due temi ritieni sia scritto meglio. Non esistono risposte giuste o sbagliate: conta semplicemente quello che pensi! Tieni presente che i temi di una stessa coppia possono trattare argomenti diversi, ma questo non deve influire sul tuo giudizio. La tua partecipazione al sondaggio è completamente libera. Se in qualsiasi momento dovessi cambiare idea e volessi interrompere il test, potrai farlo liberamente. Un’ultima cosa: prima di iniziare il sondaggio, ti chiediamo di darci alcune tue informazioni anagrafiche, che serviranno solo a fini statistici. I dati rimarranno completamente anonimi e in nessun modo le risposte verranno associate alla tua persona. Se hai dubbi, curiosità o proposte di miglioramento, scrivimi all’indirizzo: a.cerulli1@studenti.unipi.it. Buona lettura!

11For the sake of completeness, we also report an English translation of the same guidelines:

Hello! This survey is addressed to Italian native speakers. Its submission requires about 20 minutes. By completing it, you give your consent to participation. Before going on, we explain to you what it consists of. In the following pages you will read ten pairs of essays written by Italian L1 learners during the first two years of lower secondary school. The essays may contain linguistic errors. For each pair, you are asked to choose the best written of the two essays. No answers are right or wrong: you only have to express your opinion! Bear in mind that the essays of a pair can concern different topics, but this must not affect your judgment. Your participation to the survey is completely free. You may withdraw from it at any time. Before starting the survey, we ask you to provide some personal information that will be used for statistical purposes. Data will remain completely anonymous and will not be connected to you in any way. If you have doubts, curiosities or improvement proposals, please write me to the address: a.cerulli1@studenti.unipi.it. Have a good read!

12Personal data entry form. The surveys were obviously anonymous. However, as we mentioned before, we asked the annotators to entry some personal information (age, sex, education) for statistical purposes.

13Essays evaluation. The third section comprised ten pages, each occupied by two side by side essays and a field to give the answer (Figure 1). The user had to choose the label ‘1’ if they had preferred the first essay, ‘2’ otherwise.

14After carrying out a pilot study to test the adequacy of the structure as well as the completeness and clearness of the instructions, we started collecting evaluations. Using Linktree4 we added the ten questionnaires links to a single web page and shared its link through WhatsApp, Facebook and Instagram: clicking on it, users were redirected to the page and could access every survey.

3. Analysis of Human Judgments

15We collected 223 annotations distributed quite homogeneously among the ten surveys, except for the first one, submitted 28 times. It is worth to focus on the heterogeneous composition of the readers sample. Concerning sex, the large majority of answers (183 units, equal to 82.1%) were given by women, against the 38 (17%) by men; just two people preferred not to specify their gender.

16Regarding age, we divided the group into six bins (Figure 2). The most frequent class (97 units) was ‘20-24 years’, followed by ‘25-29 years’ (64 units). This means that most readers (72.5%) ranged from 20 to 29 years of age. 35 evaluations (15.8%) were made by natives between 30 and 39 years of age. People belonging to the remaining bins contributed to the task for an overall 11.7%.

Figure 2: Distribution of annotations with respect to readers’ age bins

17Finally, Figure 3 shows the distribution of submissions with respect to readers’ education: 91.9% of annotations were given by people holding an academic degree (118 units, equal to 53.2%) or a high school diploma (86 units, equal to 38.7%). 12 annotators (5.4%) had a middle school certificate; 4 (1,8%) held a doctoral degree; the last two indicated a non-specific ‘Other’.

Figure 3: Distribution of annotations with respect to readers’ education

3.1 Inter-Annotator Agreement

18At this point, we defined a selection function to discard inaccurate annotations and obtain the same number of coherent annotations for each survey. Thus, we firstly built the average vector of every survey as the set of ten values ‘1’ or ‘2’ chosen according to the most assigned label to each pair of essays; then, we calculated the distance between each survey average vector and all its annotations. We implemented the euclidean metric generalized to the n-dimensional space that computes the distance between two vectors as the square root of the sum of their sizes squared difference:

19                (1)

20To give relevance to the deviating degree of answers differing from the average, we assigned every pair a weight (w_k) equal to the number of times in which the ‘winning’ essay was chosen; then, we computed the weighted distance between annotations and average vectors.

21                 (2)

22Finally, we ranked weighted and unweighted distance values of each survey in ascending order and calculated the Inter-annotator agreement (IAA) of the first 15 and 20 annotations. We implemented Krippendorff’s alpha (α), a coefficient that expresses IAA in terms of observed (D_o) and casual (D_e) disagreement (Krippendorff 2011):

23              (3)

24We noticed that IAA values of the first 15 submissions ordered by their increasing weighted distance were the highest. Thus, we took them into account (150 total annotations) for the analysis and discarded the remaining 735. It is noteworthy that the selection led us to an average IAA of 0.26, that is a much higher value than the initial 0.12. Relying on the selected annotations, we established the ‘winning’ and ‘loser’ essay of each pair.

4. Data Analysis

25We carried out two evaluations: a first one was meant to identify which linguistic features impact more on the human assessment of the writing quality; a second one focused on the impact of students errors on annotators’ judgments. In what follows we describe the approach underlying the two perspectives and discuss our most interesting findings.

4.1 Linguistic Profiling and Stylistic Analysis

26The first analysis relies on linguistic profiling, a NLP-based methodology in which a large set of linguistically-motivated features automatically extracted from annotated texts are used to obtain a vector-based representation of it. Such representations can be then compared across texts representative of different textual genres and varieties to identify the peculiarities of each (Montemagni 2013; Halteren, n.d.). To perform the analysis, we relied on Profiling-UD6, a recently introduced tool that allows the extraction of a wide set of lexical, morpho-syntactic and syntactic features from texts linguistically annotated according to the Universal Dependencies (UD)7 formalism. These features, described in details in Brunato et al. (2020), have been shown to be involved in many tasks, all related to modeling the form rather than the content of a text, such as the assessment of text readability and linguistic complexity and the identification of stylistic traits of an author or groups of authors.

27We thus split our annotated corpus into two sections: one comprised all ‘winning’ essays and the other all ‘loser’ ones. Using Profiling-UD, we extracted for each text of the two subsets a feature-based vector representation. For each considered feature we calculated the average value, the standard deviation and the coefficient of variation (\frac{SD}{Avg}) in the two subsets and we assessed whether the variation between mean values was significant using the Wilcoxon rank sum test.

28Table 2 shows the seven linguistic features whose variation turned out to be statistically significant (p-value<0.05), ordered by increasing p-values. It emerges that ‘winning’ essays are on average longer (32.2 tokens more) than the ‘losers’ (n_tokens), a finding that may suggest that longer compositions are evaluated as more reasoned, structured and content-rich. Interestingly, this also reflects the students’ perception of school writing: Barbagli et al. (2015) showed that two of the most frequent suggestions contained in essays that respond to ‘common prompts’ are Leggi/scrivi molto (“Read/write a lot") and Lavora sodo, fai vedere che ti impegni (“Work hard, show your dedication"). Thus, pupils possibly write more so as to show their dedication and get higher grades. Secondly, we noticed that a richer vocabulary (ttr_form_chunks_100) plays a crucial role in native’s judgment. This is in line with another advice of the just mentioned ranking, Usa un vocabolario ricco ed espressivo (“Use a rich and expressive vocabulary"), that reflects teachers’ encouragement to use synonyms in order to write clearer and more readable compositions. Values related to the third feature (upos_dist_NOUN) reveal that ‘loser’ essays present a slightly higher distribution of nouns. A predominant use of nouns is typical of highly informative texts (e.g. newspaper articles, laws), while genres closer to speech contain more verbs (Montemagni 2013). Belonging to the second category, a school essay with fewer nouns is probably perceived as more coherent with its genre. Concerning verbal inflection, ‘better’ productions include, on average, 0.28% more future verbs (verbs_tense_dist_Fut), 0.81% more gerund verbs (verbs_form_dist_Ger) and 1.93% more subjunctive auxiliary verbs (aux_mood_dist_Sub). Verbal tenses differing from present and moods differing from indicative require elevated linguistic skills, which positively influence annotators’ choices. The last feature significantly varying between the two groups is the number of prepositional chains (n_prepositional_chains): ‘winning’ compositions have, on average, 1.2 more of them.

29A further study was focused on the variability degree of linguistic features in the two essay groups. For each subset, we ordered the features by their increasingly coefficients of variation; then, we calculated the difference between the two rankings in order to identify the features that were maximally uniformly distributed in ‘better’ essays as compared to the ‘worse’ ones (Table 3). It can be noticed that future verbs (verbs_tense_dist_Fut) are very uniformly distributed among ‘better’ essays. We have previously commented that their frequency is higher in the ‘winners’; it proves again that natives interpret the use of complex verbal forms as an indicator of higher skills. Also parataxis distribution (dep_dist_parataxis) is quite uniform in ‘winning’ essays; however, its average value is higher in the ‘loser’ ones. It can be deduced that annotators prefer hypotaxis but this is not surprising: hypotactic periods are more structured and elegant and require refined abilities to be built. The same evidence is given on the morphosyntactic level (upos_dist_CCONJ), since ‘better’ compositions include 0.34% less coordinating conjunctions. Curiously, ‘better’ essays have, on average, 0.1% more foreign terms (dep_dist_flat:foreign); this may suggest that annotators appreciate these expressions. Finally, it is worth highlighting a higher and more uniform percentage of verbs with few modifiers in the ‘winning’ essays (verb_edges_dist_0, verb_edges_dist_1).

4.2 Students Errors Impact

30The last analysis was aimed at assessing whether and in what measure students errors impact on human judgments. We counted the pairs of essays whose ‘winning’ composition had a lower number of errors, those in which the ‘loser’ one had more mistakes and those with an equal number of errors. We noticed that essays with fewer errors had won in 56% cases, reaching the 79% if including pairs with the same number of errors. This procedure gave a first empirical answer to our starting question: errors substantially affect human assessment.

31At this point, we focused on error categories to identify which ones affect more the perception of writing quality. For each category, we calculated the average number of errors and their standard deviation in both subsets; then, relying on Wilcoxon rank sum test, we found out that grammatical and orthographic mistakes vary significantly between the two groups (Table 4). As expected, ‘loser’ essays have, on average, 1.29 more grammatical errors and 0.85 more orthographic errors. It is worth to add that orthographic mistakes variation (p-value=0.007) is more significant than the other (p-value=0.029). This could mean that natives judge deviations in orthography worse than those in grammar. Once again, our findings are in line with Barbagli et al. (2015): Usa una corretta ortografia (“Use correct orthography") is the of the most frequent suggestions given in the second year; moreover, Errori di ortografia (“Orthography errors") occupies the and the position among the most salient terms respectively of the first and the second year. The non-significant variations of lexical (p-value=0.581) and punctuation errors (p-value=0.617) are probably due to their scarce amount in the analysed essays.

5. Conclusions

32We presented a pilot study towards the identification of the linguistic features that are own of well written perceived essays. We collected Italian natives’ preferences on 100 pairs of essays written by L1 students, that we analysed in terms of linguistic profiling and errors distribution. Our results reveal an interesting correspondence between annotators’ judging criteria and writing instructions that L1 learners receive by teachers. Our findings could be interpreted as an indicator of the reliability of our data and, more in general, could suggest the effectiveness of crowdsourcing methods to quickly build large and reliable datasets. Considering the lack of Italian corpora of graded essays, such datasets could be valuable resources for the development of Computer-Assisted Learning Systems.

33The limited size of our dataset certainly reduced the amount of results. Thus, we have to expand it (i) by collecting more annotations for the already existing surveys and (ii) by creating and distributing new surveys in order to gather judgments on new pairs of essays. Analysis on the enlarged dataset could provide more features that are own of good essays. Following the model of Miaschi, Brunato, and Dell’Orletta (2021), we could use the results to train a classifier that, given a pair of essays, recognizes the best written one.

34The tool would not presume to replace teachers, but it could be a valuable teaching aid. Students could use it to get an immediate and preliminary self-assessment on their written productions so as to better understand their mistakes and hopefully avoid repeating them. Such tools can be very useful if integrated into educational processes based on distance learning paradigms, which need adequate technological infrastructures to be really efficient.