1 Introduction

1In this work, we present an extensive study on the evolution of high-school students’ writing skills, taking into account essays spanning 15 years (from 2001 to 2016). In particular, we are interested in tracking the presence of expressions and constructions typical of neo-standard Italian (Berruto, 2012), in the light of the recent public discussion on the ‘decline of Italian in schools’ 1. The Italian neo-standard is the current linguistic register in Italy, in which forms previously considered colloquial have become widely accepted in the national language.

2We analyse more than 2,500 essays written by students from different high-schools in the Autonomous Province of Trento during the exit exam (the so-called Maturità). The study is the outcome of a project comprising different steps: i) digital acquisition and transcription of thousands of essays balancing their distribution across school years and school types; ii) computer-assisted annotation of some linguistic traits of interest; iii) diachronic analysis of the traits. While the first step has been carried out by the Istituto provinciale per la Ricerca e la Sperimentazione educativa (IPRASE), we led steps ii) and iii), which are discussed in the next sections. Beside an in-depth and diachronic study of the evolution of students’ writing skills, a major contribution of this paper is also the release of the corpus in the form of embeddings and n-grams.

2 Corpus Collection

3The staff of IPRASE have digitized and transcribed essays stored in the archives of 21 secondary schools located in different areas of Trentino Province. These areas include both the two major cities, Trento and Rovereto, but also other communities in the valleys (Val di Fiemme, Val di Non, Valsugana) and Riva del Garda. Nine different types of schools were involved: liceo classico, liceo scientifico, liceo artistico, liceo linguistico, liceo musicale e coreutico, liceo delle scienze umane, istituto tecnico tecnologico, istituto tecnico economico and istituto professionale. Six school years were chosen between 2000-2001 and 2015-2016, thus having a temporal span of 15 years for a total of 2,544 essays and almost 1.5 million words. Table 2 shows the distribution of essays per year with the corresponding number of words. These essays are of the so-called type B, that requires students to write a short essay or a newspaper article. Students can choose between 4 areas: artistic-literary, socio-economic, technical-scientific, historical-political. For each area, a title is given together with a set of reference materials. For example, students writing an essay of type B with historical-political content in 2014 were asked to comment some excerpts from Hannah Arendt, Ghandi and Martin Luther King about violence and non-violence in the XX Century.

4Due to privacy reasons, we are not allowed to distribute the full texts of the corpus. However, we release both word vectors and n-grams of the essays. We build three types of embeddings with 300 dimensions: the GloVe embeddings based on linear bag-of-words contexts (Pennington et al., 2014), Levy and Goldberg’s ones using dependency parse-trees (Levy and Goldberg, 2014), and fastText embeddings with bag of character n-grams (Bojanowski et al., 2017). As for the n-grams, we generated both case-sensitive and case-insensitive sequences per school year, considering the range [1,5]. N-grams and pre-trained word embeddings in text format are available for download on our website2. In addition, word vectors are visualized through a dedicated stand-alone version of the TensorFlow embedding projector (Smilkov et al.,2016)3.

3 Description of Linguistic Traits

5Around 20 teachers have been involved in the annotation of essays using the CAT platform (Bartalesi Lenzi et al., 2012), through which they had to annotate between 100 and 150 essays each. We also organised 2 preliminary training sessions with the teachers to show the tool functionalties, explain the annotation process and make sure that everyone followed the guidelines4. Note that the teachers knew neither the name of the student writing the essay nor his/her school. Moreover, for all of them, it was the first time using an electronic platform for text annotation.

6We briefly present in Table 2 the traits that the teachers had to mark on each essay. The goal of the annotation is to detect the presence of linguistic traits that were deemed relevant to diachronically study style and complexity evolution by IPRASE experts and teachers. This approach is therefore rather different from the standard essay correction that is usually performed by teachers, and for this reason the training phase was particularly relevant.

7The list of traits to include in the project was mainly inspired by the work of (D’Achille, 2003) and (Boscolo and Zuin, 2015). The goal of this annotation was to cover all levels of linguistic analysis, including lexical choices (e.g. trait 8 and 20), grammar (e.g. trait 1 and 2), semantics (e.g. trait 15) and discourse structure (e.g. trait 24 and 25).

8In the first Table column, we mark traits that were identified in a fully automatic way (A), those that were annotated semi-automatically (S), and the manual ones (M). For those marked with S, we pre-processed the essays using the Tint NLP tool (Aprosio and Moretti, 2018) enriched with a set of new modules developed to add all information needed to speed up annotation. For example, for traits 21 and 23 we matched the essay n-grams with pre-defined lists of politically correct expressions and cliché expressions provided by IPRASE, so that teachers could see in the CAT interface the corresponding markables already highlighted, and they just had to validate them. For other traits, for example 10 and 11, they had to add attributes to the markables. For some traits, we performed pre-annotation using available external resources, for example the list of affixes included in the derIvaTario5 for trait 13 (Talamo et al., 2016).

9After the initial training phase, the average annotation time for each essay through the web interface was 30 minutes. We roughly estimate that the same task would take at least one hour on a standard Word document. Another advantage of using the CAT interface was the possibility to have all annotations in a consistent format, easily export them to compute statistics and make comparisons.

4 Linguistic Analysis

10We present here an analysis of some traits of interest. We focus in particular on traits that are, at least in part, automatically annotated and counted (marked with A or S in Table 2), because the work of those requiring a manual annotation is still in progress. For each trait we compute the observed relative frequency per 10,000 words. This normalization has allowed us to have more easily comparable and legible numbers. Furthermore, we calculate the Gulpease index to monitor writing complexity (Lucisano and Piemontese, 1988). This score has been specifically defined for measuring the readability of Italian texts based on proficiency level and it combines two linguistic variables: the average length of the words and of the sentences in a document. Its value determines the level of readability of a text: the higher the score, the easier the text is to understand.

11To extract reliable measures of students’ language use, we removed from the texts the quotations present in the essays citing the reference material provided together with the topic. This pre-processing step was performed by adopting the FuzzyWuzzy package6, a Java fuzzy string matching implementation, and the Stanford CoreNLP quote annotator7. These tools allow us to recognize text reuse both when it is explicitly signaled by quotes and when there is no overt signal. The average percentage of quotations within the corpus is 1.9% but it varies a lot among the essays, reaching up to 46% of the content in some cases. The following is an example taken from an essay about the pursuit of happiness in 2010 for the socio-economic area. The snippet in bold, containing one of the complex connectives of trait 16, was automatically removed: La riflessione di Zygmunt Bauman sembra essere una risposta: “L’incertezza è l’habitat naturale della vita umana, sebbene la speranza di sfuggire ad essa sia il motore delle attività umane.”

12After removing quotations, we obtain the following results for the automatically annotated traits:

13Trait 8 - Generic Words. We trace the presence of semantically generic and polysemic words, which are frequently used in neo-standard Italian (Fig. 1). In particular, lemmas ‘fare’, ‘dire’, and ‘cosa’ (to make, to say, thing) show a decrease in occurrence in the last two school years considered (2012-2013 and 2015-2016). For example, the relative incidence of ‘fare’ every 10,000 tokens goes from 42.013 in 2000-2001 to 26.857 in 2015-2016 indicating an effort to use more specific and differentiated expressions. Liceo classico has the lowest ratio for ‘fare’ and ‘dire’, whereas istituto professionale has an occurrence above the average for ‘fare’ and ‘cosa’.

Figure 1: Observed relative frequency of three generic words per 10,000 tokens.

14Trait 14 - Nominal Sentences. Sentences without a verbal predicate are a typical feature of news style and juvenile writing, to make the text dramatic and concise (Dardano, 1986; Ardrizzo and Gambarara, 2003). This tendency is present also in our corpus with an impact of 6.1% over the total amount of sentences, after removing the title of the essays. The trait is particularly relevant in liceo classico with an above-average percentage of 7.7%.

15Trait 16 - Complex Connectives. The lack of complex connectives is another indicator of neo-standard Italian. As shown in Figure 2, ‘nondimeno’ is never used by students and also ‘qualora’ and ‘giacché’, used mostly in liceo classico, disappear in the last two school years from all the essays. ‘Affinché’ is adopted in all school types with the only exception of liceo artistico, in which complex connectives are barely used.

16Trait 17 - Punctuation. Over the last two school years considered in our analysis, there has been an overall decline in the use of punctuation with the exception of question marks (see Figure 3). The frequent use of question marks is inherited from the style of news (Buroni, 2009); however, the peak in 2009-2010 is also due to the presence of a question in the title of an essay (Siamo soli?), which led students use the same rhetorical device in their texts. The presence of punctuation not suitable for medium-high style such as multiple exclamation marks and suspension points is also decreasing.

17Trait 27 - Euphonic ‘d’. Following a recent grammatical rule8, the euphonic ‘d’ should be introduced only when the conjunction ‘e’ or the preposition ‘a’ are followed by a word starting with the same vowel: e.g., ed ecco, ad andare. However, this rule is not followed in the essays and the presence of ‘d’ between two different vowels is higher than the one between the same vowels (33.8 versus 17.6 of relative frequency). Besides, while the disappearance of this trait is considered a characteristic of neo-standard Italian (D’Achille, 2003), this trend is not found in our corpus, where the relative frequency of euphonic ‘d’ is only 6 points lower than the same conjunction without ‘d’ preceding a vowel.

18Gulpease. We computed the Gulpease index to see whether there has been a decrease of complexity, i.e. an increase in readability, over time. Contrary to our expectations, the average readability of essays has slightly decreased in the last two years considered, with a drop of 1.8 points, bringing it below 50. This corresponds to texts that are quite difficult to read for a person with a medium school degree (diploma di scuola media in the Italian school system), but not too challenging for a person with a high school degree. Moreover, values do not change much across different school types.

19These preliminary analyses show that the impact of neo-standard Italian is multi-faceted and, while some traits confirm that students’ language is getting simpler and less formal (e.g. overall decline of punctuation), some others seem to contradict this finding (e.g. decline in the use of ‘fare’, ‘dire’, ‘cosa’). Also the differences across school types are not clear-cut and consistent.

5 Related Work

20While several works in the past have focused on the creation and analysis of corpora to study students’ mistakes, their writing quality and their rate of progress over the year (Parr, 2010; McNamara et al., 2010), they have mainly dealt with English essays. A notable exception are two corpora in German, the KoKo corpus of argumentative essays to study pupils’ writing competences (Abel et al., 2016) and the corpus collected by Berkling et al. (2014) to study different error categories.

Figure 2: Observed relative frequency of complex connectives per 10,000 words.

21As for Italian, a relatively small number of studies has been carried out with various goals. The projects TIscrivo (2011-2014) and TIscrivo 2.0 (2014-2017)9 have been launched to investigate the writing skills of primary schools and lower secondary schools in Southern Switzerland (Cignetti et al., 2016), and have led to the creation of a corpus of 1,735 essays. Another research deals with the analysis of oral and written productions of Italian children in primary schools, and 200 texts have been collected in the ISACCO corpus (Brunato and dell’Orletta, 2015). Another corpus, called CItA (Barbagli et al., 2016), includes texts written in the first and second year of lower secondary school, tracking L1 writing competence of the same group of students over two school years.

22Compared to previous works, our analysis is different in several ways. First, none of the previous studies considers a text span of 15 years. Then, the traits to be annotated are different: we do not focus on mistakes, but on indicators of neo-standard Italian. Finally, our interest lies also in the annotation workflow, studying how NLP can support the identification of such traits and implementing the necessary processing modules to speed up annotation.

Figure 3: Observed relative frequency of punctuation per 10,000 words.

6 Conclusions

23In this work, we have presented a project aimed at tracking the evolution of students’ writing skills over time. The goal of this work was not only to introduce the corpus collection and annotation activities, but also to show how this kind of projects can benefit from NLP by speeding up annotation and increasing data consistency. In the future we will complete the analysis of all the traits for a more comprehensive view of the role of neo-standard Italian in students’ essays. We will also use some of the manual annotations to train new NLP modules performing the same task automatically.

Acknowledgments

24We would like to thank Chiara Motter from IPRASE for coordinating the corpus transcription, and the high-school teachers for annotating the essays.