1 Introduction

1The study of linguistically annotated learner corpora has received a growing interest over the past 20 years (Granger et al., 2013). In learner corpus linguistics, such corpora are usually defined as “systematic computerized collections of texts produced by language learners” (Nesselhauf, 2005). Unlike most learner corpora focusing on L2/FL learners (i.e. learners learning a foreign language), the KoKo corpus focuses on advanced L1 speakers that are still learning their mother tongue, which typically happens in educational contexts.

2This paper describes an extended version of the KoKo corpus (Abel et al., 2014a), a corpus created for the purposes of the KoKo project which aims at investigating the writing skills of German-speaking secondary school pupils. The creation of the corpus was guided by two goals: on the one hand to describe writing skills at the end of secondary school, on the other hand to consider external socio-linguistic factors (e.g. gender, socio-economic background etc.).

3The previous description focused on the data collection, the data processing, the annotation of orthographic and grammatical features as well as on aspects regarding annotation quality (Abel et al., 2014a). This paper, however, introduces the new textual metadata and lexical annotations.

4The paper is structured as follows. In section 2, key facts are briefly reported, including references to related work. The new textual metadata and lexical annotations are then described in section 3, alongside with the methods adopted for their manual annotation and linguistic analyses and some examples of linguistic findings. In section 4, future works are discussed right before concluding in section 5.

2 Key Information about the Corpus

5The KoKo corpus is a collection of 1,503 authentic argumentative essays, and the corresponding survey information about their authors, produced in classrooms under standardized conditions by learners of 85 classes of 66 schools from three different German-speaking areas: South Tyrol in Italy, North Tyrol in Austria and Thuringia in Germany¹. Such areas are particularly suitable for comparative studies because of differences regarding the German standard varieties, the use of dialectal vs. standard varieties and the monolingual vs. plurilingual environments (Abel et al., 2014a).

6The corpus is roughly equally distributed over the three regions and amounts to 824,757 tokens (punctuation excluded). All writers were attending secondary schools one year before their school-leaving examinations. 83% of the pupils were native speakers of German. The corresponding L1 part of the corpus amounts to 726,247 tokens. Metadata annotations amount to 52,605 annotations whereas manual annotations amount to 117,422 annotations. Furthermore, 366 features to measure linguistic complexity² (Hancke et al., 2012; Hancke and Meurers, 2013) were automatically calculated per text (550,098 in total) and added as metadata.

7Previous evaluation showed high accuracy of manual transcriptions (> 99%), and automatic tokenization (> 99%), sentence splitting (> 96%) and POS-tagging (> 96%) (Glaznieks et al., 2014).

8As it is among the first accessible richly linguistically annotated German L1 learner corpora, the KoKo corpus is particularly relevant to L1 learner language researchers, and for the field of didactics of German as L1. Other comparable language resources are either not accessible (Berg et al., 2010; DESI-Konsortium, 2006; Nussbaumer and Sieber, 1994), or although accessible, have not been enriched with linguistic information (Augst et al., 2007; Fix and Melenk, 2002) or are only partly annotated (Thelen, 2010). Some other corpora include L1 data, but as reference for L2/FL learner corpus research (Reznicek et al., 2010;Zinsmeister and Breckle, 2012).

3 New Metadata and Annotations

9This section describes the main features of the latest corpus version KoKo4 (Dec. 2015) that have been added to the version KoKo3 (Dec. 2014). It thus focuses on a new set of textual metadata and a new layer of lexical annotations which is, due to the selected features and the degree of granularity, a novelty in (corpus-based) modeling of L1-writing competences for German .

3.1Textual Metadata

10In the KoKo corpus, two kinds of Metadata in-formation are available: (1) non-linguistic, i.e. person-related information provided by each participant via a questionnaire survey in class that is available for the whole sample and (2) linguistic, i.e. text-related information provided for a subsample of the corpus (569 texts, equally distributed over the three regions involved) through an online evaluation form by three different specially trained raters originating from the different participating regions.

11While type (1) metadata allow for sociolinguistic analyses in order to detect relations between linguistic features (e.g. text length, sentence length, orthographic errors, grammatical errors, etc.) and non-linguistic person-related information, type (2) metadata constitute a further expansion of our analysis by including textual features as well. Text analysis was done holistically using an evaluation form and detailed guidelines that were elaborated on the basis of recent findings in writing research and text analyses (Brinker, 2010; Feilke, 2010; Augst et al., 2007; Böttcher and Becker-Mrotzek, 2006; Jechle, 1992; Augst and Faigel, 1986) and the curricula in the participating regions. The text evaluation form distinguishes four categories : (A) formal completeness, (B) content, (C) formal and linguistic means of text arrangement and (D) overall impression.

12For category A, 10 questions of the online evaluation form focused on the presence of obligatory text parts (introduction, main part, closing part) and explicitly requested constituents of argumentative essays (opinion of the author, conclusion). The 25 questions of category B belong to two subcategories: (B1) the topics of the essay (9 questions), (B2) patterns of topic development (16 questions). B1 comprises evaluations on e.g. the topics of each text part, gaps, and the overall coherence of the text. B2 refers to the main pattern of topic development (argumentative, etc.), the argumentation strategies (point of view, concessive or not), and the motivation of arguments (objective vs. subjective stance, quality of arguments). Formal and linguistic means of text arrangement (category C, 7 questions) focus on the use of paragraphs, the explicit announcement of and commitment to the function of the essay, and the use of linguistic means to structure the text with regards to content. Finally, category D (20 questions) aims for an overall impression and therefore focuses on the completion of the task (successful or not), the overall quality of the text and the overall consistency of both the quality and coherence. Of all 62 questions of the entire online evaluation form, we used 57 for each document of the subcorpus (alltogether 33,972 annotations).

13The analyses revealed, among other things, that the text quality is classified as quite satisfactory on a 5 point Likert-scale³. More specifically, there are significant correlations between text quality assessment and other linguistic variables: thus, a lower number of e.g. lexical errors is connected to a higher text quality score⁴, and, finally, a variety of group differences could be detected (e.g. concerning school type: lower text quality scores within vocational schools compared to general high schools⁵).

3.2 Lexical Annotations

14As for the manual annotations of orthographic and grammatical features added to previous corpus versions (Abel et al., 2014a), a specifically crafted tag set and annotation manual were used for the annotation of lexical features. 61,728 lexical annotations were manually performed by trained annotators on a subcorpus of 980 texts, almost equally distributed over the three regions.

15The analyses of lexical features focuses on lexical knowledge as a central part of lexical competence which includes the dimensions of lexical breadth (quantitative aspect) and lexical depth (qualitative aspect) (Steinhoff, 2009; Böttcher and Becker-Mrotzek, 2006; Mukherjee, 2005; Read and Nation, 2004; Read, 2000; Nation, 2001). Whereas the analyses of quantitative aspects of lexical knowledge were performed automatically by using different measures (e.g. lexical diversity measures such as MTDL and Yule’s K, or lexical frequency scores based on dlexDB (Hancke and Meurers, 2013)), the analyses of qualitative aspects were done by means of manual annotations. We focus hereafter exclusively on the manual annotations allowing us to model qualitative aspects of lexical knowledge.

16Table 1: Quantitative figures for the 980 documents annotated with the new lexical annotations.

17For annotating lexical features, we developed a new hierarchically-structured linguistic classification scheme inspired by previous work that focused on L2 learner languages (Abel et al., 2014b; Konecny et al., 2016). The classification scheme takes both into account occurrences of selected lexical phenomena and defective as well as nondefective particularities of learner languages considering two dimensions: (1) the linguistic subcategory, e.g. collocations and idioms, and (2) a target modification classification, e.g. omission, addition (Díaz-Negrillo and Domínguez, 2006; Abel et al., 2014b). Furthermore, we formulated target hypotheses for those categories that we annotated as defective in order to make the error interpretation transparent (Lüdeling et al., 2005). The corresponding annotation scheme contains 77 different tags including a set of further attributes.

18In a multi-stage annotation procedure, all occurrences of phenomena on both single words and formulaic sequences (FS) were annotated (Wray, 2005). Annotations for particularities were subsequently added in order to distinguish between errors concerning correctness, errors concerning appropriateness of usage (Eisenberg, 2007; Schneider, 2013), non-defective modifications (to capture, for example, creative use of language), and diasystematic markedness. At the single word level, we considered all out-of-vocabulary tokens of the part-of-speech tagger (Schmid, 1994) as candidates of neologisms or occasionalisms. In addition, we captured a variety of tokens relevant for the text genre of an argumentative essay (i.e. argumentative adverbs and conjunctions). At the level of FS, we applied a function-based approach distinguishing between three main categories of phrasemes (Burger, 2007), each of them with further subcategories (Abel et al., 2014b; Konecny et al., 2016; Granger and Paquot, 2008; Burger, 2007; Stein, 2007; Steinhoff, 2007), as well as a “mixed classification” (Burger, 2007):

19Referential phrasemes include collocations⁶ and idioms⁷, distinguished among other things with respect to their degree of idiomaticity. Communicative phrasemes are subdivided into those bound to specific situations⁸, and those not bound to specific situations⁹. Finally, structural phrasemes comprise complex conjunctions and prepositions¹⁰and concessive constructions¹¹.

20For particularities, we considered four main categories, each with further subcategories:

21On a semantic dimension a distinction is made between denotative errors concerning cor-rectness or appropriatness of use¹², and connotative markedness or appropriatness of use¹³. The stytlistic dimension considers repetition, and redundancy. The form dimension focusses on word formation errors (concerning single word units only¹⁴), and on omission, choice, position and addition errors as well as creative modifications (concerning FS). Concerning metalinguistic markers the appropriateness of the use of quotation marks for highlighting units is considered.

22An overview of the number of annotations is provided in Table 1.

23Results of the analyses showed, among others, that pupils use different types of FS quite frequently, on average 5.12 constructions per 100 words: with 62%, non idiomatic referential phrasemes constitute the major part, followed by idiomatic referential phrasemes (19%), and, finally, structural (10%) and communicative phrasemes (9%). However, lexical errors in general affect more often FS than single word units (10% of the FS vs. 1.04% of the single words). The latter are most frequently form errors (5.50% of FS affected, especially choice errors: 4.17%).

4 Future Work

24The KoKo project was completed and presented to the public in December 2015. We will start releasing the data via the corpus exploration interface ANNIS3 (Krause and Zeldes, 2016) and for download on request, after signing a license agreement.¹⁵ Aside from the aforementioned data, future versions will also include additional metadata information about the authors integrated for the purposes of future socio-linguistic analyses.

25Consensus in the annotations among annotators, and as such an indication of its reliability, will be evaluated on sub-sets of texts that were annotated for this purpose by more than one annotator. Three annotators independently annotated the text level metadata annotations on 27 texts, and six annotators independently annotated the lexical level annotations on the same 27 texts. Inter-annotator agreement will be calculated for annotations and segmentation, i.e. the agreement on the decision which word sequence needs to be tagged vs. what annotation needs to be assigned to it, and will be evaluated and reported in the form of Fleiss Multik and boundary similarity (Artstein and Poesio, 2008; Fournier, 2013).

26Finally, thanks to its relatively large size and its richly annotated nature, potential additional uses of the KoKo corpus in Natural Language Processing and Corpus Linguistics are being considered. Regarding Natural Language Processing, the error annotations paired with target hypothesis annotations allow for creating an aligned corpus. Such corpora can be used to improve machine translation for automatically correcting learner texts (Ng et al., 2014). Regarding Corpus Linguistics, machine learning methods can be used (e.g. as being done in WebAnno (Yimam et al., 2014)) to drive linguistic intuitions when performing annotations or analyses. Because of the richness of its annotation schemes, the KoKo corpus constitutes a challenging but at the same time promising dataset to test if the developed methods are able to uncover relevant correlations that have already been investigated, or to uncover even new ones that are worth considering for future linguistic analyses.

Conclusion

27This paper described the most recent version of the KoKo corpus, a collection of richly annotated German L1 learner texts, and focused on the new textual metadata and lexical annotations.

28Because other comparable language resources are either not accessible, or have not been enriched with linguistic information or are only partly annotated, the corpus is a valuable resource for research on L1 learner language, in particular for the research on writing skills, and for teachers of German as L1, in particular for the teaching of L1 German writing skills.