1. Introduction

1A cloze test is a reading comprehension assessment where participants are presented with a text in which selected tokens have been replaced with blanks. The goal is for the participant to choose tokens (often from a list) and use them to replace the blanks based on the overall context. Typically, one every 5-10 tokens is replaced with a blank.

2Cloze tests are one of the most common linguistic tests in use for formative and summative purposes, along with written responses, multiple-choice tests, matching tests, ordering tests, summarizing tests etc. (Lugarini 2010). Cloze tests were originally introduced in the United States in the 1950s to measure the readability of texts (Taylor 1953) and involved the random and not pre-determined deletion of words that appeared at pre-defined intervals. This method was too general for didactic and evaluation purposes, but it was quickly adapted and became very widespread as a teaching and testing technique (Radice 1978). In education, cloze tests have become more targeted: words are deleted according to various criteria, depending on the specific testing goals. In general, cloze tests are designed to evaluate one of the following:

• field-specific knowledge acquisition, by asking to insert appropriate words about a topic or a discipline;

• text comprehension, by asking for information that can be inferred from the text (with no prior domain knowledge);

• linguistic aspects, typically with respect to L1, L2 and FL (foreign language) acquisition at different levels (i. e. vocabulary, specific parts of speech etc.).

3If carefully designed, cloze tests can be a very effective tool at all educational levels; on the other hand, cloze tests may also show some limits and issues in assessing linguistic competence (Chiari 2002), as they necessarily offer a partial and contextual view. However, the long tradition of study and use in the fields of educational linguistics and linguistic makes it very interesting to compare human and automatic performances in dealing with cloze tests.

2. Methodology

4We tackle the cloze tests in our dataset with pre-trained language models based on the Transformer architecture (Vaswani et al. 2017). We employ both autoencoding and autoregressive models. Given the very small number of datapoints at our disposal, model fine-tuning is not a viable option; therefore, we use pre-trained versions of such models, all of which are publicly available through Huggingface at the time of writing (summer 2021).

Dataset

5Our dataset contains eleven cloze tests focusing on general linguistic competence that were administered to incoming first year students at the University of Eastern Piedmont in the cities of Alessandria and Vercelli in northwestern Italy between 2017 and 2019. Each cloze test was taken by a number of students in the low three digits, ranging from 130 to 390. As these are university-level tests, all students had at least a high school diploma. Most of the students were L1. The tests were offered on-site (in information technology classrooms) through the Moodle Learning Platform.

6Our dataset contains two types of cloze tests: nine restricted tests where a list of options is provided for each blank to be filled, and two unrestricted tests where a global list of options is provided for all blanks with no token subgrouping (i.e., with no information about which tokens are supposed to go where). In the two unrestricted tests and three of the nine restricted ones, the list(s) contain single token options. In the other six restricted tests, the lists contain at least one multiple token option (e.g., il quale or con l’utilizzo). These cloze tests involved both function words as well as content words with both lexical and grammatical meanings

Autoencoding models

7Our choices for autoencoding models are BERT (Devlin et al. 2019), RoBERTa (Liu et al. 2019), DistilBERT (Sanh et al. 2019), and ELECTRA (Clark et al. 2020).

8BERT is a natural choice because one of its two pre-training tasks is masked language modeling: a fraction of tokens in the pre-training data are masked so that BERT can be pre-trained to reconstruct them. Viewed as an NLP task, a cloze test is a special case of masked language modeling task where tokens are masked in an adversarial fashion: instead of choosing tokens to be masked uniformly at random, tokens are masked to challenge the test taker to reconstruct the meaning of the original text. Because a cloze test is functionally equivalent to a masked language modeling task, it is reasonable to use pre-trained BERT with no further task-specific fine-tuning.

9RoBERTa improves on the original BERT by focusing on the aforementioned masked language modeling task and removing the other pre-training task (next sentence prediction). UmBERTo1 is a RoBERTa-based model that contains some interesting optimization such as SentencePiece and Whole Word Masking. UmBERTo has been shown to perform very well compared to other BERT-based models (Tamburini 2020).

10DistilBERT (Sanh et al. 2019) is a more compact language model pre-trained with knowledge distillation(Hinton, Vinyals, and Dean 2015), a technique that uses the output of a larger teacher network to train a smaller student network. BERTino (Muffo and Bertino 2020) is an Italian DistilBERT model that was recently proposed as a lightweight alternative to BERT specifically for the Italian language.

11ELECTRA is pre-trained with replaced token detection: instead of being masked, tokens are replaced with plausible alternatives sampled from a generator network; the model is then pre-trained to discriminate whether each token was replaced by a generator sample or not. At the outset of this study, the authors posited that replaced token detection is enough to make ELECTRA reasonably ready to tackle cloze tests with no further task-specific fine-tuning; this is indeed the case, as confirmed by the results shown in Table 1.

12To summarize, we employ the following autoencoding models (all cased, as the cloze tests in our dataset contain case-sensitive options):

• multilingual BERT-base2 (BERT multi), which serves as a baseline for autoencoding models;

• the Bayerische Staatsbibliothek’s Italian BERT model3 (BERT it);

• a smaller version of multilingual BERT-base4 (BERT it LWYN) based on the Load What You Need concept described in (Abdaoui, Pradel, and Sigel 2020);

• UmBERTo5 as the representative of the RoBERTa family.

• BERTino6 as the representative of the DistilBERT family;

• the Bayerische Staatsbibliothek’s Italian ELECTRA model7.

Autoregressive models

13The key limitation of masked language modeling as a proxy for cloze test is the focus on single token masking. Therefore, autoencoding models are not applicable to the six cloze tests in our dataset that feature at least one multiple token option. (In some cases, the multiple token options are consistently among the incorrect options; using our autoenconding models in such cases would therefore skew the results in the models’ favor.) For these tests, we employ a simple strategy based on autoregressive models: we iterate over all possible substitutions given the options offered by a test and choose the one with the lowest perplexity as determined by each of our autoregressive language models, all of which are from the GPT-2(Radford et al. 2019) family and include the following:

• a standard GPT-2 model8, which serves as a performance lower bound (Vanilla GPT-2);

• a recycled version of GPT-29 transferred to the Italian language(Vries and Nissim 2020) (Recycled GPT-2);

• GePpeTto10(Mattei et al. 2020), the first generative language model for Italian, also built using the GPT-2 architecture.

3. Results

14The results of our study are summarized in Table 1. We report the results obtained by the human test takers and the models for each of the eleven cloze tests in our dataset as well as aggregates (mean values) over the whole dataset. For each cloze test, we report the number of blanks to be filled (Questions, which varies from 4 to 6), the number of human test takers (Human count), as well as with the mean and the standard deviation of the scores. Each test is identified by the initial of its topic (S=Science, L=Legal, G=Geometry, R=Reasoning, E=Education, H=History, T=Technology) along with a numeral to disambiguate multiple tests on the same topic. As previously mentioned, two tests are unrestricted (all the provided options can go anywhere in the text) and the others are restricted (there are specific option lists for each blank to be filled). As previously explained, six tests (L2, G2, E, H1, H2, T) contain at least one multi-token option and are only tackled with autoregressive models. On average, we observe that:

• humans do better than the best model (Electra) by eight percentage points;

• Electra, UmBERTo, and GePpeTto are the top three performers;

• Vanilla GPT-2 aside, BERT it LWYN comes in last and underperforms BERT it multilingual.

15Averages, however, hide the enormous gap between restricted and unrestricted tests. We illustrate this gap in Table 2, which compares these two categories of tests model by model and also shows averages across autoencoding and autoregressive models (computed over the best models for each category, i.e., without BERT-base-it LWYN and BERT-base-multi for autoencoding models and without Vanilla GPT-2 for autoregressive models). This leads us to the following observations:

• our best autoencoding models outperform the human average;

• as expected, our models perform much better in restricted tests (we see a gap of 30 percentage points for autoencoding model and 10 points for autoregressive models);

• autoregressive models outperform autoenconding models in unrestricted tests, while the converse holds in restricted tests;

• humans perform similarly on both our restricted and unrestricted tests (and so does our performance lower-bound, Vanilla GPT-2).

16In our restricted tests, UmBERTo and Electra outperform the human average and emerge as the top performers among our models. Though far below the human average, GePpeTto and Recycled GPT-2 are the two top performers in unrestricted tests, where none of the autoencoding model reach the pass threshold of 0.6. Vanilla GPT-2 aside, BERT it LWYN comes in last and underperforms BERT it multilingual in restricted tests while matching its baseline performance in unrestricted tests.

4. Case Studies

17In this section, we focus on two specific examples of cloze tests from our dataset that serve as case studies to shed further light on our results. Let us consider the following restricted cloze test (G1 in Table 1).

18The replacements are reported in Table 3 and show that this specific cloze test focuses solely on function words.

19UmBERTo offers the best performance. UmBERTo’s only mistake is at blank 5, where Tra is chosen instead of Per. We note that this is a typical mistake made by the students who took this cloze test. The correct answer, Per, ranks second among UmBERTo’s top picks, with a probability of approximately 2.9~10^-3 as opposed to 3.3~10^-2 for Tra. The second best models, BERTino, BERT-base, and ELECTRA-base, make an additional mistake at blank 2.

20Let us now consider the following unrestricted cloze test (L1 in Table 1).

21The replacements are reported in Table 4 and show that this specific cloze test focuses primarily on content words.

22Autoregressive models ace this test. GePpeTto offers the best performance (no incorrect replacements). Recycled GPT-2 is second best, with only one incorrect replacement out of 6: giorno is chosen instead of the correct token quinquennio. This replacement requires a level of contextual understanding that cannot be realistically expected from a language model at this point in time; our conjecture is that, in this specific instance, GePpeTto’s correct replacement is most likely fortuitous (its performance range across all of our tests seems to validate our conjecture). Autoenconding models fare substantially worse, even though ELECTRA and BERT-base are fairly close to the average human performance.

5. Conclusion

23While these results are based on as few as eleven cloze tests (and only two unrestricted ones), the key takeaway is that existing pre-trained Italian language models with no task-specific fine-tuning can successfully tackle (and pass) relatively sophisticated tests designed for Italian students who have successfully completed their high school education. These results, though preliminary in nature, suggest various research questions, which could be answered based on a larger set of cloze tests. Such questions include whether there exists a pattern to the incorrect replacements made by the models, how the models fare with different parts of speech and with function words as opposed to content words, and how much their performance would improve with task-specific fine-tuning.