1. Introduction

1The introduction of contextualized embedding models, such as ELMo (Peters et al. 2018) and BERT (Peters et al., 2018), allows the construction of vector representations of lexical items that adapt to the context in which words appear. It has been shown that the upper layers of these models contain semantic information (Jawahar et al., 2019) and are more diversified than lower layers (Ethayarajh 2019). These word representations overcame the meaning conflation deficiency that affects static word embedding techniques (Camacho-Collados and Pilehvar 2018; Tripodi and Pira 2017), such as word2vec (Mikolov et al. 2013) or GloVe (Pennington et al., 2014) thanks to the adaptation to the context of use.

2The evaluation of these models has been conducted mainly on downstream tasks (Wang et al., 2018; Wang et al. 2019). With extrinsic evaluations, the models are fine-tuned, adapting the vector representations to specific tasks. The resulting vectors are then used as features in classification problems. This hinders a direct evaluation and analysis of the models because the evaluation also takes into account the ability of the classifier to learn the task. A model trained for this kind of task may learn only to discriminate among features that belong to each class with poor generalization.

3The interpretability of neural networks is an emerging line of research NLP that aims at analyzing the properties of pre-trained language models (Belinkov and Glass, 2019). Different studies have been conducted in recent years to discover what kind of linguistic information is stored in large neural language models. Many of them are focused on syntax (Hewitt and Manning, 2019; Jawahar et al., 2019) and attention (Michel, Levy, and Neubig 2019; Kovaleva et al. 2019). For what concerns semantics, the majority of the studies focus on common knowledge (Petroni et al. 2019) and inference and role-based event prediction (Ettinger 2020). Only a few of them have been devoted to lexical semantics, for example, Reif et al.(2019) show how different representations of the same lexical form tend to cluster according to their sense.

4In this work, we propose an in-depth analysis of the properties of the vector spaces induced by different embedding models and an evaluation of their word representations. We present how the properties of the vector space contribute to the success of the models in two tasks: sense induction and word sense disambiguation. In fact, even if contextualized models do not create one representation per word sense (Ethayarajh 2019), their contextualization create similar representations for the same word sense that can be easily clustered.

2. Related Work

5Given the success (and the opacity) of contextualized embedding models, many works have been proposed to analyze their inner representations. These analyses are based on probing tasks (Conneau et al., 2018) that aim at measuring how the information extracted from a pre-trained model is useful to represent linguistic structures. Probing tasks involve training a diagnostic classifier to determine if it encodes desired features. Tenney et al. (2019) discovered that specific BERT’s layers are more suited for representing information useful to solve specific tasks and that the ordering of its layers resembles the ordering of a traditional NLP pipeline: POS tagging, parsing, NER, semantic role labeling, and coreference resolution. Hewitt and Manning (2019)evaluated whether syntax trees are embedded in a linear transformation of a neural network’s word representation space. Hewitt and Liang (2019) raised the problem of interpreting the results derived from probing analysis. In fact, it is difficult to understand whether high accuracy values are due to the representation itself or, instead, they are the result of the ability to learn a specific task during training.

6Our work is more in line with works that try to find general properties of the representations generated by different contextualized models. For example, Mimno and Thompson (2017) demonstrated that the vector space produced by a static embedding model is concentrated in a narrow cone and that its concentration depends on the ratio of positive and negative examples. Mu and Viswanath (2018) explored this analysis further, demonstrating that the embedding vectors share the same common vector and have the same main direction. Ethayarajh (2019) demonstrated how upper layers of a contextualizing model produce more contextualized representations. We built on top of these works analyzing the vector space generated by contextualized models and evaluating them.

3. Construction of the Vector Spaces

7We used SemCor (Miller et al. 1993) as reference corpus for our work. This choice is motivated by the fact that it is the largest dataset manually annotated with sense information and it is commonly used as training set for word sense disambiguation. It contains 352 documents whose content words (about 226,000) have been annotated with WordNet (Miller 1995) senses. In total there are 33,341 unique senses distributed over 22,417 different words. The sense distribution in this corpus is very skewed, and follows a power law (Kilgarriff 2004). This makes the identification of senses challenging. The dataset is also difficult due to the fine granularity of WordNet (Navigli 2006).

8To construct the vector space A from SemCor we collected all the senses S_i of a word w_i and for each sense  we recovered the sentences  in which this particular sense occurs. These sentences are then fed into a pre-trained model and the token embedding representations of word w_i, , are extracted from the last hidden layer. This operation is repeated for all the senses in S_i, and for all the tagged words in the vocabulary, V. The vector space corresponds to all the representations of the words in V.

9A t-SNE visualization of the different embeddings in SemCor for the word foot is presented in Figure 1. In this Figure, we can see that the three main senses of foot (i.e., human foot, unit of length and lower part) occupy a definite position in the vector space, suggesting that the models are able to produce specific representations for the different senses of a word and that they lie on defined subspaces. In this work we want to test to what extent this feature is present in language models.

Figure 1: t-SNE representations for the word foot in SemCor, grouped by sense.

Implementations details

10The pre-trained models1 used in this study are: two BERT (???) models, base cased (12-layer, 768-hidden, 12-heads, 110M parameters) and large cased (24-layer, 1024-hidden, 16-heads, 340M parameters); three GPT-2 (Radford et al. 2019) models, base (12-layer, 768-hidden, 12-heads, 117M parameters), medium (24-layer, 1024-hidden, 16-heads, 345M parameters) and large (36-layer, 1280-hidden, 20-heads, 774M parameters); two RoBERTa (Liu et al. 2019) models, base (12-layer, 768-hidden, 12-heads, 125M parameters) and large (24-layer, 1024-hidden, 16-heads, 355M parameters); two XLNet (Yang et al. 2019) models, base (12-layer, 768-hidden, 12-heads, 110M parameters) and large (24-layer, 1024-hidden, 16-heads, 340M parameters); one XLM (Lample et al. 2019) model (12-layer, 2048-hidden, 16-heads) and one CTRL (Keskar et al. 2019) model (48-layer, 1280-hidden, 16-heads, 1.6B parameters). The main features of these models are summarized in Table 1. We averaged the embeddings of sub-tokens to obtain token-level representations.

3.1 Analysis

11The first objective of this work is to analyze the vector space produced with the models. This analysis is aimed at investigating the properties of the contextualized vectors. A detailed description of the embedding spaces constructed with the pre-trained models is presented in Table 2. We computed the norm for all the vectors in the vector space A, and averaged them:

12(1)

13This measure gives us an intuition on how diverse the semantic space constructed with the different models is. In fact, we can see that the magnitude of the vectors constructed with BERT, RoBERTa, XLNet, and XLM is low while those of GPT-2 and CTRL are very high.

14We computed also the norm of the vector resulting in averaging all the vectors in the semantic space V, as:

15    (2)

16All the semantic spaces have non-zero mean and the mean norm is high. This result suggests that the vectors contain redundant information and share a common nonzero vector. This is not only because the vector space contains representations of the same sense. In fact, if we create a new semantic space, Â, averaging all the representations of the same word sense, the MeanVecNorm of this space is still high for all the models.

17We used the Maximum Explainable Variance (MEV) for the representations of each word in V. This measure corresponds to the proportion of the variance in the embeddings that can be explained by their first principal components and was computed as:

(3)

where  is the first principal component of the vector space œ. It can give an upper bound on how contextualized representations can be replaced by a static embedding (Ethayarajh 2019). The model with the lowest MEV is BERT_large and XLNet_large.

18The other measures that we used for the evaluation of the vector space are based on the very notion of a cluster, which imposes that the data points inside a cluster must satisfy two conditions: internal similarity and external dissimilarity (Pelillo 2009). To this end, we used the senses of each word in the vocabulary of SemCor as clusters and extracted the corresponding vectors from V. We then computed the internal similarity of a cluster, c, as:

      (4)

where n is the number of data points in the cluster. We computed also the external similarity of a cluster c by computing the cosine similarity among each point in c and all the points in the subspace S induced by the senses of a word that has c as one of its senses:

    (5)

where m is the total number of data points in the subspace S (excluding those in c) and n is the number of points in the cluster c. Our hypothesis is that good representations should have high internal similarity and low external similarity and that the difference between them should be high.

19As it can be seen from Table 2 the internal similarity is higher than the external for all the models. Despite this, the scores are in a wide range. The lowest IntSim is given by BERT_large and the highest by RoBERTa_large and XLNet_base. The lowest ExtSim is given by BERT_large and the highest by XLNet_base. The largest difference between the two measures is given by BERT_large. RoBERTa_large gives has also a large gap between the two measures, furthermore, their standard deviation is very low. As we will see in Section 4 these last two models perform better than others in clustering and classification tasks.

Figure 1: The first 500 principal components computed on A and Â.

4. Evaluation

Sense Induction

20This task is aimed at understanding if representations belonging to different senses can be separated using an unsupervised approach. We hypothesize that a good contextualization process should produce more discriminative representations that can be easily identified by a clustering algorithm.

21We used the sense clusters extracted from SemCor as ground truth for this experiment (see Section 3) and grouped them if they are senses of the same word (with a given part of speech). We retained only the groups that have at least 20 data points and we discarded also monosemous words for the evaluation on k-means. The resulting datasets consist of 1871 (entire) and 1499 (without monosemous words) sub-datasets with 141,074 and 116,019 data points in total, respectively. We computed the accuracy on each sub-dataset computing the number of data points that have been clustered correctly and averaged the results to measure the performance of each model.

22The first algorithm is k-means (Lloyd 1982). It is a partitioning, iterative algorithm whose objective is to minimize the sum of point-to-centroid distances, summed over all k clusters. We used the k-means++ heuristic (Arthur and Vassilvitskii 2007) and the cosine distance metric to determine distances. We selected this algorithm because it is simple, non-parametric, and is widely used. It is important to notice that k-means requires the number of clusters to extract, for this reason, we restricted the evaluation only to ambiguous words.

23The second algorithm used is dominant-set (Pavan and Pelillo 2007). It is a graph-based algorithm that extracts compact structures from graphs generalizing the notion of maximal clique defined on unweighted graphs to edge-weighted graphs. We selected this algorithm because it is non-parametric, requires only the adjacency matrix of a weighted graph as input, and, more importantly, does not require the number of clusters to extract. The clusters are extracted from the graph sequentially using a peel-off strategy. This feature allows us to include in the evaluation also unambiguous words and to see if their representations are grouped into a single cluster or partitioned into different ones. We used cosine similarity to weigh the edges of the input graph.

24The results of this evaluation are presented in Table 3. RoBERTa and BERT have the overall best performances on this task using both algorithms. In particular, RoBERTa_large performs consistently well on all parts of speech and across algorithms, while other models perform well only in combination with one of the two algorithms. This is presumably owing to the big gap between the internal and the external similarity produced by this model, as explained in Section 3.1.

25This evaluation tends to confirm the claim that larger versions of the same model achieve better results. From Table 3, we can also see that the models have more difficulties in identifying the different senses of verbs, while nouns and adverbs have higher results. This is probably due to the different distribution of these word classes in the training sets of the models and WordNet’s fine-granularity. The performances of the models with dominant-set are surprisingly high, considering that the setting of this experiment is completely unsupervised. Furthermore, this algorithm is conceived to extract compact clusters and this feature could drive it to over partition the vector space of monosemous words. Instead, the results suggest the opposite: that the models are able to produce representations with high internal similarity, positioning their representations on a defined sub-space.

Word Sense Disambiguation

26We used the method proposed in to create sense vectors from contextualized word vectors. This method consists in averaging all the representations of a given sense. The resulting vector space corresponds to  (see Section 3.1). We evaluated the generated vectors on a standard benchmark (Raganato, Camacho-Collados, and Navigli 2017) for WSD. It consists of five datasets that were unified to the same WordNet version: Senseval-2 (S2), Senseval-3 (S3), SemEval-2007 (S7), SemEval-2013 and SemEval-2015, having in total 10,619 target words.

27The identification of word senses is conducted by feeding the entire texts of the datasets into a pre-trained model and extracting, for each target word w_i, its embedding representation  as was done for the construction of the semantic space. Once these representations are available, we compute the cosine similarities among  and the embeddings in  constructed with the same model and selected the sense with the highest similarity. We did not use more sophisticated models such as WSD-games (Tripodi and Navigli 2019; Tripodi, Vascon, and Pelillo 2016) because we wanted to keep the evaluation as simple as possible as not to influence the evaluation of the results.

28The results of this evaluation are presented in Table 4. The first trend that emerges from the results is the big gap between precision and recall. This is due to the absence of many senses in our training set. We did not want to use back-off strategies or other techniques usually employed in the WSD literature, to not influence the performances and the analysis of the results. Despite the simplicity of the approach, it performs surprisingly well. In particular, BERT, RoBERTa, and XLNet (three bidirectional models) have very high results. The low performances of CTRL are probably due to its large vocabulary and to its objective, designed to solve different tasks.

5. Conclusion and Future Work

29We conducted an extensive analysis of the semantic capabilities of contextualized embedding models. We analyzed the vector space constructed using pre-trained models and found that their vectors contain redundant information and that their first two principal components are dominant.

30The results on sense induction are promising. They demonstrated the effectiveness of contextualized embeddings to capture semantic information. We did not find higher performances from more complex models, rather, we found that RoBERTa, a model that was developed by simplifying a more complex model, BERT, was one of the best performers. Neither the dimension of the hidden layers, the size of the training data, nor the size of the vocabulary seems to play a big role in modeling semantics. As stated in previous works, inserting an anisotropy penalty to the objective function of the models could improve directly the representations. We also noticed that, even if BERT models and XLNet have different objectives and are trained on different data, they have similar performances. It emerged that these models are less redundant than others.

31The conclusion that we can draw from our analysis and evaluation is that pre-trained language models can capture lexical-semantic information and that unsupervised models can be used to distinguish among their representations. On the other hand, these representations are redundant and anisotropic. We hypothesize that reducing these aspects can lead to better representations. This operation can be carried out post-hoc but we think that training new models keeping this point in mind could lead to the development of better models.