1. Introduction

1When faced with the same situation, different humans behave differently. This is, of course, due to different backgrounds, education paths, and life experiences, but according to psychologists there is another important aspect: personality (Snyder 1983; Parks and Guay 2009).

2Human Personality is a psychological construct aimed at explaining the wide variety of human behaviours in terms of a few, stable and measurable individual characteristics (Vinciarelli and Mohammadi 2014).

3Such characteristics are formalised in Trait Models, and there are currently two of these models that are widely adopted: Big Five (John and Srivastava 1999) and Myers-Briggs Type Indicator (MBTI) (Myers and Myers 1995). The first examines five dimensions (Openness to experience, Conscientiousness, Extroversion, Agreeableness and Neuroticism) and for each of them assigns a score in a range. The second one, instead, considers 16 fixed personality types, coming from the combination of the opposite poles of 4 main dimensions (Extravert-Introvert, iNtuitive-Sensing, Feeling-Thinking, Perceiving-Judging). Examples of full personality types are therefore four letter labels such as ENTJ or ISFP.

4The tests used to detect prevalence of traits include human judgements regarding semantic similarity and relations between adjectives that people use to describe themselves and others. This is because language is believed to be a prime carrier of personality traits (Schwartz et al. 2013). This aspect, together with the progressive increase of available user-generated data on social media, has prompted the task of Personality Detection, i.e., the automatic prediction of personality from written texts (Youyou, Kosinski, and Stillwell 2015; Argamon et al. 2009; Litvinova et al. 2016; Whelan and Davies 2006).

5Personality detection can be useful in predicting life outcomes such as substance use, political attitudes and physical health. Other fields of application are marketing, politics and psychological and social assessment.

6As a contribution to personality detection in Italian, we present Personal-ITY, a new corpus of YouTube comments annotated with MBTI personality traits, and some preliminary experiments to highlight its characteristics and test its potential. The corpus is made available to the community1.

2. Related Work

7There exist a few datasets annotated for personality traits. For the shared tasks organised within the Workshop on Computational Personality Recognition (Celli et al. 2013), two datasets annotated with the Big Five traits have been released in 2013 (Essays (Pennebaker and King 2000) and myPersonality2) and two in 2014 (YouTube Personality Dataset (Biel and Gatica-Perez 2013) and Mobile Phones interactions (Staiano et al. 2012)).

8For the 2015 PAN Author Profiling Shared Task (Pardo et al. 2015), personality was added to gender and age in the profiling task, with tweets in English, Spanish, Italian and Dutch. These are also annotated according to the Big Five model.

9Still in the Big Five landscape, collected a dataset of FaceBook comments (700 millions words) written by 136.000 users who shared their status updates. Interesting correlations were observed between word usage and personality traits.

10If looking at data labelled with the MBTI traits, we find a corpus of 1.2M English tweets annotated with personality and gender (Plank and Hovy 2015), and the multilingual TwiSty (Verhoeven, Daelemans, and Plank 2016). The latter is a corpus of data collected from Twitter annotated with MBTI personality labels and gender for six languages (Dutch, German, French, Italian, Portuguese and Spanish) and a total of 18,168 authors. We are interested in the Italian portion of TwiSty.

11Table 1 contains an overview of the available Italian corpora labelled with personality traits. We include our own, which is described in Section 3.

12Regarding detection approaches, tested the usefulness of different sets of textual features making use of mostly SVMs.

13At the PAN 2015 challenge (see above) a variety of algorithms were tested (such as Random Forests, decision trees, logistic regression for classification, and also various regression models), but overall most successful participants used SVMs. Regarding features, participants approached the task with combinations of style-based and content-based features, as well as their combination in n-gram models (Pardo et al. 2015).

14Experiments on TwiSty were performed by the corpus creators themselves Experiments on TwiSty were performed by the corpus creators themselves Experiments on TwiSty were performed by the corpus creators themselves Experiments on TwiSty were performed by the corpus creators themselves Experiments on TwiSty were performed by the corpus creators themselves corpus creators themselves using a LinearSVM with word (1-2) and character (3-4) n-grams. Their results (reported in Table 2 for the Italian portion of the dataset) are obtained through 10-fold cross-validation; the model is compared to a weighted random baseline (WRB) and a majority baseline (MAJ).

3. Personal-ITY

15First, we explain two major choices that we made in creating Personal-ITY, namely the source of the data and the trait model. Second, we describe in detail the procedure we followed to construct the corpus. Lastly, we provide a description of the resulting dataset.

Data

16YouTube is the source of data for our corpus. The decision is grounded on the fact that compared to the more commonly collected tweets, YouTube comments can be longer, so that users are freer to express themselves without limitations. Additionally, there is a substantial amount of available data on the YouTube platform, which is easy to access thanks to the free YouTube APIs.

Trait Model

17Our model of choice is the MBTI. The first benefit of this decision is that this model is easy to use in association with a Distant Supervision approach (just checking if a message contains one of the 16 personality types; see Section 3.1). Another benefit is related to the existence of TwiSty. Since both TwiSty and Personal-ITY implement the MBTI model, analyses and experiments over personality detection can be carried out also in a cross-domain setting.

Ethics Statement

18Personality profiling must be carefully evaluated from an ethical point of view. In particular, often, personality detection involves ethical dilemmas regarding appropriate utilization and interpretations of the prediction outcomes (Weiner and Greene 2017). Concerns have been raised regarding the inappropriate use of these tests with respect to invasion of privacy, cultural bias and confidentiality (Mehta et al. 2019).

19The data included in the Personal-ITY dataset were publicly available on the YouTube platform at the time of the collection. As we will explain in detail in this Section, the information collected are comments published under public videos on the YouTube platform by authors themselves. For a major protection of user identities, in the released corpus only the YouTube usernames of the authors are mentioned which are not unique identifiers. The YouTube IDs of the corresponding channels, which are the real identifiers in the platform, allowing to trace the identity of the authors, are not released. Note also that the corpus was created for academic purposes and is not intended to be used for commercial deployment or applications.

3.1 Corpus Creation

20The fact that users often self-disclose information about themselves on social media makes it possible to adopt Distant Supervision (DS) for the acquisition of training data. DS is a semi-supervised method that has been abundantly and successfully used in affective computing and profiling to assign silver labels to data on the basis of indicative proxies (Go, Bhayani, and Huang 2009; Pool and Nissim 2016; Emmery, Chrupała, and Daelemans 2017).

21Users left comments to some videos on the MBTI theory in which they were stating their own personality type (e.g. Sono ENTJ...chi altro? [en: "I’m ENTJ...anyone else?"]). We exploited such comments to create Personal-ITY with the following procedure.

22First, we searched for as many Italian YouTube videos about MBTI as possible, ending up with a selection of ten with a conspicuous number of comments as the ones above3.

23Second, we retrieved all the comments to these videos using an AJAX request, and built a list of authors and their associated MBTI label. A label was associated to a user if they included an MBTI combination in one of their comments. Table 3 shows some examples of such associations. The association process is an approximation typical of DS approaches. To assess its validity, we manually checked 300 random comments to see whether the mention of an MBTI label was indeed referred to the author’s own personality. We found that in 19 cases (6.3%) our method led to a wrong or unsure classification of the user’s personality (e.g. O tutti gli INTJ del mondo stanno commentando questo video oppure le statistiche sono sbagliate :-)). We can assume that our dataset might therefore contain about 6-7% of noisy labels.

24Using the acquired list of authors, we meant to obtain as many comments as possible written by them. The YouTube API, however, does not allow to retrieve all comments by one user on the platform. In order to get around this problem we relied on video similarities, and tried to expand as much as possible our video collection. Therefore, as a third step, we retrieved the list of channels that feature our initial 10 videos, and then all of the videos within those channels.

25Fourth, through a second AJAX request, we downloaded all comments appearing below all videos retrieved through the previous step.

26Lastly, we filtered all comments retaining those written by authors included in our original list. This does not obviously cover all comments by a relevant user, but it provided us with additional data per author.

3.2 Final Corpus Statistics

27For the final dataset, we decided to keep only the authors with a sufficient amount of data. More specifically, we retained only users with at least five comments, each at least five token long.

28Personal-ITY includes 96,815 comments by 1048 users, each annotated with an MBTI label. The average number of comments per user is 92 and each message has on average 115 tokens.

29The amount of the 16 personality types in the corpus is not uniform. Figure 1 shows such distribution and also compares it with the one in TwiSty. The unbalanced distribution can be due to personality types not being uniformly distributed in the population, and to the fact that different personality types can make different choices about their online presence. for example, observed that there is a significant correlation between online–offline choices and the MBTI dimension of Extravert-Introvert: extroverts are more likely to opt for offline modes of communication, while online communication is presumably easier for introverts. In Figure 1, we also see that the four most frequent types are introverts in both datasets. The conclusion is that, despite the different biases, collecting linguistic data in this way has the advantage that it reflects actual language use and allows large-scale analysis (Plank and Hovy 2015).

Figure 1: Distribution of the 16 personality types in the YouTube corpus and in the Italian section of TwiSty

30Figure 2 shows more in detail, trait by trait, the distribution of the opposite poles through the users in Personal-ITY and in TwiSty. As we might have expected, in line with what is observed in Figure 1, the two datasets present very similar trends. Such similarities between Personal-ITY and TwiSty are these similarities are a further confirmation of the reliability of the data we collected.

Figure 2: Comparison of the distributions of the four MBTI traits between Personal-ITY and the Italian part of TWISTY

4. Preliminary Experiments

31We ran a series of preliminary experiments on Personali-ITY which can also serve as a baseline for future work on this dataset. We pre-processed texts by replacing hashtags, urls, usernames and emojis with four corresponding placeholders. We adopted the sklearn (Pedregosa et al. 2011) implementation of a linear SVM (LinearSVM), with standard parameters. We tested three types of features. At the lexical level, we experimented with word (1-2) and character (3-4) n-grams, both as raw counts as well as tf-idf weighted. Character n-grams were tested also with a word-boundary option. At a more stylistically level, we considered the use of emojis, hashtags, pronouns, punctuation and capitalisation. Lastly, we also experimented with embeddings-based representations, by using, on the one hand, YouTube-specific (Nieuwenhuis and Nissim 2019) pre-trained models, on the other hand, more generic embeddings, such as the Italian version of GloVe (Pennington, Socher, and Manning 2014), which is trained on the Italian Wikipedia4. We looked for all the available embeddings of the words written by each author, and used the average as feature. If a word appeared more than once in the string of comments, we considered it multiple times in the final average.

32We used 10-fold cross-validation, and assessed the models using macro f-score. Note that the original TwiSty paper uses micro f-score. Thus, for the sake of comparison, we include also micro-F in Table 5 for the MAJ baseline and our lexical n-gram model. Table 4 shows the results of our experiments with different feature types.5 Overall, lexical features (n-grams) perform best. Combining different feature types did not lead to any improvement. Classification was performed with four separate binary classifiers (one per dimension), and with one single classifier predicting four classes, i.e, the whole MBTI labels at once. In the latter case, we observe that the results are quite high considering the increased difficulty of the task. Table 5 reports the scores of our models on TwiSty. As for Personal-ITY, best results were achieved using lexical features (tf-idf n-grams); stylistic features and embeddings are just above the baseline. Our model outperforms the one in (Verhoeven, Daelemans, and Plank 2016) for all traits (micro-F).

33To test compatibility of resources and to assess model portability, we also ran cross-domain experiments on Personal-ITY and TwiSty. In the first setting, we tested the effect of merging the two datasets on the performance of models for personality detection, maintaining the 10-fold cross-validation setting and by using the model performing better on average for YouTube and Twitter data (a character n-grams model). Table 6 contains the result of such experiments6. Scores are almost always lower compared to the in-domain experiments (excepts for NS as regards Twitter scores reported in Table 5: 46.15 48.31), but quite increased compared to the majority baseline.

34In the second setting, instead, we divided both corpora in fixed training and test sets with a proportion of 80/20 and ran the models using lexical features, in order to run a cross-domain experiment. For direct comparison, we run the model in-domain again using this split. Results are shown in Table 7. Cross-domain scores are obtained with the best in-domain model.7 They drop substantially compared to in-domain, but are always above the baseline.

5. Conclusions

35The experiments show that there is no single best model for personality prediction, as the feature contribution depends on the dimension considered, and on the dataset. Lexical features perform best, but they tend to be strictly related to the context in which the model is trained and so to overfit.

36The inherent difficulty of the task itself is confirmed and deserves further investigations, as assigning a definite personality is an extremely subjective and complex task, even for humans.

37Personal-ITY is made available to further investigate the above and other issues related to personality detection in Italian. The corpus can lend itself to a psychological analysis of the linguistic cues for the MBTI personality traits. On this line, it is interesting to investigate the presence of evidences linking linguistic features with psychological theories about the four considered dimensions (Extravert-Introvert, iNtuitive-Sensing, Feeling-Thinking, Perceiving-Judging). First results in this direction are presented in (Bassignana, Nissim, and Patti 2020).