1 Introduction

1Modern social networks allow users to express themselves, writing their opinions about facts, things and events. In social posting, people often adopt figurative languages, e.g. Irony and Sarcasm. These communication mechanism must be carefully considered in the automatic processing of texts in social media: as an example, they may be used to convey the opposite of literal meaning and thus just intentionally sketching a secondary or extended meaning (Grice, 1975). On Twitter, users can express themselves with very short messages. Given the short length, the information useful to detect figurative uses of natural language is very limited or missing. Irony and sarcasm detection represents challenging tasks within Sentiment Analysis and Opinion Mining often undermining the overall system accuracy. There is not a clear separation between irony and sarcasm, but the former is often considered to include the latter. In particular sarcasm is defined as sharp or cutting ironic expressions towards a particular target with the intention to offend (Joshi et al., 2016).

2This paper presents and describes the UNITOR system participating in the Irony Detection in Italian Tweets (IronITA) task (Cignarella et al., 2018) within the EvalIta 2018 evaluation campaign. The system faces both the proposed subtasks within IronITA: Irony Classification and Sarcasm Classification. In a nutshell, the former subtask aims at evaluating the performance of a system in capturing whether a message is ironic or not. The second subtask is intended to verify if, given an ironic tweet, a system is able to detect sarcasm within the message.

3The classification of each tweet is carried out by applying a cascade of kernel-based Support Vector Machines (Vapnik, 1998). In particular, two binary SVM classifiers (one per subtask) are designed to adopt specific combinations of different kernel functions, each operating over a task-specific tweet representation. This work extends the modeling proposed in (Castellucci et al., 2014) that was proved to be beneficial within the Irony Detection subtask within SENTIPOLC 2014. The UNITOR system here presented ranked 1^st and 2^nd in the Sarcasm Detection subtask, while it ranked 6^th and 7^th within the Irony Detection subtask.

4In Section 2 the SVM classifiers, their features and the underlying kernels are described and the adopted workflow is presented. In Section 3 the performance measures of the system are reported, while Section 4 derives the conclusions.

2 System Description

5The UNITOR system adopts a supervised learning setting where a multiple kernel-based approach is adopted to acquire two binary Support Vector Machine classifiers (Shawe-Taylor and Cristianini, 2004): a first classifier discriminates between ironic and non ironic tweets, while a second one decides whether an ironic tweet is sarcastic or not. In the rest of this section, we first summarize the pre-processing stage as well as the adopted linguistic resources (e.g. word embeddings or lexicons). Then, the feature modeling designed for the two steps is discussed.

2.1 Tweet processing and resources

6Each tweet is linguistically processed through an adapted version of the Chaos parser (Basili and Zanzotto, 2002) in order to extract the information required for feature modeling, e.g. the Part-of-speech tags and lemmas of individual words. A normalization step is applied before the standard Natural Language Processing activity is carried out. A number of actions is performed: fully capitalized words are converted into their lowercase counterparts; hyperlinks are replaced by a special token, i.e. LINK; characters repeated more than three times are cleaned, as they increase lexical data sparseness (e.g. “nooo!!!!!” is converted into “noo!!”); all emoticons are replaced by special tokens¹.

7In the feature modeling activities, we relied on several linguistic resources, hereafter reported.

8First, we used a Word Space model (or Word Embedding) to generalize the lexical information of the (quite small) training material: this semantic space is obtained starting from a corpus of Italian tweets downloaded in July 2016 of about 10 millions of tweets (same used in Castellucci et al. (2016a)) and it is a 250-dimensional embedding generated according to a Skip-gram model (Mikolov et al., 2013)².

9Moreover, we adopted a large scale sentiment specific lexicon, i.e., the Distributional Polarity Lexicons (DPL) (Castellucci et al., 2016b)³. Distributional Polarity Lexicon (DPL) is introduced to inject sentiment information of words in the learning process through a large-scale polarity lexicon that is automatically acquired according to the methodology proposed in (Castellucci et al., 2015). This method leverages on word embeddings to model lexical polarity by transferring it from entire sentences whose polarity is known. The process is based on the capability of word embeddings to represent both sentences and single words in the same space (Landauer and Dumais, 1997). First, sentences (here tweets) are labeled with some polarity classes: in (Castellucci et al., 2015) this labeling is achieved by applying simple heuristics, e.g. Distant Supervision (Go et al., 2009). The labeled dataset is projected in the embedding space by applying a simple but effective linear combination of the word vectors composing each sentence. Then, a polarity classifier is trained over these sentences in order to emphasize dimensions of the space that are more related to the polarity classes. The DPL is generated by classifying each word (represented in the embedding through a vector) with respect to each targeted class, using the confidence level of the classification to derive a word polarity signature. For example, in a DPL the word ottimo is 0.89 positive, 0.04 negative and 0.07 neutral. For more details, please refer to (Castellucci et al., 2015).

10Finally, we also adopted an Irony specific Corpus to capture terms and patterns that are often used to express irony (e.g., “non lo riconosceresti neanche se ti cascasse" or “… allora piove "): it is a corpus composed by a set of Italian tweets automatically extracted using Distance Supervision (Go et al., 2009). In particular the Irony specific Corpus is composed by a set of 6,000 random tweets in Italian, freely available, assumed to be ironic, as they contain hashtags such as #irony or #ironia.

2.2 Modeling irony and sarcasm in kernel-based learning

11UNITOR is based on kernel functions operating on vector representations of tweets, described hereafter. After the language processing stage, each tweet allows generating one of the following representations⁴ , later exploited by the kernel-based SVM in the training/classification steps.

2.2.1 Irony-specific Features

12The aim of this set of features is to capture irony by defining a set of irony-specific features inspired by the work of (Castellucci et al., 2014).

13Word Space Vector (WS) is a 250-dimensional vector representation of the average semantic meaning of a tweet according to a Word space model. It is used to generalize the lexical information of tweets. We can summarize it as , where T is the set of nouns, verbs, adjectives, adverb and hashtag in a tweet t and We(t) is a function that returns the 250-dimensional word embedding of the word t. Other words, such as articles and preposition are discarded as they do not convey useful information within a word space.

14Irony Specific BOW (ISBOW) is a BoW vector representing the lexical information expressed in a message. The main difference with respect to a conventional BOW representation is the adopted weighting scheme. In fact, in this case we leverage on the Word Space previously described. For each dimension representing a lemma/part-of-speech pair, its weight is computed as the cosine similarity between the word embedding vector of the considered word and the vector obtained from the linear combination of all the other words in the message (WS)⁵. This vector aims at capturing how much odd is the occurrence of a given word in a sentence aiming at capturing its unconventional uses: it should be an indicator of potential ironic mechanisms, as suggested in (Castellucci et al., 2014).

15Irony Specific BOW(Adjective, Noun, Verb) (ISBOW-A), (ISBOW-S), (ISBOW-V) are three BoW vectors that use the same weighting scheme specified in ISBOW. Each vector represents one individual part of speech (i.e. adjective, noun and verb), as words belonging to different POS-tag categories may be characterized by quite different distributions.

16Irony Specific Mean and Variance (ISMV) is a 4-dimensional vector representation that summarized the information captured by the previous representations. It contains mean and variance of the cosine similarity, calculated between the words in a tweet in the ISBOW representation, and the maximum and minimum of the cosine similarity per tweet. This vector aims at summarizing the distribution and potential "spikes" of unusual patterns of use for words in a sentence.

17Irony Specific Mean and Variance (Adjective, Noun, Verbs) (ISMV-A), (ISMV-S), (ISMV-V) are three distinct 4-dimensional vectors that are the same specified in ISMV, with the only difference that each representation works on one specific part of speech, respectively adjectives, nouns and verbs.

18Char n-gram BOWs (n-CHARS) is a representation expressing the char n-grams contained in a message. We used 4 n-CHARS representations: 2-CHARS BoW vector representing 2-char-ngrams contained in a message, 3-CHARS BoW vector representing 3-char-ngrams, 4-CHARS BoW vector representing 4-char-ngrams, 5-CHARS BoW vector representing 5-char-ngrams. The aim of this representation is to capture the usage of specific textual patterns, e.g., hihihihi often used to express irony.

19Synthetic Features (SF) is a 7-dimensional vector containing the following synthetic features, traditionally used in Sentiment Analysis: percentage of the number of uppercase letters in the tweet, number of exclamation marks, number of question marks, number of colons, number of semicolons, number of dots, number of commas. It has been inspired by works on irony detection of (Carvalho et al., 2009; Reyes et al., 2012).

2.2.2 Features based on Distribution Polarity Lexicons

20The aim of this group of features is to exploit the negative evaluation towards a target typical of sarcasm mechanism (Joshi et al., 2016) using a polarity lexicon, here a Distribution Polarity Lexicon (DPL).

21Distributional Polarity Lexicon Sum (DSUM) is a 15-dimensional vector representation made by the concatenation of 5 different representations, i.e. where N_t, V_t, Adj_t, Adv are the nouns, verbs, adjectives and adverbs occurring in the tweet, and w^p expresses the 3-dimensional polarity lexicon entry⁶ for the word w. This feature summarize the a-priori sentiment of words according to the different morphological categories. We speculate that the regularities or contrasts between these distributions may suggest the presence of irony or sarcasm.

22Distributional Polarity Lexicon BoW (DBOW) is a BoW vector representing, for each word in a message, its polarity (positive, negative and neutral) as a three dimensional score derived from the DPL.

2.2.3 Irony Corpus Features

23Generalizing linguistic information useful for Irony or Sarcasm detection is a very challenging tasks, as the adoption of these figurative languages mainly concern extra-linguistic phenomena. The idea underlying the following features is to define a tweet representation that is not directly connected to their (possibly limited) linguistic material, but that is connected with respect to a larger set of information derived from a Irony specific Corpus, i.e., a large scale collection of a Ironic tweets. This is used to extract an Irony specific Lexicon: a set of words and patterns occurring in such corpus with a high frequency.

24Irony Corpus BOW (ICBOW) is a BoW vector representing lemmas of Nouns, Verbs, and Adjective in a message. Again, the main difference with respect to a conventional BoW representation is the adopted weighting scheme: a word is weighted 1.0 if that particular word was in the Irony specific Corpus, otherwise is weighted 0.

25Irony Corpus weighted BOW (ICwBOW) is a BoW vector representing lemmas of Nouns, Verbs, and Adjective in a message. A word is weighted log(f+1) where f is the frequency of that particular word in the Irony Corpus.

26Irony Corpus weighted Mean (ICM) is a 2-dimensional vector representation that summarize the mean words weight observed in a ICBOW representation and the mean over the ICwBOW. These scores indicate how a words or patterns in a tweet occur also in the Irony specific corpus. This information is very interesting as it is not tied to the lexical information from a tweet, so allowing a more robust generalization.

27Irony Corpus BOW (bi-grams, three-grams) (IC2BOW), (IC3BOW) are two distinct BoW vector respectively representing bi-grams and three-grams of surface words in a message. The weighting scheme is the same explained in ICBOW.

28Irony Corpus weighted BOW (bi-grams, three-grams) (IC2wBOW), (IC3wBOW) are two distinct BoW vectors respectively representing bi-grams and three-grams of terms in a message. The weighting scheme is the same explained in ICwBOW.

29Irony Corpus weighted Mean (bi-grams, three-grams) (IC2M), (IC3M) are two distinct 2-dimensional vector representations that contain means that are the same specified in ICM, with the only difference that the first representation works on bi-grams (IC2BOW, IC2wBOW), while the second works on three-grams (IC3BOW, IC3wBOW).

Figure 1: The UNITOR classifier workflow

3 Experimental evaluation and results

30The cascade of SVM classifiers implemented in UNITOR is summarized in Figure 1. After the linguistic processing stage and the feature extraction stage, each tweet is classified by a binary classifier, the so-called irony classifier. If a message is judged as not ironic, we assume that it is also not sarcastic (according to the task guidelines) and a label 0 0 is assigned to it. Otherwise, if the tweet is judged as ironic, the second binary classifier, devoted to Sarcasm Detection, is invoked. If positive, the tweet is sarcastic and the message is labeled with 1 1, otherwise, 1 0.

31Separated representations are considered in the constrained and unconstrained settings, according to the guidelines in (Cignarella et al., 2018). In the constrained setting only feature vectors using tweet information or public available lexicons are considered (Irony-specific Features and Features derived from a DPL). In the unconstrained setting, feature vectors are derived also using the Irony specific Corpus.

32In our experiments, we train the SVM classifiers using the same kernel combination for Irony Detection and Sarcasm Detection. Even if this is not a general solution (different tasks may require different representations) we adopted this greedy strategy, leaving the SVM to select the most discriminative information.

33A normalized linear combination of specific kernel functions is used in both subtasks. In the linear combination, a specific linear kernel is applied to the following sparse representations: ISBOW, ISBOW-A, ISBOW-S, ISBOW-V, DBOW, 2BOW, 3BOW, 4BOW, 5BOW, ICBOW, IC2BOW, IC3BOW, ICwBOW, IC2wBOW, IC3wBOW; in the same combination a RBF kernel (Shawe-Taylor and Cristianini, 2004) is applied to the following dense representations WS, SF, ICM, IC2M, IC3M, DSUM, ISMV, ISMV-A, ISMV-S, ISMV-V⁷.

34Each SVM classifier is built by using the KeLP framework⁸ (Filice et al., 2018).

35Figure 1 reflects also the learning strategy that has been set up during the training phase: the Irony Classifier was trained on the complete training dataset composed by the entire training set (made of 3,977 tweets) while the Sarcasm Classifier is trained only on the ironic tweets in the training dataset (made of 2,023 tweets). A 10-fold cross validation strategy was applied to optimize the SVM parameters, while the linear combination of the kernel assigns the same weights to each kernel function.

36In Table 1 the performances of the Irony Classification task are reported: in the constrained run the UNITOR system ranks 7^th, while in 6^st position in the unconstrained one. For this task the adopted representations were able to correctly determine whether a message is ironic with good precision. However, the winning system (about 3 points ahead) results more effective in the detection of non-ironic messages. In fact, according to the F1-score on the Ironic class, the system would have been ranked 2^nd. We also evaluated a slightly different modeling with two additional features vector, i.e., a classic BoW composed of lemmas derived from the input tweet, and a BoW of bi-grams. These features have been excluded from our official submission to keep the model simple. However, these simple features would have been beneficial and the system would have ranked 2^nd. Performances on the Sarcasm Classification are in Table 2: UNITOR here ranks in 1^st or in 2^nd position, in the constrained and unconstrained run, respectively. Differences between the two results are not significant. Nevertheless the further features derived from the Irony specific corpus allow improving results (especially in terms of recall) in the Sarcasm Detection task. For this latter task, results achieved by UNITOR suggest that the proposed modeling, in particular the contribution of Polarity Features, seem to be beneficial. To prove it, we decided to evaluate a run with the same winning features, except Polarity Features. In this case the UNITOR system would have been ranked 4^th. These Polarity Features seem to exploit the negative bias typical of sarcasm (Joshi et al., 2016).

4 Conclusions

37In this paper we described the UNITOR system participating to the IronITA task at EvalIta 2018. The system won 1 of the 2 evaluations carried out in the task, and in the worst case it ranked in the 6^th position. The good results in constrained and unconstrained settings suggest that the proposed irony and sarcasm specific features were beneficial to detect irony and sarcasm also in short messages. However, further work is needed to improve the non ironic F1 scores. The nature of the task seems to be non trivial also for a human reader, as some tweets extracted from the test set suggest: “@beppe_grillo Beppe..tu sei un grande..questi si stanno finendo di mangiare l’Italia..", “scusa hai ancora posti liberi nella app di braccialetti rossi?"; here the interpretation of irony goes beyond the textual information and it is very difficult to state if these messages are ironic or not. Since tweets are very short, useful information for detecting irony is often out of the message, like this ironic tweet extracted from the test set may suggest: “immagine perfetta ed esplicita che descrive la realtá della ”buona scuola” a renzopoli"; in this case the system may fail without a proper representation for the meaning of the neologism “renzopoli". So we think that the contextual approach suggested in (Vanzo et al., 2014) will be explored in future research.