1 Introduction

1In recent years, many websites started offering a high level interaction with users, who are no more a passive audience, but can actively produce new content. For instance, platforms like Amazon¹ or TripAdvisor² allow people to express their opinions on products, such as food, electronic items, clothes, and services, such as hotels and restaurants.

2In such a social context, Sentiment Analysis (SA) is the task of automatically extract subjective opinions from a text. In its most basic form, a SA system takes in input a text written in natural language and assign it a label indicating whether the text is expressing a positive or negative sentiment, or neither (neutral, or objective, text). However, reviews are often quite detailed in expressing the reviewer’s opinion on several aspects of the target entity. Aspect-based Sentiment Analysis (ABSA) is an evolution of Sentiment Analysis that aims at capturing the aspect-level opinions expressed in natural language texts (Liu, 2007).

3At the international level, ABSA was introduced as a shared task at SemEval, the most prominent evaluation campaign in the Natural Language Processing field, in 2014 (SE-ABSA14), providing a benchmark dataset of reviews in English (Pontiki et al., 2014). Datasets of computer laptops and restaurant reviews were annotated with aspect terms (both fine-grained, e.g. “hard disk”, “pizza”, and coarse-grained, e.g., “food”) and their polarity (positive or negative).

4The task was repeated in SemEval 2015 (SE-ABSA15) and 2016 (SE-ABSA16), aiming to facilitate more in-depth research by providing a new ABSA framework to investigate the relations between the identified constituents of the expressed opinions and growing up to include languages other than English and different domains (Pontiki et al., 2015; Pontiki et al., 2016).

5ABSITA (Aspect-based Sentiment Analysis on Italian) aims at providing a similar evaluation with respect to texts in Italian. In a nutshell, participants are asked to detect within sentences (expressing opinions about accommodation services) some of the aspects considered by the writer. These aspects belongs to a close set ranging from the cleanliness of the room to the price of the accommodation. Moreover, for each detected aspect, participants are asked to detect a specific polarity class, expressing appreciation or criticism towards it.

6During the organization of the task, we collected a dataset composed of more than 9,000 sentences and we annotated them with aspects and polarity labels. During the task, 20 runs were submitted by 7 teams comprising 11 individual participants.

7In the rest of the paper Section 2 provides a detailed definition of the task. Section 3 describes the dataset made available in the evaluation campaign, while Section 4 reports the official evaluation measures. In Section 5 and 6, the results obtained by the participants are reported and discussed, respectively. Finally, Section 7 derives the conclusions.

2 Definition of the task

8In ABSITA, Aspect-based Sentiment Analysis is decomposed as a cascade of two subtasks: Aspect Category Detection (ACD) and Aspect Category Polarity (ACP). For example, let us consider the sentence describing an hotel:

9I servizi igienici sono puliti e il personale cordiale e disponibile. (Toilets are clean but the staff is not friendly nor helpful.)

10In the ACD task, one or more “aspect categories” evoked in a sentence are identified, e.g. the pulizia (cleanliness) and staff categories in sentence 2. In the Aspect Category Polarity (ACP) task, the polarity of each expressed category is recognized, e.g. a positive category polarity is expressed concerning the pulizia category while it is negative if considering the staff category.

11In our evaluation framework, the set of aspect categories is known and given to the participants, so the ACD task can be seen as a multi-class, non-exclusive classification task where each input text has to be classified as evoking or not each aspect category. The participant systems are asked to return a binary vector where each dimension corresponds to an aspect category and the values 0 (false) and 1 (true) indicate whether each aspect has been detected in the text. Table 1 shows examples of annotation for the ACD task.

12For the ACP task, the input is the review text paired with the set of aspects identified in the text within the ACD subtask, and the goal is to assign polarity labels to each of the aspect category. Two binary polarity labels are expected for each aspect: POS an NEG, indicating a positive and negative sentiment expressed towards a specific aspect, respectively. Note that the two labels are not mutually exclusive: in addition to the annotation of positive aspects (POS:true, NEG:false) and negative aspects (POS:false, NEG:true), there can be aspects with no polarity, or neutral polarity (POS:false, NEG:false). This is also the default polarity annotation for the aspects that are not detected in a text. Finally, the polarity of an aspect can be mixed (POS:true, NEG:true), in cases where both sentiments are expressed towards a certain aspect in a text. Table 2 summarizes the possible annotations with examples.

13The participants could choose to submit only the results of the ACD subtask, or both tasks. In the latter case, the output of the ACD task is used as input for the ACP. As a constraint on the results submitted for the ACP task, the polarity of an aspect for a given sentence can be different than (POS:false, NEG:false) only if the aspect is detected in the ACD step.

3 Dataset

14The data source chosen for creating the ABSITA datasets is the popular website booking.com³. The platform allows users to share their opinions about hotels visited through a positive/negative textual review and a fine-grain rating system that can be used for assigning a score to each different aspect: cleanliness, comfort, facilities, staff, value for money, free/paid WiFi, location. Therefore, the website provides a large number of reviews in many languages.

15We extracted the textual reviews in Italian, labeled on the website with one of the eighth considered aspects. The dataset contains reviews left by users for hotels situated in several main Italian cities such as Rome, Milan, Naples, Turin, Bari, and more. We split the reviews into groups of sentences which describe the positive and the negative characteristics of the selected hotel. The reviews have been collected between the 16th and the 17th of April 2018 using Scrapy⁴, a Python web crawler. We collect in total 4,121 distinct reviews in Italian language.

16The reviews have been manually checked to verify the annotation of the aspects provided by booking.com, and to add missing links between sentences and aspects. We started by annotating a small portion of the whole dataset split by sentences (250 randomly chosen sentences) using four annotators (the task organizers) in order to check the agreement of the annotation. For the ACD task, we asked the annotators to answer straightforward questions in the form of “Is aspect X mentioned in the sentence Y?” (Tab. 1).

17The set of italian aspects is the direct translation of those booking.com: pulizia (cleanliness), comfort, servizi (amenities), staff, qualita-prezzo (value), wifi (wireless Internet connection) and posizione (location). Similarly, for the ACP subtask, the annotation is performed at sentence level, but with the set of aspects already provided by the ACD annotation, and checkboxes to indicate positive and negative polarity of each aspect (Tab. 2). The result of the pilot annotation has been used to compute an inter-annotator agreement measure, in order to understand if it was possible to allow annotators to work independently each other on a different set of sentences. We found agreement ranging from 82.8% to 100% with an average value of 94.4% obtained counting the number of sentences annotated with the same label by all the annotators.

18In order to complete the annotation, we assigned different 1,000 reviews to each annotator (about 2,500 sentences on average). We split the dataset among the annotators so that each of them received a uniformly balanced distribution of positive and negative aspects, based on the scores provided by the original review platform. Incomplete, irrelevant, and incomprehensible sentences have been discarded from the dataset during the annotation. At the end of the annotation process, we obtained the gold standard dataset with the associations among sentence, sentiment and aspect. The entire annotation process took a few weeks to complete. The positive and negative polarities are annotated independently, thus for each aspect the four sentiment combination discussed in Section 2 are possible: positive, negative , neutral and mixed. The resulting classes are: cleanliness_positive, cleanliness_negative, comfort_positive, comfort_negative, amenities_positive, amenities_negative, staff_positive, staff_negative, value_positive, value_negative, wifi_positive, wifi_negative, location_positive, location_negative, other_positive, other_negative. For each aspect, the sentiment is encoded in two classes:

• negative = (*_positive = 0, *_negative = 1)

• positive = (*_positive = 1, *_negative = 0)

• neutral = (*_positive = 0, *_negative = 0)

• mixed = (*_positive = 1, *_negative = 1)

19Please note that the special topic, other has been added for completeness, to annotate sentences with opinions on aspects not among the seven considered by the task. The aspect other is provided additionally and it is not part of the evaluation of results provided for the task.

20We released the data in Comma-separated Value format (CSV) with UTF-8 encoding and semicolon as separator. The first attribute is the id of the review. Note that in booking.com the order of positive and negative sentences is strictly defined and this can make too easy the task. To overcome this issue, we randomly assign for each sentence a new position in the review. As a consequence, the final positional id showed in the data file do not reflect the real order of the sentences in the review. The text of the sentence is provided at the end of the line and delimited by”. It is preceded by three binary values for each aspect indicating respectively: the presence in the sentence (aspectX_presence:0/1), the positive polarity for that aspect (aspectX_pos:0/1) and finally the negative polarity (aspectX_neg:0/1). Fig. 1 shows an example of the annotated dataset in the proposed format.

21The list of the datasets released for the task is provided in Tab. 3 and the distribution of the sentences among aspects and polarity is provided in Tab. 4. The subdivision adopted for it is respectively 0.34%, 69.75%, 29,91% for trial, training and test data. The datasets can be freely downloaded from http://sag.art.uniroma2.it/absita/ and reused in non-commercial projects and researches. After the submission deadline, we also distributed the gold standard test set and evaluation script.

4 Evaluation measures and baselines

22We evaluate the ACD and ACP subtasks separately by comparing the classifications provided by the participant systems to the gold standard annotations of the test set. For the ACD task, we compute Precision, Recall and F₁-score defined as: , where Precision (P_a) and Recall (R_a) are defined as: . Here S_a is the set of aspect category annotations that a system returned for all the test sentences, and G_a is the set of the gold (correct) aspect category annotations. For instance, if a review is labeled in the gold standard with the two aspects G_a = {cleanliness, staff}, and the system predicts the two aspects S_a = {cleaness, comfort}, we have that , and |S_a| = 2 so that , and . For the ACD task the baseline will be computed by considering a system which assigns the most frequent aspect category (estimated over the training set) to each sentence.

23For the ACP task we evaluate the entire chain, thus considering both the aspect categories detected in the sentences together with their corresponding polarity, in the form of (aspect,polarity) pairs. We again compute Precision, Recall and F₁-score now defined as . Precision (P_p) and Recall (R_p) are defined as , where S_p is the set of (aspect,polarity) pairs that a system returned for all the test sentences, and G_a is the set of the gold (correct) pairs annotations. For instance, if a review is labeled in the gold standard with the pairs G_p = {(cleanliness, POS), (staff, POS)}, and the system predicts the three pairs S_p = {cleanliness, POS), (cleanliness, NEG), (comfort, POS)}, we have that , |G_p| = 2 and |S_p| = 3 so that , and F1_a = 0.28.

24For the ACP task, the baseline is computed by considering a system which assigns the most frequent (aspect,polarity) pair (estimated over the training set) to each sentence.

Figure 1: Sample of the annotated dataset in CSV format

25We produced separate rankings for the tasks, based on the F₁ scores. Participants who submitted only the result of the ACD task appear in the first ranking only.

5 Results

26We received submissions from several teams that participated in past editions of EVALITA, in particular to the SENTIPOLC (Sentiment Polarity Classification (Barbieri et al., 2016)) and NEEL-it (Named Entity Recognition (Basile et al., 2016)), but also some new entries in the community. In total, 20 runs were submitted by 7 teams comprising 11 individual participants. The task allowed participant teams to send up to 2 submissions from each team. In particular, 12 runs were submitted to ACD task and 8 runs to the ACP task. We also provide the result of a baseline system that assigns to each instance the most frequent class in each task, i.e., the aspect (Comfort) and polarity (positive) for that aspect, according to the frequency of classes in the training set. The results of the submissions for the two tasks, and the baseline (namely mfc baseline), are reported in Tab. 5 and Tab. 6. Of the seven teams who participated to the ACD task, five teams also participated to the ACP task.

27The results obtained by the teams largely outperform the baseline demonstrating the efficacy of the solutions proposed and the affordability of all two tasks. The results obtained for the ACD task (Tab. 5) show a small range of variability, at least in the first part of the ranking (the top results are concentrated around a F1 score value of 0.80). On the contrary, the values of precision and recall show higher variability, indicating significant difference among the proposed approaches.

6 Discussion

28The teams of the ABSITA challenge have been invited to describe their solution in a technical report and to fill in a questionnaire, in order to gain an insight on their approaches and to support their replicability. Five systems (ItaliaNLP, gw2017, X2Check, UNIPV, SeleneBianco) are based on supervised machine learning, that is, all the systems for which we have access to the implementation details, with the exception of VENSES, which is a rule-based unsupervised system. Among the system that use supervised approaches, three systems (ItaliaNLP, gw2017, UNIPV) employ deep learning (in particular LTSM networks, often in their bi-directional variant).

29All runs submitted can be considered “constrained runs”, that is, the systems were trained on the provided data set only.

30Besides additional training data, some systems employ different kind of external resources. Among these, pre-trained word embeddings are used as word representations by UNIPV (Fasttext⁵) and gw2017 (word embeddings provided by the SpaCy framework⁶). The system of ItaliaNLP employs word embedding created from the ItWaC corpus (Baroni et al., 2009) and corpus extracted from Booking.com.

31Some of the systems are ABSA extensions built on top of custom or pre-existing NLP pipelines. This is the case for ItaliaNLP, VENSES and X2Check. Other systems make use of off-the-shelf NLP tools for preprocessing the data, such as SpaCy (gw2017, UNIPV) and Freeling⁷ (SeleneBianco).

32Finally, additional resources used by the systems often include domain-specific or affective lexicons. ItaliaNLP employed the MPQA affective lexicon (Wilson et al., 2005), and further developed an affective lexicon from a large corpus of tweets by distant supervision. The UNIPV system makes use of the affective lexicon for Italian developed in the framework of the OpeNER project⁸.

33In the ACD task, the precision of the second ranked system (gw2017) is significantly higher than that of the first system (ItaliaNLP), although the latter ranks at the top because of a higher recall. This unbalance between precision and recall is mainly due to the high number of aspect that can be assigned at the same time to a sentence: a system returning too many aspects is exposed to low precision but higher recall, while a more conservative system would achieve the opposite situation. Further details about the systems developed for the task can be found in the technical reports of the partecipants: ItaliaNLP (Cimino et al., 2018), UNIPV (Nicola, 2018), VENSES (Delmonte, 2018), X2Check (Di Rosa and Durante, 2018), gw2017 (Bennici and Portocarrero, 2018)

7 Conclusion

34The large availability of user-generated contents over the Web that characterizes the current tendencies of virtually sharing opinions with others has promoted the diffusion of platforms able to analyze and reuse them for personalized services. A challenging task is the analysis of the users’ opinions about a product, service or topic of discussion. In particular, the ABSA (Aspect-based Sentiment Analysis) task concerns the association of a polarity (positive, negative, neutral/objective) to the piece of the sentence that refers to an aspect of interest. In ABSITA, we proposes to automatically extract users’ opinions about aspects in hotel rewievs. The complexity of the task has been successfully faced by the solutions submitted to the task. Systems that used supervised machine learning approaches, based on semantic and morphosyntactic features representation of textual contents, demonstrate encouraging performances in the task. Good results have also been obtained using rule-based systems, even though they suffer from generalization issues and need to be tailored on the set of sentences to classify. The decision to use additional resources as additional lexicons in conjunction with semantic word embeddings have been demonstrated to be successful. More details about the implementation of the systems that participated in the task can be found in their specific reports. In conclusion, we consider the ABSITA 2018 task a success and an improvement of state of the art for the ABSA task in the Italian language.