1. Introduction

1People are practically reviewing anything in on-line sites and understanding the polarization of a comment through automatic sentiment classifier is a tantalizing challenge. In recent years, the number of virtual reviewers has drastically increased and there are many products and services, which can be reviewed. Each person, before buying a product or a service, searches into reviews from people who have already had experienced the product or the service. Review portals are usually linked to the leisure or business activities such as the world of tourism, e-commerce or movies. However, there are topics where these reviews and the associated automatic computed sentiment may induce to select wrong services, which may dramatically affect personal life.

2When dealing with health-related services, the effect of positive or negative reviews on hospitals and doctors can have a potential catastrophic impact on the health of who is using this piece of information. QSalute1 is one of the most important Italian portals of reviews about hospitals, nursing homes and doctors. It is very important for patients to seek the best hospital for their condition based on the past experience of other patients. Reviews in the world of health benefit both patients and hospitals because they are a means to discover problems and solve them (Greaves et al. 2013; Khanbhai et al. 2021).

3Automatic sentiment analyzer have then a big responsibility in the context of health-related services. In these sensitive areas, it is important to design AI systems whose decisions are transparent (Doshi-Velez and Kim 2017), that is, the systems must give the motivation for the choice made so that people can trust. If the users do not trust a model or a prediction, they will not use it (Ribeiro, Singh, and Guestrin 2016).

4In this article, we investigate a model that can mitigate the responsibility of sentiment analyzers for health-related services. The model we are using exploits syntactic information within neural networks to provide a clear visualisation of the internal decision mechanism of the model that produced the decision. We propose KERMIT_HC (KERMIT for HealthCare) based on KERMIT (Zanzotto et al. 2020) to solve the sentiment analysis task introduced by Bacco el al.. We use KERMIT_HC on QSalute Italian portal reviews in order to include symbolic knowledge as a part of the architecture and visualize the internal decision-making mechanism of the neural model, using KERMIT-viz (Ranaldi, Fallucchi, and Zanzotto 2021).

5In the rest of paper, Section 2 gives details about the dataset and methods, while Section 3 and 4 describe the experiments, the results obtained and their discussion. Finally, in Section 5 we present the final conclusions and future goals.

2. Data & Methods

6To explore our hunch that syntactic interpretation may help in Healthcare reviews recognition, we leverage: (1) a Healthcare training corpus (Sec. 2.1); (2) a KERMIT_HC, which is based on syntactic interpretation and it can explain its decisions; and finally, (3) some challenges solved due to KERMIT_HC (Sec. 2.2).

2.1 Dataset

7In order to investigate reviews in healthcare area, we selected the QSalute portal, one of the most important health websites in Italy. This portal can be defined as the TripAdvisor of hospital facilities, indeed it talks about: Expertise, Assistance, Cleaning and Services. In addition to the reviews, there are some associated metadata such as: user id, hospital name, review title and patient pathology. To ensure privacy we do not consider sensitive data such as user id and hospital name.

8We used a free available scraper on GitHub2 to download the dataset. Then, to model this data to a sentiment analysis task, we followed the indications provided by Bacco et al. - in detail, a review is: (1) negative if the average of its scores is less than or equal to 2, (2) positive if the average of its scores is greater than or equal to 4 (3) neutral otherwise.

9The resulting dataset is composed of 47,224 reviews consisting of: 40,641 reviews in the positive class, 3,898 in the neutral class and 2,685 in the negative class.

10In this work, we solely consider positive and negative classes, so our final dataset is composed of 43,326 reviews. The dataset is heavily skewed (93,80% positive class - 6,20% negative class) favoring reviews labeled as positive.

2.2 KERMIT 4 Healthcare

11KERMIT_HC (KERMIT for HealthCare) architecture is composed of 3 major parts: (1) a KERMIT model described in Zanzotto et al. , (2) a Transformers model and (3)  a decoder layer that combines the results obtained from the previous two sub-parts. In figure Fig.1 we show a graphical representation of the architecture of KERMIT_HC, pointing the parts that compose it.

Figure 1

12The architecture of KERMIT_HC makes it a particular model, because it combines the syntax offered by KERMIT with the versatility of a Transformer-model. We use KERMIT because it allows the encoding of universal syntactic interpretations in a neural network architecture. KERMIT component is itself composed of two parts: KERMIT encoder, which converts parse tree T into embedding vectors and a multi-layer perceptron that exploits these embedding vectors.

13The second sub-part of our architecture is composed of a Bidirectional Encoder Representations from Transformers, - as known as BERT - to classify the sentiment of the reviews. BERT is a pre-trained language model developed by Devlin et al. at Google AI Language. In particular, since the task concerns sentences in the Italian language, we have used a special BERT version pretrained on that language called AlBERTo (Polignano et al. 2019).

3. Experiments

14We used KERMIT_HC architecture to examine if it is possible to answer the research questions showed in KERMIT (Zanzotto et al. 2020) also in healthcare domain using the Italian language. Those research questions are: (1) Can the symbolic knowledge provided by universal symbolic syntactic interpretations, make a difference and it be used effectively in neural networks? (2) Do universal symbolic syntactic interpretations encode different syntactic information than those encoded in “embeddings of universal sentences"? (3) Can the universal symbolic syntactic interpretations provided by KERMIT_HC, supply a better and clearer way to explain the decisions of neural networks than those provided by transformers?

15To provide a comprehensive answer to these questions, we tested the architecture in a completely universal setting where both KERMIT and AlBERTo are trained only in the last decision layer.

16The rest of the Section describes the experimental set-up, the quantitative experimental results and discusses how we can use the KERMIT-viz to explain decisions of neural network inferences over examples.

3.1 Experimental Set-up

17This section describes the general experimental set-up of our experiments and the specific configurations adopted.

18The parameters used for the KERMIT encoder are those proposed in Zanzotto et al., paper. The constituency parse trees used for KERMIT sub-part are obtained using our freely available script on GitHub3.

19We tested several different BERT version pre-trained on Italian language in order to get the best model for our task. In particular, we tested the following transformers: (1) UmBERTo (Parisi, Francia, and Magnani 2020); (2) AlBERTo (Polignano et al. 2019); (3) BERT multilingual (Devlin et al. 2018) and (4) ELECTRA_ita: an Italian version of ELECTRA model (Clark et al. 2020) implemented by Schweter on a work of Chan et al. . All the models were implemented using Huggingface’s transformers library (Wolf et al. 2019) and all were used in the uncased setting with the pre-trained version. The input text for BERT has been preprocessed and tokenized as specified in respectively work (Parisi, Francia, and Magnani 2020; Polignano et al. 2019; Devlin et al. 2018; Schweter 2020).

20Since our experiments are text classification task, the decoder layer of our KERMIT_HC architecture is a fully connected layer with the softmax activation function applied to the concatenation of the KERMIT sub-part output and the final [CLS] token representation of the selected transformer model. Finally, the optimizer used to train the whole architecture is AdamW (Loshchilov and Hutter 2019) with the learning rate set to 2e^-5. For reproducibility, the source code of our experiments is publicly available on our GitHub repository4.

Figure 2: The visualizations offered by KERMIT-viz. Both examples have the target class positive but in the first one, it is easy to state the positivity. In the second one, who wrote the review, makes disquisitions about the medical staff but at the same time lauds the head of the department

4. Results and Discussion

21Syntactic information is useful to significantly increase performances to classify Healthcare reviews (see Table 2). KERMIT_HC uses AlBERTo which is the best BERT-italian version model according to our experiments, showed in Table 1. Especially KERMIT_HC outperforms the solely AlBERTo sub-part model (ref. to Table 2).

22As in the work proposed by Bacco et al., (2020) we chose to divide the dataset by “Site” and evaluate the models using accuracy and F1-score metrics. Despite this division, the dataset is still very unbalanced favoring the class 1 (positive reviews). We reports results in terms of the accuracy, Macro F1 and Weighed F1. Observing Table 2, we can see that the performance obtained by KERMIT_HC always exceeds the best configuration of BERT: AlBERTo. Hence, trained on the Healthcare review dataset (Bacco et al. 2020) (see Section 2.1) KERMIT_HC seems to be a good candidate to analyze sentiment of hospital patients.

23Using the KERMIT-viz visualiser, we analysed how important the contribution of symbolic knowledge provided by KERMIT can be. In many cases it makes all the difference. Looking at the Figure 2, these are two sentences with a positive target. The first sentence (shown in Fig. 2a) is clearly positive while the sentence shown in the Fig. 2b could be ambiguous as the patient makes bad remarks about the service but praises the head of the department. We can observe how some words have been colored in red (therefore they have received a greater weight during the classification phase) emphasizing the positive aspects of the sentence and causing it to be labeled as “positive review”. In this way the explainability is guaranteed and in very delicate topics - like sentiment in health reviews - we can have more “trust” on sentiment analysers.

5. Conclusion

24In this article, we investigated a model that can mitigate the responsibility of sentiment analyzers for health-related services. Our model KERMIT_HC exploits syntactic information within neural networks to provide a clear visualisation of its internal decision mechanism. KERMIT_HC is based on KERMIT (Zanzotto et al. 2020) and we worked in a sentiment analysis task introduced by Bacco el al..

25We studied several versions of pre-trained BERT models on the Italian language and found out that AlBERTo is, among them, the best model for this task. However, KERMIT_HC, which is composed of KERMIT+AlBERTo, outperforms better than AlBERTo model alone. Additionally, via KERMIT-viz, we visualized the reasons why KERMIT_HC classifies the dataset. We observed how KERMIT_HC captures relevant syntactic information by catching the keywords in each sentence giving them more weight in the decision phase, mitigating and capturing possible errors of the sentiment analysers. Our future goal is to be able to have full control of the sentiment analysers by injecting human rules (Onorati et al. 2020) in order to mitigate possible errors.