1. Introduction

1Recently, the presence of digital technologies in our lives has grown enormously, with a strong impact on our daily lives. Digital spaces and social media have become a privileged channel for communication, information and socialization, frequented by millions of people at the same time. Along with the new relational opportunities and access to knowledge, even misbehaviour have acquired new visibility and virality, such as hate speech. In spite of a causal link between hate speech and crime is hard to demonstrate, the risk of offences and effects on psychological and physical well-being of the victims are clear in psychological and social studies (Nadal et al. 2014; Fulper et al. 2014). The extreme consequence of these effects might be suicide, especially considering the adolescents, as suggested by recent studies investigating the link between cyberbullying and suicidal behaviors of U.S. youth (Nikolaou 2017). To prevent such scenarios, few awareness-raising projects in schools are activated by NGOs in Italy, such as Amnesty International1 or Cifa ONLUS2.

2The Commissione Orientamento e Informatica nelle scuole3 supports a manifold of activities with the main goal of creating a link between schools and academia, also in the context of the national project Piano Lauree Scientifiche (PLS). The members of the CCC (Content-Centered Computing) group of the Computer Science Department of the University of Turin, active in the investigation of hate speech online4, have led and participated in several hate-speech-related projects, including “Contro l’odio”5 (Capozzi et al. 2020) a joint work with non-profit entities and University of Bari that aims at monitoring hate speech against minorities in Italy. Within the current experience, we created a data annotation platform specifically dedicated to support educational activities and aimed at reflecting on the importance of a conscientious communication. In this perspective, the idea of #Deactivate takes hold. This laboratory, addressed at students of secondary schools, is articulated in three main modules with the purpose of:

1. raising awareness about this social problem, encouraging the reflection on microaggressions, hate speech, stereotypes, prejudices;

2. stimulating the so-called computational thinking and the study of linguistic elements that are exploited by users to offend or to express hate against a victim online (hashtags, emoticons, or figures of speech);

3. introducing high schoolers to how tools based on NLP work to incentive a more conscious use of technology.

3We designed a series of educational activities that include: analysis of the online problem by means of an investigation on own social networks personal profiles; linguistic analysis of the hateful messages during the annotation of tweets on the “Contro l’odio” annotation platform; manual identification of hate speech in Italian texts playing the role of ‘being an automatic classifier’; translation of this task in a real automatic task, coding two types of classifiers in Python. These activities, delivered online due to the pandemic restrictions, have been distributed in 5 meetings (lasting 2 hours each) for each class, between April and June 2021, for a total of 10 hours per class.

2. Related Work

4A popular workshop series on the topic of “Teaching NLP” has been recently held on its fifth edition at NAACL-HLT 2021 (Jurgens et al. 2021), where the participants discussed and shared experiences on a variety of important issues such as: teaching guidelines, teaching strategies, adapting to different student audiences, resources for assignments, and course or program design. The main lesson learned has been that of highlighting the importance of creating materials describing NLP, not only for learners at a university/college level, but also for those learners who are younger and have diverse educational backgrounds. On this regard, a great inspiration for starting to work with schools in Italy derives from the experience of Sprugnoli et al. (2018), where the authors – although with different goal in mind than ours – started a project involving NLP and pupils from Italian schools, aged 12-13. That experience was chiefly dedicated to the study of cyberbullying among pre-teens and the creation of a corpus of WhatsApp threads in the context of the CybeRbullying EffEcts Prevention activities (CREEP) project. Our idea of starting a project that could bring NLP among high schoolers and that, at the same time, could introduce the themes of hate speech, microaggressions, and discriminations by eliciting personal experiences and students’ opinions, is somehow in continuity with that experience.

5A second work of great relevance for the creation of our experience, has been the reading of Pannitto et al. (2021), in which the authors point out, for the first time, the fact that no high school curricula in Italy includes any (computational) linguistics education and that the lack of this kind of exposure makes choosing computational linguistics as a university degree unlikely. Furthermore, the authors highlight that NLP is, indeed, at the core of many tools young people use in their everyday life, and having almost zero knowledge of this field makes the use of such tools less responsible than it could be. The authors have been the first to create a dedicated workshop for Italian aimed at raising awareness of Italian students aged between 13 and 18 years regarding the subject of NLP (Messina et al. 2021).

6Additionally, the idea of creating some playful and meaningful activities regarding NLP and the themes of hate speech for high schoolers, are in line with the concept of ‘gamification’, which lately has been applied to many linguistic annotation tasks, as an alternative to crowdsourcing platforms to collect annotated data in an inexpensive way (Bonetti and Tonelli 2020), such as our “Contro l’odio” annotation platform.

3. #DeactivHate

7The aims of #DeactivHate are: 1) raising awareness about misbehaviour online, such as hate speech, eliciting also personal experiences, 2) stimulating computational thinking and linguistic observation of hateful messages, and 3) encouraging a conscious use of technologies discovering how they work. To reach these objectives we articulated three modules as described below.

3.1 Hate Speech: Introduction

8The first module is aimed at introducing at introducing a definition of hate speech to students. Hate speech is often mistaken for a generic insult rather than a specific phenomenon “connected with hatred of members of groups or classes of persons identified by cer-tain ascriptive characteristics (e.g., race, ethnicity, nationality)” (Brown, 2015).

9The session started with an icebreaking activity in which students presented themselves through an image found online, depicting an aspect of their identity (see Figure 1). We then asked them to tell whether they were ever attacked or stigmatized for this characteristic.

Figure 1: Example of Jamboard of Google

10In this way, we guided the class in drawing a distinction between non-ascriptive identity traits (e.g., political belief, style of dressing) and ascriptive6 ones (e.g., ethnicity, sexual orientation, skin colour) (Reskin, 2005). The idea behind this activity is twofold: i) it links issues such as hate speech and racial microaggression (Sue, 2010) to students’ lives; ii) it helps distinguishing the spread-ing of discriminatory contents7 from generic insults. The module ended with an assignment: students had to find at least one public figure who had been a victim of online discrimination, providing one or more hateful messages as an example, and a counter-narrative response..

3.2 “If I Were a Classifier...”

11The second module is organized in two meetings and focuses on the importance of manually annotated corpora for online hate speech detection and what are the peculiarities of hateful messages.

12Within the first meeting, each student presented the found messages and try to define the type of attack and the linguistic characteristics of the text that make it hateful or a counter-narrative. The variety of examples led to the introduction of a deeper taxonomy of discrimination (e.g., misogyny, homophobia, sexism, etc...). As expected, the following group discussion brought out a considerable subjectivity in perceiving these phenomena, thus highlighting the need of adopting a shared annotation schema to identify hate speech in messages.

13After a brief introduction on what corpora are and how they are used in new technologies, students have been involved in an annotation task of hate speech, asking them to evaluate at least 30 tweets.

14For this purpose, we created the data annotation platform8 within the “Contro l’odio” project. This web application, built using PHP, MySQL, and JavaScript, preserves the student’s annotation history by using a passwordless authentication link sent to the email choose during the login. This method has the double advantage of not requiring the student to register to the platform and of preserving ourselves to save the student’s email or other personal data. It then ensures the annotation anonymity and satisfies the requirements of General Data Protection Regulation (GDPR), as a desired consequence.

15The home page of the web application consists of a dashboard that provides the annotation guideline and shows basic information about the student’s activity. Indeed, the student could know the number of sessions they completed (each session consists of annotating 15 tweets) and the level of agreement (expressed in percentage) between their annotation and the annotation performed by the automatic model realized in the “Contro l’odio” project. Gamifying the task through this comparison, we provide the basis for a discussion about the fallibility of automatic systems. Furthermore, we also allow the student to compare their annotation with the annotation of their classmates in order to introduce the measures of annotator agreement. When a session starts, the student could annotate the level of hatefulness of a tweet through a 7 square scale filled with a color scale from Watusi to Sangria as shown in Figure 2. Two additional squares, respectively filled with White and Mid Gray, allow to state the absence of hate or to consider off-topic the content of the tweet. Finally, three toggle switches (on/off button) were added to check the presence of ‘irony/sarcasm/humor’, ‘offensiveness’, and ‘stereotype’, giving them the possibility to reflect about the ways in which users spread hate online.

Figure 2: Data Annotation Platform

16During the annotation task, students were asked to fill a shared spreadsheet with the tweets that impressed them the most for its offensiveness, for its humorous intention, or the most difficult to annotate. By discussing with them annotation results, we introduced the latest core concept of the module: the agreement. We presented some metrics that are typically adopted to calculate it among annotators and outlined some good practice recently emerged in Corpus Linguistics, such as ensuring the involvement of minorities in corpora development in order to avoid biases (Basile 2020).

3.3 My First Classifier

17In this module the main idea is to stimulate computational thinking by translating linguistic observations coming from the annotation procedure in a proper computational task. The activity of annotation has, indeed, given the opportunity of reflecting on how users tend to verbally express hate online, and on how minorities are represented through stereotypes. To incentivize this transition, we proposed two activities:

1. to mark in each tweet the textual span that could make a classifier aware of the presence of hate speech creating a list of word n-grams;

2. to develop two automatic classifiers (supervised and unsupervised) exploiting the list of word n-grams.

18Before starting with the first activity, we asked students to motivate their choice of the tweets selected during the previous exercise. Some tweets triggered a discussions on what should be considered hate speech or not, and the doubts were later solved by looking at the provided definitions of hate speech and at the annotation guidelines. The most controversial tweets report aggressive events or racial propositions; and, for this reason, they were perceived as hurtful by the majority of the students:

          (i) Autobus per i bianchi e altri per i migranti. Non si parla dell’apertheid del Sudafrica né del periodo di segregazione negli Stati Uniti, ma di una proposta della Lega per la provincia di Bergamo. L’Italia non è un paese razzista ma nel 2020 questo è ciò di cui si discute. URL9

19Others triggered interesting linguistic reflections, such as:

          (ii) Peccato che non sbarcano povere famiglie africane, ma solo mafia nigeriana, ex galeotti tunisini, stupratori senegalesi, terroristi dell’Isis dalla libia, tutti criminali robusti 1.80 di altezza, pronti a spacciare droga, violentare le nostre donne, cannibali e assassini.10

20In these, the students retrieved specific figures of speech such as sarcasm, rhetorical questions and analogies, and also strong words that reflect the social biases towards the minorities. In activity A, all the words and expressions that could make the message hurtful have been collected in a list of n-grams of words called our_lexicon (Table 1). Following, the items of such list have been exploited by the classifiers to predict if a tweet contains hate speech or not.

21For activity B, we created an interactive Python notebook using the Colaboratory platform provided by Google, as a similar initiative had successfully been carried out by with a similar educational tool. To allow the students to use the notebook in spite of their computer skills, we elaborated some guidelines explaining even how to create a folder in Google Drive and how to import all the necessary materials inside of it. Among the required materials, we prepared the dataset using the tweets previously annotated by the students.

22We proposed two types of classifiers:

1. unsupervised classifier based on the list our_lexicon for which if one of the selected grams are inside the text, the text is predicted as hateful;

2. supervised classifier based on Support Vector Machine algorithm using the list our_lexicon as main feature of the classification task.

23The coding of the first classifier allowed students to gain confidence with some basics of Python; whereas the second one introduced them to core of new technologies based on machine learning (see Figure 3). At the end of the activity, we observed together the performances of automatic systems and analyzed some of the tweets that were wrongly classified. This final step helped students to reflect on the limitations of machines and the important role of the linguistics in language-related technologies.

Figure 3: Supervised Classifier Section on Python Notebook

4. What We Learned

24Due to pandemic restrictions, we thought the entire laboratory through remote modality (DAD)13 between April and June 2021 to 2 classes of one secondary school of Turin, with students aged 16-20. As described above, various resources and tools have been used (and created ex novo) to bring forward the educational activities in distant teaching mode. However, we believe that the same activities/materials could be proposed even for lessons in presentia exploiting the computer rooms of the schools.

25For each class, we organized the activities of the three modules in 5 meetings of about 2 hours. Despite the shortness of the laboratory, we found that realizing specific activities for each session helped us manage efficiently the available time. We resorted to web applications to make up for the different devices and operating systems used by the students at their homes. And, in particular, we used Google Meet, as it offers interactive tools such as virtual blackboard, and Moodle, a learning platform provided by the University of Turin that gave us the possibility to organize our activities making available the necessary materials to students. Moreover, each meeting was supported by the use of slides for having visual and descriptive support. The classes assisted in this short period were composed of a total of 35 adolescents, coming from different countries. From the first meeting they showed a general interest in the treated subject, and we were surprised especially by the profoundness of some observations raised during the discussions. The students, indeed, were encouraged to share their opinions, doubts, and perspectives. These discussions made clear that the students face these problems related to technology and communication every day, sometimes suffering even the consequences. Hate speech is, indeed, a very sensitive issue and the perception of what is abusive or not, depends on the cultural back-ground of each student. This fact, on the one side stimulated the debates, however, on the other side, it made it difficult for us to find the ideal way to share complex concepts and manage specific situations.

26At the end of the laboratory, we provided a survey in order to collect the impressions and the opinions of students. Analyzing these surveys, we noticed that the majority of students considered interesting the content of #DeactivHate, but it appears clear that the format online of the laboratory was perceived from students less interactive and fluent, due especially to technical problems when a part of students were in class and other part at home14. From our perspective, we noticed an interesting difference between younger and older students. The older were more active during the activities and discussions than the younger. Moreover, we thought that the number of students affected the flow of the debates, especially in the DAD context. We expect that in presentia the proposed activities could have a better impact facilitating the interaction.

5. Conclusion

27#DeactivHate represents for Italian high schoolers a first step towards the introduction to subjects such as Linguistics and NLP, that are, for the most part, unknown in Italian high schools, in spite of their relevance in everyday technology. Indeed, this kind of laboratory reveal which are the multiple applications of the Computer Science degree course at a university level, and even of Linguistics, commonly considered as opening fewer employment opportunities. Looking at the future, we would like to enhance the proposed activities in order to make them more interactive even in an online context (such as the DAD) following the example of Hiippala (2021).

28A final remark needs to be made regarding the lack of evaluative strategies that could allow us to understand the impact of #DeactivHate in students’ online behaviors or their knowledge of technologies. Therefore, following the example of and Bioglio et al. (2018) and Athanasiades et al.(2015), in the next editions we have planned to employ: surveys before and after the intervention to evaluate the activity online of the students and their experiences about misbehavior (caused or suffered); and interviews to teachers after the conclusion of the laboratory to understand if some changes were perceived with respect to the class group. Future activities will integrate also basic evaluations to assess the degree of learning with respect to the contents of the course, such as computational thinking, annotation methodologies, automatic text processing, as well as a final evalua-tion of the proposed teaching activities collecting the personal impressions of the students.

29In addition, to validate also the impact of #DeactivHate in the society and, in particular, in the city context we think to measure the detection of the amount of hateful message online by means of monitoring platforms, such as “Contro l’odio” map15.