1 Introduction

1In the field of communication, the importance of addressing your audience with a common language in which the customer can recognize and identify with each other is fundamental. In social interactions, an increasing amount of communication occurs in a non-verbal way, as with emoji.

2Being able to predict the best emoji to use in a message can increase the perception of the same and give strength to the message itself.

3In the context of the Italian Emoji Prediction task called ITAmoji, we have tried to predict one of 25 possible emojis from different tweets.

4Despite the knowledge of how a system of SVM could be the best solution for the problem, as per the previous context SemEval 2018 (Rama & Çöltekin, 2018), a different approach was chosen to focus on the effectiveness of a Neural Network based model

2 Description of the system

5We first started by cleaning the given data from all the noise information. All the punctuation marks were removed from the text of tweets, and we focused on cleaning the text and removing ambiguities such as shortened words and abbreviations. We substituted all the hyperlinks with a more generic word “LINK” and we did the same with the usernames preceded by ‘@’ (users’ tags), after seeing that it was not relevant in the prediction of the most likely emoji for the tweet.

6We tried removing the stop words from the tweets’ text to leave only the words with relevant meaning in it, but the results were poor.

7Then we converted every word of the tweet’s text into its lemma, and while doing the lemmatization, we saw that sometimes the username was misleading in the text, so we chose to remove it and substitute it with a more generic word ‘USERNAME’.

8We used two different fastText² vectors created in the 2016 and the other created in 2017, all with Italian tweets containing at least one emojis. The idea is to analyze if different fastText vectors created with tweets published in different periods could discover the use of the emojis and its evolution over the time.

9The system created is an Ensemble of two different models to replicate the result obtained in the emotion classification (Akhtar at al., 2018).

10The first model is a bi-directional Long Short-Term Memory (BI-LSTM) implemented in Keras³.

11A dropout and a recurrent_dropout of 0.9.
The optimizer is the RMSProp. The embedding is trainable.

12The second is a LightGBM⁴, where the following properties are extracted from the tweet text:

• length of the tweet

• percentage of special characters

• the number of exclamation points

• the number of question marks

• the number of words

• the number of characters

• the number of spaces

• the number of stop words

• the ratio between words and stop words

• the ratio between words and spaces

• the ratio between words and hashtags

and are joined to the vector created by the bigram and the trigram of the tweet itself at word and character level.

13The number of leaves is 250, the learner set as ‘Feature’, and the learning rate at 0.04.

14The ensemble is done in the weighted average when the BI_LSTM decide the 60% of the vote and the LightGBM the 40%.

15It was also tried to add a linear classifier but the attempt did not provide any advantage. The cross-validation task to find a good weight was ineffectual and the provision was insignificant.

3 Results

16The results of the Bi-LSTM were:

17The model trained with the data published during the 2017 is quite similar to the model trained with the data published on the 2016.

18The results of the LightGBM were:

19The LightGBM model was also tested by adding to the already mentioned properties additional information such as the user ID and information extracted from the tweet date such as day, month, the day of the week and time.

20The results obtained also indicate here that there is a correspondence between the use of emojis, the user, the time and the day. For example the Christmas tree in December or the heart emoji in the evening hours.

21The level of Precision obtained in this way was very high even if the F1 score is still lower than the BI-LSTM model.

22To avoid the unbalancing of the emojis present in the training dataset various undersampling and oversampling operations were performed without any appreciable results.

23Turning to the result of the ensemble of the two models we had a marked increase in the F1 score thanks to the substantial growth of the Recall in both cases.

24In the tables 5 and 6 there are the results from the minimum and the maximum F1 score obtained during the process of the ensemble.

25The result of the validation was however very far from that obtained during the training phase. It will be necessary to evaluate if, as in the research Exploring Emoji Usage and Prediction Through a Temporal Variation Lens (Barbieri et al., 2018), it was the time of the publication of the tweets is to be distant from the date of the tweets analyzed.

26If the tweets analyzed were too different from those of the training dataset, if the users in the test dataset have different behaviors, or if the system suffered from some kind of overfitting (visible in the third submission, gw2017_pe).

27In table 8 we can observe the result of the three submissions split for each emoji.

Table 8: Precision, Recall, F1 Score, and quantity in the test set of the 25 most frequent emojis

Runs	gw2017_e			gw2017_p			gw2017_pe
Label	precision	recall	f1-score	precision	recall	f1-score	precision	recall	f1-score	quantity
	0.2150	0.0224	0.0405	0.1395	0.0642	0.0879	0.0242	0.0107	0.0148	1028
	0.4429	0.1917	0.2676	0.3608	0.2075	0.2635	0.0215	0.0178	0.0195	506
	0.3142	0.3417	0.3274	0.2726	0.3681	0.3133	0.0343	0.0468	0.0396	834
	0.3624	0.3540	0.3582	0.3204	0.3850	0.3498	0.0107	0.0155	0.0127	387
	0.3137	0.0360	0.0646	0.1608	0.0518	0.0784	0.0077	0.0023	0.0035	444
	0.3533	0.8357	0.4967	0.4185	0.6104	0.4965	0.2024	0.2648	0.2294	4966
	0.3902	0.1535	0.2203	0.3257	0.2038	0.2507	0.0263	0.0264	0.0263	417
	0.2917	0.0554	0.0931	0.2190	0.0678	0.1035	0.0328	0.0102	0.0155	885
	0.0800	0.0053	0.0099	0.0581	0.0132	0.0215	0.0380	0.0079	0.0131	379
	0.5143	0.2581	0.3437	0.4464	0.2688	0.3356	0.0044	0.0036	0.0039	279
	0.3144	0.1635	0.2152	0.1895	0.2520	0.2163	0.0135	0.0134	0.0135	373
	0.7567	0.7497	0.7531	0.7803	0.7358	0.7574	0.2101	0.2016	0.2058	5069
	0.1714	0.0110	0.0207	0.1053	0.0183	0.0312	0.0137	0.0018	0.0032	546
	0.3769	0.1849	0.2481	0.3439	0.2038	0.2559	0.0142	0.0113	0.0126	265
	0.3137	0.4109	0.3558	0.2952	0.4824	0.3663	0.0904	0.1583	0.1151	2363
	0.2384	0.1607	0.1920	0.2068	0.1747	0.1894	0.0526	0.0546	0.0536	1282
	0.3174	0.1043	0.1570	0.2432	0.1157	0.1568	0.0317	0.0243	0.0275	700
	0.4667	0.1579	0.2360	0.3239	0.1729	0.2255	0.0096	0.0075	0.0084	266
	0.6735	0.3103	0.4249	0.6221	0.3354	0.4358	0.0106	0.0063	0.0079	319
	0.3204	0.1220	0.1767	0.2101	0.2680	0.2356	0.0193	0.0185	0.0189	541
	0.4278	0.1199	0.1873	0.3043	0.1526	0.2033	0.0249	0.0171	0.0203	642
	0.3220	0.0548	0.0936	0.2368	0.0778	0.1171	0.0187	0.0086	0.0118	347
	0.3590	0.0411	0.0737	0.2537	0.0499	0.0833	0.0161	0.0059	0.0086	341
	0.2082	0.1181	0.1507	0.1584	0.2451	0.1924	0.0369	0.0419	0.0392	1338
	0.2609	0.0248	0.0454	0.1860	0.0331	0.0562	0.0336	0.0083	0.0133	483
avg / total	0.4071	0.4219	0.3690	0.3870	0.4009	0.3781	0.1051	0.1195	0.1097	25000

28It is important to note that despite the significant presence of the dataset the has a meager final

29F1 score. On the other hand, the has a high F1 score even if only present in 319 items.

4 Discussion

30In the study of the dataset, three critical issues emerged.

31• The first is that the use of similar emojis seems more dictated by a personal choice of the user.
There are not many pieces of evidence because the use of one emoji is preferred.
In particular for the following emoji:

32• The second is that, especially in cases where a tweet begins by indicating a USERNAME, or in a mention or a direct response, the use of emoji takes on a sub-language value. That is, the use of a specific word or emoji has a meaning that only the tweet recipients know. Use of emoji and could be irony or just references to previous pasted experiences in common.

33• Thirdly, the strong imbalance of the training dataset is not the only reason for the unbalanced prediction of some emojis, as in the case of and .

5 Conclusion

34The result of the ensemble was pretty good and demonstrate the validity of this kind of approach. The use of emoji is personal and also depends on the context and the people in the discussion. A system with the emojis with the same meaning merged could be more proficient and ready for the production.

35In the near future, we will evaluate the speed and effectiveness of a CNN model in which the operation of the BI-LSTM and the features extrapolation used in the LightGBM model can be merged during the same training session.

36We will also focus on the creation of fastText vectors of different size containing tweets for specific contexts and published in different periods to identify the periodicity and variation in the use of particular emoji. The intent is to discover other hidden patterns, more than the obvious that has emerged for the holiday periods.