1I moderni personal assistant richiedono di accedere ad informazioni non strutturate per soddisfare con successo le richieste degli utenti. In questo articolo, abbiamo studiato l’uso dell’ apprendimento automatico per progettare due componenti di un personal assistant: classificazione degli intenti, per comprendere la richiesta dell’utente, e la selezione della frase di risposta per rispondere alle domande con testo non strutturato. I risultati della valutazione derivati da cinque diversi datasets del mondo reale, associati a diverse società, mostrano un’elevata precisione per entrambi i modelli. Ciò suggerisce che la moderna tecnologia di question answering e dialogo è efficace per attività reali.

1. Introduction

2Help-desk applications use Machine Learning to classify user’s request into intents. The information owned by companies generally is in free text form, from company’s documents or websites. For example, corporate knowledge is typically encoded within documents in an unstructured way. This poses limitations on the effectiveness of standard information access. For example, searching documents by keywords is not a viable solution for the users, as they seldom can find an answer to their questions. The possibility of using QA systems to search for information on a corpus of documents, also through a dialogue system, offers an attractive solution for extracting the best information from the company knowledge bases.

3IMSL company offers virtual agents that can be retrained based on the customer needs. The agent is composed of many Natural Language Understanding components, such as classifiers that map each user utterance in input to their corresponding intent. However, since it is not possible to forecast all the intents corresponding to the questions that the user are going to ask – which are potentially infinite – it is of paramount importance to have an automated QA system able to automatically provide the best answer (paragraph) extracted from a company owned knowledge base.

4Information access is becoming an increasingly critical issue. Traditional Information Retrieval systems, used in industry, help the user in accessing information, but are often imprecise and impractical. Current search engines are an example of this. Searching for information on the web often requires a double effort for the user: first it is necessary to understand how to formulate a query in the most effective manner, and then filter out the proposed results in order to find the most relevant information.

5In this paper, we described our QA system based on answer sentence selection and intent detection, and how we integrate them in a Conversational agent.

2. Related Work

6As today, the largest part of general-purpose QA services are provided by big tech companies such as Amazon Alexa, Google Home, Ask Yahoo!, Quora and many others. Unfortunately, these types of applications are not easily accessible for smaller companies, as the offered QA service cannot be easily adapted to handle corporate knowledge, which is in form of unstructured text. To build their own solutions SMEs can exploit QA components such as Answer Sentence Selection.

7In recent years, deep learning approaches have been successfully applied for automatically modeling text pairs, e.g., (Lu and Li 2013; Yu et al. 2014). Additionally, a number of deep learning models have been recently applied to QA, e.g., Yih et al. (2013) applied CNNs to open-domain QA; Bordes et al. (2014) propose a neural embedding model combined with the knowledge base for open-domain QA. Iyyer et al. (2014) applied recursive neural networks to factoid QA over paragraphs. Miao et al. (2016) proposed a neural variational inference model and a Long-short Term Memory network for the same task. Yin et al. (2016) proposed a siamese convolutional network for matching sentences that employ an attentive average pooling mechanism, obtaining state-of-the-art results in various tasks and datasets.

8The work closest to this paper is by Yu et al.(2014) and Severyn and Moschitti (2015). The former presented a CNN architecture for answer sentence selection that uses bigram convolution and average pooling, whereas the latter use convolution with k-max pooling.

9Nowadays, supporting customers in their activities across applications and websites is becoming always more demanding, due a large number of customers and the variety of topics that have to be covered.

10New tools, such as chatbots, able to answer frequently asked questions, i.e., FAQs, are rising in response to this needs. Classifying the user need expressed in a natural question, into a predefined set of categories, allow conversational agents to recognize which users are asking which types of questions and to react accordingly.

11Traditional approaches to this problem include the use supervised approaches such as Support Vector Machines (SVM) (Cortes and Vapnik 1995), Boosting (Iyer et al. 2000; Schapire and Singer 2000), Kernel machines operating on input structured objects (Moschitti 2006; Lodhi et al. 2002) and Maximum Entropy models (Yaman et al. 2008).

12In the latest years, new models such as Recurrent Neural Network (RNN), Long Short Term Memory (LSTM) (Cortes and Vapnik 1995), Gated Recurrent Unis (GRU) (Chung et al. 2014) and Convolutional Neural Networks (CNN) (Lecun et al. 1998; Kim 2014) were established as state-of-the-art approaches for text classification.

3. System Description

13Our QA system allows for extracting portions of text from company documents or from websites. This information is then organized into paragraphs, which are then used to provide an answer to the user’s questions.

14One practical problem is the fact that not all PDF files encode text, and many fail to preserve the logical order of the text. Thus, in order to extract paragraphs, we used pdf2text.

15Another practical problem we need to solve was to keep portions of text separated by punctuation together: such as bullet lists or very structured paragraphs. Our designed tool automatically assigns a reference index or summary to each paragraph to improve subsequent searches (see Figure 1).

Figure 1: Paragraphs extraction

16Subsequently, each question and answer pair must be annotated with correctness (label TRUE/FALSE). This allows us to create a training set to train the re-ranking network (see Figure 2).

Figure 2: Annotation data for training

17The final system, shown in Figure 3, therefore allows for using the target company data, appropriately reorganized into paragraphs, to provide answers to the user’s request. On average we provide from 3 to 5 answers for each question. However, we also provide the reference to the document and the summary which the paragraph refers to.

Figure 3: system architecture

4. Answer Sentence Selection (AS2)

18The AS2 goal is to rank a list of answer candidates by their similarity with respect to an input question q_i. We design a network that includes relational information between questions and answers. Our results show that CNNs reach better performance than traditional IR models based on bag of words.

4.1 Model

19The architecture of the network used for mapping sentences in embedding vectors is showed in Figure 4 and is inspired to the CNNs employed by Severyn and Moschitti (2015) to perform many classification activities over sentences. It includes two main components:

(i) an encoder that map an input document s_i into a vector and (ii) a feed-forward network that computes the similarity between input sentences.

Figure 4

20Our network takes two sentences in input, i.e., a question and a text paragraph that may contain an answer, and it represents each of them into vectors of fixed-size dimension .

21The sentence model is composed of a sequence of convolutional maps followed by some pooling operations. Such model achieves the state of the art in many NLP tasks (Kalchbrenner, Grefenstette, and Blunsom 2014; Kim 2014).

22Then, the sentence vectors, corresponding to the questions and answers, are concatenated together and passed to the following neural network layers. These are composed of a non-linear hidden layer and an output layer with a sigmoid activation unit. At the end, the network returns a value between 0 and 1 corresponding to the relevancy of the answer with respect to the question.

23Finally, we included word overlap embeddings encoding relational information between words in questions and answers (Severyn and Moschitti 2016).

5. Intent Classification

24We adopted advanced techniques, such as deep learning models, to classify the user need, which is semantically expressed by the user question, into a predefined set of categories, i.e., intents.

25We used some common deep learning models for solving the intent detection task. The main point of our study is to test those models and observe how they perform on datasets containing real user questions addressed to a virtual agent, operating in the banking/financial sector.

26At this stage, we dot not consider novel methods based on transformer architecture such as BERT (Devlin et al. 2019), which require a a large amount of resources, typically not avialable to SMEs. Instead, we focused on lighter approaches that can run on small GPUs. We report our experiments and discuss the obtained results using such lighter models.

5.1 Models

27SVM (baseline) fed with word features, derived from the text of the utterances.

28LSTM using recurrent units that take in input the embedding x_t of the current word at time step t and the hidden vector encoding the sub-phrase at previous step, i.e., h_t-1, and return the vector representation of the phrase at step h_t

29CNN uses a set of convolutional filters of different size and max pooling operations to extract the most important features, e.g., bigrms, trigams, etc…, which represent the sentence meaning.

30LSTM + CNN based on an architecture composed of two layers: an LSTM layer that builds a fixed-size vector representation of the sentence at each word, and a convolutional layer. The latter applies a set of convolutional operations on the representations returned by the first layer.

31CNN + CNN composed of two CNN layers, where the second layer takes the previous layer representation as input, and applies a set of convolutional filters and pooling operation to compute the final vector representation of the sentence.

6. Experiments

32In this section, we first describe the datasets we used in our experiments, then we provide the results on the answer sentence selection and the intent classification tasks. Finally, we report an end-to-end evaluation of our system.

6.1 Data Description

33We built our datasets by collecting samples of questions asked by users to conversational agent for either Credit Institution or Bank websites. We collected two intent corpora from each data provider, resulting in a total of four datasets.

34Istituto Credito - synthetic (IC_s): This corpus was created by expert dialog engineers. It contains a set of utterances annotated with their corresponding intents. The subject of questions are diverse and spans over many topics. For example, some questions seek information over the bank branch locations, problems regarding how to cash checks, and requests of availability of finance products. It contains 2,305 training examples, and 593 test examples, for a total of 2,898 examples.

35Istituto Credito - full (IC_f): This dataset is composed of synthetic questions, generated by language engineers. Subsequently, it has been augmented to take into account also real sentences, retrieved from website chat-bot of a well known Credit Institution operating in Italy. It contains 2,898 training examples and 770 test examples, for a total of 3,668 examples.

36Banca - Area Informativa (Banca_AI): This dataset contains real questions asked by users about the Area Informativa of a bank. It includes 3,947 training examples, and 987 test examples, for a total of 4,934 examples divided in 282 intents.

37Banca - Internet Banking (Banca_IB): This dataset includes real questions asked by users about the iBanking service offered from a well known Italian bank. It includes 4,380 training instances and 1,906 test instances divided in 251 intents.

38Answer Sentence Selection data: We used an in-house dataset called IMSL-WIKI, which contains a list of question and answer regarding some of the products and services sold by IM Service Lab. For each question, a paragraph list was collected using an off-the-shelf search engine, i.e., Lucene, and manually annotated as either relevant or irrelevant. The dataset is divided into two parts, i.e., a training and test sets, which contain a total of 5,190 and 1,240 QA pairs, respectively. For each question, we retrieved a list of 10 candidate answers.

6.2 Model results

39In this section we report the performance of our two main machine learning components of our system: Answer Sentence Selection and Intent Classification.

6.2.1 Answer Sentence Reranking

40Table 1 reports the performance of the neural network and the baseline system. The first row, i.e., BM25, shows the baseline system, while the second row shows the performance of the CNN. The systems are evaluated according to the Mean Average Precision (MAP), Mean Reciprocal Rank (MRR) and Precision at 1 (P@1). The final results reported at the bottom is obtained as the average of 5 different models trained and evaluated on the test set. For each measure in the table, we report both mean and standard deviation computed on dev. and test sets.

41We used a small fraction of the training set, i.e., 15% of the data, for early stopping. As it can be seen from the table, CNN performs about 1 point more than the baseline algorithm (BM25) in terms of MAP on the dev. set, and almost 10 absolute points more of MAP on the test set.

42In addition, we observe an increase of 9.8 absolute points in terms of MRR, and 10.65 absolute points of P@1 on the test set. The difference between results on dev. and test sets can be explained by the fact that the used dev. set is very small: only 124 list of questions and 1,239 Q/A pairs, which made it difficult to optimize the three ranking metrics at the same time, so we focused on MAP.

6.2.2 Intent Classification

43We ran state-of-the-art neural classifiers described in Section 6.2.1 on Credit Institute and Bank datasets. To choose the best performance, we used 30% of training data as validation set and select the best hyperparameters. We compare the performance of neural models with respect to strong baseline classifiers, i.e., SVMs, and report the results in terms of Accuracy (Table 2) and F₁ (Table 3). The tables show that the final performance heavily depends on the used dataset and models.

44Istituto Credito (IC) datasets. Regarding the IC synthetic dataset, the best model, i.e., LSTM+CNN, obtains Accuracy of 77.37 and a micro-avg F₁ of 0.7742. This is about one absolute point of Accuracy higher than the base SVM model (77.37 vs. 76.22) and 1.47 absolute points of F₁ more than the base model (77.42 vs. 75.95). Similarly, on the IC full dataset, the performance of the best model, i.e, LSTM+CNN, achieved an accuracy of 82.31%, which is 0.66% absolute points better than the base model (82.31 vs. 80.65) and a micro-avg F₁ of 82.52, which is about one point better than the base SVM model (82.52 vs 81.51).

45Banca datasets. Regarding Banca AI dataset, the best model, i.e., LSTM obtained accuracy of 85.29, which is about 4 absolute points better than the base SVM model (85.29 vs. 81.97). Also, in terms of F₁, the best model obtained 3.77 absolute points more than the baseline (82.86 vs 80.09). Regarding the Banca IB dataset, the best model, i.e., LSTM, obtained around 6 points more both in terms of Accuracy (78.43 vs 72.35) and F₁ (76.91 vs 71.08).

6.3 End-to-End system evaluation

46We trained and evaluated our system using samples of data collecting from IMSL customers.

47We noted that the accuracy of the system improved because more answers are generally provided (from 3 to 5) to the user’s question, thus allowing to almost certainly provide the correct answer.

48The only point of attention is the fact that there is not always a valid answer to the user’s request in company knowledge. Indeed, the questions related to the user’s personal profile or data cannot be precisely answered by the company documentation.

49Furthermore, it often happens that the company policy prevents to provide explicit answers to specific user problems. In all these cases, it is therefore necessary to support the QA system with operators, who can provide personal answers or those not coded in the corporate knowledge.

7. Conclusions

50In this paper, we have presented a modern dialog system for real-world applications. We have tested advanced technology for QA and intent classification on several datasets derived from company data, such as Banks and Credit Institutions. The results show a promising direction for SMEs to build their own effective access to unstructured data.