1. Introduction

1Natural Language Understanding (NLU) in task-oriented dialogue systems is responsible for parsing user utterances to extract the intent of the user and the arguments of the intent (i.e. slots) into a semantic representation, typically a semantic frame (Tur and De Mori 2011). For example, the utterance “Play Jeff Pilson on Youtube" has the intent PlayMusic and “Youtube" as value for the slot Service. As more skills are added to the dialogue system, the NLU model frequently needs to be updated to scale to new domains and languages, a situation which typically becomes problematic when labeled data are limited (data scarcity).

2One way to combat data scarcity is through data augmentation (DA) techniques performing label preserving operations to produce auxiliary training data. Recently, DA has shown potential in tasks such as machine translation (Fadaee, Bisazza, and Monz 2017), constituency and dependency parsing (Şahin and Steedman 2018; Vania et al. 2019), and text classification (Wei and Zou 2019; Kumar, Choudhary, and Cho 2020). As for slot filling (SF) and intent classification (IC), a number of DA methods have been proposed to generate synthetic utterances using sequence to sequence models (Hou et al. 2018; Zhao, Zhu, and Yu 2019), Conditional Variational Auto Encoder (Yoo, Shin, and Lee 2019), or pre-trained NLG models (Peng et al. 2020). To date, most of the DA methods are evaluated on English and it is not clear whether the same finding apply to other languages.1

3In this paper, we study the effectiveness of DA on several non-English datasets for NLU in task-oriented dialogue systems. We experiment with existing lightweight, non-gradient based, DA methods from that produces varying slot values through substitution and sentence structure manipulation by leveraging syntactic information from a dependency parser. We evaluate the DA methods on NLU datasets from five languages: Italian, Hindi, Turkish, Spanish, and Thai. The contributions of our paper are as follows:

1. We assess the applicability of DA methods for NLU in task-oriented dialogue systems in five languages.

2. We demonstrate that simple DA can improve performance on all languages despite different characteristic of the languages.

3. We show that a large pre-trained multilingual BERT (M-Bert) (Devlin et al. 2019) can still benefit from DA, in particular for slot filling.

Figure 1

3.1 Slot Filling and Intent Classification

4The NLU component of a task-oriented dialogue system is responsible in a parsing user utterance into a semantic representation, such as semantic frame. The semantic frame conveys information, namely the user intent and the corresponding arguments of the intent. Extracting such information involves slot filling (SF) and intent classification (IC) tasks.

5Given an input utterance of n tokens, x = (x₁, x₂,.., x_n), the system needs to assign a particular intent y^intent for the whole utterance x and the corresponding slots that are mentioned in the utterance . In practice, IC is typically modeled as text classification and SF as a sequence tagging problem. As an example, for the utterance “Play Jeff Pilson on Youtube", y^intent is PlayMusic, as the intent of the user is to ask the system to play a song from a musician and y^slot = (O, B-ARTIST, I-ARTIST, O, B-SERVICE) in which the artist is “Jeff Pilson" and the service is “Youtube"". Slot labels are in BIO format: B indicates the start of a slot span, I the inside of a span while O denotes that the word does not belong to any slot. Recent approaches for SF and IC are based on neural network methods that models SF and IC jointly (Goo et al. 2018; Chen, Zhuo, and Wang 2019) by sharing model parameter among both tasks.

Data Augmentation (DA) Methods

6DA aims to perform semantically preserving transformations on the training data to produce auxiliary data . The union of and is then used to train a particular NLU model. For each utterance in , we produce N augmented utterances by applying a specific augmentation operation. We adopt a subset of existing augmentation methods from , that has shown promising results on English datasets. We describe the augmentation operations in the following sections.

3.1 Slot Substitution (Slot-Sub)

7Slot-Sub (Figure 1 left) performs augmentation by substituting a particular text span (slot-value pair) in an utterance with a different text span that is semantically consistent i.e., the slot label is the same. For example, in the utterance “Quali film animati stanno proiettando al cinema più vicino”, one of the spans that can be substituted is the slot value pair (più vicino, spatial relation). Then, we collect other spans in  in which the slot values are different, but the slot label is the same. For instance, we found the substitute candidates SP'= {(“distanza a piedi”, spatial relation), (“lontano”, spatial relation), (“nel quartiere”, spatial relation), …}, and then we sample one span to replace the original span in the utterance.

Statistics on the datasets. #train indicates our limited training data setup (10% of full training data). is produced by tuning the number of augmentations per utterance (N) on the dev set.

3.2 Crop and Rotate

8In order to produce sentence variations, we apply the crop and rotate operations proposed in , which manipulate the sentence structure through its dependency parse tree. The goal of Crop (Figure 1 middle) is to simplify the sentence so that it focuses on a particular fragment (e.g. subject/object) by removing other fragments in the sentence. Crop uses the dependency tree to identify the fragment and then remove it and its children from the dependency tree.

9The Rotate (Figure 1 right) operation is performed by moving a particular fragment (including subject/object) around the root of the tree, typically the verb in the sentence. For each operation, all possible combinations are generated, and one of them is picked randomly as the augmented sentence. Both Crop and Rotate rely on the universal dependency labels (Nivre et al. 2017) to identify relevant fragments, such as NSUBJ (nominal subject), DOBJ (direct object), OBJ (object), IOBJ (indirect object).

4. Experiments

10Our primary goal is to verify the effectiveness of data augmentation on Italian, Hindi, Turkish, Spanish and Thai NLU datasets with limited labeled data. To this end, we compare the performance of a baseline NLU model trained on the original training data () with a NLU model that incorporates the augmented data as additional training instances (+ ). To simulate the limited labeled data situation we randomly sample 10% of the training data for each dataset.

Baseline and Data Augmentation (DA) Methods

11We use the state of the art BERT-based joint intent slot filling model (Chen, Zhuo, and Wang 2019) as the baseline model. We leverage the pre-trained multilingual BERT (M-Bert), which is trained on 104 languages. During training, M-Bert is fine tuned on the slot filling and intent classification tasks. Given a sentence representation , we use the hidden state h_[CLS] to predict the intent, and to predict the slot label. As for DA methods, in addition to the methods described in Section 2, we add one configuration Combine, which combines the result of Slot-Sub and Rotate, as Rotate obtains better results than Crop on the development set.

Settings

12The model is trained with the BertAdam optimizer for 30 epochs with early stopping. The learning rate is set to 10^-5 and batch size is 16. All the hyperparameters are listed in Appendix A. For Slot-Sub the number of augmentation per sentence N is tuned on the development set. To produce the dependency tree, we parse the sentence using Stanza (Qi et al. 2020). For both Crop and Rotate we follow the default hyperparameters from . We did not experiment with Thai for Crop and Rotate as Thai is not supported by Stanza. The number of augmented sentences () for each method is listed in Table 1. For evaluation metric, we use the standard CoNLL script to compute F1 score for slot filling and accuracy for intent classification.

Datasets

13For the Italian language, we use the data from , translated from the English SNIPS dataset (Coucke et al. 2018). SNIPS has been widely used for evaluating NLU models and consists of utterances in multiple domains. As for Hindi and Turkish, we use the ATIS dataset from , derived from . ATIS is a well known NLU dataset on flight domain. As for Spanish and Thai we use the FB dataset from that contains utterances in alarm, weather, and reminder domains. The overall statistics of the datasets are shown in Table 1.

5. Results

14The overall results reported in Table 2 show that applying DA improves performance on slot filling and intent classification across all languages. In particular, for SF, the Slot-Sub method yields the best result, while for IC, Rotate obtains better performance compared to Crop in most cases. These results are consistent with the finding from on the English dataset, where Slot-Sub improves SF and Crop or Rotate improve IC. In general, Rotate is better than Crop for most cases on IC, and we think this is because Crop may change the intent of the original sentence. Intents typically depend on the occurrence of specific slots, so when the cropped part is a slot-value, it may change the sentence’s overall semantics.

15We can see that languages with different typological features (e.g. subject/verb/object ordering)2 benefit from Rotate operation for IC. This result suggests that augmentation can produce useful noise (regularization) for the model to alleviate overfitting when labeled data is limited. When we use Combine, it still helps the performance of both SF and IC, although the improvements are not as high as when only one of the augmentation method is applied. The only language that gets the benefits the most from Combine is Turkish. We hypothesize that as Turkish has a more flexible word order than the other languages it benefits the most when Rotate is performed.

Figure 2

Performance on varying data size

16To better understand the effectiveness of Slot-Sub, we perform further analysis on different training data size (see Figure 2). Overall, we observe that as we increase the training size, the benefit of Slot-Sub is decreasing for all datasets. For some datasets, namely ATIS-HI and FB-ES, Slot-Sub can cause performance drop for larger data size, although it is reasonably small (less than 1 F1 point). FB-TH consistently benefits from Slot-Sub even when full training data is used. Until which training data size the improvement is significant vary across datasets3. For SNIPS-IT, improvement is clear for all training data size and they are statistically significant up until the training data size is 80%. For ATIS-HI improvements are significant until data size of 40%. As for FB datasets, improvements are significant only until the training data size is 10%. Overall, we can see that Slot-Sub is effective for cases where data is scarce (5%, 10%), while it is still relatively robust for larger data size on all datasets.

Performance on different numbers of augmentation per utterance (N).

17We examine the effect of a larger number of augmentations per utterance (N) to the model performance, specifically for SF (see Figure 3). For FB-ES, similarly to the results in Table 2, increasing N does not affect the performance. For the other datasets, increasing N brings performance improvement. For ATIS-HI, SNIPS-IT, and FB-TH the trend is that, as we increase N, performance goes up and plateau. For ATIS-TR, changing N does not really affect the gain of the performance as the performance trend is quite steady across number of augmentations. For most combinations of N in each dataset (except FB-ES), the difference between the performance of model that using Slot-Sub and the model that does not use Slot-Sub is significant.4

Figure 3

6. Related Work

18Data augmentation methods that has been proposed in NLP aims to automatically produce additional training data through different kinds of methods ranging from simple word substitution (Wei and Zou 2019) to more complex methods that aims to produce semantically preserving sentence generation (Hou et al. 2018; Gao et al. 2020). In the context of slot filling and intent classification, recent augmentation methods typically apply deep learning models to produce augmented utterances.

19proposes a two-stages methods to produce the delexicalized utterances generation and slot values realization. Their method is based on a sequence to sequence based model (Sutskever, Vinyals, and Le 2014) to produce a paraphrase of an utterance with its slot values placeholder (delexicalized) for a given intent. For the slot values lexicalization, they use the slot values in the training data that occur in similar contexts. trains a sequence to sequence model with training instances that consist of a pair of atomic templates of dialogue acts and its sentence realization. proposes a solution by extending Variational Auto Encoder (VAE) (Kingma and Welling 2014) into a Conditional VAE (CVAE) to generate synthetic utterances. The CVAE controls="true" the utterance generation by conditioning on the intent and slot labels during model training. Recent work from make use of Transformer (Vaswani et al. 2017) based pre-trained NLG namely GPT-2 (Radford et al. 2019), and fine-tune it to slot filling dataset to produce synthetic utterances. We consider these deep learning based approaches as heavyweight as they often require several stages in the augmentation process namely generating augmentation candidates, ranking and filtering the candidates before producing the final augmented data. Consequently, the computation time of these approaches is generally more expensive as separate training is required to train the augmentation and joint SF-IC models. Recent work from apply a set of lightweight methods in which most of the augmentation methods do not require model training. The augmentation methods focus on varying the slot values through substitution mechanisms and varying sentence structure through dependency tree manipulation. While the methods are relatively simple it obtains competitive results with deep learning based approaches on the standard English slot filling benchmark datasets namely ATIS (Hemphill, Godfrey, and Doddington 1990), SNIPS (Coucke et al. 2018), and FB (Schuster et al. 2019) datasets.

20Existing methods mostly evaluate their approaches on English datasets, and little work has been done on other languages. Our work focuses on investigating the effect of data augmentation on five non-English languages. We apply a subset of lightweight augmentation methods from that do not require separate model training to produce augmentation data.

7. Conclusion

21We evaluate the effectiveness of data augmentation for slot filling and intent classification tasks in five typologically diverse languages. Our results show that by applying simple augmentation, namely slot values substitutions and dependency tree manipulations, we can obtain substantial improvement in most cases when only small amount of training data is available. We also show that a large pre-trained multilingual BERT benefits from data augmentation.