MEDA: Meta-Learning with Data Augmentation for Few-Shot Text Classification

1 Introduction

Deep learning has achieved great success in many areas, including computer vision, speech recognition, and natural language processing. However, supervised deep learning models require a large amount of labeled data to achieve competitive performance. This requirement limits the generalizability of models to novel classes with few examples. In contrast, humans can learn a novel concept with very few examples. Inspired by this observation, researchers have begun to study few-shot learning problem.

Recent efforts to address the few-shot learning problem have leveraged a meta-learning paradigm [Snell et al., 2017; Finn et al., 2017]. Meta-learning methods train a metalearner, which extracts knowledge from many related tasks during meta-training and leverages that knowledge to learn new tasks during meta-testing quickly. Meta-learners are trained by sampling a small training set (support set) and test set (query set) from a large universe of labeled examples, feeding the support set to the learner to obtain a classifier, and then computing the loss of the classifier on the query set. The goal of meta-learner is to classify novel classes with few examples.

∗Contact Author

Figure 1: An illustration of ball generator. The first block shows the two classes of samples in the feature space. The second and third blocks illustrate two key parts. Firstly, we calculate the smallest enclosing ball [Welzl, 1991] of each class according to the existing samples. After that, we synthesize new samples in the smallest enclosing ball. Secondly, a transformation module is proposed to make synthetic samples close to their own ball center and away from other ball centers. The last block shows the results of synthesis and transformation.

Although these methods develop fast, most of these works concentrate on image classification, and less attention has been paid to text classification. Different from images, text in the same class has diverse expressions, which makes the performance of meta-learner more easily limited by the number of samples per class (i.e., the number of shots). Thus, the current meta-learning methods might be difficult to directly extend to few-shot text classification.

To tackle this problem, we introduce data augmentation for meta-learning, which brings in the benefits that we can produce augmented samples to increase the shot numbers for novel classes and diversify training samples. However, this poses a new challenge that producing high-confidence samples is non-trivial in the few-shot settings.

In order to address this issue, we propose a novel data augmentation method named ball generator, and Figure 1 shows a simple illustration for it. Firstly, we calculate the smallest enclosing ball [Welzl, 1991] of the support set and synthesize samples in this ball. We argue that samples in this ball have higher confidence because all support set samples are contained in this ball, and their farthest distance from the ball center is minimized. Secondly, to avoid the effect of synthetic sample bias, we introduce the transformation module to make synthetic samples close to their own ball center and away from other ball centers. The whole process is called "ball generator". Besides, we also propose a novel framework MEta-Learning with Data Augmentation (MEDA) to jointly train the ball generator and the meta-learner, and make them evolve synergistically. The experimental results show that compared with vanilla meta-learning, increasing the number of shots for novel classes can effectively improve the generalization ability of meta-learner on the few examples.

In summary, the main contributions of our work are as follows:

• We propose a novel data augmentation method named ball generator to increase the number of shots for novel classes. Notably, ball generator is agnostic to the choice of the meta-learning methods. We have successfully applied it to prototypical networks and relation networks, and verified the effectiveness of it.
• We propose a novel framework MEta-Learning with Data Augmentation (MEDA) for few-shot text classification, which jointly optimizes the ball generator and the meta-learner, such that the ball generator can learn to produce augmented samples that best fit the metalearner.
• We conduct a lot of experiments to prove the benefits of our proposed method. Experiments on two benchmark datasets demonstrate the effectiveness and superiority of the model.

2 RelatedWork

2.1 Meta Learning

Meta-learning intends to design models that can learn new skills or adapt to new environments rapidly with a few training examples. Metric-based meta-learning is one of the classical techniques used, where learns a distance function between data points so that it classifies test instances by comparing them to the K labeled examples. Typical work includes siamese networks [Koch et al., 2015], matching networks [Vinyals et al., 2016], prototypical networks [Snell et al., 2017], relation networks [Sung et al., 2018]. Although these works acquire fine results, most achievements are concentrated on computer vision field, the research and applications in text classification field are limited. Zhang et al. [2019] propose a meta-learning model which are the first to bridge the pretraining strategy with meta-learning methods for few-shot text classification. Unlike previous studies, our method equips meta-learning with data augmentation, which increases the number of shots by augmenting the minimally available samples and generates more diverse samples to extend to new tasks more effectively.

2.2 Data Augmentation

Data augmentation methods are explored in the field of text processing for improving the performance of models. At present, the mature methods of data augmentation can be divided into two categories: one is original text-oriented augmentation research; another is feature space data augmentation research. The former is mainly augmented by synonym replacement or deletion of words in the raw text. Such as Easy Data Augmentation (EDA) [Wei and Zou, 2019] augments the raw text based on four operations: synonym replacement, random insertion, random swap, and random deletion. Although these operations can augment text data, they may alter the text purport, and the noise brought will affect the performance of the model. More generally, a recent class of methods tries to explore augmentation in the feature space. They mainly process feature space of raw text, and inject noise into the text representation through different methods to improve the generalization ability of the model [Malandrakis et al., 2019; Fedus et al., 2018]. Although these methods have achieved good results, in the case of few examples, there are some problems such as the confidence of the augmented samples is not high, and the model training is difficult.

For few-shot text classification, Kumar et al. [2019] study six feature space data augmentation methods to improve classification performance in few-shot setting in combination with both supervised and unsupervised representation learning methods. Our approach also follows this line of work, but we redesign the model to make it simpler and easier to train. Meanwhile, our approach generates additional training samples by combining a meta-learner module with a generator module. These aforementioned modules are trained in an end-to-end manner to facilitate training few-shot text classifiers.

3 Preliminaries

3.1 ProblemFormulation

In this paper, we establish few-shot text classification in a meta-learning framework [Vinyals et al., 2016]. Specifically, we are given labeled examples from a set of classes Ctrain. Our goal is to learn a classification algorithm on Ctrain, so that we can make predictions over novel classes, which have only a few examples. These novel classes belong to a set of classes Ctest and disjoint from Ctrain.

To emulate the few-shot scenario, meta-learning algorithms learn from a group of N-way, K-shot tasks sampled from Ctrain and are then evaluated in a similar way on Ctest. During meta-training, we sample N classes (N-way) from Ctrain to create a single training episode. For each of these N classes, we sample K examples (K-shot) as the support set S and T examples as the query set Q, i.e., S = {(Xi, yi)}N×K i=1 and Q = {(Xj, yj)}N×T j=1 . We update our model based on loss over the query set Q. During meta-testing, we apply the same episode-based mechanism to test whether our model can indeed adapt quickly to novel classes Ctest.

3.2 Meta-learner

In meta-learning , meta-learners are trained according to episodes. In each episode, meta-learners calculate the probability distribution of each query instance ˆX ∈ Q under the support set S, denote as P(ˆ y| ˆX, S;φ). The difference between different meta-learners is how to implement P (ˆ y| ˆX, S;φ). Specifically, we choose prototypical networks and relation networks as meta-learners for MEDA.

Prototypical Networks. The prototypical networks [Snell et al., 2017] use support set S to calculate a prototype for each class, and then classify the query instance according to the distance between the query instance and the prototypes. More specifically, given a query instance ˆX, it belongs to the probability distribution of class c:

where d(·, ·) represents the distance calculation function, and the Euclidean distance is used here. pc represents the prototype, and its calculation formula is as follows:

where Sc ⊂ S represents the subset corresponding to class c in the support set S.

Relation Networks. Compared with the prototypical networks, relation networks [Sung et al., 2018] measure the distance between an unlabeled query instance and a few labeled instances by constructing neural networks. It consists of two parts: a feature extractor f(·) and a relation module g(·). First, the feature extractor generates representations of the query and training instances. Then the probability of the query instance ˆX belonging to class c is calculated by the relation module. The details are as follows:

where C(·, ·) is concatenating operation. pc represents the feature vector corresponding to class c, its calculation is as follows:

4 MEDA

This section introduces the general architecture and principal modules of the proposed model. Figure 2 shows the overall architecture of MEDA. MEDA has two modules: one is ball generator module; the other is meta-learner module. The generator module generates the samples based on the feature vector of the support S samples (Section 4.1). We take the set of augmented samples S′ and add it to the support set S to produce an extended support set ˜ S = S ∪ S′ . The metalearner module is to make predictions over the query set Q, after learning from the extended support set ˜ S (Section 3.2). These aforementioned modules are trained in an end-to-end manner (Section 4.2 and Section 4.3).

4.1 Ball Generator

Our method is a kind of feature space data augmentation. The key idea is to synthesize samples in feature space, and then alleviate the synthetic sample bias. To achieve so, we design a generator network, which is composed of the synthesis module and transformation module. As shown in Figure 2, we first learn data representation using a powerful pre-training language model BERT [Devlin et al., 2019]. Then, all samples are projected into the D-dimensional feature space through BERT.

Synthesis module. In feature space, we restrict the sampling space into smallest enclosing ball of the support set. Specifically, for support set Si, we use the randomization method proposed by Welzl [1991] to calculate the smallest enclosing ball B(Si) = {Ci, Ri} corresponding to each class yi, where Ci ∈ RD is the center of the ball and Ri is the

radius. After obtaining B(Si), we synthesize the sample X′ ∈ RD using the formula below:

where u is uniformly sampled in the range of [0-1]. Z is from standard normal distribution, i.e. Z ∼ N(0, I).

Transformation module. To avoid the effect of synthetic sample bias, we apply a transformation in feature space to make synthetic samples close to their own ball center and away from other ball centers. We therefore write the transformation as a function X′′ = G(X′ ; θ) that takes a synthetic point X′ as input, and produces an augmented sample X′′ as output. θ is the parameters of the transformation. We use a MLP with three fully connected layers for G(·; θ).

We call the whole process the ball generator. For each class, we use the generator to generate M augmented samples and get the set of augmented samples S′ . These augmented samples are added to the support set S to produce an extended support set ˜ S = S ∪ S′ , where S = {(Xp,i, yp,i)}N,K p=1,i=1 and S′ = {(X′′ p,j, yp,j)}N,M p=1,j=1 . Therefore, it is easy to integrate data augmentation into meta-learning by rewriting meta-learner from P(ˆ y| ˆX, S;φ) to P(ˆ y| ˆX, ˜ S; φ).

4.2 Hybrid Loss

To jointly optimize the ball generator and the meta-learner, we introduce the hybrid loss. With the help of hybrid loss, the ball generator can produce more suitable samples for the meta-learner. Therefore, the hybrid loss is defined as follows:

where λ is a trade-off parameter to control the relative importance of each term. LGEN is the generator loss, and LML is the meta-learner loss.

Generator Loss. To generate higher confidence augmented samples, we design the generator loss to make augmented samples close to their own ball center and away from other ball centers. The specific definition is as follows:

where X′′ is the augmented sample by the generator, Ci and Cj are the ball center corresponding to the i-th and jth classes respectively, d(·, ·) is the Euclidean distance and r is a margin.

Meta-learner Loss. To maximize learning, we design different loss functions for different meta-learners. Specifically, we use cross entropy loss for prototypical networks and mean square error loss for relation networks. Empirically, we find that using the generator loss speeds up the convergence and improves the classification performance than using the metalearner loss alone.

Algorithm 1 Training Strategy

Input: The number of ways, shots, queries, augmented samples N, K, T , M, labeled dataset Ctrain Output: θ,φ

• 1: for each episode iteration do
• 2: Randomly sample N classes fromCtrain
• 3: for i = 1 to N do
• 4: Randomly sample K examples for the class i-th as support set Si = {(Xs, ys)}Ks=1
• 5: Randomly sample T examples for the class i-th as query set Qi = {(Xq, yq)}Tq=1
• 6: B(Si) = {Ci,Ri} # Calculate the smallest enclosing ball using Si
• 7: S′ = ∅ # Initialize the extended support set
• 8: for j = 1 to M do
• 9: Compute the synthetic sample X′ by Eq. 5
• 10: Use G(X′ ; θ) to generate the augmented sample X′′
• 11: S′ = S′ ∪ {(X′′ , yi)}
• 12: end for
• 13: ˜ S = S ∪ S′
• 14: end for
• 15: Use the augmented support set S′ to calculate the generator loss by Eq. 7
• 16: Feed the extended support set ˜ S to the meta-learner, and calculate the meta-learner loss in the query set Q
• 17: Update the parameters θ and φ by minimizing the hybrid loss
• 18: end for

4.3 Training Strategy

Algorithm 1 summarizes the end-to-end training strategy. We construct the episode to calculate the gradient and update the parameters in each training iteration. We first sample the N-way-K-shot training episode to produce S and Q from Ctrain. We use the generator to take the set of augmented samples S′ and add it to the support set S to produce an extended support set ˜ S = S ∪ S′ . The augmented support set S′ is used to calculate the generator loss by Eq. 7. We then feed the extended support set ˜ S to the meta-learner, and calculate the meta-learner loss in the query set Q. We minimize the hybrid loss to update the parameters. In this way, we can optimize the MEDA in an end-to-end manner.

5 Experiments

5.1 Datasets

To prove the effectiveness of our proposed model, we evaluate the MEDA on two publicly datasets for the few-shot scenario: SNIPS 1 [Coucke et al., 2018] and ARSC 2 [Yu et al., 2018]. SNIPS is a personal voice assistant dataset which is collected in a crowd sourced fashion. ARSC is a sentiment classification dataset which is constructed by Amazon product reviews. Table 1 summarizes the statistics of the above mentioned datasets.

1 https://github.com/snipsco/nlu-benchmark/

2 https://github.com/Gorov/DiverseFewShot Amazon

SNIPS. As SNIPS is not a benchmark for few-shot learning, we first construct few-shot splits to simulate the fewshot scenario. We divide the original 7 intents into 5 intents as Ctrain and 2 intents (AddToPlaylist, RateBook) as Ctest. The Ctrain and Ctest are used as training set and test set respectively. Thus, we evaluate the performance on Ctest, i.e., 2-way-K-shot settings, where K=3,5,10.

ARSC. For ARSC dataset, we partition datasets following [Yu et al., 2018]. The dataset comprises English reviews for 23 types of products on Amazon. For each product domain, there are three different binary classification tasks. These buckets then form 23 × 3 = 69 tasks in total. Following [Yu et al., 2018], we also select 12 (4×3) tasks from four domains (Books, DVD, Electronics, Kitchen) as the test set, with only five examples as support set for each label in the test set. We create 5-shot learning models on this dataset.

5.2 Baselines

We compare against several baselines and competitors as follows. (1) From the perspective of data augmentation, our method is a kind of feature space data augmentation. Therefore, in order to illustrate the effectiveness of the proposed method, we compare it with other feature space data augmentation methods. There are mainly Extrapolation (EXTRA) [DeVries and Taylor, 2017] and Random Perturbation (RP) [Kumar et al., 2019]. In order to ensure fairness, we also increase the number of shots by the above methods and then classify them by prototypical networks. (2) For meta-learner, our overall framework is generic, and we can employ different meta-learning algorithms as meta-learner. Therefore, we compare the two methods described in Section 3.2, including prototypical networks (PN) and relation networks (RN). (3) From the perspective of few-shot text classification, in order to illustrate the effectiveness of MEDA, we also compare it with other state-of-the-art models, including ROBUSTTCFSL [Yu et al., 2018] and Induction Networks (IN) [Geng et al., 2019].

5.3 Implement Details

Our implementation is based on Pytorch 3 . We experiment with pre-trained BERT [Devlin et al., 2019] using Sentence-Transformers codebase [Reimers and Gurevych, 2019]. Adam [Kingma and Ba, 2015] is used to train the MEDA in an end-to-end manner. The initial learning rate is 1e-3. In the loss function, we set λ = 1 and r = 1. To avoid overfitting, we use dropout with 0.2 dropout rate. We generate 10 samples using different augmentation methods for the given class. We evaluate the model performance after each epoch and stop the training when the error on the validation set starts increasing. All hyper-parameters of the MEDA are cross-validated on the validation set using a coarse grid search. For EXTRA, we use λ = 0.5. For RP, we sample noise from a normal distribution with µ = 0, σ = 1.

3 https://pytorch.org/

6 Results

Overall Performance. Experiment results on all two datasets are presented in Table 2 and Table 3. From the results, we can clearly see that our proposed MEDA outperforms all other models on standard few-shot settings. Particularly, on SNIPS, the classification accuracy of MEDA is superior to the prototypical networks by more than 2.5% when K=3;5, showing the effectiveness of MEDA for fewshot text classification. With more data (e.g., K=10), although all baselines have performed very well, our MEDA is still improved by about 2.5% compared with others. For ARSC, the improvements of MEDA on this dataset are consistent with SNIPS. This further validates the general effectiveness of our proposed MEDA in addressing few-shot text classification.

MEDA vs. meta-learner. The comparison results between MEDA versions and vanilla versions of prototypical networks and relation networks are summarized in Table 2 and Table 3. For ARSC, we find that MEDA versions improve the classification accuracy significantly, by up to 17% and 4% compared with prototypical networks and relation networks, re- Figure 3: The mean accuracy on SNIPS with varying number of spectively. For SNIPS, MEDA versions are also effective, especially when K=10, they increase by 3.33% and 2.45%, respectively. This shows that the performance of conventional meta-learners can be further boosted when equipped with our data augmentation. Meanwhile, this also suggests that our proposed method is general and can work with different metalearners.

MEDA vs. data augmentation. We also compare our generation strategy with common forms of data augmentation (EXTRA and RP) in Table 2 and Table 3. From the results, we can clearly see that our MEDA outperforms other data augmentation approaches. This confirms that our generation strategy produces more diverse and useful training examples than simple data augmentation. It is interesting to note that all data augmentation methods have achieved good results. This further shows that on the basis of meta-learning, data augmentation can expand prior knowledge and improve model performance.

7 Analysis

K-shot evaluations. In this section, we analyze the effectiveness of MEDA from the perspective of K-shot in different settings. So we compare our model's accuracy with other models such as prototypical networks and relation networks on the 2-way SNIPS task, and the shot number ranges from 5 to 20. As shown in Figure 3, we could observe that MEDA achieves the best classification performances in all settings. Notably, the margin of MEDA increases as the shot number decreases, showing that the model achieves performance improvement when the shot number is relatively small.

Figure 5: Accuracy of different versions of L on ARSC validation set.

Number of augmented samples. We also study how the accuracy of the model changes with the number of augmented samples on SNIPS. We therefore plot the 5-shot and 10-shot accuracy in Figure 4. The results show that as the number of augmented samples increases, the accuracy of the model goes up. The performance saturates after 10 samples. For this reason, we use 10 augmented samples in all of our experiments. Comparing the 5-shot and 10-shot settings, we find that adding five augmented samples has almost the same effect as using five real samples. This may suggest that our proposed MEDA is not simply copying samples, but generating meaningful samples for the model.

Hybrid loss analysis. As described in Section 4.2, we use the hybrid loss L to guide the training of our MEDA. To demonstrate its superiority, we compare it with the alone meta-learner loss LML, i.e., without the generator loss LGEN. Figure 5 shows the accuracy curves of different loss functions with the number of iterations on ARSC. Comparing the different plots in Figure 5, we can see that using the hybrid loss function has a faster convergence speed than using the meta-learner loss function alone. At the same time, in Figure 5, we also analyze the change of the accuracy with the number of iterations. We find that, using the hybrid loss function has higher accuracy than using the meta-learner loss alone. This further shows that using the hybrid loss speeds up the convergence and improves the classification performance than using the meta-learner loss alone.

8 Conclusion

In this paper, we propose a novel framework MEDA for fewshot text classification that uses ball generator to increase the number of shots for meta-learner. MEDA is an end-to-end framework that jointly optimizes ball generator and metalearner. The proposed model achieves state-of-the-art performance on two publicly datasets. In the future, we will explore the effectiveness of MEDA in other domains.

Acknowledgements

We thank all anonymous reviewers who provided valuable feedback. Our research was supported by the NSFC (No. 61936012,61976114) and the National Key R&D Program of China (No. 2018YFB1005102).