Interpretable Adversarial Perturbation in Input Embedding Space for Text

1 Introduction

The existence of (small) perturbations, which induce prediction error in machine learning models, was first discovered and discussed in [Szegedy et al., 2014]. They called the perturbed inputs adversarial examples. Such perturbations can be easily found by optimizing the input to maximize the prediction error. After this discovery, a framework called adversarial training (AdvT) was proposed [Goodfellow et al., 2015] whose basic idea was to train models that can correctly classify both the original training data and adversarial examples generated based on the training data. Using AdvT, we can further improve the generalization performance of models. This improvement implies that the loss function of adversarial examples works as a good regularizer during model training. Currently, a technique for generating adversarial examples is crucial to neural image processing for both improving the task performance and analyzing the behaviors of black-box neural models.

∗This work was conducted when the first author worked at Nara Institute of Science and Technology and RIKEN AIP.

†His current affiliation is Tohoku University.

1 Our code for reproducing our experiments is available at https: //github.com/aonotas/interpretable-adv

Unlike its great success in the image processing field, AdvT cannot be straightforwardly applied to tasks in the natural language processing (NLP) field. This is because we cannot calculate the perturbed inputs for tasks in the NLP field since the inputs consist of discrete symbols, which are not a continuous space used in image processing. A novel strategy was recently proposed to improve AdvT for NLP tasks [Miyato et al., 2017] whose key strategy is simple and straightforward: applying AdvT to continuous word embedding space rather than the discrete input space of texts. Their method preserves a theoretical background developed in the image processing field and works well as a regularizer. In fact, this method significantly improved the task performance and achieved the state-of-the-art performance on several text classification tasks. Another notable merit of this method is succinct architecture. It only requires the gradient of the loss function to obtain adversarial perturbations (see Eq. 9). Note that the gradient calculation is a standard calculation procedure for updating the model parameters during training. We can obtain adversarial perturbations in the embedding space with a surprisingly small calculation cost without incorporating any additional sophisticated architecture.

In contrast, the main drawback of this method is that it abandons the generation of adversarial examples interpretable by people since how to appropriately reconstruct perturbed inputs in the input word embedding space to actual texts is not trivial. This implies that this approach lacks interpretability. In fact, they declared that since their training strategy is no longer intended as a defense against adversaries, they exclusively proposed it as a regularizer to stabilize the model [Miyato et al., 2017]. It is often desirable for researchers and developers to generate adversarial examples (adversarial texts) to understand the behavior of black-box neural models. Therefore, a trade-off exists between wellformed and low-cost (gradient-based) approaches and the interpretability of the AdvT methods used in the NLP field.

The main topic of this paper is the reduction of this tradeoff gap. This paper restores interpretability while preserving the good ability of regularizer. Our main idea is to only restrict the directions of the perturbations toward the locations of existing words in the word embedding space. Fig. 1 illustrates an intuitive explanation of our idea. With our method, we can straightforwardly interpret each input with a perturbation as an actual sentence by considering the perturbations to be substitutions of the words in the sentence. To the best of our knowledge, our study is the first trial that discusses the interpretability of AdvT based on adversarial perturbation applied to tasks in the NLP field.

2 RelatedWork

Several studies have applied the ideas of AdvT to certain NLP tasks. A method was proposed that fooled reading comprehension systems by adding sentences to the ends of paragraphs using crowdsourcing [Jia and Liang, 2017]. Random character swaps can break the output of neural machine translation systems [Belinkov and Bisk, 2018; Hosseini et al., 2017], and thus they proposed AdvT methods that generate random character swaps and utilize the generated input sentences as additional training data for their models. Moreover, a method generated a large number of input sentences by replacing a word with its synonym [Samanta and Mehta, 2017]

The primary strategy for generating adversarial examples in the NLP field clearly differs from those developed in the image processing field, which are rather ad-hoc, e.g., using human knowledge [Jia and Liang, 2017], dictionaries [Samanta and Mehta, 2017], or require such costly procedures as exhaustive searches [Samanta and Mehta, 2017]. These methods are not essentially based on the previously discussed idea of perturbation that was first discussed [Szegedy et al., 2014] for generating adversarial examples.

In contrast, our baseline method [Miyato et al., 2017] preserves a theoretical background developed in the image processing field. Thus, note that the methods discussed in this paper borrow a distinct strategy from the current primal strategy taken in the NLP field as described above.

3 Target Tasks and Baseline Models

This section briefly explains the formal definitions of our target tasks, text classification and sequence labeling, and the baseline neural models for modeling these tasks. Fig. 2 shows the architecture of the baseline neural models.

3.1 Common Notation

Let X represent an input sentence. V denotes the vocabulary of the input words. x(t) ∈ V is the t-th word that appears in given input sentence X, where X = (x(1) , . . . , x(T) ) if the number of words in X is T . Here we introduce the following short notation of sequence (x(1) , . . . , x(T) ) as (x(t) )Tt=1 . Y denotes a set of output classes. To explain the text classification and the sequence labeling tasks in a single framework, this paper assumes that output Y denotes sequence of class labels Y = (y(t) )Tt=1 , where y(t) ∈ Y for all t in the case of sequence labeling, and class label Y = y, where y ∈ Y for the text classification case.

Let w(t) be a word embedding vector that corresponds to x(t) whose dimension is D, where w(t) ∈ RD . Thus, sequence of word embedding vectors ˜X that corresponds to X can be written as ˜X = (w(t) )Tt=1 . Then for text classification, ˜ y denotes a corresponding class ID of y in Y. ˜ y always takes one integer from 1 to |Y|, where ˜ y ∈ {1, . . . , |Y|}. ˜ y(t) also denotes a corresponding class ID of ˜ y(t) in Y for sequence labeling. Finally, ˜ Y represents ˜ Y = ˜ y for text classification and ˜ Y = (˜ y(t) )Tt=1 for sequence labeling.

Here, without loss of generality, we formulate a text classification task or a sequence labeling task whose inputs and outputs are respectively ˜X and ˜ Y instead of X and Y . This is because we can uniquely convert from X to ˜X and from ˜ Y to Y . Thus, training data D can be represented as a set of ˜X and ˜ Y pairs, namely, D = {( ˜X(n) , ˜ Y (n) )}Nn=1 , where N represents the amount of training data.

3.2 Baseline Model for Text Classification

We encode input ˜X with a recurrent neural network (RNN)- based model, which consists of an LSTM unit [Hochreiter and Schmidhuber, 1997]. The (forward) LSTM unit calculates a hidden state in each step t as h(t) = LSTM(w(t) , h(t-1) ), where h(0) is assumed to be a zero vector. Then we model the (conditional) probability of output ˜ Y given input ˜X as follows:

where qm is the m-th factor of q whose dimension is |Y|. q is calculated through a standard feed-forward neural network from T-th final hidden state h(T) , where q = FFNN(h(T) ). Here we omit an explanation of the detailed configurations of LSTM and FFNN, but they will be described in our experiment section, since the selection of their configurations affects none of this paper's discussion.

3.3 Baseline Model for Sequence Labeling

We employ a bi-directional LSTM to encode input ˜X. The hidden state of each step t, that is, h(t) , can be obtained by the concatenation of two hidden states from forward and backward LSTMs: h(t) = concat(h(t) f , h(t) b ), where h(t) f = LSTM(w(t) , h(t-1) f ), and h(t) b = LSTM(w(t) , h(t+1) b ). We assume that h(0) f and h(T+1) b are always zero vectors. We also assume that probability p(˜ Y | ˜X,W) can be decomposed into each step t. This means that probability p(˜ Y | ˜X,W) can be calculated:

where q(t) m is the m-th factor of q(t) whose dimension is |Y|. q(t) is calculated through a standard feed-forward neural network from t-th final hidden state h(t) : q(t) = FFNN(h(t) ).

3.4 Training

For training both the text categorization and the sequence labeling, we generally find the optimal parameters of an RNNbased model that can minimize the following optimization problem:

where W represents the overall parameters in the RNN-based model. J (D,W) is the loss function over entire training data D, and ℓ( ˜X, ˜ Y ,W) is the loss function of individual training sample ( ˜X, ˜ Y ) in D:

4 Adversarial Training in Embedding Space

Adversarial training (AdvT) [Goodfellow et al., 2015] is a novel regularization method that improves the robustness of misclassifying small perturbed inputs. To distinguish the AdvT method in image processing, this paper specifically refers to AdvT that is applied to input word embedding space for NLP tasks as AdvT-Text, which was first introduced in [Miyato et al., 2017].

Let r(t) AdvT be a (adversarial) perturbation vector for tth word x(t) in input ˜X. We assume that r(t) AdvT is a Ddimensional vector whose dimension always matches that of word embedding vector w(t) . Fig. 3 shows the AdvT-Text architecture and our baseline neural models by applying AdvT. See also Fig. 2 for a comparison of the architecture with our baseline neural models. Let r represent a concatenated vector of r(t) for all t. We introduce ˜X+r that denotes ˜X with additional small perturbations, where ˜X+r = (w(t) +r(t) )Tt=1 . To obtain (worst case) perturbations r(t) AdvT for all t for maximizing the negative log-likelihood (equivalent to minimizing the log-likelihood), we seek optimal solution rAdvT by maximizing the following equation:

where ϵ is a tunable hyper-parameter that controls the norm of the perturbation and rAdvT represents a concatenated vector of r(t) AdvT for all t that resemble r. Then based on adversarial perturbation rAdvT, the loss function for AdvT-Text can be defined:

where ˜X+rAdvT = (w(t) +r(t) AdvT)Tt=1 , similar to ˜X+r.

It is generally infeasible to exactly estimate rAdvT in Eq. 7 for sophisticated deep neural models. As a solution, an approximation method was proposed by linearizing ℓ( ˜X, ˜ Y ,W) around ˜X [Goodfellow et al., 2015]. For our RNN-based models, the approximation method induces the following non-iterative solution for calculating r(t) AdvT for all t:

where g is a concatenated vector of g(t) for all t.

Finally, we jointly minimize objective functions J (D, W) and JAdvT(D, W):

where λ is a coefficient that controls the balance of two loss functions.

5 Interpretable Adversarial Perturbation

As described above, we extended Adv-Text to restore the ability to generate adversarial texts that are interpretable by people while maintaining the task performance. We only restrict the directions of the perturbations in the embedding space toward existing words in the input word embedding space. The intuition behind our method is that the directions to other words can be interpreted as the substitution of another word in the sentence, which may reconstruct adversarial texts. We refer to our AdvT-Text extension as interpretable AdvT-Text or iAdvT-Text.

5.1 Definition of Interpretable AdvT-Text

Suppose wk denotes a word embedding vector that corresponds to the k-th word in vocabulary V. We define direction vector d(t) k that indicates the direction fromw(t) to wk in the input word embedding space:

Note that d(t) k for all t and k is always a unit vector, ||d(t) k ||2 = 1. If the t-th word in the given input sentence is the k-th word in the vocabulary, then wk = w(t) , and thus, d(t) k becomes a zero vector 2 .

Next let α(t) be a |V|-dimensional vector, and let α(t) k be the k-th factor of α(t) , where α(t) = (α(t) k )|V| k=1 . We define r(α(t) ) that denotes the perturbation generated for the t-th word in ˜X, which is parameterized by α(t) :

α(t) k is a weight for the direction from the t-th word in the input to the k-th word in the vocabulary. Then, similar to the definition of ˜X+r, we also introduce ˜X+r(α) as follows:

Similar to Eq. 7, we seek the worst case weights of the direction vectors that maximize the loss functions as follows:

Then we define the loss functions of our method, iAdvT-Text, based on αiAdvT:

We substitute JAdvT(D,W) in Eq. 10 with JiAdvT(D,W) for our method, where the form of the optimization problem can be simply written:

To reduce the calculation cost, we also introduce an update formula derived by applying the same idea of the approximation method explained in Eq. 9:

Similar to r(t) AdvT, the intuitive interpretation of α(t) iAdvT is the (normalized) strength of each direction d(t) k about howmuch to increase the loss function. Thus, we expect to evaluate which direction of words is a good adversarial perturbation.

5.2 Practical Computation

The most time-consuming part of our method is its calculation of the summation of all the words that appeared in Eq. 12, which includes the calculation of directions d(t) k for all the words from each word w(t) , as shown in Eqs. 11. At most, this creates a computational cost of |V|2 , which might be unacceptable compared with the small computational cost of AdvT-Text (the previous method). Here we introduce V (t) as individual vocabularies of step t, where V (t) ⊆ V for all t and |V (t) | ≪ |V|, i.e., |V (t) | = 10. In our method, we select the |V (t) | nearest neighbor word embeddings around each w(t) for all t in each iteration during the training. This approximation is equivalent to treating α(t) k = 0 for all k if wk /∈ V (t) for all t. The intuition behind this approximation is that words with a large distance can be treated as nearly unrelated words.

5.3 Extension to Semi-Supervised Learning

Suppose D′ denotes a set of labeled and unlabeled data. Virtual adversarial training (VAT) [Miyato et al., 2016] is a (regularization) method closely related to AdvT. VAT, a natural extension of AdvT to semi-supervised learning, can also be applied to tasks in NLP fields, which we refer to as VAT-Text. We borrow this idea and extend it to our iAdvT-Text for a semi-supervised setting, which we refer to as iVAT-Text.

VAT-Text uses the following objective function for estimating the loss of adversarial perturbation rVAT:

where KL(·||·) denotes the KL divergence. To obtain rVAT, we solve the following optimization problem:

Then instead of solving the above optimization problem, an approximated method was proposed [Miyato et al., 2017]:

By using the same derivation technique to obtain the above approximation, we introduce the following equation to calculate α(t) iVAT for an extension to semi-supervised learning:

Then the objective function for iVAT-Text can be written:

6 Experiments

We conducted our experiments on a sentiment classification (SEC) task, a category classification (CAC) task, and a grammatical error detection (GED) task to evaluate the effectiveness of our methods, iAdvT-Text and iVAT-Text. SEC is a text classification task that classifies a given text into either a positive or a negative class. GED is a sequence labeling task that identifies ungrammatical words.

6.1 Datasets

For SEC, we used the following well-studied benchmark datasets, IMDB [Maas et al., 2011], Elec [Johnson and Zhang, 2015], and Rotten Tomatoes [Pang and Lee, 2005]. In our experiment with the Rotten Tomatoes dataset, we utilized unlabeled examples from the Amazon Reviews dataset 3 . For CAC, we utilized DBpedia [Lehmann et al., 2015] and RCV1 [Lewis et al., 2004]. Since the DBpedia dataset has no additional unlabeled examples, the DBpedia results are only for the supervised learning task. Following [Miyato et al., 2017], we split the original training data into training and development sentences. For GED, we utilized the First Certificate in the English dataset (FCE-public) [Yannakoudakis et al., 2011]. Table 1 summarizes the information about each dataset.

6.2 Model Settings

To fairly compare our methods with previous methods, we followed previously described model configurations [Miyato et al., 2017] for SEC and [Rei and Yannakoudakis, 2016; Kaneko et al., 2017] GED, shown in Fig. 3: left for SEC and right for GED. Moreover, following [Miyato et al., 2017], we initialized the word embeddings and the LSTM weights with a pre-trained RNN-based language model [Bengio et al., 2000] that was trained on labeled training and unlabeled data if they were available. To reduce the computational cost of softmax loss, we use the Adaptive Softmax [Grave et al., 2017] for training language model. We utilized an early stopping criterion [Caruana et al., 2000] based on the performance measured on development sets. The hyper-parameters are summarized in Table 2, with dropout [Srivastava et al., 2014] and Adam [Kingma and Ba, 2014]. In addition, we set ϵ = 5.0 for both AdvT-Text and VAT-Text and ϵ = 15.0 for our method. We also set λ = 1 for all the methods. To find the best hyper-parameter, we picked models whose performances were best measured on development data.

3 http://snap.stanford.edu/data/web-Amazon.html

In addition, we implemented our methods (iAdvT-Text and iVAT-Text) and re-implemented the previous methods (AdvTText and VAT-Text) using Chainer [Tokui et al., 2015] with GPU support. All four methods share sub-modules, such as RNN-based models, in our implementation. Therefore, our internal experiments are fairly compared under identical conditions.

6.3 Evaluation by Task Performance

Table 3 shows the IMDB performance evaluated by the error rate. Random perturbation (Labeled) is the method with which we replaced r(t) AdvT with a random unit vector, and Random Perturbation (Labeled + Unlabeled) is the method with which we replaced r(t) VAT with a random unit vector. We tried two simple methods, iAdvT-Rand and iAdvT-Best. iAdvRand is the method with which we replaced r(t) AdvT with the nearest ten, randomly picked words vectors. iAdv-Best is the method from which we picked the best direction based on Eq. 13. Surprisingly, iAdvT-Text outperformed AdvTText, and iVAT-Text achieved the same performance level and slightly outperformed VAT-Text 4 . Note here that our method was mainly designed to restore the interpretability, not to improve the task performance. Before evaluating our methods, iAdvT-Text and iVAT-Text, we assumed that they would respectively degrade the AdvT-Text and VAT-Text performances, since our methods strongly restrict the degrees of freedom for the direction of the perturbations for interpretability. This suggests that the directions of the existence of actual words in the word embedding space provide useful information for improving the generalization performance. The performance of two simple methods (iAdvT-Rand and iAdvT-Best) is poor. Tables 4 and 5 show the performance on the other datasets 5 .

Table 6 shows the test performance on the GED task. We used F0.5 as an evaluation measure for GED, which was adopted in the CoNLL-14 shared task [Rozovskaya et al., 2014] 6 . Other reported results [Rei and Yannakoudakis, 2016; Kaneko et al., 2017] are around the current state-of-theart performance on this dataset. In our experiments, AdvTText achieved the highest F0.5. Our experiments revealed that AdvT-Text can further outperform the current state-ofthe-art methods. Moreover, our methods successfully reached performances that almost matched AdvT-Text and VAT-Text. Again, we emphasize that these results are substantially positive for our methods since they did not degrade the performance even when we added a strong restriction for calculating the perturbations.

4 The AdvT-Text and VAT-Text scores were obtained by our reimplemented code, which outperformed the original scores [Miyato et al., 2017] (Adv:6.21 %, VAT:5.91 %).

5 For RCV1 and DBpedia, our baselines were slightly weak due to the resource limitation of constracting the large-scale pretrainined language models.

6 A previous study [Nagata and Nakatani, 2010] suggested that since accurate prediction is more important than coverage in error detection applications, F0.5 was selected rather than F1.

In addition, in contrast to SEC, VAT-Text did not outperform AdvT-Text. Since the GED dataset does not contain a large amount of unlabeled data, we confirmed that it is hard for VAT-Text to improve the performance.

6.4 Visualization of Sentence-Level Perturbations

We visualized the perturbations computed by our method (iAdvT-Text) in Fig. 4 for understanding its behavior. We also visualized the perturbations by the previous method (AdvTText) for comparison. The words at the left of each (blue or green) bar indicate the words in the (true) sentences in the dataset. We selected the highest direction toward a word from each word in the sentence. In our method, it can be easily obtained by selecting the maximum values of α(t) k for all t.

For AdvT-Text, we calculated the cosine similarities between the perturbation and direction to every word wk and selected a word with the highest cosine similarity. Each word written in the (blue or green) bar represents the selected word by the above operations, and shades of color are the relative strengths of the obtained perturbations toward the selected words.

For the SEC task, the correct label of the sentence in the blue bar is positive. The iAdvT-Text successfully found the directions for replacing better with worse to increase the loss. In other words, the direction might change the class label from positive to negative. For the GED task, the sentence in the green bar contains a grammatical error word (practise), which should be replaced with play. The iAdvT-Text also found directions for replacing practise with play.

In contrast, the perturbations of AdvT-Text (Previous) were uninterpretable (replacing <eos> with Analyze, and replacing practise with UNFORTUNETELY). This is mainly because the perturbations of AdvT-Text barely matched the direction toward any existence points of word embeddings, and we just visualized the most cosine similar words with perturbation.

These results revealed that the directions of the perturbations in iAdvT-Text are understandable by humans, and thus, offer a chance for researchers to interpret black-box neural models, regardless whether the model properly learned certain phenomena that the researchers are interested in. We believe that such interpretability is critical, especially for sophisticated neural models. The usefulness of this visualization is the main claim of our proposed methods.

6.5 Adversarial Texts

We reconstructed adversarial examples, which misclassified the trained models, from the adversarial perturbations in the input word embedding space given by iAdvT-Text. To obtain adversarial texts, we first identified the largest perturbation and replaced the original word with one that matches the largest perturbation.

Table 7 shows typical examples, where the top two rows show an example for SEC, and the bottom two rows show an example for GED. For example, the second example in Table

7 was generated by changing this to that. Even though this example does not alter the meaning, the prediction was changed from Negative → Positive. The generated adversarial texts for GED still contain grammatical error; however the model predicts that they are grammatically correct. Thus, these two examples are adversarial texts.

Note that the previous methods, AdvT-Text and VAT-Text, hardly reconstruct such effective adversarial texts. Thus, this is a clear advantage of our methods compared with the previous ones.

7 Conclusion

This paper discussed the interpretability of adversarial training based on adversarial perturbation that was applied to tasks in the NLP field. Our proposal restricted the directions of perturbations toward the locations of existing words in the word embedding space. We demonstrated that our methods can successfully generate reasonable adversarial texts and interpretable visualizations of perturbations in the input embedding space, which we believe will greatly help researchers analyze a model's behavior. In addition, we confirmed that our methods, iAdvT-Text and iVAT-Text, maintained or improved the state-of-the-art performance obtained by our baseline methods, AdvT-Text and VAT-Text, in well-studied sentiment classification (SEC), category classification (CAC), and grammatical error detection (GED) benchmark datasets.

Acknowledgments

We thank four anonymous reviewers for their helpful comments. We also thank Takeru Miyato who suggested that we reproduce the result of a previous work [Miyato et al., 2017] as well as Masahiro Kaneko and Tomoya Mizumoto who provided helpful comments.