Reading selectively via Binary Input Gated Recurrent Unit

1 Introduction

Recurrent Neural Networks (RNNs) have become the most popular approach to address machine learning tasks involving sequential data, such as machine translation [Bahdanau et al., 2015], document classification [Le and Mikolov, 2014], sentiment analysis [Socher et al., 2011], language recognition [Graves et al., 2013], and image generation [Villegas et al., 2017].

However, the vanilla RNN suffers from the gradient exploding/vanishing problem during training. To address this problem, long short-term memory (LSTM) [Hochreiter and Schmidhuber, 1997b] was proposed by introducing gate functions to control the information flow within a unit. LSTM has shown impressive results in several applications, but it has much higher computational complexity, which means longer inference time and more power consumption [Cheng et al., 2018a; Cheng et al., 2018b; Wang and Cheng, 2016; He and Cheng, 2018]. Therefore, K. Cho et al. proposed Gated Recurrent Unit (GRU) [Cho et al., 2014]. Compared with LSTM that has three gate functions, GRU has only two gates: a reset gate function to determine how much previous information can be ignored, and an update gate function to select whether the hidden state is to be updated with the current information. Thus GRU can better balance the relationship between performance and computational complexity, which makes it very widely used. For ease of optimization, GRU uses sigmoid as the gate function, whose outputs are soft values between 0 and 1. However, the sigmoid gate function is easy to overfit [Zaremba et al., 2014], and the information flow behind this gate function is not graspable to human. No matter LSTM or GRU, the model reads all the text available to them. But inspired by the human reading, we know that not all inputs are equally important. Therefore, the fact that texts are usually written with redundancy makes us think about the possibility of reading selectively. If the network can distinguish the important words from the decorated or connection words in a document, we can better understand how the network works internally.

∗Contact Author

In this paper, we want to explore a new GRU based RNN structure that can read only the important information and ignore the redundancies. The reset gate function in GRU is replaced by our proposed binary input gate function. With the hard binary gate, the model learns which word should be skimmed. One benefit of our model is that the binary input gate can clearly show the flow of information. As shown in our experiments, only significant information flows into the network. Another advantage is that while a model lies in a flat region of loss surface, it is likely to generalize well [Hochreiter and Schmidhuber, 1997a], because any small perturbation to the model makes little fluctuation to the loss. As training a binary gate means pushing the value to 0 and 1 residing in the flat region of the sigmoid function, small disturbances of the output value of sigmoid functions are less likely to change the binary gate output. The model ought to be more robust.

However, training binary gate function is also very challenging. The binary gate is obviously not differentiable, a common method for optimizing functions involving discrete variables is REINFORCE algorithm [Williams, 1992], but it uses the reward signal, increasing floating point operations [Bengio et al., 2013]. Another way is to use the estimator [Bengio et al., 2013; Courbariaux et al., 2016; Hu et al., 2018] which has been successfully applied in different works. By using the estimator, all the model parameters can be trained to minimize the target loss function with standard backpropagation and without defining any additional supervision or reward signal. We conduct document classification task and language modeling task on 6 different datasets to verify our model and our model achieves better performance. In summary, this paper makes the following contributions:

• We propose a novel interpretable RNN structure based on GRU: We replace the reset gate functions of GRU by the binary input gate functions, and retain the update gate functions.
• Our model can read the input sequences selectively: In our model, we can find more clearly whether the current information is passed into the network or not. In the experimental analysis, we show the gates in our learned model are meaningful and intuitively interpretable.
• The proposed method achieves better results compared to the baseline model: In the experiments, with the same amount of parameters and computational complexity, our model outperforms the baseline algorithms on all tasks.

The reminder of the paper is organized as follows. Section 2 introduces other works related to our research. Section 3 describes the model of binary input gated recurrent unit and how to train it. The experiments and analysis are shown in section 4. Finally, we conclude our model and discuss the future work in the last section.

2 RelatedWork

As all texts have emphasis and redundancy, making neural networks identify the importance of words has drown much attention recently. LSTM-Jump [Yu et al., 2017] augments an LSTM cell with a classification layer that decides how many words to jump after reading a fixed number of words. Moreover, the maximum jump distance and the number of jumps allowed all need to be chosen in advance. The model is trained with the REINFORCE algorithm, where the reward is defined based on whether the model predicts correctly or not. These hyperparameters define a reduced set of subsequences that the model can sample, instead of allowing the network to learn any arbitrary sampling scheme. Skim-RNN [Seo et al., 2018] contains two RNNs, a "small" RNN and a "big" RNN. At every time step the model determines to use the small RNN or the big RNN, which is based on the current input and the previous states. When the word is considered unimportant, the small model will be chosen. Then only a few dimensions in word vector will be updated and the other parts are retained. Instead of the REINFORCE algorithm, this network uses Gumble softmax to handle the non-differentiable problem. Skip RNN [Campos Camunez et al., 2018] is a RNN with a binary state update gate, selecting whether the state of RNN will be updated or copied from the previous time step. This can be considered as completely skipping a word. This network use the straight-through estimator to solve the discrete problem, which guarantees that the amount of calculation does not increase. Structural-Jump-LSTM [Hansen et al., 2018] introduces a new RNN structure with a skip agent and a jump agent. In every time step, the skip agent determines whether to skip the current word, and the jump agent selects a jump to the next word, or the next sub-sentence separator sympol (,;), or the next end of sentence sympol (.!?), or to the end of the text (which is also an instance of end of sentence). G2-LSTM [Li et al., 2018] proposes a new way for LSTM training. It uses the developed Gumbel-Softmax trick to push the output of sigmoid function towards 0 and 1, and the gate functions are still continuous. Although this network reads all the text, the word is considered less important if the value of its input gate function is closer to 0.

Overall, all the above state-of-art models can identify the importance of words and read the text selectively. However, except G2-LSTMwhich reads all the text, other networks that read selectively have shown promising results only on specific temporal tasks: they targeted merely on the document classification task or the question answering task on small datasets. These tasks are easier than language modeling task and require less understanding of every word. And so far, there are no such reading selectively models that can match the performance of vanilla RNN on the word-level language modeling task. In this paper, we present the gated recurrent unit with binary input gate functions which has superior performances to the baseline algorithm on the documents classification task and the word-level language modeling task as well.

3 Model Description

In this section, we present a new structure of GRU, which we call it Binary Input Gated Recurrent Unit (BIGRU), containing binary input gate functions and update gate functions. The binary input gate functions are in charge of controlling the information input and the update gate functions are responsible for the prediction. Therefore, BIGRU has the ability to read selectively on both documents classification task and wordlevel language modeling task.

3.1 Gated Recurrent Unit

Recurrent Neural Network (RNN) takes an input sequence {x1, x2, · · · , xT} and generates a hidden state sequence {h1, h2, · · · , hT}. In recurrent neural networks, the hidden states in layer k are used as inputs to layer k + 1. The hidden states in last layer are used for prediction and making decision. The hidden state sequence is generated by iteratively applying a parametric state transition model S from time t = 1 to T :

where Wh,Wx are weights, b is bias and f is the activation function. In general, we use hyperbolic tangent (tanh) as the activation function.

Despite the remarkable success of RNNs in processing variable-length sequences, traditional RNNs suffer from the exploding and vanishing gradient problem that occurs when learning long-term dependencies, which limits the wide application of RNN. To alleviate this issue, Gated Recurrent Unit (GRU) [Cho et al., 2014] and Long Short-Term Memory (LSTM) [Hochreiter and Schmidhuber, 1997b] were introduced. In general, LSTM has the state-of-art performance, but it needs longer inference time and more energy consumption. While GRU is a carefully designed recurrent structure which makes a good trade-off between performance and speed. Therefore, GRU has been prevalently used in both academia and industry. The recurrent transition of GRU is obtained by:

where {Wz,Wr,W} denote the recurrent weights and ht, ˜ ht are hidden states. The logistic sigmoid function and Hadamard product are denoted as σ and ⊙, respectively. The update gate zt selects whether the hidden state ht is to be updated with a new hidden state ˜ ht. Small value of the update gates causes the cell to remember most of its previous values and use little current information. The reset gate rt decides whether the former hidden state can be ignored. The smaller the value of rt, the more previous information is forgotten.

3.2 Reading Selectively by Binary Input Gated Recurrent Unit

In Figure 1(a) we can see the structure of GRU, the reset gate is responsible for the current information input and the memory of previous information. However, the update gate also decides how much previous information to ignore, which means the two gates have duplicate functions. If we could separate the function of these two gates, the cell may have a better performance. Therefore, we design the binary input gate function which only decides whether the current information should flow into the network or not. It is similar to the input gate functions of LSTM. Thus, ˜ ht can be considered as the current input. As a result, the binary input gate function and update gate function in our model are responsible for current information input and previous information forgetting, respectively.

Our model can be seen in Figure 1(b), the binary input gate only decides whether the current information flows into the network or not and we retain the update gate to obtain hidden states. Thus the division of labor between two gates is more clear, and the task can be better accomplished without increasing the computational complexity. The proposed model, Binary Input Gated Recurrent Unit (BIGRU), works as follows during training:

where Eqn.(5) is the update gate whose function is similar to the one of GRU, Eqn.(6) is the binary input gate whose output value is 0 or 1. If the current information is not important to the task, the value of it will be 0 and the current input information will not be passed into the network. As the input value of the binary function is between 0 and 1, we use the stochastic binary method. The probability of getting 0 or 1 for the binary function can be computed by:

Afterwards, we stochastically sample from the Bernoulli distribution to obtain the binary value. However, the binary input gate functions create a tricky problem. Obviously, the whole model is differentiable except for fbinary and the discrete value cannot directly calculate the derivative. To solve this problem, we generally have two solutions. The first method for optimizing the discrete output value functions is REINFORCE algorithms [Williams, 1992], but it has some disadvantages. The reward is difficult to design and it will increase the computational complexity. In recent years, several estimators have been proposed for the particular case of neurons with binary outputs [Bengio et al., 2013], and they have been successfully applied in different works [Courbariaux et al., 2016]. Since we want to keep the same inference time as GRU, we choose to use the second method. We use the straight-through estimator, which consists of approximating the step function by the identity when computing gradients during the backward pass:

By using the straight-through estimator as the backward pass for fbinary, all the model parameters can be trained to minimize the target loss function with standard backpropagation and without defining any additional supervision or reward signal. So we can ensure that the computational complexity does not increase.

There are several advantages in using the binary input gate functions. First, compared with vanilla GRU, BIGRU can decide to pay how much attention to different words like human reading. The binary input gate functions only select the important information to read and the redundant information will be discarded, leading to faster convergence in some tasks involving long sequences. Second, unlike some other reading selectively models [Campos Camunez et al., 2018; Hansen et al., 2018; Yu et al., 2017], whose update gate functions do not work when the word is skipped, the update gate functions of BIGRU still work even if the output value of the input gate functions is 0. Therefore, the hidden state is updated at every time step, which is significant in the language modeling tasks. Besides, BIGRU can decide how much information can be forgotten at every time step. As a result, our model performs better than baseline GRU in all our test tasks. In the next section, we will present in detail the model performance in the document classification task and language modeling task.

4 Experiments

We evaluate the effectiveness of BIGRU in both document classification task and word-level language modeling task on six different datasets. We compare our model with the baseline GRU.

4.1 Document Classification

In the language classification task, the input is a sequence of words and the output is the vector of categorical probabilities. The evaluation criteria are generally the accuracy (Acc). We choose four different datasets, including Stanford Sentiment Treebank (SST), IMDb, AGNews and DBPedia. The detail of the datasets can be seen in Table 1.

For both GRU and BIGRU, we use a stacked three-layer RNN. Each word is embedded into a 100-dimensional vector. We make a linear transformation on the last hidden state of the network and then apply softmax function to obtain the classification probabilities. All models are trained with Adam, with the initial learning rate of 0.0001. We set gradient clip to 2.0.

We use batch size of 32 for SST and 128 for the remaining. For both models, we set an early stop if the validation accuracy does not increase for 1000 global steps.

4.2 Language Modeling

For further evaluating the impact of our model, we perform word-level language modeling over a preprocessed version of the Penn Treebank (PTB) and WikiText-2 dataset. The Penn Treebank dataset has long been a central dataset for experimenting with language modeling. The dataset is heavily preprocessed and does not contain capital letters, numbers, or punctuation. The vocabulary is also capped at 10,000 unique words. Compared to the preprocessed version of PTB, WikiText-2 is over 2 times larger. The WikiText-2 dataset also features a far larger vocabulary and retains the original case, punctuation and numbers. As it is composed of full articles, the dataset is well suited for models that can take advantage of long term dependencies. The word-level language modeling task is to predict every next word in condition on the previous words. A model is evaluated by the prediction perplexity: smaller the perplexity, better the performance.

We follow the experimental settings in [Merity et al., 2017] for GRU and our model. For each dataset we use a small model (one-layer network) and a big model (a stacked threelayer network with drop-connect on recurrent weights) for testing. For training the model, we choose averaged stochastic gradient descent (ASGD) [Polyak and Juditsky, 1992] for optimization. We use an initial learning rate of 10 for all experiments and carry out gradient clipping with maximum norm 0.25. We use a batch size of 80 for WikiText-2 and 40 for PTB. We train 1000 epochs for the small model and 2000 epochs for the large model. For all the models, all other experimental settings, including random BPTT length, the word embedding size and dropout probability, are the same as [Merity et al., 2017].

4.3 Experimental Results and Analysis

Document Classification

Table 2 shows the accuracy of our model compared with the baseline GRU. The models are evaluated by accuracy (Acc) and reading rate (Rer). The reading rate is computed as follows. First, we get the output vectors of binary input gate functions in the first layer of the model. The output vector are called word input vector. We calculate the proportion of 1 in all word input vectors as the reading rate. The reading rate is considered as the ratio of reading information to all information. Absolutely, standard GRU uses the reset gate and reads the whole input sequence, so its reading rate can be seen as 1. In all four datasets, our model has a better performance than the baseline, about two percentage points higher, although it reads significantly less information. In these four datasets, our model reads no more than 43% information of all input sequence, but it has at least 1.9% promotion in terms of accuracy. For further comparison, we remove the stopwords (a, the, that, this, which, what, why, when, where, while, who, whom) in the datasets, and use the pre-processed dataset to train GRU. Compared with the dataset without being pre-processed, the experimental performance has a certain decline. This is because redundant words are difficult to know from the priori, and pre-defined stopwords are not always redundant. The same word has different meanings in different contexts, e.g. "that" is meaningless in some cases but in some other cases "that" refers to the subject and we cannot ignore it. Our model can learn which words are redundant based on context information, so BIGRU has a better performance. In order to find out whether this is the case, we make further analysis.

For each word, we specify that the word is skimmed if no less than 90% dimensions of its word input vector are 0. As the embedding size is 100, the skimmed word input vector has no more than 10 nonzero dimensions, which is a fairly strict criterion. Then we choose two comments in the IMDb test set, calculating the word input vector of every word. Table 3 shows the result: the skimmed words are shown in bold. Almost all the skimmed words are conjunctions, adverbs and articles, which have no special effect on the understanding of the whole sentence. These words have no contribution to our judgment of the category of this comment. This also illustrates that our model is more interpretable than GRU. We know what is input into the network, and our model has a better performance.

1 Use the pre-processed dataset.

Language Modeling

The results of two models on PTB and WikiText-2 are shown in Table 4 and Table 5, respectively. On PTB dataset, the small and large BIGRU outperform the baseline algorithm by 5.17 and 4.09 points in terms of test perplexity, respectively. On the more difficult dataset WikiText-2, the perplexity upgrade can even reach 5.90/5.38 points in the test set. It is easy to find that the improvement is more significant in language modeling tasks than in text classification tasks, so we want to explore why the task is more difficult, the more obvious the promotion, and we do further analysis.

Language modeling task is more sensitive and difficult than classification task. We can judge the category of the article by a few words, but when we want to predict every next word, we need the context information. This is why the former reading selectively model performed poorly on word-level language modeling task. However, there must be important words and meaningless words in a document, so it is also important to distinguish them in order to avoid distractions. Similar to human intensive reading and skimming, BIGRU can focus on the more important words, while other unimportant words information will not be input completely to the network. In order to prove this, we do the visualization of words. We calculate the average value of the word input vectors in BIGRU. As a comparison, we also calculate the average output value of the reset gate functions in GRU. We plot the heatmap of an example sentence on the PTB test dataset in Figure 2.

We can see that our model pays less attention to the articles and adverbs, besides almost all of the subjects and verbs are completely read into the network. In contrast, the output of the reset gate functions in GRU has no obvious propensity for all words. In other word, the GRU is more like a black box to human. It's hard to explain how it works internally. On the contrary, our model is more interpretable. BIGRU learns the importance of each word like a human, which means it has a stronger learning ability, so its promotion in hard tasks is more obvious. We then plot the perplexity of validation dataset during training in Figure 3. We can see that on both PTB and WikiText-2 datasets, our model converges faster at the beginning of training. Therefore, if we want to get a model with the same effect, our model needs less training time than standard GRU.

5 Conclusion

In this paper, we present BIGRU, a GRU based RNN model that can read selectively. BIGRU is inspired by human reading, and can distinguish the important part from redundant part to avoid the unnecessary interference. In BIGRU we use binary input gate functions instead of the reset gate functions of the standard GRU and retain the update gate functions. Thus, the two gate functions in our model are responsible for current information input and previous information forgetting, respectively. As a result, our model can learn what is the important part of the input sequence based on the task, and ignore dynamically the redundancy. Through extensive experimental analysis, we demonstrate that our model ignores mostly the meaningless adverbs, conjunctions and articles and reads the main information of the text. In other word, BIGRU reads document like human, paying different attention to different words. After removing the interference of unnecessary information, BIGRU has a better performance. In both the document classification task and the word-level language modeling task, our model performs better than standard GRU on six different datasets. As the previous reading selectively RNN are all based on LSTM and they are not effective on word-level language modeling task, we contribute the first reading selectively GRU which has better results without increasing the amount of calculation.

In future work, we are going to investigate similar structure in LSTM, making LSTM read selectively and spend less time during inference. We also want to apply our model to hardware such as FPGA to further increase the inference speed while maintaining the performance.

Acknowledgments

This work was supported in part by National Natural Science Foundation of China (No.61876182, 61872364), the Strategic Priority Research Program of Chinese Academy of Science(No.XDB32050200).