Densely Connected CNN with Multi-scale Feature Attention for Text Classification

1 Introduction

Text classification is a fundamental problem in natural language processing (NLP). In recent years, deep learning models have been widely applied to this task, including recurrent neural networks (RNN), recursive autoencoders, convolutional neural networks (CNN). In particular, thanks to the great success of CNN in computer vision, CNN has also been successfully applied to NLP applications such as text classification [Kim, 2014; Blunsom et al., 2014; Lei et al., 2015; Zhang et al., 2015; Conneau et al., 2017].

However, most existing CNN models employ convolution filters of fixed size: there is a fixed-size window sliding from the beginning to the end of a text to produce feature maps, which is equivalent to extracting fixed-size ngram features [Zhang et al., 2015; Conneau et al., 2017; Wang et al., 2017]. This has apparent drawbacks in that text classification needs to extract variable-size features such as phrases to form a better representation [Zhang et al., 2018]. For example, sentiment classification for sentence - "He's nice to talk to without being patronizing" (as shown in Figure 1) requires to extract a unigram feature "nice" and a trigram feature "without being patronizing", which are both positive words or phrases. However, when applying filters of size 1 (see the first row of Figure 1), we obtain one positive feature "nice" (green dot) and one negative feature "patronizing" (red dot), which is not friendly for classification. When using filters of a larger size (e.g., 3, see the third row of Figure 1), we can obtain an ideal neural response for phrase "without being patronizing" (a positive phrase). However, the response for the unigram "nice" is undesirably decreased because the model includes additional unnecessary information in either "He's nice to" or "nice to talk" (see the third row of Figure 1). We term this issue as the inability to adaptively select multi-scale features in a CNN model for text classification. Multi-scale features refer to n-gram features with variable n, such as unigram (n = 1), bigram (n = 2), trigram (n = 3) and so on.

∗Corresponding author (Zhidong Deng).

A possible solution to this issue is to utilize filters with different window sizes to extract multi-scale features. However, such a solution requires much experimental efforts to find the optimal combinations of different filter sizes. Moreover, this solution is essentially to widen the network with different filter sizes, equivalent to learning several separate, disconnected networks in parallel. Thus, the interactions of feature maps from different filter sizes are not fully exploited. We argue that a deeper model is more elegant than a wider solution in that: a deeper model is more compact because it can construct hierarchical representations by multi-level abstraction and there is no need to re-learn redundant feature maps. As shown in Figure 2(a), the model can construct trigram features equivalently in the third layer (e.g., f ′ (x1, x2, x3)) by stacking another convolutional layer (window size w = 2) to the second layer which extracts bigram features (e.g., f(x1, x2) and f(x2, x3)). Thus, feature maps at upstream layers can be effectively re-used at downstream layers. Furthermore, features can be reused more flexibly by adding dense connections between layers (denoted by the red dotted lines in Figure 2). For instance, a trigram feature may be composed not only from f(x1, x2) and f(x2, x3), but also possibly from x1 and f(x2, x3) (i.e., x1(x2x3)). Consequently, such a deep model offers more flexibility to construct multi-scale features than a wide model.

In this paper, we propose a densely connected CNN model with multi-scale feature attention for text classification. To equip the model with the ability to generate multi-scale features, we adopt dense connections between different convolutional layers, inspired by [Huang et al., 2017a]. Through the dense connections, the model can compose features of larger scale from those of smaller scale more flexibly. To select taskfriendly features from all the possible multi-scale features, we design an attention module before the classification layer. The two characteristics enable the proposed model to obtain competitive results on several text classification datasets.

In summary, the contributions of this paper include:

• We propose a new CNN model that is equipped with dense connections between convolutional layers and with a multi-scale feature attention mechanism. Through these two design considerations, the model is able to adaptively select multi-scale features for text classification.
• We extensively evaluate the model on six text classification tasks. The model obtains competitive performance compared with state-of-the-art baselines on several benchmark datasets.

2 RelatedWork

Deep Neural Networks

Recently, deep neural networks have achieved great success in many NLP applications. There are a variety of representation learning models for text representation. Recurrent neural networks (RNN), including long short-termmemory (LSTM) and gated recurrent units (GRU) have been widely used in text processing because of their strong performance in processing the structure of text. Several variants are also proposed, such as [Tai et al., 2015; Huang et al., 2017b].

Another type of popular models are undoubtedly CNNs. [Collobert et al., 2011] utilized convolution filters on sliding windows for a text sequence and applied a max operation to capture the most useful local features. [Kim, 2014] adopted multiple filters with different window sizes to extract multiscale convolutional features for text classification. A dynamic k-max pooling mechanism was proposed in [Blunsom et al., 2014]. [Lei et al., 2015] proposed a novel feature mapping operator to generate non-consecutive n-gram features. [Wang et al., 2017] used a large taxonomy knowledge base to improve performance. Different from taking word vectors as input, [Johnson and Zhang, 2015] applied CNN to learn the embeddings of text regions directly. Based on the text region embeddings, [Johnson and Zhang, 2017] aims to deepen CNNs without increasing much computational cost. All the above approaches are based on word embeddings, while [Zhang et al., 2015] proposed character-level CNN models and reported competitive results. [Xiao and Cho, 2016] utilized both convolutional and recurrent layers to encode character inputs. [Conneau et al., 2017] adopted very deep convolutional neural networks , i.e. ResNet [He et al., 2016] to text classification.

Attention Mechanism

Attention becomes an effective mechanism for selecting significant information to obtain superior results. Deep neural networks, including CNNs and RNNs, can obtain better results if equipped with attention mechanisms. Among many proposed attention mechanisms, there are some noticeable examples including soft and hard attention [Xu et al., 2015], global and local attention [Luong et al., 2015], and source-target attention and self attention [Lin et al., 2017]. Specifically, [Yang et al., 2016] developed a two-level attention mechanism on GRU (i.e. word attention and sentence attention). [Yin et al., 2016] proposed a CNN based attention module for jointly performing attention between two CNN hierarchies. [Lin et al., 2017] developed an attention mechanism for extracting an interpretable sentence embedding.

3 Method

3.1 Overview

The framework of our model is shown in Figure 2. The left panel shows the model in an intuitive view while the right one gives more technical details. The model begins with a text input x1x2 · · · xm, and generates multi-scale features by convolution blocks and through dense connections (denoted by the red dotted lines). The upstream convolutional blocks construct features for small n-grams, and downstream ones for large n-gram features 1 . The dense connections enable the model to compose features of larger scale from those of smaller scale flexibly because downstream blocks have access to all upstream feature maps. To select task-friendly features, an attention mechanism is presented to re-weigh these multiscale feature maps through the two processes: filter ensemble

1 Upstream blocks are close to input while downstream close to output.

(Fensem(·)) and scale reweight (Freweight(·)). A final representation is then built for text classification.

3.2 Densely Connected CNN

Preliminary. The proposed model is illustrated in Figure 2. Let xi ∈ Rd be the d-dimensional pretrained word vector of the i-th word in a text and the input text can be represented as a matrix:

where m is the number of words in a text. In addition to the input X, the outputs of each intermediate layer can be similarly represented as Xl = [x1l , x2l , ..., xml ]m×k, where l (1 ≤ l ≤ L) is the layer index. L denotes the total number of convolutional blocks and k is the dimension of the transformed feature representation.

A convolutional block generates intermediate feature representations, which can be formulated as below:

where f(·) is a composite function consisting of three cascaded operations: a convolution, a batch normalization (BN) [Ioffe and Szegedy, 2015], and a rectified linear unit (ReLU) [Nair and Hinton, 2010]. Xl is produced by receiving one preceding features Xl-1 with learnable weights Wl ∈ Rk×w×k . The weight matrix Wl consists of k filters, each of which is of size w × k, convolving w adjacent vectors (e.g., xil-1 and xi+1 l-1 when w = 2). This is denoted by "Convolutional Block, w, k" in Figure 2. Notice that we use zero padding to the two sides of the input to ensure that the resulting feature map has the same size as the input. For instance, we pad 0 ∈ Rk after xml when w = 2; and pad 0 before x1l and after xml when w = 3.

Traditional CNN models stack convolutional blocks sequentially to form hierarchical representations. For text processing, convolutions can be viewed as to extract n-gram features over a word sequence, and the hierarchical structure enables CNNs to produce features of large scale incrementally. However, only conventional connections may not deal with language composition properly, as illustrated in the introduction. Inspired by [Huang et al., 2017a], we adopt dense connections between convolutional layers to equip the model with the ability to compose representations from multi-scale features (i.e., variable n-grams), which is critical for language processing.

Dense connections. Motivated by the above observation, we adopt a densely connected CNN for text classification, following DenseNet for computer vision [Huang et al., 2017a]. The model takes as input the outputs of all upstream layers X1,X2, · · · ,Xl-1, and generates feature maps for the current layer l, as formulated below:

where [X1,X2, ...,Xl-1] refers to the concatenation of the feature-maps produced by layers 1, ..., l - 1 and each Xi ∈ Rm×k . Weight Wl ∈ R(l-1)×k×w×k . As shown in the bottom of the legend of Figure 2, the model produces X1 and X2, then generates features by applying k filters of size w×k onX1 andX2 respectively, resulting in twom×k matrices. X3 is obtained by element-wise addition of the two resulting matrices. Notice that we first apply one layer of k filters of size 1 × d to convert the initial feature dimension from d to k. The size of all the feature maps is preserved to the same (m× k) because they will be fed into the attention module which requires the same dimension for multi-scale features.

There is a remarkable advantage in such densely connected CNN: the model computes downstream feature maps (corresponding to larger-scale n-grams) by considering upstream features (smaller-scale n-grams). In this manner, words or shorter phrases will be fully used in obtaining feature maps of longer phrases, resulting in the flexibility of extracting multiscale features.

3.3 Multi-scale Feature Attention

Through dense connections, the downstream layers of the CNN model has access to the features generated at the upstream layers. However, effective use of these features (some are redundant) is still a key issue. In this subsection, we present a multi-scale feature attention mechanism to effectively use these features for classification. The mechanism adaptively selects features of different scales at each position of a text. As shown in Figure 2(a), at position i, the attention mechanism computes a weight distribution over xi, f(xi, xi+1) (features from the second layer), and f(xi, xi+1, · · · , xi+L-1) from the L-th layer. These feature values indicate the responses of multi-scale n-grams, i.e., unigram xi, bigram xixi+1,..., and L-gram xixi+1 · · · xi+L-1.

The attention mechanism contains two operations: filter ensemble and scale reweight. Filter ensemble aims to develop scalar descriptors sil to represent features of each scale xil at position i. Scale reweight uses the descriptors sil as input and outputs a softmax distribution of attention weights to reweigh these features of different scales, i.e., xi1 , xi2 , · · · , xiL.

Filter ensemble. As shown in Figure 2, we use k filters in each convolutional block, generating Xl = [x1l , x2l , ..., xml ]m×k. Each xil ∈ Rk , indexed by i, denotes the k-dimensional features in the i-th position of a text at layer l. We then use a scalar to express each feature vector xil as

where Fensem(·) indicates a function that sums all k elements of the input vector. The scalar can be used as a descriptor of the feature vector, because xil is yielded by applying k filters on the preceding feature maps and each value in xil is a neural response value. Thus, the sum of all the values in xil can be viewed as feature salience.

Scale reweight. After obtaining sil through filter ensemble, we will use them as input to generate attention weights in order to adaptively re-weight the features from different scales. We define the final representation xiatten and attention weights αi l as follows:

where xil , xiatten ∈ Rk , and αi l are attention weights. Note that feature maps at different layers have correspondence to the scale of features. For instance, when w = 2, X1 corresponds to unigram features, and X3 to trigram features.

The attention weights are produced as follows:

where sil is defined by Eq. (4), and MLP is a multi-layer perceptron.

After being processed by the attention module, the final representation Xatten = [x1atten , x2atten , ..., xmatten ] ∈ Rm×k is generated, which will be fed into the classification layer.

3.4 Objective Function

The training objective is to minimize the cross-entropy loss:

where h(W,Xatten) is the predicted output distribution based on the final representation Xatten and y is the referenced distribution. L(p, q) is the cross entropy function between two distributions p and q.

Note that supervision signals are propagated to upstream blocks more straightforwardly through the shortcutd or dense connections. Such connections enforce the upstream layers to learn task-friendly features, also known as "deep supervision" [Huang et al., 2017a].

4 Experiments

4.1 Datasets

We evaluated our model on six public datasets: MR from [Pang and Lee, 2005] and the others (AG, DBPedia, Yelp P./F., and Amazon F.) from [Zhang et al., 2015]. The statistics of the datasets are summarized in Table 2.

4.2 Implementation details

Input. Word vectors were adopted from [Pennington et al., 2014]. The word embedding was of size 300. The input text was padded to a fixed length - maxlen, where maxlen was chosen 50 for MR, 100 for AG and DBPedia, and 300 for Yelp P., Yelp F. and Amazon F., respectively.

Architecture configuration. The model was implemented with Caffe [Jia et al., 2014]. Due to the different lengths of input texts and the number of training samples, we adopted 5 convolutional blocks for MR and AG, and 6 convolutional blocks for the rest datasets. We chose window size w = 3

and feature dimension k = 128. The classification layer is a two-layer fully connected MLP with ReLU activation function and softmax output. We made slight modifications to the framework shown in Figure 2: pooling layers are added with stride 2 after each convolutional block, which can merge some redundant features and improve computational efficiency. Additional pooling operations with an appropriate stride are required in order to keep the multi-scale features from different layers having the same size.

Training settings. We used stochastic gradient descent (SGD) with a mini-batch of 256. The learning rate is initially set to 0.01 and then gradually decreased to e - 5. The training process lasts at most 30 epoches on all the datasets. We applied "L1" regularization and the momentum was set to 0.9.

4.3 Baselines andMain Results

We compared our model with several genres of popular models: linear models [Grave et al., 2017]; RNNs including LSTM and its variants Discriminative-LSTM [Yogatama et al., 2017]; CNNs including classical CNN [Kim, 2014], CNN with dynamic pooling [Blunsom et al., 2014], character-level CNN [Zhang et al., 2015], and very deep CNN [Conneau et al., 2017]; and attention-based models such as [Lin et al., 2017].

Main results. The results are listed in Table 1. The proposed model outperforms the linear models and RNNs on all these datasets. Compared with those shallow but wider CNNs [Kim, 2014], our model has advantages on large datasets (Yelp P./F. and Amazon F.) since the proposed architecture can learn hierarchical and multi-scale features flexibly. Compared to other deep CNNs, the densely connected structure which uses fewer parameters (5-6 convolutional blocks) outperforms the model [Conneau et al., 2017] with 29 convolutional layers. Our model is better than the attention-based model [Lin et al., 2017]. At last, the last two rows show the performance difference with/without multi-scale feature attention, thereby demonstrating the effectiveness of the proposed attention mechanism.

4.4 Attention Visualization and Phrase Analysis

Attention Visualization: Through attention visualization analysis, we made two observations:

1) The same word in different sentences has different optimal feature scales depending on the context. As shown in Figure 4(a), each row indicates a weight distribution over different feature scales xil (1 ≤ l ≤ L) and i is the position for word "interesting". When w = 3, Xl corresponds to feature maps of (2l - 1)-gram. We presented two negative and two positive sentences. For the negative sentences, the optimal feature scale for the positive word "interesting" is larger and includes negators such as "not" or "nothing", which is more adequate for sentiment classification. Whereas in the other two positive sentences, the optimal feature scale is smaller (X2 ∼ trigram) to form positive phrases. Thus, the model can adaptively select task-friendly feature scales for classification.

2) Different words in the same sentence have different op-

(b) Different words in the same sentence have different optimal scales to form meaningful representation. Self-contained words such as "unfortunately" form smaller-scale features, while long phrases form larger-scale features.

Figure 4: Multi-scale attention visualization. The values in a row are a distribution over the feature scales at a word position (see Eq. 5/6).

timal scales to form meaningful representations. As shown in Figure 4(a), the optimal feature scale for each word in a sentence tends to be different. For example, in this review "not so much farcical as sour", the model retains a smaller scale for "not" and chooses larger scales for "much farcical as sour". Similar phenomena can be found in the other sentence: a smaller scale for "unfortunately" and "," since such words are self-contained, and a larger scale for "the picture failed to" to formmeaningful representations.

To summarize, the visualization results show that the model is able to adaptively select multi-scale features to form more meaningful representations. Furthermore, these cases show an agreement with our intuition.

Phrase Analysis: Table 3 lists a few phrases discovered by our model on the MR dataset. The phrases are ranked by the product of normalized neural responses and the logarithm of word frequency within each scale. Results show that our model has the ability to select task-relevant phrases, such as "very funny." for positive reviews while "a complete waste of time" for negative reviews. The model may produce bad phrase boundaries, however, this is extremely difficult without structure annotations [Zhang et al., 2018].

4.5 Tuning of Hyperparamters

Window size: We justified how performance is influenced by the window size of convolution filters applied in each block. We varied the window size w among {2,3,5,7}. As shown in Figure 5(a), the accuracy is not significantly influenced by the window size. This is in line with our claim that the model is able to construct high-level features flexibly from low-level features through dense connections and multi-scale feature attention.

Network Depth: We assessed how the network depth influences the performance by varying the number of convolutional blocks. The results are shown in Figure 5(b). For the larger datasets (Yelp.F/Amazon.F), as the network depth increases, the performance is further improved since the model is equipped with more learnable weights. For AG, a relatively small dataset, the performance drops slightly when increasing the depth from 5 to 7 blocks, possibly due to over-fitting with too many parameters. In general, increasing network depth is able to obtain better performance on larger datasets, but not that remarkably because of the dense connections we applied. Note that the parameters of the 4th - 7th blocks grow almost linearly, corresponding to 0.74M, 0.97M, 1.25M and 1.60M, respectively.

5 Conclusion

In this paper, we present a new CNN model equipped with dense connections and multi-scale feature attention for text classification. Through dense connections, the model is able to flexibly generate larger n-gram features from variable smaller n-gram features. By multi-scale feature attention, the model can adaptively select task-friendly yet effective features from many multi-scale features for text classification. The model demonstrates competitive performance on six benchmark datasets. Attention visualization reveals that our model can select the optimal scale to form meaningful representation for text classification.

Acknowledgments

This work was supported partly by the National Key Research and Development Program of China under Grant No. 2017YFB1302200, by research fund of Tsinghua University - Tencent Joint Laboratory for Internet Innovation Technology, and by the National Science Foundation of China (NSFC) under Grant No. 91420106, 90820305, and 60775040.