Dynamically Route Hierarchical Structure Representation to Attentive Capsule for Text Classification

1 Introduction

Text classification is one of the fundamental tasks in natural language processing (NLP). In recent years, many successful deep learning models have been widely applied to this task. Mainstream deep learning methods usually contain two important components: representation learning and feature aggregation. Given a sentence or document in the classification task, the longstanding challenges for deep learning models are (1) how to enumerate task-relevant structure representations of the sequence automatically via representation learning and (2) how to leverage latent label-oriented abstractions of each extracted structure feature via feature aggregation.

According to different representation learning approaches, existing models for text classification can be roughly categorized into four types. Bag-of-words representation models represent the text sequence by taking the average of different words [Grave et al., 2017], but unable to consider word order.

Sequence representation models consider word order using convolutional neural network [Kim, 2014] or recurrent neural network [Chung et al., 2014], but do not involve structure information. Structure representation models, such as treestructured LSTM [Tai et al., 2015], utilize pre-specified parsing trees to take structure information into account. Attentionbased models [Yang et al., 2016] use attention mechanism to build representations by weighting input words differently.

Currently, the structure features in most structure representation models are either provided as input from pre-defined parsers or learned end-to-end from feature detectors, such as CNN and RNN. However, these representation methods are not automatical and unable to learn adaptive text structure sufficiently. Focusing on CNN encoder, the traditional pipeline is using convolutional layers with geometrically fixed filters to extract spatial features (n-gram structure) in the last level of output, whatever for shallow or deep encoders. Then a following pooling or attention layer is utilized to aggregate prominent features for downstream tasks. The main limitations include that little structure information is explicitly learned from the encoder, and the aggregation strategies perform selection directly over structure representations, which can't fully utilize their latent semantic abstractions.

More recently, a promising work named capsule network [Sabour et al., 2017] provides novel viewpoint of feature aggregation. The dynamic routing process learns partwhole relation between adjacent capsule layers. However, existing capsule networks determine the category either according to the vector length of capsules or just flatten all capsules to a fixed-length vector for the classifier, which lost potential and diverse information embedded in different capsules.

In this paper, to address the aforementioned challenges and issues, we propose a novel model (dubbed as HAC) that dynamically route Hierarchical structure feature to Attentive Capsule for text classification. We employ a deep dilated CNN to extract hierarchical structure features. Different from conventional CNNs, we denote the intermediate feature maps at each level of deep dilated CNN as hierarchical structure representations. Dilated convolutions exponentially grow the receptive field from upstream layers to downstream layers, which can effectively form various linguistic units of input text, such as word, phrase, clause, sentence and other specific levels. To aggregate and fully utilize information within hierarchical structure features, we design a capsule module using dynamic routing and equip it with an attention mechanism. During the dynamic routing process, information is embedded into capsules and iteratively distilled into task-relevant categories, which can be viewed as information distillation and feature clustering. Considering that target capsules contain different viewpoints and contribute unequally to the text category, we build an attention mechanism upon the aggregation of target capsules to adaptively select significant parts for classification. Extensive experiments on several benchmarks demonstrate the effectiveness of the proposed architecture, surpassing the state-of-the-art methods remarkably.

Specifically, our contributions are of three-folds:

• We propose a novel HAC architecture for improving the structure representation learning and feature aggregation within the text classification task, by dynamically routing hierarchical structure feature to attentive capsule.
• We design an effective CNN-based module to automatically extract hierarchical representations of text structure, and a variant capsule-based module equipped with an attention mechanism to make full use of information.
• We conduct extensive experiments on both small and large datasets, covering a wide range of text classification tasks. Experiments demonstrate that our proposed approach outperforms a number of competitive baselines and achieve a few state-of-the-art results.

2 RelatedWork

Related work can be divided into two threads and we briefly review each of these areas in this section.

2.1 Structure Representation

Structure representation is an attractive representation method for sentence modeling. In the classification task, existing works can be concluded into three types. Hierarchical models include [Zhou et al., 2015], combining CNN and RNN to learn structure representation. Tree-based models include [Tai et al., 2015], involving pre-defined parsers to construct sentence tree structure. Reinforcement learning models include [Tianyang et al., 2018], using policy network and classification network to discover optimized structures.

However, previous models only select the final level output of encoder (CNN/RNN) as sentence representations and pass it to the following classifier. To utilize multi-scale structure features within the sentence, [Zhao et al., 2015; Wang et al., 2018b] use intermediate information obtained during the encoding process. [Zhao et al., 2015] concatenates different representations of the sentences at different levels of abstractions, by performing a pooling operation over each level of recurrent and recursive networks. [Wang et al., 2018b] introduces a densely connected CNN and an attention mechanism to choose each level of CNN features.

2.2 Feature Aggregation

Feature aggregation aims at aggregating the extracted features into a fixed-length vector as sentence representation. Bag-of-word (BoW) [Grave et al., 2017] strategy represents the sentence by taking the average of embedding vectors. Max-pooling is employed to extract the most salient feature within each feature map in CNNs [Kim, 2014; Zheng et al., 2017] or within each hidden state in RNNs. Dynamic k-max pooling [Kalchbrenner et al., 2014] strategy selects the k most active features to prevent missing valuable information. Attention mechanism becomes popular by calculating attention distribution and performing a weighted average on the output features of CNN/RNN layer [Yang et al., 2016], for classification [Chen et al., 2018] and transfer learning [Shaoan et al., 2018].

Recently, capsule network with dynamic routing is proposed to improve the representational limitations of CNNs and RNNs. Capsule network is first applied in image [Sabour et al., 2017], and recently in text [Zhao et al., 2018; Gong et al., 2018]. Unlike pooling operation, the capsule network collects the whole information after transformation, instead of discarding unrelated information.

3 Method

The overall architecture of our model is depicted in Figure 1. Following sections elaborate the components and workflow.

3.1 Word Embedding Layer

Consider a text sentence S = w1 , ..., wN with length N. The goal of this layer is to represent each word in S with a d-dimensional vector. The word embedding is a fixed vector for each individual word, which is projected from the pre-trained embedding lookup-table. The output of this layer is the sequence of word vectors X = [x1, ..., xN ] ∈ RN×d .

3.2 Context Representation Layer

The goal of this layer is to incorporate contextual information into the representation of each input time step. Meanwhile, the word vectors will be compressed into a lower dimension for the following layers. We utilize a bi-directional Gated Recurrent Unit (Bi-GRU) [Cho et al., 2014] to produce contextual representation ht of a word by concatenating forward

→

←

hidden state output ht and backward hidden state output ht .

Thus, the output of Bi-GRU encoder are a sequence of vectors H = [h1, ..., hN ] ∈ RN×d′ .

3.3 Deep Dilated Convolution Layer

This is one of the core layers within our model. The purpose of this deep dilated convolution layer is to extract hierarchical multi-granularity features as textual structure representation.

Unlike conventional CNN which apply convolution operation upon pre-trained word embeddings, our dilated CNN is built upon Bi-GRU output vectors which contain contextual information. For the first convolution block, we denote U0 = H = [h1, ..., hN] ∈ RN×d′ as the input representation matrix, where d′ is the input vector dimension. Similarly,

ൌ ௅ ܃

ൌ ܵ

௅ ସܝ ௅ ଷܝ ௅ ଶܝ ௅ ଵܝ

௜ݓ ସݓ ଷݓ ଶݓ ଵݓ

௅

ଶǦேܝ

ଷǦேܝ

ଷǦேݓ

௅

௅

௅ ேܝ

ேݓ

ଵǦேܝ

ൌ ௅ ܄

௅ ெܞ ௅ ௝ܞ ௅ ଵܞ

ଶǦேݓ

ଵǦேݓ

the outputs of each intermediate block and the final block can be represented as Ul = [ul1 , ..., ulN ] ∈ RN×k (l ∈ [1,L]), where L is the number of total convolution block and k is the number of filters in each block.

Consider the l-th convolution block, let Wl ∈ Rk×w×k (W1 ∈ Rk×1×d′ is the exception) be the filter matrix for convolving w input vectors with k filter kernels. The transformation between two adjacent blocks can be formulated as

where f is an affine function that sliding filters over the w-length input window.

Specifically, ult ∈ Ul is produced by calculating

where [] is vector concatenation operator, ⊕ is convolution operator and r is dilated rate. Rectified linear units (ReLU) function is adopted for activation.

We use a dilation scheme whereby the dilation rates are doubled every block up to a maximum rate 2L-2 and thus the receptive fields are increased every block up to maximum width (w-1)2L-1 , which can be fine-tuned according to specific dataset. Notice that in the standard convolution, r = 1.

Dilation [Yu and Koltun, 2016] makes the receptive field grow exponentially in terms of the depth of the networks, as opposed to linearly. The receptive field can be computed by (w-1)2l-1 . Using larger dilation rate or larger filter size enable the output at each intermediate level to represent a wider range of input, which can be regarded as a larger n-gram feature. Intuitively, variant size of n-gram features corresponding to multi-granularity structure representation of the input text sequence, where small n-gram match word/phrase-level and large n-gram match clause/sentence-level in general.

Zero padding is used on either side of the convolution layer's input to ensure that the output features have the same size as the input features. Finally, we extract hierarchical feature maps U1 , U2 , ..., UL.

3.4 Dynamic Routing Capsule Layer

This is another core layer within our model. The goal of this layer is to summarize the raw convolutional features generated at the previous layer into several higher abstract aspects. We propose a dynamic routing policy, a variant of capsule network first proposed by [Sabour et al., 2017], to implement this process for feature clustering. In this layer, upstream row features and downstream abstract features are both represented in capsules. Note that for each convolution block, an unshared capsule layer is adopted to categorize their output features, with regard to a specific linguistic unit.

As shown in Figure 2, we take one convolution block's output features for example and formally introduce the dynamic routing process in detail. Recall that Ul = [ul1 , ..., uli , ..., ulN ] ∈ RN×k denotes the output of l-th convolution block. Each ul represents a n-gram feature generated by k filter maps and we call it as source capsule. And we call Vl = [vl1 , ..., vlj , ..., vlM ] ∈ RM×dv as target capsules, where M denotes the target capsule number and dv denotes the target capsule dimension. We aim at routing Ul to Vl , as an information distillation and feature clustering process. For simplicity of notation, we omit the superscript l in the following formulas.

Consider ui in source capsule layer and vj in target capsule layer, a routing vector mi→j is computed to indicate the information to be transferred from ui to vj

Prediction vector (vote) ˆuj|i indicates the viewpoint of raw feature to be transferred and is computed from ui by multiplying a transformation matrix Wj

Coupling coefficients cij indicates the proportion of prediction vector to be transferred and is determined by a

ଵܛ

ெܛ ௝ܛ

௜ܝ ସܝ ଷܝ ଶܝ ଵܝ

ଷǦேܝ

ெܞ ௝ܞ ଵܞ

"routing softmax" function

where bij represents the log prior probability that capsule i should be coupled to capsule j and is calculated by the iterative dynamic routing process.

Then, the target capsule vj is produced with two steps. A collection of incoming routing vectors is calculated firstly

and then be activated by applying the squash function

The squash function enlarges small vectors and shrinks large vectors to unit vectors, which makes the dynamic routing process more stable and the information interaction more efficient.

Between two capsule layers, we use an iterative dynamic routing policy to compute the capsule output by calculating an intermediate value cij. Within the "routing softmax" function, bij is initialized with zero and updated with an agreement scale aij. The agreement aij is calculated by a scale product between capsules in two layers

By default, the target capsule number M is the number of text categories plus an orphan category. The orphan category inspired by [Sabour et al., 2017] can capture the "background" information such as stop words and the words that are unrelated to the task. Introducing the orphan category

ଶǦேܝ

ଵǦேܝ

ேܝ

Algorithm 1 Dynamic Routing Algorithm

Input: Source capsules u1, ..., uN ; Maximum iterations R Output: Target capsules v1, ..., vM

• 1: Initialize bij = 0 for all i and j.
• 2: for r = 1 to R do
• 3: Compute cij for all i and j by Eq. (8)
• 4: Update all target capsules vj by Eq. (10)
• 5: Update bij for all i and j by Eq. (12)
• 6: end for
• 7: return v1, ..., vj, ..., vM

reduces the interference for normal categories and makes the dynamic routing more effective.

Our dynamic routing algorithm is summarized in Algorithm 1. Each separated target capsule layer yields Vl = [vl1 , ..., vlj , ..., vlM ] for the l-th convolutional feature block. We concatenate target capsules outputs into a united

and pass it to the attentive aggregation layer.

3.5 Attentive Aggregation Layer

The goal of this layer is producing a fixed-length representation vector o by taking all target capsules as input. For each target capsule vi ∈ Rdv in V, we compute an attention score αi for it, which indicates its importance and contribution to the classification task,

where q is a task-specific trainable pattern vector and k denotes the number of capsules in the capsule pool V.

Finally, we obtain a fix-length aggregation vector o by calculating the weighted sum over all target capsules for text representation and pass it to the downstream classifier.

3.6 Prediction Layer

The purpose of this layer is to estimate the probability distribution p(y|S), where y is the classification targets. We feed the fixed-length aggregation vector o to a multi-layer perceptron (MLP) classifier using softmax activation.

4 Experiments

4.1 Datasets

Literature usually tests their model on either small datasets or large datasets, which could not provide extensive results. Here, we test our model on both. For small datasets, seven widely-studied datasets [Kim, 2014] include: movie reviews (MR), Stanford Sentiment Treebank (SST-1 and SST-2), subjectivity classification (SUBJ), question dataset (TREC), customer review (CR), opinion polarity (MPQA). For large datasets, four widely-studied datasets [Zhang et al., 2015] include: AG's news corpus (AG), DBPedia ontology (DBP), Yelp reviews (Yelp.P and Yelp.F).

The detailed statistics are listed in Table 1. These datasets cover a wide range of text classification tasks, with varying numbers of documents and varying document length. Following the evaluation scheme in existing literatures, for MR, CR, Subj, MPQA, we use nested 10-fold cross-validation, for TREC, AG, DBP, Yelp.P, Yelp.F, 10-fold cross-validation, and for SST-1, SST-2, standard validation.

4.2 Implementation Details

Input Embedding

We use two versions of embedding for fair comparisons. The first one is 300-dimensional GloVe [Pennington et al., 2014] word vectors trained on Common Crawl corpus. Another is 1024-dimensional word vectors extracted from ELMo [Peters et al., 2018] pre-trained on Word Benchmark corpus. For both small and large datasets, GloVe embedding is adopted. Besides, ELMo embedding is adopted on small datasets for fairly comparing with results in existing transfer learning literatures. Word embeddings are fixed during training.

Model Configuration

We are interested in our model's robustness across a diverse set of tasks. To this end, if not mentioned otherwise, we use the same network parameters in all classification tasks without specific fine-tuning. In detail, we set the hidden state dimension of Bi-GRU to be 100 for each direction. We adopt 5 dilated convolutional blocks, with filter window size 2 and filter number 100. We set the target capsule dimension to be 50, and use 3 iterations of routing for all datasets. The MLP classifier has a hidden layer of size 50 using ReLU activation.

Training Protocol

The training objective is to minimize the cross-entropy loss. Dropout regularization is employed on the input embedding layer, with the dropout rate 0.5. We don't impose L2 regularization at each layer. We train our model's parameters using gradient-based optimizer Adam, with an initial learning rate 1e-3. We halve the learning rate if the dev accuracy doesn't increase in 3 training epochs, and set the minimum rate to be 1e-4. We conduct mini-batch with size 8 for small datasets, and size 128 for large datasets. The training process lasts at most 30 epochs on all the datasets.

4.3 Competitors

To comprehensively evaluate the performance of our proposed approach, we give a variety of baseline methods and state-of-the-art models for comparison, listed in different genres (RNNs variants, CNNs variants, and others).

For small datasets, the competitors include: BiLSTM [Cho et al., 2014], tree-structured LSTM (TreeLSTM) [Tai et al., 2015], multichannel CNN (MCCNN) [Kim, 2014], multichannel variable-size CNN (MVCNN) [Yin and Sch¨utze, 2015], capsule network (CapsuleB) [Zhao et al., 2018], structured self-attentive model (Self-ATT) [Lin et al., 2017]. Additionally, transfer learning methods include bag-of-word with ELMo (BoW+ELMo) [Perone et al., 2018], sentence representations from inference (InferSent) [Conneau et al., 2017a], transformer-based universal sentence encoder (USET+CNN) [Cer et al., 2018]. For large datasets, the competitors include: LSTM and its variant discriminative LSTM (Dis-LSTM) [Yogatama et al., 2017], very deep CNN (VD-CNN) [Conneau et al., 2017b], densely connected CNN (DC-CNN) [Wang et al., 2018b], label-embedding attentive model (LEAM) [Wang et al., 2018a], word-context region embedding (WC-RE) [Qiao et al., 2018].

4.4 Main Results

Classification results show the strong generalization capability of our model, across different datasets and different tasks.

Small Datasets Results

The results of small datasets are listed in Table 2. From the results, we observe that our model achieves best classification accuracy on 5 out of 7 benchmarks under the non-transfer scheme, and 6 of 7 benchmarks under the transfer scheme. Particularly, our model outperforms previous deep neural network models with a large margin and beats advanced models involving capsule or self-attention. Notice that our model under the non-transfer scheme has surpassed some state-ofthe-art transfer learning approaches on several datasets.

Large Datasets Results

The results of large datasets are listed in Table 3. From the results, we observe that our model achieves best classification accuracy on all four benchmarks. Particularly, our model significantly outperforms RNNs and other advanced approaches on all datasets. While comparing with those deep and sophisticated CNNs, our model achieves analogical improvement, especially for datasets with larger average word length.

4.5 Ablation Study

More investigations are conducted to study the independent effect of each module in our proposed architecture. We replace intermediate features with the final-layer feature, capsule routing with max-pooling, attentive aggregation with concatenation, respectively. Table 4 shows the accuracy on the SST-2 (small dataset) test set and the AG (large dataset) test set. We can observe that eliminating any of the proposed modules would hurt the performance significantly. Ablation results reveal the effectiveness of each proposed component and the elaborate full architecture among different tasks.

4.6 Visualization

To intuitively understand our model, we visualize the extracted most salient structure feature and attention weight for each target capsule. Figure 3 shows the visualization result for a sentence sampled from SST-2 dataset. Red denotes the target capsule attention weight according to Equation (14), and blue denotes the source capsule coupling coefficient towards current target capsule according to Equation (8). We sort different levels of positive/negative categories from top to down. Due to limited space, we omit orphan categories, which always have small attention weights and unrelated structure. From this figure, we can see that the detected most salient text structure feature mostly represents a linguistic unit and carries corresponding sentiment polarity to their target capsule. In addition, the aggregation mechanism is able to select the informative target capsules containing strong sentiment corresponding to the true label for classification.

5 Conclusion

In this paper, we propose a novel architecture that dynamically route hierarchical structure feature to attentive capsule for text classification. The main idea of the proposed model is to adaptively form multi-granularity structure representations of text and fully leverage the categorized abstract of features with attention. Experiments on various datasets demonstrate that the proposed approach outperforms competitors and achieve several state-of-the-art results. Ablation and visualization analyses also reveal the effectiveness of our model for structure representation learning and feature aggregation.

Acknowledgments

This paper was supported by the National Key Research and Development Program (2016YFB1000101), the National Natural Science Foundation of China (61722214, 61573386, 11801595) and the Natural Science Foundation of Guangdong (2018A030310076, 2016A030313292). Corresponding authors are Hai Wan and Chuan Chen.