--- title: "Hierarchical Inter-Attention Network for Document Classification with Multi-Task Learning" authors: - "Bing Tian" - "Yong Zhang" - "Jin Wang" - "Chunxiao Xing" doi: "10.24963/ijcai.2019/495" url: "https://ijcai.org/proceedings/2019/495" pdf: "https://ijcai.org/proceedings/2019/0495.pdf" venue: "Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence" year: 2019 pages: "3569-3575" track: "Main track" keywords: - "Machine Learning: Transfer, Adaptation, Multi-task Learning" - "Natural Language Processing: Text Classification" - "Natural Language Processing: Sentiment Analysis and Text Mining" --- # Hierarchical Inter-Attention Network for Document Classification with Multi-Task Learning Bing Tian, Yong Zhang, Jin Wang, Chunxiao Xing *Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence*, pages 3569-3575. DOI: [10.24963/ijcai.2019/495](https://doi.org/10.24963/ijcai.2019/495) ## Abstract Document classification is an essential task in many real world applications. Existing approaches adopt both text semantics and document structure to obtain the document representation. However, these models usually require a large collection of annotated training instances, which are not always feasible, especially in low-resource settings. In this paper, we propose a multi-task learning framework to jointly train multiple related document classification tasks. We devise a hierarchical architecture to make use of the shared knowledge from all tasks to enhance the document representation of each task. We further propose an inter-attention approach to improve the task-specific modeling of documents with global information. Experimental results on 15 public datasets demonstrate the benefits of our proposed model. ## Full text ## 1 Introduction Document classification is a fundamental task in Natural Language Processing (NLP). The goal is to assign proper class labels to documents. It is essential in many real world applications, such as sentiment analysis [Maas et al., 2011], topic labeling [Wang and Manning, 2012] and financial analysis [Luo et al., 2018]. Recently deep neural networks have shown great success in learning distributed representation of texts. Given a varilength text, they represent it as a fixed-length vector using different approaches, such as Convolutional Neural Network (CNN) [Wang et al., 2017], Recurrent Neural Network (RNN) [Sutskever et al., 2014] and syntactic composition models [Socher et al., 2012]. There are also some studies utilizing deep neural networks for document classification [Yang et al., 2016]. Since deep learning models always contain huge number of parameters, they require a large volume of labeled corpus to learn a good document representation. However, in many situations, it is difficult to construct large training sets because acquiring manually labeled documents is very expensive. In this case, the performance of such models would be limited due to the lack of training data. Multi-Task Learning (MTL) [Caruana, 1997] can jointly train several tasks with the goal of mutual benefits. It is inattention weights merely from a document itself, we design a global attention layer which can learn the attention weights from other tasks. This idea comes from the observation that some semantic patterns are very rare in one task. In this case, the intra-attention mechanism would fail to recognize such patterns. But if similar semantic patterns appear in other tasks, we can recognize them and make the correct predication with the help of multi-task learning techniques. An example is shown in Figure 1. Case 1 to 4 are Amazon product reviews coming from different domains. For the Magazine review "There's really great fashion coverage that is not at all cookie-cutter and formulaic like some of the other women's books. " in Case 1, it is essential to understand the informative word cookie-cutter for sentiment classification. However, it is hard for intra-attention based model to capture this kind of expression since it is rare in this corpus. Nevertheless, as this kind of expression is common in datasets related to movies such as Video and IMDB as shown in Case 2 to 4, our inter-attention based model could take advantage of inter-attention layer to recognize it and finally obtain the right prediction. Moreover, we also propose a global shared layer with another LSTM network to utilize the global information among all tasks to enhance each task-specific representation. We conduct a comprehensive evaluation on 15 real world datasets. Experimental results show that our model outperforms previous approaches by an obvious margin on most datasets. Contributions of this paper are summarized as following: - We propose MT-HIA, a hierarchical model with multitask learning for document classification. Compared with previous studies, our proposed model can take advantage of both inherent document structure and the common knowledge from multi-tasks. - We introduce a hierarchical inter-attention approach to enhance the document representation of each single task. Moreover, we also devise a global sharing mechanism to utilize the knowledge frommultiple tasks. - We conduct extensive experiments on 15 real world datasets. The results demonstrate the effectiveness of our proposed methods. ## 2 RelatedWork ## 2.1 Document Classification Recently neural network models have been widely applied in the task of document classification. Wang et al. [2017] integrated information from knowledge bases into CNN models to improve the performance. Sutskever et al. [2014] and Liu et al. [2015] utilized variants of RNN to capture the sequence information in the texts. Socher et al. [2012] proposed Recursive Neural Network by leveraging the syntax information. Zhang et al. [2015] adopted a CNN model in character level for text classification. Yang et al. [2016] proposed HAN, a hierarchical model to learn the document structure and enrich the representation. ## 2.2 Sentiment Analysis Sentiment analysis is an important task in text classification which focuses on inferring the sentiment polarity of the text. Figure 2: Overall Architecture of MT-HIA ![Image](https://ijcai.org/proceedings/2019/fulltext/495/image_000000_fa8d0b9110c120e6e6980873bad17add583ace03471c57a1a38fadb1acb4d30e.png) Recently neural network has been widely applied in the task of sentiment analysis. Tang et al. [2015] proposed a neural network based framework which can make use of extra information, i.e., product and user to improve sentiment classification. Johnson et al. [2016] explored region embeddings via one-hot LSTM to improve the performance. McCann et al. [2017] introduced an approach for transferring knowledge from an encoder pretrained on machine translation to sentiment analysis. ## 2.3 Multi-Task Learning Multi-task learning [Caruana, 1997] is an approach to learn multiple related tasks simultaneously, aiming at yielding performance gains by leveraging potential correlations and common features among related tasks. It has been widely adopted in many applications, such as speech recognition [Rao et al., 2018], computer vision [Zhang et al., 2012] and natural language processing [Sutskever et al., 2014]. Multi-task learning techniques have already been widely adopted in NLP tasks. For example, Collobert et al. [2008] utilized the word inputs across multiple tasks to improve POS tagging. Firat et al. [2016] adopted multi-task learning in the task of machine translation. Liu et al. [2019] proposed a graph based model to capture the relations between different tasks. Recently there have been some studies using multi-task neural network for text classification. Liu et al. [2016a; 2016b] proposed a multi-task RNN model for text classification with 3 mechanisms for sharing information. Liu et al. [2017] further employed the adversarial training mechanism to reduce the noise from different tasks. Chen et al. [2018] improved the multi-task RNN models by extracting meta-knowledge from different tasks. These methods just focus on general text classification tasks and cannot make use of the hierarchical structure of documents. ## 3 Methodology Figure 2 displays the overall architecture of our proposed model. Here, we take the case of two tasks as an example. It consists of two components: the task specific encoder and the global shared layer. The task specific encoder is a hierarchical network which involves multi-task driven inter-attention mechanism from multiple granularities, i.e. both in wordlevel and sentence-level, to enhance the representation of documents. The global shared layer enables multi-task learning: it adopts a global shared memory to learn the common knowledge from all tasks. Next we will first introduce the task specific encoder and then the global shared layer in details. ## 3.1 Task Specific Encoder: Hierarchical Inter-Attention Network The task specific encoder is a hierarchical attention network shown in Figure 3. It consists of three components: a BiLSTM layer, a multi-task driven inter-attention layer and an output layer. The Bi-LSTM and attention layers are applied in both word and sentence levels. Next we will describe the details of different components. ## Bi-LSTM Based Sequence Encoder We adopt LSTM as the basic building block for sequence encoder. Given input xt and previous hidden state ht-1, the current hidden state ht can be updated by: Here we choose the LSTM architecture devised in [J´ozefowicz et al., 2015] as our encoder and use the function LSTM(·, ·, ·) as a shorthand for the encoding process. And θ refers to all parameters of LSTM. To better utilize the contextual information, we use Bi-LSTM to learn the hidden states. Based on Bi-LSTM, we construct the hierarchical model as following. Given a sentence i with words wit, t ∈ [0, T] (T is the number of words in the sentence), we first embed the words to low-dimensional vectors through an embedding matrix We, i.e. xit = Wewit. And then we construct the word-level encoder by feeding xit into a Bi-LSTM network to obtain the hidden state of each word. The final representation of the tth word in ith sentence is the concatenation of output in both directions: In sentence level, we feed the output of word-level encoders [⃗s1,⃗s2, ...,⃗sL] (L is the number of sentences in the document) into the Bi-LSTM and concatenate -→hi and ←- hi , i ∈ [1, L] to get a representation of ith sentence hi in a similar way. ## Multi-task Driven Inter-Attention As different words and sentences make different contributions to the composition of a document, it is necessary to assign different importance to them. To reach this goal, we design an inter-attention mechanism driven by multi-task learning to select the informative words and sentences. The intuition is that as external knowledge, information from the global shared component can provide common and taskinvariant knowledge, which can help disambiguate the semantics of words and sentences in the document of each single task. Specifically, we utilize a multi-task based MultiLayer Perceptron (MLP), which is a global layer shared by all tasks, to help compute attention weights. And we then involve information from this layer into both word-level and sentence-level encoder to help assign weights for each word and sentence. Figure 3: Hierarchical Inter-Attention Network ![Image](https://ijcai.org/proceedings/2019/fulltext/495/image_000001_e6f37dd85da45314437266231e255bad05817b6d0c2ada7e082bd0e367f6ee72.png) That is, we first feed the word vector xit into a Word-level Shared MLP layer (Word Shared MLP) which consists of a MLP to get the shared representation x(s) it for computing attention weights in Eq. 3. Then we get a hidden representation u(s) it of x(s) it containing common and task-invariant knowledge in Eq. 4. We measure the importance of tth word in ith sentence by an attention weight αit which can be computed as the inner product of u(s) it and aword level weight vector uw followed by a softmax layer in Eq. 5. The word level weight vector uw is randomly initialized and jointly learned during the training process. Finally, we obtain the sentence representation by computing the weighted sum of the word hidden states based on the weights in Eq. 6. Similarly, in order to obtain a better document representation with selected information, we also adopt multi-task driven inter-attention mechanism and propose a Sentencelevel Shared MLP layer (Sentence Shared MLP). We first feed the sentence vector si into the Sentence Shared MLP to get the representation s(s) i . Then we can correspondingly get the attention weights and the document representation d following the same route with the word-level encoder. ## Output Layer Since the document vector d is the high level representation of the document, it can be then fed into a softmax layer for classification: Figure 4: Word Level Shared Layer of Multi-Task Learning ![Image](https://ijcai.org/proceedings/2019/fulltext/495/image_000002_f43b8a4e630434d97affd158b81e4b6cf0f37c7b9319c68088142e87afc6e566.png) ## 3.2 Global Shared Layer for Multi-Task Learning Based on the task-specific model, we then describe our multitask learning based architecture. The task of document level classification is to assign a label Y to a document X. Supposing there are K related tasks, we denote the corpus of the kth task with Nk samples as: where X(k) i and Y (k) i are ith sample and its label in the kth task. The key factor of multi-task learning is the sharing scheme among different tasks. Considering the hierarchical structure of documents, we design the global shared layer to determine the common features in both word-level and sentence-level. Figure 4 is the illustration of word-level shared layer. Specifically, for each word x(k) t at time step t in task k, we first compute its shared representation h(s) t by an LSTMnetwork: And then we utilize the gating mechanism to control the portion of information flows from the global shared layer to each task: where g(k) controls the portion of information flows from the word-level global shared layer to task k, based on the correlation strength between x(k) t and h(s) t at the current time step. Similarly, in sentence-level encoder, there also exists a global shared LSTM layer (sentence shared LSTM) which captures the shared information for all the tasks and can be conveyed to each task. In this way, the hidden states and memory cells of each single task can benefit from extra information from all other tasks. ## 3.3 Training The objective of training process is to minimize the crossentropy of the predicted and true distributions for all tasks. For a single task k, we adopt cross-entropy to compute its loss L( ˆ y(k) , y(k) ) as shown in Eq. 14 where Nk refers to the number of training samples and C is the class number. y(k) ij is the ground-truth label and ˆ y(k) ij is the predicted probability. The overall loss of all tasks is shown in Eq. 13, where λk is the weight for the kth task and Θ is the set of all trainable parameters. For multi-task learning, the labeled data for training each task can come from completely different datasets. Following the previous study [Collobert and Weston, 2008], the training process is conducted in a stochastic manner by looping over the tasks: 1. Select a random task. 2. Select a mini-batch of examples from this task. 3. Update the parameters for this task and the global shared layer with respect to this mini-batch. 4. Go to 1. After the joint learning phase, we can use a fine tuning strategy to further optimize the performance for each task. ## 4 Evaluation In this section, we introduce the empirical results of our MTHIA model. We use the accuracy of classification as the evaluation metrics. ## 4.1 Experiment Setup ## Datasets We evaluate the effectiveness of our model on 15 documentlevel corpus. The first 14 datasets are Amazon product reviews coming from different domains such as Books, Music, Baby, etc. These datasets are collected based on the dataset 1 provided by Blitzer et al. [2007]. The last IMDB dataset contains movie reviews with binary classes [Maas et al., 2011]. All these datasets are document-level in which each review consists of several sentences. And the goal is to assign the positive or negative label to each document. Following previous studies, we randomly split these datasets into training sets, development sets and testing sets with the proportion of 70%, 10% and 20% respectively. The detailed statistics about these datasets are displayed in Table 1. To enable multi-task learning, we jointly train all 15 datasets simultaneously. ## Competitor Methods We compare our model with several state-of-the-art methods, including single-task learning based models and multi-task learning based models. For single-task learning models, we compare with four methods: - LSTM: the standard LSTM [Hochreiter and Schmidhuber, 1997]. - HyperLSTM: a model which uses a small network to generate weights for a larger network [Ha et al., 2016]. 1 https://www.cs.jhu.edu/∼mdredze/datasets/sentiment/ Table 1: Statistics of 15 Datasets for Document Classification | Datasets | Train Size | Dev. Size | Test Size | Class | Avg. Length | Voc Size | |------------|--------------|-------------|-------------|---------|---------------|------------| | Books | 1400 | 200 | 400 | 2 | 159 | 62K | | Elec | 1398 | 200 | 400 | 2 | 101 | 30K | | DVD | 1400 | 200 | 400 | 2 | 173 | 69K | | Kitchen | 1400 | 200 | 400 | 2 | 89 | 28K | | Apparel | 1400 | 200 | 400 | 2 | 57 | 21K | | Camera | 1397 | 200 | 400 | 2 | 130 | 26K | | Health | 1400 | 200 | 400 | 2 | 81 | 26K | | Music | 1400 | 200 | 400 | 2 | 136 | 60K | | Toys | 1400 | 200 | 400 | 2 | 90 | 28K | | Video | 1400 | 200 | 400 | 2 | 156 | 57K | | Baby | 1300 | 200 | 400 | 2 | 104 | 26K | | Mag | 1370 | 200 | 400 | 2 | 117 | 30K | | Soft | 1315 | 200 | 400 | 2 | 129 | 26K | | Sports | 1400 | 200 | 400 | 2 | 94 | 30K | | IMDB | 1400 | 200 | 400 | 2 | 269 | 44K | - MetaLSTM: a model which uses a meta-network to capture the meta-knowledge of semantic composition [Chen et al., 2018]. - HAN: a hierarchical attention network for single task document classification [Yang et al., 2016]. For multi-task learning, we compare our model with five models. - ASP-MTL: an adversarial MTL framework, alleviating the shared and private latent feature spaces from interfering with each other [Liu et al., 2017]. - Meta-MTL: a function-level sharing scheme for MTL with a shared meta-network proposed in [Chen et al., 2018]. - DA-MTL: a MTL framework with a shared sentence encoding layer and a dynamic task-attentive mechanism [Zheng et al., 2018]. - CG-MTL: a graph based MTL framework, which is the up-to-date method [Liu et al., 2019]. - MT-HA: a method proposed by ourselves. It is implemented by just replacing the inter-attention with intraattention mechanism. Although there are some other related approaches, such as single-task ones [Tang et al., 2015; Zhang et al., 2015] and multi-task ones [Liu et al., 2016a; Liu et al., 2016b], previous studies have shown that they cannot outperform the above selected methods. Therefore, we do not report the results of them here due to the space limitation. ## Hyper-Parameter Settings We initialize the word embeddings with pre-trained GloVe vectors [Pennington et al., 2014]. For out of box words, we randomly initialize their embeddings from uniform distribution in (-0.01, 0.01). The dimension of word embeddings is 200. And the models are trained with backpropagation using Adam optimizer with mini-batch size 30. The detailed settings of hyper-parameters are shown in Table 2. Table 2: Hyper-Parameter Settings | Word embedding size | d = 200 | |------------------------------------------|-------------| | Size of word-level basic-LSTM layer | hw = 50 | | Size of word-level shared-LSTM layer | h(s) w = 50 | | Size of sentence-level basic-LSTM layer | hs = 50 | | Size of sentence-level shared-LSTM layer | h(s) s = 50 | | Initial learning rate | 0.001 | | Regularization | 1E - 5 | ## 4.2 Analysis of Results Table 3 shows the classification accuracies on 15 datasets. The column "Single Task" refers to the results of 4 single task models. The column of "Multiple Tasks" shows the results achieved by six multi-task models including ours. Besides MT-HIA, we also report the results obtained by MT-HA. It stands for the multi-task hierarchical intra-attention network which assigns the attention weights merely from the document itself like attention mechanism in HAN. From the results, we have the following observations. Firstly, the overall performance of multi-task learning based models is much better than single-task ones. This result is consistent with previous studies [Chen et al., 2018] and demonstrates the benefits of multi-task learning techniques. Among them our MT-HIA obtains the best performance. Specifically, it outperforms HAN, the most competitive single-task model by 4.3% on average. Secondly, our model also achieves a better performance than competitors in most cases for multi-task learning. For example, our method shows an average improvement of accuracy 1.8% to CG-MTL, the up-to-date multi-task model and 0.4% to DA-MTL, the most competitive multi-task model. Lastly, we can see that MT-HIA outperforms MT-HA in 13 out of 15 datasets. This demonstrates the advantage of our inter-attention mechanism over the intra-attention one. Next we look at the specific performance gain of MT-HIA against single-task methods. The column "Single Avg" shows the average accuracy of 4 single-task based models. From this table, we can see that the performance of most tasks can be improved with a large margin with the help of MTL. And our MT-HIA is able to achieve the state-of-the-art performances on each single task in most cases, indicating the effectiveness and robustness of our model. Among all single-task learning methods, HAN performs best since it takes hierarchical structure of documents into consideration and proposes a hierarchical attention architecture. However, as the attention weights of HAN only comes from the document itself, it might be difficult for HAN to assign proper weights for infrequent semantic patterns. As our proposed model utilizes multi-task learning, each task can extract both common and task-invariant knowledge from global shared layers. And we adopts inter-attention mechanism, which can learn the attention weights from the global sharing information contributed by multiple tasks. Therefore, our MT-HIA is more powerful in discovering informative sentences and words in a document and thus achieves better results. Finally, we make detailed analysis on the results. Compared with other MTL methods, our MT-HIA model can better capture the hierarchical structure of documents. Other latest multi-task learning methods, such as Meta-MTL and DA-MTL focus on how to make good use of the shared information from text corpus. However, they all ignore the hierarchical structure of documents and will definitely fail to capture sufficient signals for document classification. Thus their performance are not as good as ours. Among all baseline methods, DA-MTL performs the best. The reason is that it proposes a new scheme of information sharing for MTL: all tasks share the same sentence representation and each task can select the task-specific information from the shared representation with attention mechanism. Compared with DAMTL, our model can also reach the same goal with the help of inter-attention mechanism. As our method can also capture the hierarchical structure, it can clearly beat DA-MTL. Table 3: Accuracy of Our Models on 15 Datasets against Baselines | Task | Single Task - LSTM | Single Task - Hyper LSTM | Single Task - Meta LSTM | Single Task - HAN | Single Multiple Tasks - Avg | Single Multiple Tasks - ASP- MTL | Single Multiple Tasks - Meta- MTL | Single Multiple Tasks - DA- MTL | Single Multiple Tasks - CG- MTL | Single Multiple Tasks - MT- HA(ours) | Single Multiple Tasks - MT-HIA (ours) | |---------------------------------------------------------------------------|---------------------------------------------------------------------|--------------------------------------------------------------------|-----------------------------------------------------------------|---------------------------------------------------------------|--------------------------------------------------------------------------|------------------------------------------------------------------------|------------------------------------------------------------------|------------------------------------------------------------------|------------------------------------------------------------------|------------------------------------------------------------------------|---------------------------------------------------------------------| | Books Electronics DVD Kitchen Apparel Camera Health Music Toys Video Baby | 79.5 80.5 81.7 78 83.2 85.2 84.5 76.7 83.2 81.5 84.7 89.2 84.7 81.7 | 78.3 80.7 80.3 80 85.8 88.3 84 78.5 83.7 83.7 85.5 91.3 86.5 82 77 | 83 82.3 82.3 83.3 86.5 88.3 86.3 80 84.3 84.3 84 92.3 88.3 82.5 | 85 81.7 83.3 85.8 85 90 85.9 82.5 85.4 85.6 86.7 92 86.7 84.2 | 81.5 81.3 81.9 81.8 85.1 88 85.2 79.4 84.2 83.8 85.2 91.2 86.6 82.6 81.6 | 87 89 87.4 87.2 88.7 91.3 88.1 82.6 88.8 85.5 89.8 92.4 87.3 86.7 85.8 | 87.5 89.5 88 91.3 87 89.7 90.3 86.3 88.5 88.3 88 91 88.5 86.7 88 | 88.5 89 88 89 88.8 91.8 90.3 85 89.5 89.5 90.5 92 90.8 89.8 89.8 | 86.7 88.5 86.5 87.7 87 89.5 89.5 85.2 89 87 89.2 92 89.7 87.7 87 | 87.5 88 88.3 91.6 86.7 91.1 88.9 85.8 88.3 88.9 88.4 92.2 90 87.5 88.9 | 89.2 87.5 89.2 91.7 87.5 92.2 91.1 88.3 85.4 90 91.7 95.6 91.7 87.5 | | Magazines | | | | | | | | | | | | | Software | | | | | | | | | | | | | Sports | | | | | | | | | | | | | IMDB | 81.7 | | 83.5 | 84.2 | | | | | | | 90 | | AVG | 82.4 | 83.0 | 84.7 | 85.6 | 83.9 | 87.8 | 88.6 | 89.5 | 88.1 | 88.8 | 89.9 | Figure 5: Visualization of Attention ![Image](https://ijcai.org/proceedings/2019/fulltext/495/image_000003_801974104c65f2e07092609f564536fcaf22484edcf8197fa99f9f093e00d2da.png) ## 4.3 Case Study In order to illustrate that our model is capable of better discovering informative sentences and words in a document, we visualize the weights of sentences and words in an hierarchical manner in Figure 5. In this example, we compare MT-HIA with inter attention mechanism with MT-HA that only assigns the attention weights merely from the document itself. The red color refers to the attention weights over sentences, blue denotes attention weights over words and the depth of color represents the size of the weights. From Figure 5, we can see that for the negative review, both models can select the words carrying strong sentiment like "attractive", "disappointed". However the intra-attention mechanism fails to capture the semantic patten "although ... I was..." since similar patterns are rare in this corpus. Therefore, it puts more attention on the fourth sentence and gives a wrong prediction. Nevertheless, as our MT-HIA is able to take advantage of common knowledge from other tasks, it will be easier to capture this pattern once it appears in other tasks. As a result, MT-HIA successfully captures this kind of pattern and chooses to ignore the fourth sentence and finally gets the right prediction. ## 5 Conclusion In this paper, we propose the Multi-Task Hierarchical InterAttention Network model for document classification. We improve the task-specific document representation by proposing an inter-attention mechanism. We further devise a global shared mechanism to smartly utilize the knowledge shared by multiple tasks. Experimental results on 15 real world datasets show that our proposed model outperforms state-of-the-art methods by a substantial margin. For the future work, we would like to explore our model on more different types of NLP tasks. ## Acknowledgements This work was supported by NSFC(91646202), National Key R&D Program of China(SQ2018YFB140235). ## References 1. [Blitzer et al., 2007] John Blitzer, Mark Dredze, and Fer- nando Pereira. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In ACL, 2007. 2. [Caruana, 1997] Rich Caruana. Multitask learning. Machine Learning, 28(1):41-75, 1997. 3. [Chen et al., 2018] Junkun Chen, Xipeng Qiu, Pengfei Liu, and Xuanjing Huang. Meta multi-task learning for se- quence modeling. In AAAI, 2018. 4. [Collobert and Weston, 2008] Ronan Collobert and Jason Weston. A unified architecture for natural language pro- cessing: deep neural networks with multitask learning. In ICML, pages 160-167, 2008. 5. [Firat et al., 2016] Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. Multi-way, multilingual neural machine translation with a shared attention mechanism. In NAACL- HLT, pages 866-875, 2016. 6. [Ha et al., 2016] David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetworks. CoRR, abs/1609.09106, 2016. 7. [Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735-1780, 1997. 8. [Johnson and Zhang, 2016] Rie Johnson and Tong Zhang. Supervised and semi-supervised text categorization using LSTM for region embeddings. In ICML, pages 526-534, 2016. 9. [Józefowicz et al., 2015] Rafal Józefowicz, Wojciech Zaremba, and Ilya Sutskever. An empirical exploration of recurrent network architectures. In ICML, pages 2342-2350, 2015. 10. [Liu et al., 2015] Pengfei Liu, Xipeng Qiu, Xinchi Chen, Shiyu Wu, and Xuanjing Huang. Multi-timescale long short-term memory neural network for modelling sen- tences and documents. In EMNLP, pages 2326-2335, 2015. 11. [Liu et al., 2016a] Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Deep multi-task learning with shared memory for text classification. In EMNLP, pages 118-127, 2016. 12. [Liu et al., 2016b] Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Recurrent neural network for text classification with multi-task learning. In IJCAI, pages 2873-2879, 2016. 13. [Liu et al., 2017] Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Adversarial multi-task learning for text classifi- cation. In ACL, pages 1-10, 2017. 14. [Liu et al., 2019] Pengfei Liu, Jie Fu, Yue Dong, Xipeng Qiu, and Jackie Chi Kit Cheung. Multi-task learning over graph structures. In AAAI, 2019. 15. [Luo et al., 2018] Ling Luo, Xiang Ao, Feiyang Pan, Jin Wang, Tong Zhao, Ningzi Yu, and Qing He. Beyond polar- ity: Interpretable financial sentiment analysis with hierar- chical query-driven attention. In IJCAI, pages 4244-4250, 2018. 16. [Maas et al., 2011] Andrew L. Maas, Raymond E. Daly, Pe- ter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In ACL, pages 142-150, 2011. 17. [McCann et al., 2017] Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in transla- tion: Contextualized word vectors. In NIPS, pages 6297- 6308, 2017. 18. [Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In EMNLP, pages 1532-1543, 2014. 19. [Rao et al., 2018] Jinfeng Rao, Ferhan Türe, and Jimmy Lin. Multi-task learning with neural networks for voice query understanding on an entertainment platform. In KDD, pages 636-645, 2018. 20. [Socher et al., 2012] Richard Socher, Brody Huval, Christo- pher D. Manning, and Andrew Y. Ng. Semantic com- positionality through recursive matrix-vector spaces. In EMNLP, pages 1201-1211, 2012. 21. [Subramanian et al., 2018] Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J. Pal. Learn- ing general purpose distributed sentence representations via large scale multi-task learning. ICLR, 2018. 22. [Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In NIPS, pages 3104-3112, 2014. 23. [Tang et al., 2015] Duyu Tang, Bing Qin, and Ting Liu. Learning semantic representations of users and products for document level sentiment classification. In ACL, pages 1014-1023, 2015. 24. [Wang and Manning, 2012] Sida I. Wang and Christopher D. Manning. Baselines and bigrams: Simple, good sentiment and topic classification. In ACL, pages 90-94, 2012. 25. [Wang et al., 2017] Jin Wang, Zhongyuan Wang, Dawei Zhang, and Jun Yan. Combining knowledge with deep convolutional neural networks for short text classification. In IJCAI, pages 2915-2921, 2017. 26. [Yang et al., 2016] Zichao Yang, Diyi Yang, Chris Dyer, Xi- aodong He, Alexander J. Smola, and Eduard H. Hovy. Hi- erarchical attention networks for document classification. In NAACL-HLT, pages 1480-1489, 2016. 27. [Zhang and Yang, 2017] Yu Zhang and Qiang Yang. A sur- vey on multi-task learning. CoRR, abs/1707.08114, 2017. 28. [Zhang et al., 2012] Tianzhu Zhang, Bernard Ghanem, Si Liu, and Narendra Ahuja. Robust visual tracking via multi-task sparse learning. In CVPR, pages 2042-2049, 2012. 29. [Zhang et al., 2015] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS, pages 649-657, 2015. 30. [Zheng et al., 2018] Renjie Zheng, Junkun Chen, and Xipeng Qiu. Same representation, different attentions: Shareable sentence representation learning from multiple tasks. In IJCAI, pages 4616-4622, 2018.