{"type":"ScholarlyArticle","title":"Dynamically Route Hierarchical Structure Representation to Attentive Capsule for Text Classification","authors":[{"name":"Wanshan Zheng","slug":"wanshan-zheng","given":"Wanshan","family":"Zheng","affiliation":"School of Data and Computer Science , Sun Yat-sen University , Guangzhou , China; Guangdong Key Laboratory for Big Data Analysis and Simulation of Public Opinion , The School of Communication and Design , Sun Yat-sen University , Guangzhou , China"},{"name":"Zibin Zheng","slug":"zibin-zheng","given":"Zibin","family":"Zheng","affiliation":"School of Data and Computer Science , Sun Yat-sen University , Guangzhou , China; Guangdong Key Laboratory for Big Data Analysis and Simulation of Public Opinion , The School of Communication and Design , Sun Yat-sen University , Guangzhou , China"},{"name":"Hai Wan","slug":"hai-wan","given":"Hai","family":"Wan","affiliation":"School of Data and Computer Science , Sun Yat-sen University , Guangzhou , China"},{"name":"Chuan Chen","slug":"chuan-chen","given":"Chuan","family":"Chen","affiliation":"School of Data and Computer Science , Sun Yat-sen University , Guangzhou , China; Guangdong Key Laboratory for Big Data Analysis and Simulation of Public Opinion , The School of Communication and Design , Sun Yat-sen University , Guangzhou , China"}],"abstract":"Representation learning and feature aggregation are usually the two key intermediate steps in natural language processing. Despite deep neural networks have shown strong performance in the text classification task, they are unable to learn adaptive structure features automatically and lack of a method for fully utilizing the extracted features. In this paper, we propose a novel architecture that dynamically routes hierarchical structure feature to attentive capsule, named HAC. Specifically, we first adopt intermediate information of a well-designed deep dilated CNN to form hierarchical structure features. Different levels of structure representations are corresponding to various linguistic units such as word, phrase and clause, respectively. Furthermore, we design a capsule module using dynamic routing and equip it with an attention mechanism. The attentive capsule implements an effective aggregation strategy for feature clustering and selection. Extensive results on eleven benchmark datasets demonstrate that the proposed model obtains competitive performance against several state-of-the-art baselines. Our code is available at https://github.com/zhengwsh/HAC.","doi":"10.24963/ijcai.2019/759","url":"https://ijcai.org/proceedings/2019/759","pdf":"https://ijcai.org/proceedings/2019/0759.pdf","markdown":"https://ijcai.org/proceedings/2019/759.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/759","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Natural Language Processing","track":"Main track","number":759,"pages":{"first":5464,"last":5470},"keywords":["Natural Language Processing: Text Classification","Natural Language Processing: Sentiment Analysis and Text Mining","Natural Language Processing: Natural Language Processing"],"videos":[],"references":[{"position":1,"raw":"[Cer et al., 2018] Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Con- stant, et al. Universal sentence encoder. arXiv preprint arXiv:1803.11175, 2018.","title":"Universal sentence encoder","authors":"Daniel Cer, Yinfei Yang, Sheng-Yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2018] Chuan Chen, Jingxue Xin, et al. A semisupervised classification approach for multidomain networks with domain selection. IEEE TNNLS, 2018.","title":"A semisupervised classification approach for multidomain networks with domain selection","authors":"Chuan Chen, Jingxue Xin","year":"2018","venue":"IEEE TNNLS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP, pages 1724-1734, 2014.","title":"Learning phrase representations using rnn encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chung et al., 2014] Junyoung Chung, Caglar Gulcehre, Ky- unghyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. In NIPS 2014 Workshop on Deep Learning, 2014.","title":"Empirical evaluation of gated recurrent neural networks on sequence modeling","authors":"Chung","year":"2014","venue":"NIPS 2014 Workshop on Deep Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Conneau et al., 2017a] Alexis Conneau, Douwe Kiela, Hol- ger Schwenk, Loïc Barrault, and Antoine Bordes. Super- vised learning of universal sentence representations from natural language inference data. In EMNLP, 2017.","title":"Supervised learning of universal sentence representations from natural language inference data","authors":"Conneau","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Conneau et al., 2017b] Alexis Conneau, Holger Schwenk, Loıc Barrault, and Yann Lecun. Very deep convolutional networks for text classification. In EACL, 2017.","title":"Very deep convolutional networks for text classification","authors":"Conneau","year":"2017","venue":"EACL","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gong et al., 2018] Jingjing Gong, Xipeng Qiu, Shaojing Wang, and Xuanjing Huang. Information aggregation via dynamic routing for sequence encoding. arXiv preprint arXiv:1806.01501, 2018.","title":"Information aggregation via dynamic routing for sequence encoding","authors":"Jingjing Gong, Xipeng Qiu, Shaojing Wang, Xuanjing Huang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Grave et al., 2017] Edouard Grave, Tomas Mikolov, Ar- mand Joulin, and Piotr Bojanowski. Bag of tricks for effi- cient text classification. In EACL, page 427, 2017.","title":"Edouard Grave, Tomas Mikolov, Armand Joulin, and Piotr Bojanowski. Bag of tricks for efficient text classification","authors":"Grave","year":"2017","venue":"EACL","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kalchbrenner et al., 2014] Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. A convolutional neural network for modelling sentences. In ACL, 2014.","title":"A convolutional neural network for modelling sentences","authors":"Nal Kalchbrenner, Edward Grefenstette, Phil Blunsom","year":"2014","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kim, 2014] Yoon Kim. Convolutional neural networks for sentence classification. In EMNLP, 2014.","title":"Convolutional neural networks for sentence classification","authors":"Yoon Kim","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lin et al., 2017] Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. In ICLR, 2017.","title":"A structured self-attentive sentence embedding","authors":"Lin","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In EMNLP, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher Manning","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Perone et al., 2018] Christian S Perone, Roberto Silveira, and Thomas S Paula. Evaluation of sentence embeddings in downstream and linguistic probing tasks. arXiv preprint arXiv:1806.06259, 2018.","title":"Evaluation of sentence embeddings in downstream and linguistic probing tasks","authors":"Roberto Christian S Perone, Thomas S Silveira, Paula","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Peters et al., 2018] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL, pages 2227-2237, 2018.","title":"Deep contextualized word representations","authors":"Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer","year":"2018","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Qiao et al., 2018] Chao Qiao, Bo Huang, Guocheng Niu, Daren Li, Daxiang Dong, Wei He, et al. A new method of region embedding for text classification. In ICLR, 2018.","title":"A new method of region embedding for text classification","authors":"Chao Qiao, Bo Huang, Guocheng Niu, Daren Li, Daxiang Dong, Wei He","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sabour et al., 2017] Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In NIPS, pages 3856-3866, 2017.","title":"Dynamic routing between capsules","authors":"Sara Sabour, Nicholas Frosst, Geoffrey E Hinton","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shaoan et al., 2018] Xie Shaoan, Zheng Zibin, et al. Learn- ing semantic representations for unsupervised domain ad- aptation. In ICML, pages 5419-5428, 2018.","title":"Learning semantic representations for unsupervised domain adaptation","authors":"Zheng Xie Shaoan, Zibin","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tai et al., 2015] Kai Sheng Tai, Richard Socher, et al. Im- proved semantic representations from tree-structured long short-term memory networks. In ACL, 2015.","title":"Improved semantic representations from tree-structured long short-term memory networks","authors":"Kai Sheng, Tai, Richard Socher","year":"2015","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tianyang et al., 2018] Zhang Tianyang, Huang Minlie, and Zhao Li. Learning structured representation for text clas- sificationxvia reinforcement learning. In AAAI, 2018.","title":"Learning structured representation for text classificationxvia reinforcement learning","authors":"Zhang Tianyang, Huang Minlie, Zhao Li","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2018a] Guoyin Wang, Chunyuan Li, Wenlin Wang, Yizhe Zhang, Dinghan Shen, Xinyuan Zhang, Ri- cardo Henao, and Lawrence Carin. Joint embedding of words and labels for text classification. In ACL, 2018.","title":"Joint embedding of words and labels for text classification","authors":"Wang","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2018b] Shiyao Wang, Minlie Huang, and Zhidong Deng. Densely connected cnn with multi-scale feature attention for text classification. In IJCAI, 2018.","title":"Densely connected cnn with multi-scale feature attention for text classification","authors":"Wang","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/621"},{"position":22,"raw":"[Yang et al., 2016] Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, et al. Hierarchical attention networks for document classification. In NAACL, 2016.","title":"Hierarchical attention networks for document classification","authors":"Yang","year":"2016","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yin and Schütze, 2015] Wenpeng Yin and Hinrich Schütze. Multichannel variable-size convolution for sentence clas- sification. In CoNLL, pages 204-214, 2015.","title":"Multichannel variable-size convolution for sentence classification","authors":"Schütze Yin, Wenpeng Yin, Hinrich Schütze","year":"2015","venue":"CoNLL","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yogatama et al., 2017] Dani Yogatama, Chris Dyer, Wang Ling, and Phil Blunsom. Generative and discriminative text classification with recurrent neural networks. arXiv preprint arXiv:1703.01898, 2017.","title":"Generative and discriminative text classification with recurrent neural networks","authors":"Dani Yogatama, Chris Dyer, Wang Ling, Phil Blunsom","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yu and Koltun, 2016] Fisher Yu and Vladlen Koltun. Multi- scale context aggregation by dilated convolutions. In ICLR, 2016.","title":"Multiscale context aggregation by dilated convolutions","authors":"Koltun ; Fisher Yu, Vladlen Yu, Koltun","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2015] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS, pages 649-657, 2015.","title":"Character-level convolutional networks for text classification","authors":"Xiang Zhang, Junbo Zhao, Yann Lecun","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhao et al., 2015] Han Zhao, Zhengdong Lu, and Pascal Poupart. Self-adaptive hierarchical sentence model. In IJCAI, pages 4069-4076, 2015.","title":"Self-adaptive hierarchical sentence model","authors":"Han Zhao, Zhengdong Lu, Pascal Poupart","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/571"},{"position":28,"raw":"[Zhao et al., 2018] W Zhao, J Ye, M Yang, Z Lei, S Zhang, and Z Zhao. Investigating capsule networks with dynamic routing for text classification. In EMNLP, 2018.","title":"Investigating capsule networks with dynamic routing for text classification","authors":"J Zhao, M Ye, Z Yang, S Lei, Zhang, Zhao","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zheng et al., 2017] Zibin Zheng, Yatao Yang, et al. Wide and deep convolutional neural networks for electricity- theft detection to secure smart grids. IEEE TII, 2017.","title":"Wide and deep convolutional neural networks for electricitytheft detection to secure smart grids","authors":"Zheng","year":"2017","venue":"IEEE TII","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhou et al., 2015] Chunting Zhou, Chonglin Sun, Zhiyuan Liu, and Francis Lau. A c-lstm neural network for text classification. arXiv preprint arXiv:1511.08630, 2015.","title":"A c-lstm neural network for text classification","authors":"Zhou","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}