{"type":"ScholarlyArticle","title":"Densely Connected CNN with Multi-scale Feature Attention for Text Classification","authors":[{"name":"Shiyao Wang","slug":"shiyao-wang","given":"Shiyao","family":"Wang","affiliation":"State Key Laboratory of Intelligent Technology and Systems Beijing National Research Center for Information Science and Technology"},{"name":"Minlie Huang","slug":"minlie-huang","given":"Minlie","family":"Huang","affiliation":"State Key Laboratory of Intelligent Technology and Systems Beijing National Research Center for Information Science and Technology; Department of Computer Science , Tsinghua University , Beijing 100084 , China"},{"name":"Zhidong Deng","slug":"zhidong-deng","given":"Zhidong","family":"Deng","affiliation":"State Key Laboratory of Intelligent Technology and Systems Beijing National Research Center for Information Science and Technology"}],"abstract":"Text classification is a fundamental problem in natural language processing. As a popular deep learning model, convolutional neural network (CNN) has demonstrated great success in this task. However, most existing CNN models apply convolution filters of fixed window size, thereby unable to learn variable n-gram features flexibly. In this paper, we present a densely connected CNN with multi-scale feature attention for text classification. The dense connections build short-cut paths between upstream and downstream convolutional blocks, which enable the model to compose features of larger scale from those of smaller scale, and thus produce variable n-gram features. Furthermore, a multi-scale feature attention is developed to adaptively select multi-scale features for classification. Extensive experiments demonstrate that our model obtains competitive performance against state-of-the-art baselines on five benchmark datasets. Attention visualization further reveals the model's ability to select proper n-gram features for text classification.","doi":"10.24963/ijcai.2018/621","url":"https://ijcai.org/proceedings/2018/621","pdf":"https://ijcai.org/proceedings/2018/0621.pdf","markdown":"https://ijcai.org/proceedings/2018/621.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/621","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Natural Language Processing","track":"Main track","number":621,"pages":{"first":4468,"last":4474},"keywords":["Natural Language Processing: Embeddings","Natural Language Processing: Natural Language Processing","Natural Language Processing: Text Classification","Natural Language Processing: Natural Language Semantics"],"videos":[],"references":[{"position":1,"raw":"[Blunsom et al., 2014] Phil Blunsom, Edward Grefenstette, and Nal Kalchbrenner. A convolutional neural network for modelling sentences. In ACL, 2014.","title":"A convolutional neural network for modelling sentences","authors":"Blunsom","year":"2014","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Collobert et al., 2011] Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12(Aug):2493-2537, 2011.","title":"Natural language processing (almost) from scratch","authors":"Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, Pavel Kuksa","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Conneau et al., 2017] Alexis Conneau, Holger Schwenk, Loïc Barrault, and Yann Lecun. Very deep convolutional networks for natural language processing. In EACL, 2017.","title":"Very deep convolutional networks for natural language processing","authors":"Alexis Conneau, Holger Schwenk, Loïc Barrault, Yann Lecun","year":"2017","venue":"EACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Grave et al., 2017] Edouard Grave, Tomas Mikolov, Ar- mand Joulin, and Piotr Bojanowski. Bag of tricks for effi- cient text classification. EACL (2), pages 427-431, 2017.","title":"Edouard Grave, Tomas Mikolov, Armand Joulin, and Piotr Bojanowski. Bag of tricks for efficient text classification","authors":"Grave","year":"2017","venue":"EACL (2)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Huang et al., 2017a] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinberger. Densely con- nected convolutional networks. In CVPR, 2017.","title":"Densely connected convolutional networks","authors":"Huang","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Huang et al., 2017b] Minlie Huang, Qiao Qian, and Xi- aoyan Zhu. Encoding syntactic knowledge in neural net- works for sentiment classification. ACM Transactions on Information Systems (TOIS), 35(3):26, 2017.","title":"Encoding syntactic knowledge in neural networks for sentiment classification","authors":"Huang","year":"2017","venue":"ACM Transactions on Information Systems (TOIS)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. In ICML, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jia et al., 2014] Yangqing Jia, Evan Shelhamer, Jeff Don- ahue, Sergey Karayev, Jonathan Long, Ross Girshick, Ser- gio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. In ACM MM, pages 675-678, 2014.","title":"Caffe: Convolutional architecture for fast feature embedding","authors":"Jia","year":"2014","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Johnson and Zhang, 2015] Rie Johnson and Tong Zhang. Effective use of word order for text categorization with convolutional neural networks. In NAACL, pages 103-112, 2015.","title":"Effective use of word order for text categorization with convolutional neural networks","authors":"Johnson, Zhang ; ] Rie Johnson, Tong Zhang","year":"2015","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Johnson and Zhang, 2017] Rie Johnson and Tong Zhang. Deep pyramid convolutional neural networks for text cat- egorization. In ACL, pages 562-570, 2017.","title":"Deep pyramid convolutional neural networks for text categorization","authors":"Rie Johnson, Tong Zhang","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kim, 2014] Yoon Kim. Convolutional neural networks for sentence classification. In EMNLP, pages 1746-1751, 2014.","title":"Convolutional neural networks for sentence classification","authors":"Yoon Kim","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lei et al., 2015] Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. Molding cnns for text: non-linear, non- consecutive convolutions. pages 1565-1575, 2015.","title":"Molding cnns for text: non-linear, nonconsecutive convolutions","authors":"Tao Lei, Regina Barzilay, Tommi S Jaakkola","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lin et al., 2017] Zhouhan Lin, Minwei Feng, Cícero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. In ICLR, 2017.","title":"A structured self-attentive sentence embedding","authors":"Lin","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Luong et al., 2015] Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention-based neural machine translation. In EMNLP, 2015.","title":"Effective approaches to attention-based neural machine translation","authors":"Minh-Thang Luong, Hieu Pham, Christopher D Manning","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nair and Hinton, 2010] Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann ma- chines. In ICML, pages 807-814, 2010.","title":"Rectified linear units improve restricted boltzmann machines","authors":"Vinod Nair, Geoffrey E Hinton","year":"2010","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pang and Lee, 2005] Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In ACL, pages 115-124, 2005.","title":"Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales","authors":"Lee ; Pang, Bo Pang, Lillian Lee","year":"2005","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, volume 14, pages 1532-43, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tai et al., 2015] Kai Sheng Tai, Richard Socher, and Christopher D. Manning. Improved semantic represen- tations from tree-structured long short-term memory net- works. In ACL, pages 1556-1566, 2015.","title":"Improved semantic representations from tree-structured long short-term memory networks","authors":"Kai Sheng, Tai, Richard Socher, Christopher D Manning","year":"2015","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2017] Jin Wang, Zhongyuan Wang, Dawei Zhang, and Jun Yan. Combining knowledge with deep convolutional neural networks for short text classification. In IJCAI, pages 2915-2921, 2017.","title":"Combining knowledge with deep convolutional neural networks for short text classification","authors":"Jin Wang, Zhongyuan Wang, Dawei Zhang, Jun Yan","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/406"},{"position":21,"raw":"[Xiao and Cho, 2016] Yijun Xiao and Kyunghyun Cho. Ef- ficient character-level document classification by com- bining convolution and recurrent layers. arXiv preprint arXiv:1602.00367, 2016.","title":"Efficient character-level document classification by combining convolution and recurrent layers","authors":"Xiao, Cho, Yijun Xiao, Kyunghyun Cho","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xu et al., 2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, pages 2048-2057, 2015.","title":"Show, attend and tell: Neural image caption generation with visual attention","authors":"Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, Yoshua Bengio","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yang et al., 2016] Zichao Yang, Diyi Yang, Chris Dyer, Xi- aodong He, Alexander J. Smola, and Eduard H. Hovy. Hi- erarchical attention networks for document classification. In NAACL, pages 1480-1489, 2016.","title":"Hierarchical attention networks for document classification","authors":"Yang","year":"2016","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yin et al., 2016] Wenpeng Yin, Hinrich Schütze, Bing Xi- ang, and Bowen Zhou. Abcnn: Attention-based convolu- tional neural network for modeling sentence pairs. In ACL, pages 259-272, 2016.","title":"Abcnn: Attention-based convolutional neural network for modeling sentence pairs","authors":"Yin","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yogatama et al., 2017] Dani Yogatama, Chris Dyer, Wang Ling, and Phil Blunsom. Generative and discriminative text classification with recurrent neural networks. arXiv preprint arXiv:1703.01898, 2017.","title":"Generative and discriminative text classification with recurrent neural networks","authors":"Dani Yogatama, Chris Dyer, Wang Ling, Phil Blunsom","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2015] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS, pages 649-657, 2015.","title":"Character-level convolutional networks for text classification","authors":"Xiang Zhang, Junbo Zhao, Yann Lecun","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2018] Tianyang Zhang, Minlie Huang, and Li Zhao. Learning structured representation for text clas- sification via reinforcement learning. In AAAI, 2018.","title":"Learning structured representation for text classification via reinforcement learning","authors":"Tianyang Zhang, Minlie Huang, Li Zhao","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}