{"type":"ScholarlyArticle","title":"Recurrent Neural Network for Text Classification with Hierarchical Multiscale Dense Connections","authors":[{"name":"Yi Zhao","slug":"yi-zhao","given":"Yi","family":"Zhao","affiliation":"Department of Computer Science and Engineering , Shanghai Jiao Tong University"},{"name":"Yanyan Shen","slug":"yanyan-shen","given":"Yanyan","family":"Shen","affiliation":"Department of Computer Science and Engineering , Shanghai Jiao Tong University"},{"name":"Junjie Yao","slug":"junjie-yao","given":"Junjie","family":"Yao","affiliation":"School of Computer Science and Software Engineering , East China Normal University"}],"abstract":"Text classification is a fundamental task in many Natural Language Processing applications. While recurrent neural networks have achieved great success in performing text classification, they fail to capture the hierarchical structure and long-term semantics dependency which are common features of text data. Inspired by the advent of the dense connection pattern in advanced convolutional neural networks, we propose a simple yet effective recurrent architecture, named Hierarchical Mutiscale Densely Connected RNNs (HM-DenseRNNs), which: 1) enables direct access to the hidden states of all preceding recurrent units via dense connections, and 2) organizes multiple densely connected recurrent units into a hierarchical multi-scale structure, where the layers are updated at different scales. HM-DenseRNNs can effectively capture long-term dependencies among words in long text data, and a dense recurrent block is further introduced to reduce the number of parameters and enhance training efficiency. We evaluate the performance of our proposed architecture on three text datasets and the results verify the advantages of HM-DenseRNN over the baseline methods in terms of the classification accuracy.","doi":"10.24963/ijcai.2019/757","url":"https://ijcai.org/proceedings/2019/757","pdf":"https://ijcai.org/proceedings/2019/0757.pdf","markdown":"https://ijcai.org/proceedings/2019/757.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/757","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Natural Language Processing","track":"Main track","number":757,"pages":{"first":5450,"last":5456},"keywords":["Natural Language Processing: Text Classification","Natural Language Processing: Natural Language Processing"],"videos":[],"references":[{"position":1,"raw":"[Campos et al., 2017] Víctor Campos, Brendan Jou, Xavier Giró-i Nieto, Jordi Torres, and Shih-Fu Chang. Skip rnn: Learning to skip state updates in recurrent neural net- works. arXiv preprint arXiv:1708.06834, 2017.","title":"Skip rnn: Learning to skip state updates in recurrent neural networks","authors":"Víctor Campos, Brendan Jou, Xavier Giró-I Nieto, Jordi Torres, Shih-Fu Chang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chang et al., 2017] Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark A Hasegawa-Johnson, and Thomas S Huang. Dilated recurrent neural networks. In Advances in Neural Information Processing Systems, pages 77-87, 2017.","title":"Dilated recurrent neural networks","authors":"Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark A Hasegawa-Johnson, Thomas S Huang","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the proper- ties of neural machine translation: Encoder-decoder ap- proaches. arXiv preprint arXiv:1409.1259, 2014.","title":"On the properties of neural machine translation: Encoder-decoder approaches","authors":"Cho","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chung et al., 2014] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evalua- tion of gated recurrent neural networks on sequence mod- eling. arXiv preprint arXiv:1412.3555, 2014.","title":"Empirical evaluation of gated recurrent neural networks on sequence modeling","authors":"Chung","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chung et al., 2016] Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrent neural networks. arXiv preprint arXiv:1609.01704, 2016.","title":"Hierarchical multiscale recurrent neural networks","authors":"Chung","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[El Hihi and Bengio, 1996] Salah El Hihi and Yoshua Ben- gio. Hierarchical recurrent neural networks for long-term dependencies. In Advances in neural information process- ing systems, pages 493-499, 1996.","title":"Salah El Hihi and Yoshua Bengio. Hierarchical recurrent neural networks for long-term dependencies","authors":"El Hihi, Bengio","year":"1996","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2017] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, volume 1, page 3, 2017.","title":"Densely connected convolutional networks","authors":"Gao Huang, Zhuang Liu, Laurens Van Der Maaten, Kilian Q Weinberger","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Koutnik et al., 2014] Jan Koutnik, Klaus Greff, Faustino Gomez, and Juergen Schmidhuber. A clockwork rnn. arXiv preprint arXiv:1402.3511, 2014.","title":"A clockwork rnn","authors":"Koutnik","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lin et al., 1996] Tsungnan Lin, Bill G Horne, Peter Tino, and C Lee Giles. Learning long-term dependencies in narx recurrent neural networks. IEEE Transactions on Neural Networks, 7(6):1329-1338, 1996.","title":"Learning long-term dependencies in narx recurrent neural networks","authors":"Lin","year":"1996","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2016] Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Recurrent neural network for text classification with multi-task learning. In Proceedings of the Twenty- Fifth International Joint Conference on Artificial Intelli- gence, IJCAI 2016, New York, NY, USA, 9-15 July 2016, pages 2873-2879, 2016.","title":"Recurrent neural network for text classification with multi-task learning","authors":"Pengfei Liu, Xipeng Qiu, Xuanjing Huang","year":"2016","venue":"Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016","doi":null,"url":null,"internal_url":"/proceedings/2016/408"},{"position":13,"raw":"[Luo et al., 2018] Ling Luo, Xiang Ao, Feiyang Pan, Jin Wang, Tong Zhao, Ningzi Yu, and Qing He. Beyond polar- ity: Interpretable financial sentiment analysis with hierar- chical query-driven attention. In IJCAI, pages 4244-4250, 2018.","title":"Beyond polarity: Interpretable financial sentiment analysis with hierarchical query-driven attention","authors":"Ling Luo, Xiang Ao, Feiyang Pan, Jin Wang, Tong Zhao, Ningzi Yu, Qing He","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/590"},{"position":14,"raw":"[Maas et al., 2011] Andrew L Maas, Raymond E Daly, Pe- ter T Pham, Dan Huang, Andrew Y Ng, and Christo- pher Potts. Learning word vectors for sentiment analy- sis. In Proceedings of the 49th annual meeting of the as- sociation for computational linguistics: Human language technologies-volume 1, pages 142-150. Association for Computational Linguistics, 2011.","title":"Learning word vectors for sentiment analysis","authors":"Raymond E Andrew L Maas, Peter T Daly, Dan Pham, Andrew Y Huang, Christopher Ng, Potts","year":"2011","venue":"Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mozer, 1992] Michael C Mozer. Induction of multiscale temporal structure. In Advances in neural information pro- cessing systems, pages 275-282, 1992.","title":"Induction of multiscale temporal structure","authors":"C Michael, Mozer","year":"1992","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pascanu et al., 2013] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International Conference on Machine Learning, pages 1310-1318, 2013.","title":"On the difficulty of training recurrent neural networks","authors":"Razvan Pascanu, Tomas Mikolov, Yoshua Bengio","year":"2013","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Schmidhuber, 1991] Jürgen Schmidhuber. Neural sequence chunkers. 1991.","title":"Neural sequence chunkers","authors":"Jürgen Schmidhuber","year":"1991","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Schmidhuber, 1992] Jürgen Schmidhuber. Learning com- plex, extended sequences using the principle of history compression. Neural Computation, 4(2):234-242, 1992.","title":"Learning complex, extended sequences using the principle of history compression","authors":"Jürgen Schmidhuber","year":"1992","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Socher et al., 2013] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for seman- tic compositionality over a sentiment treebank. In Pro- ceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642, 2013.","title":"Recursive deep models for semantic compositionality over a sentiment treebank","authors":"Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, Christopher Potts","year":"2013","venue":"Proceedings of the 2013 conference on empirical methods in natural language processing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Soltani and Jiang, 2016] Rohollah Soltani and Hui Jiang. Higher order recurrent neural networks. arXiv preprint arXiv:1605.00064, 2016.","title":"Higher order recurrent neural networks","authors":"Jiang Soltani, Rohollah Soltani, Hui Jiang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tai et al., 2015] Kai Sheng Tai, Richard Socher, and Christopher D. Manning. Improved semantic represen- tations from tree-structured long short-term memory net- works. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th In- ternational Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers), pages 1556-1566. As- sociation for Computational Linguistics, 2015.","title":"Improved semantic representations from tree-structured long short-term memory networks","authors":"Kai Sheng, Tai, Richard Socher, Christopher D Manning","year":"2015","venue":"Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yang et al., 2016] Zichao Yang, Diyi Yang, Chris Dyer, Xi- aodong He, Alex Smola, and Eduard Hovy. Hierarchical attention networks for document classification. In Pro- ceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1480-1489, 2016.","title":"Hierarchical attention networks for document classification","authors":"Yang","year":"2016","venue":"Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2015] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Advances in neural information process- ing systems, pages 649-657, 2015.","title":"Character-level convolutional networks for text classification","authors":"Xiang Zhang, Junbo Zhao, Yann Lecun","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zilly et al., 2016] Julian Georg Zilly, Rupesh Kumar Sri- vastava, Jan Koutník, and Jürgen Schmidhuber. Recur- rent highway networks. arXiv preprint arXiv:1607.03474, 2016.","title":"Recurrent highway networks","authors":"Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, Jürgen Schmidhuber","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}