{"type":"ScholarlyArticle","title":"Leap-LSTM: Enhancing Long Short-Term Memory for Text Categorization","authors":[{"name":"Ting Huang","slug":"ting-huang","given":"Ting","family":"Huang","affiliation":"Key Laboratory of Machine Perception (Ministry of Education) , School of Electronics Engineering and Computer Science , Peking University"},{"name":"Gehui Shen","slug":"gehui-shen","given":"Gehui","family":"Shen","affiliation":"Key Laboratory of Machine Perception (Ministry of Education) , School of Electronics Engineering and Computer Science , Peking University"},{"name":"Zhi-Hong Deng","slug":"zhi-hong-deng","given":"Zhi-Hong","family":"Deng","affiliation":"Key Laboratory of Machine Perception (Ministry of Education) , School of Electronics Engineering and Computer Science , Peking University"}],"abstract":"Recurrent Neural Networks (RNNs) are widely used in the field of natural language processing (NLP), ranging from text categorization to question answering and machine translation. However, RNNs generally read the whole text from beginning to end or vice versa sometimes, which makes it inefficient to process long texts. When reading a long document for a categorization task, such as topic categorization, large quantities of words are irrelevant and can be skipped. To this end, we propose  Leap-LSTM, an LSTM-enhanced model which dynamically leaps between words while reading texts. \r\nAt each step, we utilize several feature encoders to extract messages from preceding texts, following texts and the current word, and then determine whether to skip the current word. We evaluate Leap-LSTM on several text categorization tasks: sentiment analysis, news categorization, ontology classification and topic classification, with five benchmark data sets. The experimental results show that our model reads faster and predicts better than standard LSTM. Compared to previous models which can also skip words, our model achieves better trade-offs between performance and efficiency.","doi":"10.24963/ijcai.2019/697","url":"https://ijcai.org/proceedings/2019/697","pdf":"https://ijcai.org/proceedings/2019/0697.pdf","markdown":"https://ijcai.org/proceedings/2019/697.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/697","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Natural Language Processing","track":"Main track","number":697,"pages":{"first":5017,"last":5023},"keywords":["Natural Language Processing: Text Classification","Natural Language Processing: Natural Language Processing"],"videos":[],"references":[{"position":1,"raw":"[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine transla- tion by jointly learning to align and translate. CoRR, abs/1409.0473, 2014.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2013] Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.3432, 2013.","title":"Estimating or propagating gradients through stochastic neurons for conditional computation","authors":"Yoshua Bengio, Nicholas Léonard, Aaron C Courville","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Campos et al., 2017] Victor Campos, Brendan Jou, Xavier Giró i Nieto, Jordi Torres, and Shih-Fu Chang. Skip RNN: learning to skip state updates in recurrent neural networks. CoRR, abs/1708.06834, 2017.","title":"Skip RNN: learning to skip state updates in recurrent neural networks","authors":"Victor Campos, Brendan Jou, Xavier Giró I Nieto, Jordi Torres, Shih-Fu Chang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the proper- ties of neural machine translation: Encoder-decoder ap- proaches. In Proceedings of SSST@EMNLP 2014, Eighth Workshop on Syntax, Semantics and Structure in Statis- tical Translation, Doha, Qatar, 25 October 2014, pages 103-111, 2014.","title":"On the properties of neural machine translation: Encoder-decoder approaches","authors":"Cho","year":"2014","venue":"Proceedings of SSST@EMNLP 2014, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Jang et al., 2016] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. CoRR, abs/1611.01144, 2016.","title":"Categorical reparameterization with gumbel-softmax","authors":"Eric Jang, Shixiang Gu, Ben Poole","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jernite et al., 2016] Yacine Jernite, Edouard Grave, Armand Joulin, and Tomas Mikolov. Variable computation in re- current neural networks. CoRR, abs/1611.06188, 2016.","title":"Variable computation in recurrent neural networks","authors":"Yacine Jernite, Edouard Grave, Armand Joulin, Tomas Mikolov","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kim, 2014] Yoon Kim. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Confer- ence on Empirical Methods in Natural Language Process- ing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1746-1751, 2014.","title":"Convolutional neural networks for sentence classification","authors":"Yoon Kim","year":"2014","venue":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kingma and Ba, 2014] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Koutník et al., 2014] Jan Koutník, Klaus Greff, Faustino J. Gomez, and Jürgen Schmidhuber. A clockwork RNN. In Proceedings of the 31th International Conference on Ma- chine Learning, ICML 2014, Beijing, China, 21-26 June 2014, pages 1863-1871, 2014.","title":"A clockwork RNN","authors":"Jan Koutník, Klaus Greff, Faustino J Gomez, Jürgen Schmidhuber","year":"2014","venue":"Proceedings of the 31th International Conference on Machine Learning, ICML 2014","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2017] Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Adaptive semantic compositionality for sentence modelling. In IJCAI, pages 4061-4067. ijcai.org, 2017.","title":"Adaptive semantic compositionality for sentence modelling","authors":"Pengfei Liu, Xipeng Qiu, Xuanjing Huang","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/567"},{"position":12,"raw":"[Maddison et al., 2016] Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A con- tinuous relaxation of discrete random variables. CoRR, abs/1611.00712, 2016.","title":"The concrete distribution: A continuous relaxation of discrete random variables","authors":"Chris J Maddison, Andriy Mnih, Yee Whye Teh","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Neil et al., 2016] Daniel Neil, Michael Pfeiffer, and Shih- Chii Liu. Phased LSTM: accelerating recurrent network training for long or event-based sequences. In Advances in Neural Information Processing Systems 29: Annual Con- ference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 3882- 3890, 2016.","title":"Phased LSTM: accelerating recurrent network training for long or event-based sequences","authors":"Daniel Neil, Michael Pfeiffer, Shih-Chii Liu","year":"2016","venue":"Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vec- tors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1532-1543, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Schwenk et al., 2017] Holger Schwenk, Loïc Barrault, Alexis Conneau, and Yann LeCun. Very deep convolu- tional networks for text classification. In Proceedings of the 15th Conference of the European Chapter of the Asso- ciation for Computational Linguistics, EACL 2017, Valen- cia, Spain, April 3-7, 2017, Volume 1: Long Papers, pages 1107-1116, 2017.","title":"Very deep convolutional networks for text classification","authors":"Holger Schwenk, Loïc Barrault, Alexis Conneau, Yann Lecun","year":"2017","venue":"Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[See et al., 2017] Abigail See, Peter J. Liu, and Christo- pher D. Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics, ACL 2017, Vancouver, Canada, July 30 -August 4, Volume 1: Long Papers, pages 1073-1083, 2017.","title":"Get to the point: Summarization with pointer-generator networks","authors":"Abigail See, Peter J Liu, Christopher D Manning","year":"2017","venue":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Seo et al., 2016] Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. Bidirectional attention flow for machine comprehension. CoRR, abs/1611.01603, 2016.","title":"Bidirectional attention flow for machine comprehension","authors":"Seo","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Seo et al., 2017] Min Joon Seo, Sewon Min, Ali Farhadi, and Hannaneh Hajishirzi. Neural speed reading via skim- rnn. CoRR, abs/1711.02085, 2017.","title":"Neural speed reading via skimrnn","authors":"Seo","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Serban et al., 2015] Iulian Vlad Serban, Alessandro Sor- doni, Yoshua Bengio, Aaron C. Courville, and Joelle Pineau. Hierarchical neural network generative models for movie dialogues. CoRR, abs/1507.04808, 2015.","title":"Hierarchical neural network generative models for movie dialogues","authors":"Iulian Vlad Serban, Alessandro Sordoni, Yoshua Bengio, Aaron C Courville, Joelle Pineau","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shen et al., 2017] Yelong Shen, Po-Sen Huang, Jianfeng Gao, and Weizhu Chen. Reasonet: Learning to stop read- ing in machine comprehension. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, NS, Canada, August 13 -17, 2017, pages 1047-1055, 2017.","title":"Reasonet: Learning to stop reading in machine comprehension","authors":"Shen","year":"2017","venue":"Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Williams, 1992] Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine Learning, 8(3-4):229-256, 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Ronald J Williams","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yogatama et al., 2017] Dani Yogatama, Chris Dyer, Wang Ling, and Phil Blunsom. Generative and discriminative text classification with recurrent neural networks. CoRR, abs/1703.01898, 2017.","title":"Generative and discriminative text classification with recurrent neural networks","authors":"Dani Yogatama, Chris Dyer, Wang Ling, Phil Blunsom","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yu et al., 2017] Adams Wei Yu, Hongrae Lee, and Quoc V. Le. Learning to skim text. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics, ACL 2017, Vancouver, Canada, July 30 -August 4, Volume 1: Long Papers, pages 1880-1890, 2017.","title":"Learning to skim text","authors":"Adams Wei, Yu, Hongrae Lee, V Quoc, Le","year":"2017","venue":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2015] Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Advances in Neural Information Pro- cessing Systems 2015, pages 649-657, 2015.","title":"Character-level convolutional networks for text classification","authors":"Xiang Zhang, Junbo, Jake Zhao, Yann Lecun","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}