{"type":"ScholarlyArticle","title":"Reading selectively via Binary Input Gated Recurrent Unit","authors":[{"name":"Zhe Li","slug":"zhe-li","given":"Zhe","family":"Li","affiliation":"National Laboratory of Pattern Recognition , Institute of Automation , Chinese Academy of Sciences"},{"name":"Peisong Wang","slug":"peisong-wang","given":"Peisong","family":"Wang","affiliation":"National Laboratory of Pattern Recognition , Institute of Automation , Chinese Academy of Sciences; Center for Excellence in Brain Science and Intelligence Technology"},{"name":"Hanqing Lu","slug":"hanqing-lu","given":"Hanqing","family":"Lu","affiliation":"National Laboratory of Pattern Recognition , Institute of Automation , Chinese Academy of Sciences"},{"name":"Jian Cheng","slug":"jian-cheng","given":"Jian","family":"Cheng","affiliation":"National Laboratory of Pattern Recognition , Institute of Automation , Chinese Academy of Sciences; Center for Excellence in Brain Science and Intelligence Technology"}],"abstract":"Recurrent Neural Networks (RNNs) have shown great promise in sequence modeling tasks. Gated Recurrent Unit (GRU) is one of the most used recurrent structures, which makes a good trade-off between performance and time spent. However, its practical implementation based on soft gates only partially achieves the goal to control information flow. We can hardly explain what the network has learnt internally. Inspired by human reading, we introduce binary input gated recurrent unit (BIGRU), a GRU based model using a binary input gate instead of the reset gate in GRU. By doing so, our model can read selectively during interference. In our experiments, we show that BIGRU mainly ignores the conjunctions, adverbs and articles that do not make a big difference to the document understanding, which is meaningful for us to further understand how the network works. In addition, due to reduced interference from redundant information, our model achieves better performances than baseline GRU in all the testing tasks.","doi":"10.24963/ijcai.2019/705","url":"https://ijcai.org/proceedings/2019/705","pdf":"https://ijcai.org/proceedings/2019/0705.pdf","markdown":"https://ijcai.org/proceedings/2019/705.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/705","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Natural Language Processing","track":"Main track","number":705,"pages":{"first":5074,"last":5080},"keywords":["Machine Learning: Interpretability","Natural Language Processing: Text Classification"],"videos":[],"references":[{"position":1,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. ICLR, 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2013] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.","title":"Estimating or propagating gradients through stochastic neurons for conditional computation","authors":"Yoshua Bengio, Nicholas Léonard, Aaron Courville","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Campos Camunez et al., 2018] Victor Campos Camunez, Brendan Jou, Xavier Giró Nieto, Jordi Torres Viñals, and Shih-Fu Chang. Skip rnn: learning to skip state updates in recurrent neural networks. In Sixth International Con- ference on Learning Representations: Monday April 30- Thursday May 03, 2018, Vancouver Convention Center, Vancouver:[proceedings], pages 1-17, 2018.","title":"Skip rnn: learning to skip state updates in recurrent neural networks","authors":"Campos Camunez","year":"2018","venue":"Sixth International Conference on Learning Representations: Monday April 30-Thursday","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cheng et al., 2018a] Jian Cheng, Pei-song Wang, Gang Li, Qing-hao Hu, and Han-qing Lu. Recent advances in effi- cient computation of deep convolutional neural networks. Frontiers of Information Technology & Electronic Engi- neering, 19(1):64-77, 2018.","title":"Recent advances in efficient computation of deep convolutional neural networks","authors":"Cheng","year":"2018","venue":"Frontiers of Information Technology & Electronic Engineering","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cheng et al., 2018b] Jian Cheng, Jiaxiang Wu, Cong Leng, Yuhang Wang, and Qinghao Hu. Quantized CNN: A uni- fied approach to accelerate and compress convolutional networks. IEEE Trans. Neural Netw. Learning Syst., 29(10):4730-4743, 2018.","title":"Quantized CNN: A unified approach to accelerate and compress convolutional networks","authors":"Cheng","year":"2018","venue":"IEEE Trans. Neural Netw. Learning Syst","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP, 2014.","title":"Learning phrase representations using rnn encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Courbariaux et al., 2016] Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Ben- gio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint arXiv:1602.02830, 2016.","title":"Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1","authors":"Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, Yoshua Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Graves et al., 2013] Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recur- rent neural networks. In Acoustics, speech and signal pro- cessing (icassp), 2013 ieee international conference on, pages 6645-6649. IEEE, 2013.","title":"Speech recognition with deep recurrent neural networks","authors":"Alex Graves, Abdel-Rahman Mohamed, Geoffrey Hinton","year":"2013","venue":"Acoustics, speech and signal processing (icassp), 2013 ieee international conference on","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hansen et al., 2018] Christian Hansen, Casper Hansen, Stephen Alstrup, Jakob Grue Simonsen, and Christina Li- oma. Neural speed reading with structural-jump-lstm. ICLR, 2018.","title":"Neural speed reading with structural-jump-lstm","authors":"Hansen","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He and Cheng, 2018] Xiangyu He and Jian Cheng. Learn- ing compression from limited unlabeled data. In Computer Vision -ECCV 2018 -15th European Conference, Mu- nich, Germany, September 8-14, 2018, Proceedings, Part I, pages 778-795, 2018.","title":"Learning compression from limited unlabeled data","authors":"Cheng ; ] Xiangyu He, Jian He, Cheng","year":"2018","venue":"Computer Vision -ECCV 2018 -15th European Conference","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hochreiter and Schmidhuber, 1997a] Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural Computation, 9(1):1-42, 1997.","title":"Flat minima","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hochreiter and Schmidhuber, 1997b] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hu et al., 2018] Qinghao Hu, Peisong Wang, and Jian Cheng. From hashing to cnns: Training binary weight net- works via hashing. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Or- leans, Louisiana, USA, February 2-7, 2018, pages 3247- 3254, 2018.","title":"From hashing to cnns: Training binary weight networks via hashing","authors":"Hu","year":"2018","venue":"AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Le and Mikolov, 2014] Quoc Le and Tomas Mikolov. Dis- tributed representations of sentences and documents. In International Conference on Machine Learning, pages 1188-1196, 2014.","title":"Quoc Le and Tomas Mikolov. Distributed representations of sentences and documents","authors":"Mikolov Le","year":"2014","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2018] Zhuohan Li, Di He, Fei Tian, Wei Chen, Tao Qin, Liwei Wang, and Tie-Yan Liu. Towards binary- valued gates for robust lstm training. arXiv preprint arXiv:1806.02988, 2018.","title":"Towards binaryvalued gates for robust lstm training","authors":"Zhuohan Li, Di He, Fei Tian, Wei Chen, Tao Qin, Liwei Wang, Tie-Yan Liu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Merity et al., 2017] Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing lstm lan- guage models. arXiv preprint arXiv:1708.02182, 2017.","title":"Regularizing and optimizing lstm language models","authors":"Stephen Merity, Nitish Shirish Keskar, Richard Socher","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Polyak and Juditsky, 1992] Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30(4):838-855, 1992.","title":"Acceleration of stochastic approximation by averaging","authors":"T Boris, Anatoli B Polyak, Juditsky","year":"1992","venue":"SIAM Journal on Control and Optimization","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Seo et al., 2018] Minjoon Seo, Sewon Min, Ali Farhadi, and Hannaneh Hajishirzi. Neural speed reading via skim- rnn. ICLR, 2018.","title":"Neural speed reading via skimrnn","authors":"Seo","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Socher et al., 2011] Richard Socher, Jeffrey Pennington, Eric H Huang, Andrew Y Ng, and Christopher D Man- ning. Semi-supervised recursive autoencoders for predict- ing sentiment distributions. In Proceedings of the confer- ence on empirical methods in natural language process- ing, pages 151-161. Association for Computational Lin- guistics, 2011.","title":"Semi-supervised recursive autoencoders for predicting sentiment distributions","authors":"Richard Socher, Jeffrey Pennington, Eric H Huang, Andrew Y Ng, Christopher D Manning","year":"2011","venue":"Proceedings of the conference on empirical methods in natural language processing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Villegas et al., 2017] Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, and Honglak Lee. Learning to generate long-term future via hierarchical pre- diction. arXiv preprint arXiv:1704.05831, 2017.","title":"Learning to generate long-term future via hierarchical prediction","authors":"Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, Honglak Lee","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang and Cheng, 2016] Peisong Wang and Jian Cheng. Accelerating convolutional neural networks for mobile ap- plications. In Proceedings of the 2016 ACM Conference on Multimedia Conference, MM 2016, Amsterdam, The Netherlands, October 15-19, 2016, pages 541-545, 2016.","title":"Accelerating convolutional neural networks for mobile applications","authors":"Wang, Cheng ; ] Peisong Wang, Jian Cheng","year":"2016","venue":"Proceedings of the 2016 ACM Conference on Multimedia Conference, MM 2016","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Williams, 1992] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8(3-4):229-256, 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Williams Ronald","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yu et al., 2017] Adams Wei Yu, Hongrae Lee, and Quoc Le. Learning to skim text. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1880-1890, 2017.","title":"Learning to skim text","authors":"Adams Wei, Yu, Hongrae Lee, Quoc Le","year":"2017","venue":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zaremba et al., 2014] Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regulariza- tion. arXiv preprint arXiv:1409.2329, 2014.","title":"Recurrent neural network regularization","authors":"Wojciech Zaremba, Ilya Sutskever, Oriol Vinyals","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}