{"type":"ScholarlyArticle","title":"ARMIN: Towards a More Efficient and Light-weight Recurrent Memory Network","authors":[{"name":"Zhangheng Li","slug":"zhangheng-li","given":"Zhangheng","family":"Li","affiliation":"School of Electronic and Computer Engineering , Peking University; Peng Cheng Laboratory"},{"name":"Jia-Xing Zhong","slug":"jia-xing-zhong","given":"Jia-Xing","family":"Zhong","affiliation":"School of Electronic and Computer Engineering , Peking University; Peng Cheng Laboratory"},{"name":"Jingjia Huang","slug":"jingjia-huang","given":"Jingjia","family":"Huang","affiliation":"School of Electronic and Computer Engineering , Peking University; Peng Cheng Laboratory"},{"name":"Tao Zhang","slug":"tao-zhang","given":"Tao","family":"Zhang","affiliation":"School of Electronic and Computer Engineering , Peking University; Peng Cheng Laboratory"},{"name":"Thomas Li","slug":"thomas-li","given":"Thomas","family":"Li","affiliation":"School of Electronic and Computer Engineering , Peking University"},{"name":"Ge Li","slug":"ge-li","given":"Ge","family":"Li","affiliation":"School of Electronic and Computer Engineering , Peking University; Peng Cheng Laboratory"}],"abstract":"In recent years, memory-augmented neural networks(MANNs) have shown promising power to enhance the memory ability of neural networks for sequential processing tasks. However, previous MANNs suffer from complex memory addressing mechanism, making them relatively hard to train and causing computational overheads. Moreover, many of them reuse the classical RNN structure such as LSTM for memory processing, causing inefficient exploitations of memory information. In this paper, we introduce a novel MANN, the Auto-addressing and Recurrent Memory Integrating Network (ARMIN) to address these issues. The ARMIN only utilizes hidden state h_t for automatic memory addressing, and uses a novel RNN cell for refined integration of memory information. Empirical results on a variety of experiments demonstrate that the ARMIN is more light-weight and efficient compared to existing memory networks. Moreover, we demonstrate that the ARMIN can achieve much lower computational overhead than vanilla LSTM while keeping similar performances. Codes are available on github.com/zoharli/armin.","doi":"10.24963/ijcai.2019/408","url":"https://ijcai.org/proceedings/2019/408","pdf":"https://ijcai.org/proceedings/2019/0408.pdf","markdown":"https://ijcai.org/proceedings/2019/408.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/408","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":408,"pages":{"first":2944,"last":2951},"keywords":["Multidisciplinary Topics and Applications: Autonomic Computing","Knowledge Representation and Reasoning: Logics for Knowledge Representation","Machine Learning: Deep Learning","Humans and AI: Cognitive Modeling","Machine Learning: Time-series;Data Streams"],"videos":[],"references":[{"position":1,"raw":"[Ba et al., 2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Ge- offrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.","title":"Layer normalization","authors":"Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E Hinton","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Campos et al., 2017] Víctor Campos, Brendan Jou, Xavier Giró-i Nieto, Jordi Torres, and Shih-Fu Chang. Skip rnn: Learning to skip state updates in recurrent neural net- works. arXiv preprint arXiv:1708.06834, 2017.","title":"Skip rnn: Learning to skip state updates in recurrent neural networks","authors":"Víctor Campos, Brendan Jou, Xavier Giró-I Nieto, Jordi Torres, Shih-Fu Chang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.","title":"Learning phrase representations using rnn encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chung et al., 2016] Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrent neural networks. arXiv preprint arXiv:1609.01704, 2016.","title":"Hierarchical multiscale recurrent neural networks","authors":"Chung","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Danihelka et al., 2016] Ivo Danihelka, Greg Wayne, Be- nigno Uria, Nal Kalchbrenner, and Alex Graves. Asso- ciative long short-term memory. In Proceedings of the 33rd International Conference on ICML-Volume 48, pages 1986-1994. JMLR. org, 2016.","title":"Associative long short-term memory","authors":"Ivo Danihelka, Greg Wayne, Benigno Uria, Nal Kalchbrenner, Alex Graves","year":"2016","venue":"Proceedings of the 33rd International Conference on ICML","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dyer et al., 2015] Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A Smith. Transition- based dependency parsing with stack long short-term memory. arXiv preprint arXiv:1505.08075, 2015.","title":"Transitionbased dependency parsing with stack long short-term memory","authors":"Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, Noah A Smith","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Elman, 1990] Jeffrey L Elman. Finding structure in time. Cognitive science, 14(2):179-211, 1990.","title":"Finding structure in time","authors":"Jeffrey L Elman","year":"1990","venue":"Cognitive science","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Grave et al., 2016] Edouard Grave, Armand Joulin, and Nicolas Usunier. Improving neural language models with a continuous cache. arXiv preprint arXiv:1612.04426, 2016.","title":"Improving neural language models with a continuous cache","authors":"Grave","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Graves et al., 2014] Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014.","title":"Neural turing machines","authors":"Alex Graves, Greg Wayne, Ivo Danihelka","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Graves et al., 2016] Alex Graves, Greg Wayne, Mal- colm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626):471, 2016.","title":"Hybrid computing using a neural network with dynamic external memory","authors":"Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Grefenstette et al., 2015] Edward Grefenstette, Karl Moritz Hermann, Mustafa Suleyman, and Phil Blunsom. Learn- ing to transduce with unbounded memory. In Advances in NIPS, pages 1828-1836, 2015.","title":"Learning to transduce with unbounded memory","authors":"Edward Grefenstette, Karl Moritz Hermann, Mustafa Suleyman, Phil Blunsom","year":"2015","venue":"Advances in NIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gulcehre et al., 2017] Caglar Gulcehre, Sarath Chandar, and Yoshua Bengio. Memory augmented neural net- works with wormhole connections. arXiv preprint arXiv:1701.08718, 2017.","title":"Memory augmented neural networks with wormhole connections","authors":"Caglar Gulcehre, Sarath Chandar, Yoshua Bengio","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gumbel, 1954] Emil Julius Gumbel. Statistical theory of extreme values and some practical applications: a series of lectures. Number 33. US Govt. Print. Office, 1954.","title":"Statistical theory of extreme values and some practical applications: a series of lectures","authors":"Emil Julius Gumbel, Gumbel","year":"1954","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ha et al., 2016] David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. arXiv preprint arXiv:1609.09106, 2016.","title":"Hypernetworks","authors":"David Ha, Andrew Dai, Quoc V Le","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Jang et al., 2016] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144, 2016.","title":"Categorical reparameterization with gumbel-softmax","authors":"Eric Jang, Shixiang Gu, Ben Poole","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Joulin and Mikolov, 2015] Armand Joulin and Tomas Mikolov. Inferring algorithmic patterns with stack- augmented recurrent nets. In Advances in neural information processing systems, pages 190-198, 2015.","title":"Armand Joulin and Tomas Mikolov. Inferring algorithmic patterns with stackaugmented recurrent nets","authors":"Mikolov Joulin","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ke et al., 2018] Nan Rosemary Ke, Anirudh Goyal, Olexa Bilaniuk, Jonathan Binas, Michael C Mozer, Chris Pal, and Yoshua Bengio. Sparse attentive backtracking: Tem- poral creditassignment through reminding. arXiv preprint arXiv:1809.03702, 2018.","title":"Sparse attentive backtracking: Temporal creditassignment through reminding","authors":"Nan Rosemary Ke, Anirudh Goyal, Olexa Bilaniuk, Jonathan Binas, C Michael, Chris Mozer, Yoshua Pal, Bengio","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Krueger et al., 2016] David Krueger, Tegan Maharaj, János Kramár, Mohammad Pezeshki, Nicolas Ballas, Nan Rose- mary Ke, Anirudh Goyal, Yoshua Bengio, Aaron Courville, and Chris Pal. Zoneout: Regularizing rnns by randomly preserving hidden activations. arXiv preprint arXiv:1606.01305, 2016.","title":"Zoneout: Regularizing rnns by randomly preserving hidden activations","authors":"David Krueger, Tegan Maharaj, János Kramár, Mohammad Pezeshki, Nicolas Ballas, Nan Rosemary Ke, Anirudh Goyal, Yoshua Bengio, Aaron Courville, Chris Pal","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kurach et al., 2015] Karol Kurach, Marcin Andrychowicz, and Ilya Sutskever. Neural random-access machines. arXiv preprint arXiv:1511.06392, 2015.","title":"Neural random-access machines","authors":"Karol Kurach, Marcin Andrychowicz, Ilya Sutskever","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Li et al., 2018a] Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao. Independently recurrent neural network (indrnn): Building a longer and deeper rnn. In Proceed- ings of the IEEE Conference on CVPR, pages 5457-5466, 2018.","title":"Independently recurrent neural network (indrnn): Building a longer and deeper rnn","authors":"Li","year":"2018","venue":"Proceedings of the IEEE Conference on CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Li et al., 2018b] Zhuohan Li, Di He, Fei Tian, Wei Chen, Tao Qin, Liwei Wang, and Tie-Yan Liu. Towards binary- valued gates for robust lstm training. arXiv preprint arXiv:1806.02988, 2018.","title":"Towards binaryvalued gates for robust lstm training","authors":"Li","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Maddison et al., 2016] Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00712, 2016.","title":"The concrete distribution: A continuous relaxation of discrete random variables","authors":"Andriy Chris J Maddison, Yee Whye Mnih, Teh","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Mujika et al., 2017] Asier Mujika, Florian Meier, and An- gelika Steger. Fast-slow recurrent neural networks. In Ad- vances in Neural Information Processing Systems, pages 5915-5924, 2017.","title":"Fast-slow recurrent neural networks","authors":"Asier Mujika, Florian Meier, Angelika Steger","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Rae et al., 2016] Jack Rae, Jonathan J Hunt, Ivo Danihelka, Timothy Harley, Andrew W Senior, Gregory Wayne, Alex Graves, and Tim Lillicrap. Scaling memory-augmented neural networks with sparse reads and writes. In Advances in NIPS, pages 3621-3629, 2016.","title":"Scaling memory-augmented neural networks with sparse reads and writes","authors":"Rae","year":"2016","venue":"Advances in NIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Rumelhart et al., 1986] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. nature, 323(6088):533, 1986.","title":"Learning representations by back-propagating errors","authors":"Geoffrey E David E Rumelhart, Ronald J Hinton, Williams","year":"1986","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Suarez, 2017] Joseph Suarez. Language modeling with re- current highway hypernetworks. In Advances in Neural Information Processing Systems, pages 3267-3276, 2017.","title":"Language modeling with recurrent highway hypernetworks","authors":"Joseph Suarez","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zoph and Le, 2016] Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578, 2016.","title":"Neural architecture search with reinforcement learning","authors":"Le Zoph","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}