{"type":"ScholarlyArticle","title":"Multiple-Weight Recurrent Neural Networks","authors":[{"name":"Zhu Cao","slug":"zhu-cao","given":"Zhu","family":"Cao","affiliation":"IIIS , Tsinghua University , Beijing , China"},{"name":"Linlin Wang","slug":"linlin-wang","given":"Linlin","family":"Wang","affiliation":"IIIS , Tsinghua University , Beijing , China"},{"name":"Gerard de Melo","slug":"gerard-de-melo","given":"Gerard","family":"de Melo","affiliation":"Rutgers University , New Brunswick , NJ , USA"}],"abstract":"Recurrent neural networks (RNNs) have enjoyed great success in speech recognition, natural language processing, etc. Many variants of RNNs have been proposed, including vanilla RNNs, LSTMs, and GRUs. However, current architectures are not particularly  adept at dealing with tasks involving multi-faceted contents. In this work, we solve this problem by proposing Multiple-Weight RNNs and LSTMs, which rely on multiple weight matrices in an attempt to mimic the human ability of switching between contexts. We present a framework for adapting RNN-based models and analyze the properties of this approach. Our detailed experimental results show that our model outperforms previous work across a range of different tasks and datasets.","doi":"10.24963/ijcai.2017/205","url":"https://ijcai.org/proceedings/2017/205","pdf":"https://ijcai.org/proceedings/2017/0205.pdf","markdown":"https://ijcai.org/proceedings/2017/205.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/205","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":205,"pages":{"first":1483,"last":1489},"keywords":["Machine Learning: Neural Networks","Machine Learning: New Problems","Machine Learning: Deep Learning","Natural Language Processing: Natural Language Processing"],"videos":[],"references":[{"position":1,"raw":"[Bengio and others, 2009] Yoshua Bengio et al. Learning deep architectures for ai. Foundations and trends R in Ma- chine Learning, 2(1):1-127, 2009.","title":"Learning deep architectures for ai","authors":"Yoshua Bengio","year":"2009","venue":"Foundations and trends R in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 1994] Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157-166, 1994.","title":"Learning long-term dependencies with gradient descent is difficult","authors":"Yoshua Bengio, Patrice Simard, Paolo Frasconi","year":"1994","venue":"IEEE transactions on neural networks","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bunescu and Mooney, 2005] Razvan C. Bunescu and Ray- mond J. Mooney. Subsequence kernels for relation extrac- tion. Advances in Neural Information Processing Systems, pages 171-178, 2005.","title":"Subsequence kernels for relation extraction","authors":"C Razvan, Raymond J Bunescu, Mooney","year":"2005","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Caruana, 1997] Rich Caruana. Multitask learning. Machine Learning, 28(1):41-75, 1997.","title":"Multitask learning","authors":"Caruana","year":"1997","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2016] Danqi Chen, Jason Bolton, and Christo- pher D Manning. A thorough examination of the cnn/daily mail reading comprehension task. arXiv preprint arXiv:1606.02858, 2016.","title":"A thorough examination of the cnn/daily mail reading comprehension task","authors":"Chen","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the proper- ties of neural machine translation: Encoder-decoder ap- proaches. arXiv preprint arXiv:1409.1259, 2014.","title":"On the properties of neural machine translation: Encoder-decoder approaches","authors":"Cho","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chung et al., 2014] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evalua- tion of gated recurrent neural networks on sequence mod- eling. arXiv preprint arXiv:1412.3555, 2014.","title":"Empirical evaluation of gated recurrent neural networks on sequence modeling","authors":"Chung","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Das and Bandyopadhyay, 2010] D. Das and S. Bandyopad- hyay. Sentence level emotion tagging on blog and news corpora : Journal of intelligent systems. Journal of Intelli- gent Systems, 19(2):145-162, 2010.","title":"Sentence level emotion tagging on blog and news corpora : Journal of intelligent systems","authors":"D Das, S Bandyopadhyay","year":"2010","venue":"Journal of Intelligent Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Degand and Bestgen, 2003] Liesbeth Degand and Yves Bestgen. Towards automatic retrieval of idioms in french newspaper corpora. Literary & Linguistic Computing, 18:249-259, 2003.","title":"Towards automatic retrieval of idioms in french newspaper corpora","authors":"Bestgen Degand, Liesbeth Degand, Yves Bestgen","year":"2003","venue":"Literary & Linguistic Computing","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dhingra et al., 2016] Bhuwan Dhingra, Hanxiao Liu, William W Cohen, and Ruslan Salakhutdinov. Gated- attention readers for text comprehension. arXiv preprint arXiv:1606.01549, 2016.","title":"Gatedattention readers for text comprehension","authors":"Bhuwan Dhingra, Hanxiao Liu, William W Cohen, Ruslan Salakhutdinov","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Eric and Manning, 2017] Mihail Eric and Christopher D Manning. A copy-augmented sequence-to-sequence archi- tecture gives good performance on task-oriented dialogue. arXiv preprint arXiv:1701.04024, 2017.","title":"A copy-augmented sequence-to-sequence architecture gives good performance on task-oriented dialogue","authors":"Mihail Eric, Christopher D Manning","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Fausett, 1994] Laurene V Fausett. Fundamentals of neural networks. Prentice-Hall, 1994.","title":"Fundamentals of neural networks","authors":"Laurene V Fausett","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Glorot et al., 2011] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Domain adaptation for large-scale sen- timent classification: A deep learning approach. In Pro- ceedings of the 28th international conference on machine learning (ICML-11), pages 513-520, 2011.","title":"Domain adaptation for large-scale sentiment classification: A deep learning approach","authors":"Xavier Glorot, Antoine Bordes, Yoshua Bengio","year":"2011","venue":"Proceedings of the 28th international conference on machine learning (ICML-11)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Greff et al., 2016] Klaus Greff, Rupesh K Srivastava, Jan Koutník, Bas R Steunebrink, and Jürgen Schmidhuber. Lstm: A search space odyssey. IEEE transactions on neu- ral networks and learning systems, 2016.","title":"Lstm: A search space odyssey","authors":"Klaus Greff, K Rupesh, Jan Srivastava, Koutník, Jürgen Bas R Steunebrink, Schmidhuber","year":"2016","venue":"IEEE transactions on neural networks and learning systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hermann et al., 2015] Karl Moritz Hermann, Tomas Ko- cisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Advances in Neural Informa- tion Processing Systems, pages 1693-1701, 2015.","title":"Teaching machines to read and comprehend","authors":"Hermann","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Jozefowicz et al., 2015] Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever. An empirical exploration of recurrent network architectures. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning -Volume 37, ICML'15, pages 2342-2350. JMLR.org, 2015.","title":"An empirical exploration of recurrent network architectures","authors":"Rafal Jozefowicz, Wojciech Zaremba, Ilya Sutskever","year":"2015","venue":"Proceedings of the 32Nd International Conference on International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kadlec et al., 2016] Rudolf Kadlec, Martin Schmid, On- drej Bajgar, and Jan Kleindienst. Text understanding with the attention sum reader network. arXiv preprint arXiv:1603.01547, 2016.","title":"Text understanding with the attention sum reader network","authors":"Kadlec","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Le et al., 2015] Quoc V Le, Navdeep Jaitly, and Geoffrey E Hinton. A simple way to initialize recurrent networks of rectified linear units. arXiv preprint arXiv:1504.00941, 2015.","title":"A simple way to initialize recurrent networks of rectified linear units","authors":"Navdeep Quoc V Le, Geoffrey E Jaitly, Hinton","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu and Lane, 2017] Bing Liu and Ian Lane. Dialog context language modeling with recurrent neural networks. arXiv preprint arXiv:1701.04056, 2017.","title":"Dialog context language modeling with recurrent neural networks","authors":"Lane ; Liu, Bing Liu, Ian Lane","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Marcus et al., 1993] Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a large annotated corpus of english: the penn treebank. Computational Linguistics, 19(2):313-330, 1993.","title":"Building a large annotated corpus of english: the penn treebank","authors":"Mary Mitchell P Marcus, Ann Marcinkiewicz, Beatrice Santorini","year":"1993","venue":"Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Mikolov and Zweig, 2012] Tomas Mikolov and Geoffrey Zweig. Context dependent recurrent neural network lan- guage model. SLT, 12:234-239, 2012.","title":"Context dependent recurrent neural network language model","authors":"Zweig Mikolov","year":"2012","venue":"SLT","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Mikolov et al., 2010] Tomas Mikolov, Martin Karafit, Lukas Burget, Jan Cernock, and Sanjeev Khudanpur. Recurrent neural network based language model. In INTERSPEECH 2010, Conference of the International Speech Communication Association, Makuhari, Chiba, Japan, September, pages 1045-1048, 2010.","title":"Recurrent neural network based language model","authors":"Mikolov","year":"2010","venue":"INTERSPEECH 2010, Conference of the International Speech Communication Association","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013.","title":"Efficient estimation of word representations in vector space","authors":"Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Mikolov et al., 2014] Tomas Mikolov, Armand Joulin, Sumit Chopra, Michael Mathieu, and Marc'Aurelio Ran- zato. Learning longer memory in recurrent neural net- works. arXiv preprint arXiv:1412.7753, 2014.","title":"Tomas Mikolov","authors":"Mikolov","year":"2014","venue":"Learning longer memory in recurrent neural networks","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Onishi et al., 2016] Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. Who did what: A large-scale person-centered cloze dataset. arXiv preprint arXiv:1608.05457, 2016.","title":"Who did what: A large-scale person-centered cloze dataset","authors":"Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, David Mcallester","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Santos and Zadrozny, 2014] Cicero Nogueira Dos Santos and Bianca Zadrozny. Learning character-level represen- tations for part-of-speech tagging. In International Con- ference on Machine Learning, pages 1818-1826, 2014.","title":"Cicero Nogueira Dos Santos and Bianca Zadrozny. Learning character-level representations for part-of-speech tagging","authors":"Zadrozny Santos","year":"2014","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang and Cho, 2015] Tian Wang and Kyunghyun Cho. Larger-context language modelling. arXiv preprint arXiv:1511.03729, 2015.","title":"Larger-context language modelling","authors":"Wang, Cho, Tian Wang, Kyunghyun Cho","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2016] Linlin Wang, Zhu Cao, Gerard de Melo, and Zhiyuan Liu. Relation classification via multi-level attention CNNs. In Proceedings of ACL 2016, 2016.","title":"Relation classification via multi-level attention CNNs","authors":"Linlin Wang, Zhu Cao, Gerard De Melo, Zhiyuan Liu","year":"2016","venue":"Proceedings of ACL 2016","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Weissenborn and Rocktäschel, 2016] Dirk Weissenborn and Tim Rocktäschel. Mufuru: The multi-function recurrent unit. arXiv preprint arXiv:1606.03002, 2016.","title":"Mufuru: The multi-function recurrent unit","authors":"Rocktäschel Weissenborn","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zeiler, 2012] Matthew D. Zeiler. Adadelta: An adaptive learning rate method. Computer Science, 2012.","title":"Adadelta: An adaptive learning rate method","authors":"D Matthew, Zeiler","year":"2012","venue":"Computer Science","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2016] Dong Zhang, Shoushan Li, Hongling Wang, and Guodong Zhou. User classification with multi- ple textual perspectives. In Proceedings of COLING, pages 2112-2121, 2016.","title":"User classification with multiple textual perspectives","authors":"Dong Zhang, Shoushan Li, Hongling Wang, Guodong Zhou","year":"2016","venue":"Proceedings of COLING","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhou et al., 2016] Guo-Bing Zhou, Jianxin Wu, Chen-Lin Zhang, and Zhi-Hua Zhou. Minimal gated unit for recur- rent neural networks. International Journal of Automation and Computing, 13(3):226-234, 2016.","title":"Minimal gated unit for recurrent neural networks","authors":"Guo-Bing Zhou, Jianxin Wu, Chen-Lin Zhang, Zhi-Hua Zhou","year":"2016","venue":"International Journal of Automation and Computing","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}