{"type":"ScholarlyArticle","title":"Convolutional-Match Networks for Question Answering","authors":[{"name":"Spyridon Samothrakis","slug":"spyridon-samothrakis","given":"Spyridon","family":"Samothrakis","affiliation":"IADS , University of Essex , Wivenhoe Park , Colchester CO4 3SQ , UK"},{"name":"Tom Vodopivec","slug":"tom-vodopivec","given":"Tom","family":"Vodopivec","affiliation":"Faculty of Computer and Information Science , University of Ljubljana , Vecna pot 113 , Ljubljana , Slovenia"},{"name":"Michael Fairbank","slug":"michael-fairbank","given":"Michael","family":"Fairbank","affiliation":"CSEE , University of Essex , Wivenhoe Park , Colchester CO4 3SQ , UK"},{"name":"Maria Fasli","slug":"maria-fasli","given":"Maria","family":"Fasli","affiliation":"IADS , University of Essex , Wivenhoe Park , Colchester CO4 3SQ , UK"}],"abstract":"In this paper, we present a simple, yet effective, attention and memory mechanism that is reminiscent of Memory Networks and we demonstrate it in question-answering scenarios. Our mechanism is based on four simple premises: a) memories can be formed from word sequences by using convolutional networks; b) distance measurements can be taken at a neuronal level; c) a recursive softmax function can be used for attention; d) extensive weight sharing can help profoundly. We achieve state-of-the-art results in the bAbI tasks, outperforming both Memory Networks and the Differentiable Neural Computer, both in terms of accuracy and stability (i.e. variance) of results.","doi":"10.24963/ijcai.2017/374","url":"https://ijcai.org/proceedings/2017/374","pdf":"https://ijcai.org/proceedings/2017/0374.pdf","markdown":"https://ijcai.org/proceedings/2017/374.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/374","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":374,"pages":{"first":2686,"last":2692},"keywords":["Machine Learning: Neural Networks","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Bordes et al., 2015] Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. Large-scale simple question answering with memory networks. CoRR, abs/1506.02075, 2015.","title":"Large-scale simple question answering with memory networks","authors":"Antoine Bordes, Nicolas Usunier, Sumit Chopra, Jason Weston","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Collobert et al., 2011] Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12(Aug):2493-2537, 2011.","title":"Natural language processing (almost) from scratch","authors":"Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, Pavel Kuksa","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[French, 1999] Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128-135, 1999.","title":"Catastrophic forgetting in connectionist networks","authors":"M Robert, French","year":"1999","venue":"Trends in cognitive sciences","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Glorot and Bengio, 2010] Xavier Glorot and Yoshua Ben- gio. Understanding the difficulty of training deep feed- forward neural networks. In International conference on artificial intelligence and statistics, pages 249-256, 2010.","title":"Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks","authors":"Bengio Glorot","year":"2010","venue":"International conference on artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Graves et al., 2014] Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. CoRR, abs/1410.5401, 2014.","title":"Neural turing machines","authors":"Alex Graves, Greg Wayne, Ivo Danihelka","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Graves et al., 2016] Alex Graves, Greg Wayne, Mal- colm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626):471-476, 2016.","title":"Hybrid computing using a neural network with dynamic external memory","authors":"Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. The Inter- national Conference on Learning Representations (ICLR), abs/1412.6980, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"The International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kumar et al., 2015] Ankit Kumar, Ozan Irsoy, Peter On- druska, James Bradbury Mohit Iyyer, Ishaan Gulrajani, and Richard Socher. Ask me anything: Dynamic mem- ory networks for natural language processing. In Neural Information Processing Systems Workshop on Reasoning, Memory and Attention, 2015.","title":"Ask me anything: Dynamic memory networks for natural language processing","authors":"Ankit Kumar, Ozan Irsoy, Peter Ondruska, James Bradbury, Mohit Iyyer, Ishaan Gulrajani, Richard Socher","year":"2015","venue":"Neural Information Processing Systems Workshop on Reasoning, Memory and Attention","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Luong et al., 2015] Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention-based neural machine translation. In Proceed- ings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412-1421, Lisbon, Portugal, September 2015. Association for Computational Linguistics.","title":"Effective approaches to attention-based neural machine translation","authors":"Minh-Thang Luong, Hieu Pham, Christopher D Manning","year":"2015","venue":"Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Maas et al., 2013] Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlinearities improve neural net- work acoustic models. In Proc. ICML, volume 30, 2013.","title":"Rectifier nonlinearities improve neural network acoustic models","authors":"Awni Y Andrew L Maas, Andrew Y Hannun, Ng","year":"2013","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In HLT-NAACL, pages 746-751, 2013.","title":"Linguistic regularities in continuous space word representations","authors":"Mikolov","year":"2013","venue":"HLT-NAACL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Peng et al., 2015] Baolin Peng, Zhengdong Lu, Hang Li, and Kam-Fai Wong. Towards neural network-based rea- soning. NIPS RAM Workshop, 2015.","title":"Towards neural network-based reasoning","authors":"Baolin Peng, Zhengdong Lu, Hang Li, Kam-Fai Wong","year":"2015","venue":"NIPS RAM Workshop","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Samothrakis et al., 2016] Spyridon Samothrakis, Tom Vodopivec, Maria Fasli, and Michael Fairbank. Match memory recurrent networks. In Neural Networks (IJCNN), 2016 International Joint Conference on, pages 1339-1346. IEEE, 2016.","title":"Match memory recurrent networks","authors":"Spyridon Samothrakis, Tom Vodopivec, Maria Fasli, Michael Fairbank","year":"2016","venue":"Neural Networks (IJCNN), 2016 International Joint Conference on","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi- nov. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Re- search, 15(1):1929-1958, 2014.","title":"Dropout: A simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sukhbaatar et al., 2015] Sainbayar Sukhbaatar, Jason We- ston, Rob Fergus, et al. End-to-end memory networks. In Advances in Neural Information Processing Systems, pages 2431-2439, 2015.","title":"End-to-end memory networks","authors":"Sukhbaatar","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tibshirani, 1996] Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B (Methodological), pages 267-288, 1996.","title":"Regression shrinkage and selection via the lasso","authors":"Robert Tibshirani","year":"1996","venue":"Journal of the Royal Statistical Society. Series B (Methodological)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Weston et al., 2015] Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. Towards ai-complete ques- tion answering: A set of prerequisite toy tasks. CoRR, abs/1502.05698, 2015.","title":"Towards ai-complete question answering: A set of prerequisite toy tasks","authors":"Jason Weston, Antoine Bordes, Sumit Chopra, Tomas Mikolov","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Xiong et al., 2016] Caiming Xiong, Stephen Merity, and Richard Socher. Dynamic memory networks for visual and textual question answering. In Maria Florina Bal- can and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, vol- ume 48 of Proceedings of Machine Learning Research, pages 2397-2406, New York, New York, USA, 20-22 Jun 2016. PMLR.","title":"Dynamic memory networks for visual and textual question answering","authors":"Xiong","year":"2016","venue":"Proceedings of The 33rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xu et al., 2015a] Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in con- volutional network. In ICML Deep Learning Workshop, 2015.","title":"Empirical evaluation of rectified activations in convolutional network","authors":"Xu","year":"2015","venue":"ICML Deep Learning Workshop","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Xu et al., 2015b] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdi- nov, Richard S. Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual at- tention. In Proceedings of the 32nd International Con- ference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, pages 2048-2057, 2015.","title":"Show, attend and tell: Neural image caption generation with visual attention","authors":"Xu","year":"2015","venue":"Proceedings of the 32nd International Conference on Machine Learning, ICML 2015","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yin et al., 2015] Pengcheng Yin, Zhengdong Lu, Hang Li, and Ben Kao. Neural enquirer: Learning to query tables with natural language. arXiv preprint arXiv:1512.00965, 2015.","title":"Neural enquirer: Learning to query tables with natural language","authors":"Pengcheng Yin, Zhengdong Lu, Hang Li, Ben Kao","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}