{"type":"ScholarlyArticle","title":"Multiway Attention Networks for Modeling Sentence Pairs","authors":[{"name":"Chuanqi Tan","slug":"chuanqi-tan","given":"Chuanqi","family":"Tan","affiliation":"† State Key Laboratory of Software Development Environment , Beihang University , China"},{"name":"Furu Wei","slug":"furu-wei","given":"Furu","family":"Wei","affiliation":""},{"name":"Wenhui Wang","slug":"wenhui-wang","given":"Wenhui","family":"Wang","affiliation":"+ Peking University , Beijing , China"},{"name":"Weifeng Lv","slug":"weifeng-lv","given":"Weifeng","family":"Lv","affiliation":"† State Key Laboratory of Software Development Environment , Beihang University , China"},{"name":"Ming Zhou","slug":"ming-zhou","given":"Ming","family":"Zhou","affiliation":"‡ Microsoft Research , Beijing , China"}],"abstract":"Modeling sentence pairs plays the vital role for judging the relationship between two sentences, such as paraphrase identification, natural language inference, and answer sentence selection. Previous work achieves very promising results using neural networks with attention mechanism. In this paper, we propose the multiway attention networks which employ multiple attention functions to match sentence pairs under the matching-aggregation framework. Specifically, we design four attention functions to match words in corresponding sentences. Then, we aggregate the matching information from each function, and combine the information from all functions to obtain the final representation. Experimental results demonstrate that the proposed multiway attention networks improve the result on the Quora Question Pairs, SNLI, MultiNLI, and answer sentence selection task on the SQuAD dataset.","doi":"10.24963/ijcai.2018/613","url":"https://ijcai.org/proceedings/2018/613","pdf":"https://ijcai.org/proceedings/2018/0613.pdf","markdown":"https://ijcai.org/proceedings/2018/613.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/613","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Natural Language Processing","track":"Main track","number":613,"pages":{"first":4411,"last":4417},"keywords":["Natural Language Processing: Text Classification","Natural Language Processing: Question Answering"],"videos":[],"references":[{"position":1,"raw":"[Bowman et al., 2015] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In EMNLP, 2015.","title":"A large annotated corpus for learning natural language inference","authors":"R Samuel, Gabor Bowman, Christopher Angeli, Christopher D Potts, Manning","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bowman et al., 2016] R. Samuel Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, D. Christopher Manning, and Christopher Potts. A fast unified model for parsing and sentence understanding. In ACL, pages 1466-1477. Asso- ciation for Computational Linguistics, 2016.","title":"A fast unified model for parsing and sentence understanding","authors":"R Samuel Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, D Christopher Manning, Christopher Potts","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2016] Danqi Chen, Jason Bolton, and Christo- pher D. Manning. A thorough examination of the cnn/daily mail reading comprehension task. In ACL, pages 2358- 2367. Association for Computational Linguistics, 2016.","title":"A thorough examination of the cnn/daily mail reading comprehension task","authors":"Chen","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2017] Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. Enhanced lstm for natural language inference. In ACL, pages 1657-1668, Vancouver, Canada, July 2017. Association for Computa- tional Linguistics.","title":"Enhanced lstm for natural language inference","authors":"Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, Diana Inkpen","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cheng et al., 2016] Jianpeng Cheng, Li Dong, and Mirella Lapata. Long short-term memory-networks for machine reading. In EMNLP, pages 551-561. Association for Com- putational Linguistics, 2016.","title":"Long short-term memory-networks for machine reading","authors":"Cheng","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP, pages 1724-1734. Asso- ciation for Computational Linguistics, 2014.","title":"Learning phrase representations using rnn encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gong et al., 2017] Yichen Gong, Heng Luo, and Jian Zhang. Natural language inference over interaction space. arXiv preprint arXiv:1709.04348, 2017.","title":"Natural language inference over interaction space","authors":"Yichen Gong, Heng Luo, Jian Zhang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Madnani et al., 2012] Nitin Madnani, Joel Tetreault, and Martin Chodorow. Re-examining machine translation met- rics for paraphrase identification. In NAACL, pages 182- 190. Association for Computational Linguistics, 2012.","title":"Re-examining machine translation metrics for paraphrase identification","authors":"Nitin Madnani, Joel Tetreault, Martin Chodorow","year":"2012","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Manning et al., 2014] Christopher D. Manning, Mihai Sur- deanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. The Stanford CoreNLP natural language processing toolkit. In ACL System Demonstrations, 2014.","title":"The Stanford CoreNLP natural language processing toolkit","authors":"D Christopher, Mihai Manning, John Surdeanu, Jenny Bauer, Steven J Finkel, David Bethard, Mcclosky","year":"2014","venue":"ACL System Demonstrations","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Munkhdalai and Yu, 2016] Tsendsuren Munkhdalai and Hong Yu. Neural tree indexers for text understanding. arXiv preprint arXiv:1607.04492, 2016.","title":"Tsendsuren Munkhdalai and Hong Yu","authors":"Yu Munkhdalai","year":"2016","venue":"Neural tree indexers for text understanding","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Parikh et al., 2016] Ankur Parikh, Oscar Täckström, Dipan- jan Das, and Jakob Uszkoreit. A decomposable attention model for natural language inference. In EMNLP. Associ- ation for Computational Linguistics, 2016.","title":"A decomposable attention model for natural language inference","authors":"Ankur Parikh, Oscar Täckström, Dipanjan Das, Jakob Uszkoreit","year":"2016","venue":"EMNLP. Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In EMNLP, pages 1532-1543. Association for Computational Linguistics, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher Manning","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Peters et al., 2018] Matthew E Peters, Neumann Mark, Iyyer Mohit, Gardner Matt, Clark Christopher, Lee Ken- ton, and Zettlemoyer Luke. Deep contextualized word rep- resentations. ICLR, 2018.","title":"Deep contextualized word representations","authors":"Neumann Matthew E Peters, Iyyer Mark, Gardner Mohit, Clark Matt, Lee Christopher, Zettlemoyer Kenton, Luke","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Rajpurkar et al., 2016] Pranav Rajpurkar, Jian Zhang, Kon- stantin Lopyrev, and Percy Liang. Squad: 100,000+ ques- tions for machine comprehension of text. In EMNLP. As- sociation for Computational Linguistics, 2016.","title":"Squad: 100,000+ questions for machine comprehension of text","authors":"Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, Percy Liang","year":"2016","venue":"EMNLP. Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Rocktäschel et al., 2015] Tim Rocktäschel, Edward Grefen- stette, Karl Moritz Hermann, Tomás Kociský, and Phil Blunsom. Reasoning about entailment with neural atten- tion. CoRR, 2015.","title":"Reasoning about entailment with neural attention","authors":"Rocktäschel","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sha et al., 2016] Lei Sha, Baobao Chang, Zhifang Sui, and Sujian Li. Reading and thinking: Re-read lstm unit for textual entailment recognition. In COLING, pages 2870- 2879. The COLING Organizing Committee, 2016.","title":"Reading and thinking: Re-read lstm unit for textual entailment recognition","authors":"Lei Sha, Baobao Chang, Zhifang Sui, Sujian Li","year":"2016","venue":"The COLING Organizing Committee","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey E. Hin- ton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 2014.","title":"Dropout: a simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tan et al., 2016] Ming Tan, Cicero dos Santos, Bing Xiang, and Bowen Zhou. Improved representation learning for question answer matching. In ACL, pages 464-473. Asso- ciation for Computational Linguistics, 2016.","title":"Improved representation learning for question answer matching","authors":"Ming Tan, Bing Cicero Dos Santos, Bowen Xiang, Zhou","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang and Jiang, 2016a] Shuohang Wang and Jing Jiang. A compare-aggregate model for matching text sequences. In ICLR, 2016.","title":"A compare-aggregate model for matching text sequences","authors":"Wang, Jiang, Shuohang Wang, Jing Jiang","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang and Jiang, 2016b] Shuohang Wang and Jing Jiang. Learning natural language inference with LSTM. In NAACL, San Diego California, USA, June 12-17, 2016.","title":"Learning natural language inference with LSTM","authors":"Wang, Jiang, Shuohang Wang, Jing Jiang","year":"2016","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2016a] Zhiguo Wang, Haitao Mi, and Abra- ham Ittycheriah. Semi-supervised clustering for short text via deep representation learning. In Proceedings of The 20th SIGNLL Conference on Computational Natural Lan- guage Learning, pages 31-39. Association for Computa- tional Linguistics, 2016.","title":"Semi-supervised clustering for short text via deep representation learning","authors":"Wang","year":"2016","venue":"Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2016b] Zhiguo Wang, Haitao Mi, and Abra- ham Ittycheriah. Sentence similarity learning by lexi- cal decomposition and composition. In COLING, pages 1340-1349. The COLING Organizing Committee, 2016.","title":"Sentence similarity learning by lexical decomposition and composition","authors":"Wang","year":"2016","venue":"The COLING Organizing Committee","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2017a] Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. Gated self-matching net- works for reading comprehension and question answering. In ACL. Association for Computational Linguistics, 2017.","title":"Gated self-matching networks for reading comprehension and question answering","authors":"Wang","year":"2017","venue":"ACL. Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2017b] Zhiguo Wang, Wael Hamza, and Radu Florian. Bilateral multi-perspective matching for natural language sentences. In IJCAI, 2017.","title":"Bilateral multi-perspective matching for natural language sentences","authors":"Wang","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/579"},{"position":25,"raw":"[Williams et al., 2017] Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge cor- pus for sentence understanding through inference. arXiv preprint arXiv:1704.05426, 2017.","title":"A broad-coverage challenge corpus for sentence understanding through inference","authors":"Adina Williams, Nikita Nangia, Samuel R Bowman","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang et al., 2015] Yi Yang, Wen-tau Yih, and Christopher Meek. Wikiqa: A challenge dataset for open-domain ques- tion answering. In EMNLP, pages 2013-2018, 2015.","title":"Wikiqa: A challenge dataset for open-domain question answering","authors":"Yi Yang, Wen-Tau Yih, Christopher Meek","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yin et al., 2016] Wenpeng Yin, Hinrich Schütze, Bing Xi- ang, and Bowen Zhou. Abcnn: Attention-based convolu- tional neural network for modeling sentence pairs. TACL, 4:259-272, 2016.","title":"Abcnn: Attention-based convolutional neural network for modeling sentence pairs","authors":"Yin","year":"2016","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yu et al., 2014] Lei Yu, Karl Moritz Hermann, Phil Blun- som, and Stephen Pulman. Deep learning for answer sen- tence selection. Proceedings of the Deep Learning and Representation Learning Workshop: NIPS, 2014.","title":"Deep learning for answer sentence selection","authors":"Lei Yu, Karl Moritz Hermann, Phil Blunsom, Stephen Pulman","year":"2014","venue":"Proceedings of the Deep Learning and Representation Learning Workshop: NIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zeiler, 2012] Matthew D. Zeiler. ADADELTA: an adaptive learning rate method. CoRR, abs/1212.5701, 2012.","title":"ADADELTA: an adaptive learning rate method","authors":"D Matthew, Zeiler","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}