{"type":"ScholarlyArticle","title":"Neural Machine Translation with Key-Value Memory-Augmented Attention","authors":[{"name":"Fandong Meng","slug":"fandong-meng","given":"Fandong","family":"Meng","affiliation":""},{"name":"Zhaopeng Tu","slug":"zhaopeng-tu","given":"Zhaopeng","family":"Tu","affiliation":""},{"name":"Yong Cheng","slug":"yong-cheng","given":"Yong","family":"Cheng","affiliation":""},{"name":"Haiyang Wu","slug":"haiyang-wu","given":"Haiyang","family":"Wu","affiliation":""},{"name":"Junjie Zhai","slug":"junjie-zhai","given":"Junjie","family":"Zhai","affiliation":""},{"name":"Yuekui Yang","slug":"yuekui-yang","given":"Yuekui","family":"Yang","affiliation":""},{"name":"Di Wang","slug":"di-wang","given":"Di","family":"Wang","affiliation":""}],"abstract":"Although attention-based Neural Machine Translation (NMT) has achieved remarkable progress in recent years, it still suffers from issues of repeating and dropping translations. To alleviate these issues, we propose a novel key-value memory-augmented attention model for NMT, called KVMEMATT. Specifically, we maintain a timely updated keymemory to keep track of attention history and a fixed value-memory to store the representation of source sentence throughout the whole translation process. Via nontrivial transformations and iterative interactions between the two memories, the decoder focuses on more appropriate source word(s) for predicting the next target word at each decoding step, therefore can improve the adequacy of translations. Experimental results on Chinese)English and WMT17 German,English translation tasks demonstrate the superiority of the proposed model.","doi":"10.24963/ijcai.2018/357","url":"https://ijcai.org/proceedings/2018/357","pdf":"https://ijcai.org/proceedings/2018/0357.pdf","markdown":"https://ijcai.org/proceedings/2018/357.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/357","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":357,"pages":{"first":2574,"last":2580},"keywords":["Machine Learning: Neural Networks","Machine Learning: Deep Learning","Natural Language Processing: Machine Translation"],"videos":[],"references":[{"position":1,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP, 2014.","title":"Learning phrase representations using rnn encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cohn et al., 2016] Trevor Cohn, Cong Duy Vu Hoang, Eka- terina Vymolova, Kaisheng Yao, Chris Dyer, and Gholam- reza Haffari. Incorporating structural alignment biases into an attentional neural translation model. In NAACL, 2016.","title":"Incorporating structural alignment biases into an attentional neural translation model","authors":"Trevor Cohn, Cong Duy Vu, Ekaterina Hoang, Kaisheng Vymolova, Chris Yao, Gholamreza Dyer, Haffari","year":"2016","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Escolano et al., 2017] Carlos Escolano, Marta R. Costa- jussà, and José A. R. Fonollosa. The talp-upc neural ma- chine translation system for german/finnish-english using the inverse direction model in rescoring. In WMT, 2017.","title":"The talp-upc neural machine translation system for german/finnish-english using the inverse direction model in rescoring","authors":"Carlos Escolano, Marta R Costajussà, José A R Fonollosa","year":"2017","venue":"WMT","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Feng et al., 2016] Shi Feng, Shujie Liu, Mu Li, and Ming Zhou. Implicit distortion and fertility models for attention- based encoder-decoder NMT model. In COLING, 2016.","title":"Implicit distortion and fertility models for attentionbased encoder-decoder NMT model","authors":"Shujie Shi Feng, Mu Liu, Ming Li, Zhou","year":"2016","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Feng et al., 2017] Yang Feng, Shiyue Zhang, Andi Zhang, Dong Wang, and Andrew Abel. Memory-augmented neu- ral machine translation. arXiv, 2017.","title":"Memory-augmented neural machine translation","authors":"Yang Feng, Shiyue Zhang, Andi Zhang, Dong Wang, Andrew Abel","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gehring et al., 2017] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convo- lutional sequence to sequence learning. arXiv, 2017.","title":"Convolutional sequence to sequence learning","authors":"Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N Dauphin","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Graves et al., 2014] Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv, 2014.","title":"Neural turing machines","authors":"Alex Graves, Greg Wayne, Ivo Danihelka","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gu et al., 2017] Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor OK Li. Search engine guided non-parametric neural machine translation. arXiv, 2017.","title":"Search engine guided non-parametric neural machine translation","authors":"Gu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hinton et al., 2012] Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhut- dinov. Improving neural networks by preventing co- adaptation of feature detectors. arXiv, 2012.","title":"Improving neural networks by preventing coadaptation of feature detectors","authors":"Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, Ruslan R Salakhutdinov","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu and Sun, 2015] Yang Liu and Maosong Sun. Con- trastive unsupervised word alignment with non-local fea- tures. In AAAI, 2015.","title":"Contrastive unsupervised word alignment with non-local features","authors":"Yang Liu, Maosong Sun","year":"2015","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Luong et al., 2015] Thang Luong, Hieu Pham, and Christo- pher D. Manning. Effective approaches to attention-based neural machine translation. In EMNLP, 2015.","title":"Effective approaches to attention-based neural machine translation","authors":"Thang Luong, Hieu Pham, Christopher D Manning","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Meng et al., 2016] Fandong Meng, Zhengdong Lu, Hang Li, and Qun Liu. Interactive attention for neural machine translation. In COLING, 2016.","title":"Interactive attention for neural machine translation","authors":"Meng","year":"2016","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mi et al., 2016] Haitao Mi, Baskaran Sankaran, Zhiguo Wang, and Abe Ittycheriah. Coverage embedding models for neural machine translation. In EMNLP, 2016.","title":"Coverage embedding models for neural machine translation","authors":"Haitao Mi, Zhiguo Baskaran Sankaran, Abe Wang, Ittycheriah","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Miller et al., 2016] Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. Key-value memory networks for directly reading documents. In EMNLP, 2016.","title":"Key-value memory networks for directly reading documents","authors":"Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein, Antoine Karimi, Jason Bordes, Weston","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Och and Ney, 2003] Franz Josef Och and Hermann Ney. A systematic comparison of various statistical alignment models. CL, 2003. [Papineni et al., 2002]","title":"A systematic comparison of various statistical alignment models","authors":"Ney ; Och, Josef Franz, Hermann Och, Ney","year":"2002","venue":"CL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for au- tomatic evaluation of machine translation. In ACL, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rikters et al., 2017] Matīss Rikters, Chantal Amrhein, Maksym Del, and Mark Fishel. C-3ma: Tartu-riga-zurich translation systems for wmt17. In WMT, 2017.","title":"C-3ma: Tartu-riga-zurich translation systems for wmt17","authors":"Rikters","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rocktäschel et al., 2017] Tim Rocktäschel, Johannes Welbl, and Sebastian Riedel. Frustratingly short attention spans in neural language modeling. In ICLR, 2017.","title":"Frustratingly short attention spans in neural language modeling","authors":"Tim Rocktäschel, Johannes Welbl, Sebastian Riedel","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Schuster and Paliwal, 1997] Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. TSP, 1997.","title":"Bidirectional recurrent neural networks","authors":"Mike Schuster, Kuldip K Paliwal","year":"1997","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sennrich et al., 2016] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL, 2016.","title":"Neural machine translation of rare words with subword units","authors":"Sennrich","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sennrich et al., 2017] Rico Sennrich, Alexandra Birch, Anna Currey, Ulrich Germann, Barry Haddow, Ken- neth Heafield, Antonio Valerio Miceli Barone, and Philip Williams. The university of edinburgh's neural MT sys- tems for WMT17. arXiv, 2017.","title":"The university of edinburgh's neural MT systems for WMT17","authors":"Sennrich","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. Sequence to sequence learning with neu- ral networks. In NIPS, 2014.","title":"Quoc VV Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tang et al., 2016] Yaohua Tang, Fandong Meng, Zheng- dong Lu, Hang Li, and Philip L. H. Yu. Neural machine translation with external phrase memory. arXiv, 2016.","title":"Neural machine translation with external phrase memory","authors":"Yaohua Tang, Fandong Meng, Zhengdong Lu, Hang Li, Philip L H Yu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tu et al., 2016] Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. Modeling coverage for neu- ral machine translation. In ACL, 2016.","title":"Modeling coverage for neural machine translation","authors":"Tu","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tu et al., 2018] Zhaopeng Tu, Yang Liu, Shuming Shi, and Tong Zhang. Learning to remember translation history with a continuous cache. TACL, 2018.","title":"Learning to remember translation history with a continuous cache","authors":"Tu","year":"2018","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2016] Mingxuan Wang, Zhengdong Lu, Hang Li, and Qun Liu. Memory-enhanced decoder for neural machine translation. In EMNLP, 2016.","title":"Memory-enhanced decoder for neural machine translation","authors":"Mingxuan Wang, Zhengdong Lu, Hang Li, Qun Liu","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2017] Xing Wang, Zhengdong Lu, Zhaopeng Tu, Hang Li, Deyi Xiong, and Min Zhang. Neural machine translation advised by statistical machine translation. In AAAI, 2017.","title":"Neural machine translation advised by statistical machine translation","authors":"Xing Wang, Zhengdong Lu, Zhaopeng Tu, Hang Li, Deyi Xiong, Min Zhang","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zeiler, 2012] Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv, 2012.","title":"Adadelta: an adaptive learning rate method","authors":"D Matthew, Zeiler","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2017] Jinchao Zhang, Mingxuan Wang, Qun Liu, and Jie Zhou. Incorporating word reordering knowl- edge into attention-based neural machine translation. In ACL, 2017.","title":"Incorporating word reordering knowledge into attention-based neural machine translation","authors":"Zhang","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zheng et al., 2018] Zaixiang Zheng, Hao Zhou, Shujian Huang, Lili Mou, Dai Xinyu, Jiajun Chen, and Zhaopeng Tu. Modeling past and future for neural machine transla- tion. TACL, 2018.","title":"Modeling past and future for neural machine translation","authors":"Zaixiang Zheng, Hao Zhou, Shujian Huang, Lili Mou, Dai Xinyu, Jiajun Chen, Zhaopeng Tu","year":"2018","venue":"TACL","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}