{"type":"ScholarlyArticle","title":"ME-MD: An Effective Framework for Neural Machine Translation with Multiple Encoders and Decoders","authors":[{"name":"Jinchao Zhang","slug":"jinchao-zhang","given":"Jinchao","family":"Zhang","affiliation":"Key Laboratory of Intelligent Information Processing , Institute of Computing Technology , CAS."},{"name":"Qun Liu","slug":"qun-liu","given":"Qun","family":"Liu","affiliation":"ADAPT Centre , School of Computing , Dublin City University; Key Laboratory of Intelligent Information Processing , Institute of Computing Technology , CAS."},{"name":"Jie Zhou","slug":"jie-zhou","given":"Jie","family":"Zhou","affiliation":"Baidu Research -Institute of Deep Learning , Baidu Inc. , Beijing , China"}],"abstract":"The encoder-decoder neural framework is widely employed for Neural Machine Translation (NMT) with a single encoder to represent the source sentence and a single decoder to generate target words. The translation performance heavily relies on the representation ability of the encoder and the generation ability of the decoder. To further enhance NMT, we propose to extend the original encoder-decoder framework to a novel one, which has multiple encoders and decoders (ME-MD). Through this way, multiple encoders extract more diverse features to represent the source sequence and multiple decoders capture more complicated translation knowledge. Our proposed ME-MD framework is convenient to integrate heterogeneous encoders and decoders with multiple depths and multiple types. Experiment on Chinese-English translation task shows that our ME-MD system surpasses the state-of-the-art NMT system by 2.1 BLEU points and surpasses the phrase-based Moses by 7.38 BLEU points. Our framework is general and can be applied to other sequence to sequence tasks.","doi":"10.24963/ijcai.2017/474","url":"https://ijcai.org/proceedings/2017/474","pdf":"https://ijcai.org/proceedings/2017/0474.pdf","markdown":"https://ijcai.org/proceedings/2017/474.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/474","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":474,"pages":{"first":3392,"last":3398},"keywords":["Natural Language Processing: Machine Translation","Machine Learning: Neural Networks"],"videos":[],"references":[{"position":1,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Proceedings of ICLR2015, 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"Proceedings of ICLR2015","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.","title":"Learning phrase representations using rnn encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chung et al., 2016] Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio. A character-level decoder without explicit segmentation for neural machine translation. In Proceed- ings of ACL2016, 2016.","title":"A character-level decoder without explicit segmentation for neural machine translation","authors":"Chung","year":"2016","venue":"Proceedings of ACL2016","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dong et al., 2015] Daxiang Dong, Hua Wu, Wei He, Dian- hai Yu, and Haifeng Wang. Multi-task learning for mul- tiple language translation. In ACL (1), pages 1723-1732, 2015.","title":"Multi-task learning for multiple language translation","authors":"Dong","year":"2015","venue":"ACL (1)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Firat et al., 2016] Orhan Firat, Kyunghyun Cho, Baskaran Sankaran, Fatos T Yarman Vural, and Yoshua Bengio. Multi-way, multilingual neural machine translation. Com- puter Speech & Language, 2016.","title":"Multi-way, multilingual neural machine translation","authors":"Orhan Firat, Kyunghyun Cho, Baskaran Sankaran, Fatos T Yarman, Yoshua Vural, Bengio","year":"2016","venue":"Computer Speech & Language","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Jean et al., 2015] Sbastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. On using very large target vocabulary for neural machine translation. In Proceedings of ACL2014, volume 1, pages 1-10, 2015.","title":"On using very large target vocabulary for neural machine translation","authors":"Jean","year":"2015","venue":"Proceedings of ACL2014","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kalchbrenner and Blunsom, 2013] Nal Kalchbrenner and Phil Blunsom. Recurrent continuous translation models. In Proceedings of EMNLP2013, pages 1700-1709, Seat- tle, Washington, USA, October 2013.","title":"Recurrent continuous translation models","authors":"Nal Kalchbrenner, Phil Blunsom","year":"2013","venue":"Proceedings of EMNLP2013","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Koehn et al., 2007] Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Con- stantin, and Evan Herbst. Moses: Open source toolkit for statistical machine translation. In Proceedings of the ACL2007 Demo and Poster Sessions, pages 177-180, Prague, Czech Republic, June 2007.","title":"Moses: Open source toolkit for statistical machine translation","authors":"Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, Evan Herbst","year":"2007","venue":"Proceedings of the ACL2007 Demo and Poster Sessions","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Luong et al., 2015a] Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention-based neural machine translation. In Proceed- ings of EMNLP2015, Lisbon, Portugal, September 2015.","title":"Effective approaches to attention-based neural machine translation","authors":"Luong","year":"2015","venue":"Proceedings of EMNLP2015","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Luong et al., 2015b] Minh Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. Ad- dressing the rare word problem in neural machine transla- tion. Proceedings of ACL2015, 27(2):82-86, 2015.","title":"Addressing the rare word problem in neural machine translation","authors":"Luong","year":"2015","venue":"Proceedings of ACL2015","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Luong et al., 2016] Minh-Thang Luong, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. Multi-task sequence to sequence learning. In International Con- ference on Learning Representations (ICLR), San Juan, Puerto Rico, May 2016.","title":"Multi-task sequence to sequence learning","authors":"Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, Lukasz Kaiser","year":"2016","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Meng et al., 2016] Fandong Meng, Zhengdong Lu, Hang Li, and Qun Liu. Interactive attention for neural machine translation. In Proceedings of COLING2016, 2016.","title":"Interactive attention for neural machine translation","authors":"Meng","year":"2016","venue":"Proceedings of COLING2016","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Och and Ney, 2003] Franz Josef Och and Hermann Ney. A systematic comparison of various statistical alignment models. Computational linguistics, 29(1):19-51, 2003.","title":"A systematic comparison of various statistical alignment models","authors":"Ney ; Och, Josef Franz, Hermann Och, Ney","year":"2003","venue":"Computational linguistics","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for auto- matic evaluation of machine translation. In Proceedings of ACL2002, pages 311-318. Association for Computational Linguistics, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"Proceedings of ACL2002","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pascanu et al., 2013] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neu- ral networks. ICML (3), 28:1310-1318, 2013.","title":"On the difficulty of training recurrent neural networks","authors":"Razvan Pascanu, Tomas Mikolov, Yoshua Bengio","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sennrich et al., 2016] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of ACL2016, pages 1715-1725, 2016.","title":"Neural machine translation of rare words with subword units","authors":"Sennrich","year":"2016","venue":"Proceedings of ACL2016","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shen et al., 2016] Shiqi Shen, Yong Cheng, Zhongjun He, Hua Wu, Maosong Sun, and Yang Liu. Minimum risk training for neural machine translation. In Proceedings of ACL2016, pages 1683-1692, 2016.","title":"Minimum risk training for neural machine translation","authors":"Shiqi Shen, Yong Cheng, Zhongjun He, Hua Wu, Maosong Sun, Yang Liu","year":"2016","venue":"Proceedings of ACL2016","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey E Hin- ton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-1958, 2014.","title":"Dropout: a simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Stolcke and others, 2002] Andreas Stolcke et al. Srilm-an extensible language modeling toolkit. In Proceedings of the international conference on spoken language process- ing, volume 2, pages 901-904, 2002.","title":"Srilm-an extensible language modeling toolkit","authors":"Andreas Stolcke","year":"2002","venue":"Proceedings of the international conference on spoken language processing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In NIPS, 2014.","title":"Quoc V Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tu et al., 2016] Zhaopeng Tu, Zhengdong Lu, yang Liu, Xi- aohua Liu, and Hang Li. Modeling coverage for neural machine translation. In Proceedings of ACL, pages 76-85, 2016.","title":"Modeling coverage for neural machine translation","authors":"Tu","year":"2016","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2016] Mingxuan Wang, Zhengdong Lu, Hang Li, and Qun Liu. Memory-enhanced decoder for neu- ral machine translation. In Proceedings of EMNLP2016, 2016.","title":"Memory-enhanced decoder for neural machine translation","authors":"Mingxuan Wang, Zhengdong Lu, Hang Li, Qun Liu","year":"2016","venue":"Proceedings of EMNLP2016","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu et al., 2016] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine translation sys- tem: Bridging the gap between human and machine trans- lation. arXiv preprint arXiv:1609.08144, 2016.","title":"Google's neural machine translation system: Bridging the gap between human and machine translation","authors":"Yonghui Wu, Mike Schuster, Zhifeng Chen, V Quoc, Mohammad Le, Wolfgang Norouzi, Maxim Macherey, Yuan Krikun, Qin Cao, Klaus Gao, Macherey","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zeiler, 2012] Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.","title":"Adadelta: an adaptive learning rate method","authors":"D Matthew, Zeiler","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhou et al., 2016] Jie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. Deep recurrent models with fast- forward connections for neural machine translation. In Proceedings of EMNLP2016, 2016.","title":"Deep recurrent models with fastforward connections for neural machine translation","authors":"Jie Zhou, Ying Cao, Xuguang Wang, Peng Li, Wei Xu","year":"2016","venue":"Proceedings of EMNLP2016","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zoph and Knight, 2016] Barret Zoph and Kevin Knight. Multi-source neural translation. In Proceedings of NAACL-HLT, pages 30-34, 2016.","title":"Multi-source neural translation","authors":"Knight Zoph, Barret Zoph, Kevin Knight","year":"2016","venue":"Proceedings of NAACL-HLT","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}