{"type":"ScholarlyArticle","title":"Improved Neural Machine Translation with Source Syntax","authors":[{"name":"Shuangzhi Wu","slug":"shuangzhi-wu","given":"Shuangzhi","family":"Wu","affiliation":"† Harbin Institute of Technology , Harbin , China"},{"name":"Ming Zhou","slug":"ming-zhou","given":"Ming","family":"Zhou","affiliation":"‡ Microsoft Research"},{"name":"Dongdong Zhang","slug":"dongdong-zhang","given":"Dongdong","family":"Zhang","affiliation":"‡ Microsoft Research"}],"abstract":"Neural Machine Translation (NMT) based on the encoder-decoder architecture has recently achieved the state-of-the-art performance. Researchers have proven that extending word level attention to phrase level attention by incorporating source-side phrase structure can enhance the attention model and achieve promising improvement. However, word dependencies that can be crucial to correctly understand a source sentence are not always in a consecutive fashion (i.e. phrase structure), sometimes they can be in long distance. Phrase structures are not the best way to explicitly model long distance dependencies. In this paper we propose a simple but effective method to incorporate source-side long distance dependencies into NMT. Our method based on dependency trees enriches each source state with global dependency structures, which can better capture the inherent syntactic structure of source sentences. Experiments on Chinese-English and English-Japanese translation tasks show that our proposed method outperforms state-of-the-art SMT and NMT baselines.","doi":"10.24963/ijcai.2017/584","url":"https://ijcai.org/proceedings/2017/584","pdf":"https://ijcai.org/proceedings/2017/0584.pdf","markdown":"https://ijcai.org/proceedings/2017/584.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/584","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Natural Language Processing","track":"Main track","number":584,"pages":{"first":4179,"last":4185},"keywords":["Natural Language Processing: Machine Translation","Natural Language Processing: Natural Language Processing"],"videos":[],"references":[{"position":1,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. ICLR 2015, 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chiang, 2005] David Chiang. A hierarchical phrase-based model for statistical machine translation. In Proceedings of ACL 2005, 2005.","title":"A hierarchical phrase-based model for statistical machine translation","authors":"David Chiang","year":"2005","venue":"Proceedings of ACL 2005","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"Cho et al., ] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase represen- tations using rnn encoder-decoder for statistical machine translation. In Proceedings of ENMLP 2014.","title":"Learning phrase representations using rnn encoder-decoder for statistical machine translation","authors":"Cho","year":null,"venue":"Proceedings of ENMLP 2014","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Eriguchi et al., 2016a] Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. Character- based decoding in tree-to-sequence attention-based neural machine translation. In Proceedings of WAT2016, 2016.","title":"Characterbased decoding in tree-to-sequence attention-based neural machine translation","authors":"Eriguchi","year":"2016","venue":"Proceedings of WAT2016","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Eriguchi et al., 2016b] Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. Tree-to-sequence attentional neural machine translation. In Proceedings of ACL 2016, August 2016.","title":"Tree-to-sequence attentional neural machine translation","authors":"Eriguchi","year":"2016","venue":"Proceedings of ACL 2016","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hashimoto and Tsuruoka, 2017] Kazuma Hashimoto and Yoshimasa Tsuruoka. Neural machine translation with source-side latent graph parsing. arXiv preprint arXiv:1702.02265, 2017.","title":"Neural machine translation with source-side latent graph parsing","authors":"Tsuruoka Hashimoto, Kazuma Hashimoto, Yoshimasa Tsuruoka","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8), 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Isozaki et al., 2010] Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh, and Hajime Tsukada. Automatic evaluation of translation quality for distant language pairs. In Proceedings of EMNLP, 2010.","title":"Automatic evaluation of translation quality for distant language pairs","authors":"Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh, Hajime Tsukada","year":"2010","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jean et al., 2015] Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. On using very large target vocabulary for neural machine translation. In Proceedings of ACL 2015, July 2015.","title":"On using very large target vocabulary for neural machine translation","authors":"Sébastien Jean, Kyunghyun Cho, Roland Memisevic, Yoshua Bengio","year":"2015","venue":"Proceedings of ACL 2015","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"Kneser and Ney, ] Reinhard Kneser and Hermann Ney. Im- proved backing-off for m-gram language modeling. In Acoustics, Speech, and Signal Processing, 1995. ICASSP- 95., 1995 International Conference on, volume 1. IEEE.","title":"Improved backing-off for m-gram language modeling","authors":"Ney Kneser, Reinhard Kneser, Hermann Ney","year":"1995","venue":"Acoustics, Speech, and Signal Processing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Koehn, 2004] Philipp Koehn. Statistical significance tests for machine translation evaluation. In EMNLP, pages 388- 395. Citeseer, 2004.","title":"Statistical significance tests for machine translation evaluation","authors":"Philipp Koehn, Koehn","year":"2004","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2006] Yang Liu, Qun Liu, and Shouxun Lin. Tree-to-string alignment template for statistical machine translation. In Proceedings of ACL 2006, 2006.","title":"Tree-to-string alignment template for statistical machine translation","authors":"Yang Liu, Qun Liu, Shouxun Lin","year":"2006","venue":"Proceedings of ACL 2006","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Luong et al., 2015a] Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention-based neural machine translation. In Proceed- ings of EMNLP 2015, September 2015.","title":"Effective approaches to attention-based neural machine translation","authors":"Luong","year":"2015","venue":"Proceedings of EMNLP 2015","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Luong et al., 2015b] Thang Luong, Ilya Sutskever, Quoc Le, Oriol Vinyals, and Wojciech Zaremba. Addressing the rare word problem in neural machine translation. In Proceedings of ACL 2015, July 2015.","title":"Addressing the rare word problem in neural machine translation","authors":"Luong","year":"2015","venue":"Proceedings of ACL 2015","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nakazawa et al., 2016] Toshiaki Nakazawa, Manabu Yaguchi, Kiyotaka Uchimoto, Masao Utiyama, Eiichiro Sumita, Sadao Kurohashi, and Hitoshi Isahara. Aspec: Asian scientific paper excerpt corpus. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC 2016), Portoroz, Slovenia, may 2016. European Language Resources Association (ELRA).","title":"Aspec: Asian scientific paper excerpt corpus","authors":"Toshiaki Nakazawa, Manabu Yaguchi, Kiyotaka Uchimoto, Masao Utiyama, Eiichiro Sumita, Sadao Kurohashi, Hitoshi Isahara","year":"2016","venue":"Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC 2016)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"Neubig et al., ] Graham Neubig, Yosuke Nakata, and Shin- suke Mori. Pointwise prediction for robust, adaptable japanese morphological analysis. In Proceedings of ACL 2011.","title":"Pointwise prediction for robust, adaptable japanese morphological analysis","authors":"Neubig","year":null,"venue":"Proceedings of ACL 2011","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for au- tomatic evaluation of machine translation. In Proceedings of ACL 2002, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"Proceedings of ACL 2002","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sag et al., 1999] Ivan A Sag, Thomas Wasow, Emily M Bender, and Ivan A Sag. Syntactic theory: A formal in- troduction, volume 92. Center for the Study of Language and Information Stanford, CA, 1999.","title":"Syntactic theory: A formal introduction","authors":"Ivan A Sag, Thomas Wasow, Emily M Bender, Ivan A Sag","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sennrich and Haddow, 2016] Rico Sennrich and Barry Had- dow. Linguistic input features improve neural machine translation. arXiv preprint arXiv:1606.02892, 2016.","title":"Rico Sennrich and Barry Haddow. Linguistic input features improve neural machine translation","authors":"Haddow Sennrich","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shen et al., 2008] Libin Shen, Jinxi Xu, and Ralph M Weischedel. A new string-to-dependency machine transla- tion algorithm with a target dependency language model. In ACL, pages 577-585, 2008.","title":"A new string-to-dependency machine translation algorithm with a target dependency language model","authors":"Libin Shen, Jinxi Xu, Ralph M Weischedel","year":"2008","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shen et al., 2016] Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. Minimum risk training for neural machine translation. In Proceed- ings of ACL 2016, August 2016.","title":"Minimum risk training for neural machine translation","authors":"Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, Yang Liu","year":"2016","venue":"Proceedings of ACL 2016","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, 2014.","title":"Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tai et al., 2015] Kai Sheng Tai, Richard Socher, and Christopher D Manning. Improved semantic representa- tions from tree-structured long short-term memory net- works. arXiv preprint arXiv:1503.00075, 2015.","title":"Improved semantic representations from tree-structured long short-term memory networks","authors":"Kai Sheng, Tai, Richard Socher, Christopher D Manning","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu et al., 2016] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine translation sys- tem: Bridging the gap between human and machine trans- lation. arXiv preprint arXiv:1609.08144, 2016.","title":"Google's neural machine translation system: Bridging the gap between human and machine translation","authors":"Yonghui Wu, Mike Schuster, Zhifeng Chen, V Quoc, Mohammad Le, Wolfgang Norouzi, Maxim Macherey, Yuan Krikun, Qin Cao, Klaus Gao, Macherey","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zeiler, 2012] Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.","title":"Adadelta: an adaptive learning rate method","authors":"D Matthew, Zeiler","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang and Nivre, 2011] Yue Zhang and Joakim Nivre. Transition-based dependency parsing with rich non-local features. In Proceedings of ACL 2011, 2011.","title":"Transition-based dependency parsing with rich non-local features","authors":"Yue Zhang, Joakim Nivre","year":"2011","venue":"Proceedings of ACL 2011","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}