{"type":"ScholarlyArticle","title":"Towards Making the Most of Context in Neural Machine Translation","authors":[{"name":"Zaixiang Zheng","slug":"zaixiang-zheng","given":"Zaixiang","family":"Zheng","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University"},{"name":"Xiang Yue","slug":"xiang-yue","given":"Xiang","family":"Yue","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University"},{"name":"Shujian Huang","slug":"shujian-huang","given":"Shujian","family":"Huang","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University"},{"name":"Jiajun Chen","slug":"jiajun-chen","given":"Jiajun","family":"Chen","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University"},{"name":"Alexandra Birch","slug":"alexandra-birch","given":"Alexandra","family":"Birch","affiliation":"ILCC , School of Informatics , University of Edinburgh; , 1 , ⋯ , y 1 , t ,"}],"abstract":"Document-level machine translation manages to outperform sentence level models by a small margin, but have failed to be widely adopted. We argue that previous research did not make a clear use of the global context, and propose a new document-level NMT framework that deliberately models the local context of each sentence with the awareness of the global context of the document in both source and target languages. We specifically design the model to be able to deal with documents containing any number of sentences, including single sentences. This unified approach allows our model to be trained elegantly on standard datasets without needing to train on sentence and document level data separately. Experimental results demonstrate that our model outperforms Transformer baselines and previous document-level NMT models with substantial margins of up to 2.1 BLEU on state-of-the-art baselines. We also provide analyses which show the benefit of context far beyond the neighboring two or three sentences, which previous studies have typically incorporated.","doi":"10.24963/ijcai.2020/551","url":"https://ijcai.org/proceedings/2020/551","pdf":"https://ijcai.org/proceedings/2020/0551.pdf","markdown":"https://ijcai.org/proceedings/2020/551.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/551","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Natural Language Processing","track":"Main track","number":551,"pages":{"first":3983,"last":3989},"keywords":["Natural Language Processing: Machine Translation"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25164/file_25164.m3u8","length_sec":299,"provider":"hls"}],"references":[{"position":1,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bawden et al., 2018] Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. Evaluating dis- course phenomena in neural machine translation. In NAACL-HLT, 2018.","title":"Evaluating discourse phenomena in neural machine translation","authors":"Rachel Bawden, Rico Sennrich, Alexandra Birch, Barry Haddow","year":"2018","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dai et al., 2019] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, and Ruslan R. Salakhut- dinov. Transformer-xl: Attentive language models beyond a fixed-length context. In ACL, 2019.","title":"Transformer-xl: Attentive language models beyond a fixed-length context","authors":"Dai","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Hassan et al., 2018] Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Fe- dermann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, et al. Achieving human parity on automatic chinese to english news translation. arXiv preprint arXiv:1803.05567, 2018.","title":"Achieving human parity on automatic chinese to english news translation","authors":"Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Jean et al., 2017] Sébastien Jean, Stanislas Lauly, Orhan Fi- rat, and Kyunghyun Cho. Does neural machine translation benefit from larger context? CoRR, abs/1704.05135, 2017.","title":"Does neural machine translation benefit from larger context?","authors":"Sébastien Jean, Stanislas Lauly, Orhan Firat, Kyunghyun Cho","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"Marcin Junczys-Dowmunt. Mi- crosoft translator at wmt 2019: Towards large-scale document-level neural machine translation. In WMT, 2019.","title":"Microsoft translator at wmt 2019: Towards large-scale document-level neural machine translation","authors":"Marcin Junczys-Dowmunt","year":"2019","venue":"WMT","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kingma and Ba, 2014] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kuang et al., 2018] Shaohui Kuang, Deyi Xiong, Weihua Luo, and Guodong Zhou. Modeling coherence for neu- ral machine translation with dynamic and topic caches. In COLING, 2018.","title":"Modeling coherence for neural machine translation with dynamic and topic caches","authors":"Kuang","year":"2018","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Läubli et al., 2018] Samuel Läubli, Rico Sennrich, and Martin Volk. Has machine translation achieved human parity? a case for document-level evaluation. In EMNLP, 2018.","title":"Samuel Läubli, Rico Sennrich, and Martin Volk. Has machine translation achieved human parity? a case for document-level evaluation","authors":"Läubli","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2019] Liangyou Li, Xin Jiang, Qun Liu, Huawei Noah', and Ark Lab. Pretrained Language Models for Document-Level Neural Machine Translation. arXiv preprint, 2019.","title":"Ark Lab. Pretrained Language Models for Document-Level Neural Machine Translation","authors":"Li","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Maruf and Haffari, 2018] Sameen Maruf and Gholamreza Haffari. Document context neural machine translation with memory networks. In ACL, 2018.","title":"Document context neural machine translation with memory networks","authors":"Haffari Maruf, Sameen Maruf, Gholamreza Haffari","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Maruf et al., 2019] Sameen Maruf, André FT Martins, and Gholamreza Haffari. Selective attention for context-aware neural machine translation. In NAACL-HLT, 2019.","title":"Selective attention for context-aware neural machine translation","authors":"Sameen Maruf, André Ft Martins, Gholamreza Haffari","year":"2019","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Miculicich et al., 2018] Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, and James Henderson. Document-level neural machine translation with hierarchical attention net- works. In EMNLP, 2018.","title":"Document-level neural machine translation with hierarchical attention networks","authors":"Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, James Henderson","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Müller et al., 2018] Mathias Müller, Annette Rios, Elena Voita, and Rico Sennrich. A Large-Scale Test Set for the Evaluation of Context-Aware Pronoun Translation in Neu- ral Machine Translation. In WMT, 2018. [Papineni et al., 2002]","title":"A Large-Scale Test Set for the Evaluation of Context-Aware Pronoun Translation in Neural Machine Translation","authors":"Mathias Müller, Annette Rios, Elena Voita, Rico Sennrich","year":"2002","venue":"WMT","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for au- tomatic evaluation of machine translation. In ACL, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sennrich et al., 2016] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL, 2016.","title":"Neural machine translation of rare words with subword units","authors":"Sennrich","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shaw et al., 2018] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representa- tions. In NAACL-HLT, 2018.","title":"Self-attention with relative position representations","authors":"Peter Shaw, Jakob Uszkoreit, Ashish Vaswani","year":"2018","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In NIPS, 2014.","title":"Quoc V Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Szegedy et al., 2016] Christian Szegedy, Vincent Van- houcke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.","title":"Rethinking the inception architecture for computer vision","authors":"Szegedy","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tan et al., 2019] Xin Tan, Longyin Zhang, Deyi Xiong, and Guodong Zhou. Hierarchical modeling of global con- text for document-level neural machine translation. In EMNLP-IJCNLP, 2019.","title":"Hierarchical modeling of global context for document-level neural machine translation","authors":"Xin Tan, Longyin Zhang, Deyi Xiong, Guodong Zhou","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tiedemann and Scherrer, 2017] Jörg Tiedemann and Yves Scherrer. Neural machine translation with extended con- text. In DiscoMT, 2017.","title":"Neural machine translation with extended context","authors":"Scherrer Tiedemann, Jörg Tiedemann, Yves Scherrer","year":"2017","venue":"DiscoMT","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tu et al., 2018] Zhaopeng Tu, Yang Liu, Shuming Shi, and Tong Zhang. Learning to remember translation history with a continuous cache. TACL, 2018.","title":"Learning to remember translation history with a continuous cache","authors":"Tu","year":"2018","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In NIPS, 2017.","title":"Attention is All you Need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Voita et al., 2018] Elena Voita, Pavel Serdyukov, Rico Sen- nrich, and Ivan Titov. Context-aware neural machine trans- lation learns anaphora resolution. In ACL, 2018.","title":"Context-aware neural machine translation learns anaphora resolution","authors":"Elena Voita, Pavel Serdyukov, Rico Sennrich, Ivan Titov","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Voita et al., 2019a] Elena Voita, Rico Sennrich, and Ivan Titov. Context-aware monolingual repair for neural ma- chine translation. In EMNLP-IJCNLP, 2019.","title":"Context-aware monolingual repair for neural machine translation","authors":"Voita","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Voita et al., 2019b] Elena Voita, Rico Sennrich, and Ivan Titov. When a good translation is wrong in context: Context-aware machine translation improves on deixis, el- lipsis, and lexical cohesion. In ACL, 2019.","title":"Ivan Titov. When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion","authors":"Voita","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2017] Longyue Wang, Zhaopeng Tu, Andy Way, and Qun Liu. Exploiting cross-sentence context for neural machine translation. In EMNLP, 2017.","title":"Exploiting cross-sentence context for neural machine translation","authors":"Wang","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xiong et al., 2019] Hao Xiong, Zhongjun He, Hua Wu, and Haifeng Wang. Modeling coherence for discourse neural machine translation. In AAAI, 2019.","title":"Modeling coherence for discourse neural machine translation","authors":"Hao Xiong, Zhongjun He, Hua Wu, Haifeng Wang","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yang et al., 2019] Zhengxin Yang, Jinchao Zhang, Fandong Meng, Shuhao Gu, Yang Feng, and Jie Zhou. Enhancing context modeling with a query-guided capsule network for document-level translation. In EMNLP-IJCNLP, 2019.","title":"Enhancing context modeling with a query-guided capsule network for document-level translation","authors":"Yang","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2018] Jiacheng Zhang, Huanbo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, Min Zhang, and Yang Liu. Improving the transformer translation model with document-level context. In EMNLP, 2018.","title":"Improving the transformer translation model with document-level context","authors":"Zhang","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}