{"type":"ScholarlyArticle","title":"Improving Context-Aware Neural Machine Translation with Source-side Monolingual Documents","authors":[{"name":"Linqing Chen","slug":"linqing-chen","given":"Linqing","family":"Chen","affiliation":""},{"name":"Junhui Li","slug":"junhui-li","given":"Junhui","family":"Li","affiliation":""},{"name":"Zhengxian Gong","slug":"zhengxian-gong","given":"Zhengxian","family":"Gong","affiliation":""},{"name":"Xiangyu Duan","slug":"xiangyu-duan","given":"Xiangyu","family":"Duan","affiliation":""},{"name":"Boxing Chen","slug":"boxing-chen","given":"Boxing","family":"Chen","affiliation":""},{"name":"Weihua Luo","slug":"weihua-luo","given":"Weihua","family":"Luo","affiliation":""},{"name":"Min Zhang","slug":"min-zhang","given":"Min","family":"Zhang","affiliation":""},{"name":"Guodong Zhou","slug":"guodong-zhou","given":"Guodong","family":"Zhou","affiliation":""}],"abstract":"Document context-aware machine translation remains challenging due to the lack of large-scale document parallel corpora. To make full use of source-side monolingual documents for context-aware NMT, we propose a Pre-training approach with Global Context (PGC). In particular, we first propose a novel self-supervised pre-training task, which contains two training objectives: (1) reconstructing the original sentence from a corrupted version; (2) generating a gap sentence from its left and right neighbouring sentences. Then we design a universal model for PGC which consists of a global context encoder, a sentence encoder and a decoder, with similar architecture to typical context-aware NMT models. We evaluate the effectiveness and generality of our pre-trained PGC model by adapting it to various downstream context-aware NMT models. Detailed experimentation on four different translation tasks demonstrates that our PGC approach significantly improves the translation performance of context-aware NMT. For example, based on the state-of-the-art SAN model, we achieve an averaged improvement of 1.85 BLEU scores and 1.59 Meteor scores on the four translation tasks.","doi":"10.24963/ijcai.2021/522","url":"https://ijcai.org/proceedings/2021/522","pdf":"https://ijcai.org/proceedings/2021/0522.pdf","markdown":"https://ijcai.org/proceedings/2021/522.md","bibtex":"https://ijcai.org/proceedings/2021/bibtex/522","volume":{"slug":"2021","title":"Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2021","url":"https://ijcai.org/proceedings/2021/","year":2021,"location":"Montreal","dates":"19-27 August 2021","editors":"Prof. Zhi-Hua Zhou","isbn":"978-0-9992411-9-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2021"},"section":"Natural Language Processing","track":"Main Track","number":522,"pages":{"first":3794,"last":3800},"keywords":["Natural Language Processing: Machine Translation"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}