{"type":"ScholarlyArticle","title":"A Unified Strategy for Multilingual Grammatical Error Correction with Pre-trained Cross-Lingual Language Model","authors":[{"name":"Xin Sun","slug":"xin-sun","given":"Xin","family":"Sun","affiliation":"MOE Key Lab of Computational Linguistics , School of Computer Science , Peking University"},{"name":"Tao Ge","slug":"tao-ge","given":"Tao","family":"Ge","affiliation":"Microsoft Research Asia"},{"name":"Shuming Ma","slug":"shuming-ma","given":"Shuming","family":"Ma","affiliation":"Microsoft Research Asia"},{"name":"Jingjing Li","slug":"jingjing-li","given":"Jingjing","family":"Li","affiliation":"The Chinese University of Hong Kong"},{"name":"Furu Wei","slug":"furu-wei","given":"Furu","family":"Wei","affiliation":"Microsoft Research Asia"},{"name":"Houfeng Wang","slug":"houfeng-wang","given":"Houfeng","family":"Wang","affiliation":"MOE Key Lab of Computational Linguistics , School of Computer Science , Peking University"}],"abstract":"Synthetic data construction of Grammatical Error Correction (GEC) for non-English languages relies heavily on human-designed and language-specific rules, which produce limited error-corrected patterns. In this paper, we propose a generic and language-independent strategy for multilingual GEC, which can train a GEC system effectively for a new non-English language with only two easy-to-access resources: 1) a pre-trained cross-lingual language model (PXLM) and 2) parallel translation data between English and the language. \r\nOur approach creates diverse parallel GEC data without any language-specific operations by taking the non-autoregressive translation generated by PXLM and the gold translation as error-corrected sentence pairs. Then, we reuse PXLM to initialize the GEC model and pre-train it with the synthetic data generated by itself, which yields further improvement. We evaluate our approach on three public benchmarks of GEC in different languages. It achieves the state-of-the-art results on the NLPCC 2018 Task 2 dataset (Chinese) and obtains competitive performance on Falko-Merlin (German) and RULEC-GEC (Russian). Further analysis demonstrates that our data construction method is complementary to rule-based approaches.","doi":"10.24963/ijcai.2022/606","url":"https://ijcai.org/proceedings/2022/606","pdf":"https://ijcai.org/proceedings/2022/0606.pdf","markdown":"https://ijcai.org/proceedings/2022/606.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/606","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Natural Language Processing","track":"Main Track","number":606,"pages":{"first":4367,"last":4374},"keywords":["Natural Language Processing: Language Generation","Natural Language Processing: Applications"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986582","length_sec":84,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984725","length_sec":327,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Boyd, 2018] Adriane Boyd. Using wikipedia edits in low resource grammatical error correction. In Proceedings of the 2018 EMNLP Workshop W-NUT: The 4th Workshop on Noisy User-generated Text, 2018.","title":"Using wikipedia edits in low resource grammatical error correction","authors":"Adriane Boyd","year":"2018","venue":"Proceedings of the 2018 EMNLP Workshop W-NUT: The 4th Workshop on Noisy User-generated Text","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bryant et al., 2017] Christopher Bryant, Mariano Felice, and Ted Briscoe. Automatic annotation and evaluation of error types for grammatical error correction. In Proc. of ACL, 2017.","title":"Automatic annotation and evaluation of error types for grammatical error correction","authors":"Christopher Bryant, Mariano Felice, Ted Briscoe","year":"2017","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chi et al., 2020] Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. In- foxlm: An information-theoretic framework for cross- lingual language model pre-training. arXiv preprint arXiv:2007.07834, 2020.","title":"Infoxlm: An information-theoretic framework for crosslingual language model pre-training","authors":"Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, Ming Zhou","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dahlmeier and Ng, 2012] Daniel Dahlmeier and Hwee Tou Ng. Better evaluation for grammatical error correction. In Proc. of ACL, 2012.","title":"Better evaluation for grammatical error correction","authors":"Dahlmeier, ; ] Ng, Hwee Tou Daniel Dahlmeier, Ng","year":"2012","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Du et al., 2021] Cunxiao Du, Zhaopeng Tu, and Jing Jiang. Order-agnostic cross entropy for non-autoregressive ma- chine translation. arXiv preprint arXiv:2106.05093, 2021.","title":"Jing Jiang. Order-agnostic cross entropy for non-autoregressive machine translation","authors":"Du","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Flachs et al., 2021] Simon Flachs, Felix Stahlberg, and Shankar Kumar. Data strategies for low-resource gram- matical error correction. In Proceedings of the 16th Work- shop on Innovative Use of NLP for Building Educational Applications, 2021.","title":"Data strategies for low-resource grammatical error correction","authors":"Simon Flachs, Felix Stahlberg, Shankar Kumar","year":"2021","venue":"Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fu et al., 2018] Kai Fu, Jin Huang, and Yitao Duan. Youdao's winning solution to the nlpcc-2018 task 2 chal- lenge: a neural machine translation approach to chinese grammatical error correction. In Proc. of NLPCC, 2018.","title":"Youdao's winning solution to the nlpcc-2018 task 2 challenge: a neural machine translation approach to chinese grammatical error correction","authors":"Kai Fu, Jin Huang, Yitao Duan","year":"2018","venue":"Proc. of NLPCC","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ge et al., 2018a] Tao Ge, Furu Wei, and Ming Zhou. Flu- ency boost learning and inference for neural grammatical error correction. In Proc. of ACL, 2018.","title":"Fluency boost learning and inference for neural grammatical error correction","authors":"Ge","year":"2018","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ge et al., 2018b] Tao Ge, Furu Wei, and Ming Zhou. Reaching human-level performance in automatic gram- matical error correction: An empirical study. arXiv preprint arXiv:1807.01270, 2018.","title":"Reaching human-level performance in automatic grammatical error correction: An empirical study","authors":"Ge","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ghazvininejad et al., 2019] Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. In Proc. of EMNLP, 2019.","title":"Mask-predict: Parallel decoding of conditional masked language models","authors":"Marjan Ghazvininejad, Omer Levy, Yinhan Liu, Luke Zettlemoyer","year":"2019","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Grundkiewicz and Junczys-Dowmunt, 2019] Roman Grundkiewicz and Marcin Junczys-Dowmunt. Minimally- augmented grammatical error correction. W-NUT 2019, 2019.","title":"Roman Grundkiewicz and Marcin Junczys-Dowmunt. Minimallyaugmented grammatical error correction","authors":"Junczys-Dowmunt Grundkiewicz","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gu et al., 2017] Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. Non- autoregressive neural machine translation. arXiv preprint arXiv:1711.02281, 2017.","title":"Nonautoregressive neural machine translation","authors":"Gu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Junczys-Dowmunt et al., 2018] Marcin Junczys-Dowmunt, Roman Grundkiewicz, Shubha Guha, and Kenneth Heafield. Approaching neural grammatical error correc- tion as a low-resource machine translation task. In Proc. of ACL, 2018.","title":"Approaching neural grammatical error correction as a low-resource machine translation task","authors":"Marcin Junczys-Dowmunt, Roman Grundkiewicz, Shubha Guha, Kenneth Heafield","year":"2018","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kaneko et al., 2020] Masahiro Kaneko, Masato Mita, Shun Kiyono, Jun Suzuki, and Kentaro Inui. Encoder-decoder models can benefit from pre-trained masked language models in grammatical error correction. In Proc. of ACL, 2020.","title":"Encoder-decoder models can benefit from pre-trained masked language models in grammatical error correction","authors":"Masahiro Kaneko, Masato Mita, Shun Kiyono, Jun Suzuki, Kentaro Inui","year":"2020","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Katsumata and Komachi, 2020] Satoru Katsumata and Mamoru Komachi. Stronger baselines for grammatical error correction using a pretrained encoder-decoder model. In Proc. of ACL, 2020.","title":"Stronger baselines for grammatical error correction using a pretrained encoder-decoder model","authors":"Komachi Katsumata, Satoru Katsumata, Mamoru Komachi","year":"2020","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kiyono et al., 2019] Shun Kiyono, Jun Suzuki, Masato Mita, Tomoya Mizumoto, and Kentaro Inui. An empir- ical study of incorporating pseudo data into grammatical error correction. In Proc. of EMNLP, 2019.","title":"An empirical study of incorporating pseudo data into grammatical error correction","authors":"Kiyono","year":"2019","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2020] Jingjing Li, Zichao Li, Lili Mou, Xin Jiang, Michael Lyu, and Irwin King. Unsupervised text genera- tion by learning from search. In Proc. of NeurIPS, 2020.","title":"Unsupervised text generation by learning from search","authors":"Jingjing Li, Zichao Li, Lili Mou, Xin Jiang, Michael Lyu, Irwin King","year":"2020","venue":"Proc. of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lichtarge et al., 2019] Jared Lichtarge, Chris Alberti, Shankar Kumar, Noam Shazeer, Niki Parmar, and Si- mon Tong. Corpora generation for grammatical error correction. In Proc. of ACL, 2019.","title":"Corpora generation for grammatical error correction","authors":"Jared Lichtarge, Chris Alberti, Shankar Kumar, Noam Shazeer, Niki Parmar, Simon Tong","year":"2019","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ma et al., 2021] Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, and Furu Wei. Deltalm: Encoder-decoder pre-training for language generation and translation by augmenting pretrained multilingual en- coders. arXiv preprint arXiv:2106.13736, 2021.","title":"Deltalm: Encoder-decoder pre-training for language generation and translation by augmenting pretrained multilingual encoders","authors":"Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Náplava and Straka, 2019] Jakub Náplava and Milan Straka. Grammatical error correction in low-resource scenarios. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), 2019.","title":"Grammatical error correction in low-resource scenarios","authors":"Straka Náplava, Jakub Náplava, Milan Straka","year":"2019","venue":"Proceedings of the 5th Workshop on Noisy User-generated Text","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ran et al., 2020] Qiu Ran, Yankai Lin, Peng Li, and Jie Zhou. Learning to recover from multi-modality errors for non-autoregressive neural machine translation. In Proc. of ACL, 2020.","title":"Learning to recover from multi-modality errors for non-autoregressive neural machine translation","authors":"Yankai Qiu Ran, Peng Lin, Jie Li, Zhou","year":"2020","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ren et al., 2018] Hongkai Ren, Liner Yang, and Endong Xun. A sequence to sequence learning for chinese gram- matical error correction. In Proc. of NLPCC, 2018.","title":"A sequence to sequence learning for chinese grammatical error correction","authors":"Liner Hongkai Ren, Endong Yang, Xun","year":"2018","venue":"Proc. of NLPCC","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rothe et al., 2021] Sascha Rothe, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. A simple recipe for multilingual grammatical error correction. arXiv preprint arXiv:2106.03830, 2021.","title":"A simple recipe for multilingual grammatical error correction","authors":"Sascha Rothe, Jonathan Mallinson, Eric Malmi, Sebastian Krause, Aliaksei Severyn","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Rozovskaya and Roth, 2019] Alla Rozovskaya and Dan Roth. Grammar error correction in morphologically rich languages: The case of russian. Transactions of the Asso- ciation for Computational Linguistics, 2019.","title":"Alla Rozovskaya and Dan Roth. Grammar error correction in morphologically rich languages: The case of russian","authors":"Roth Rozovskaya","year":"2019","venue":"Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sun et al., 2021] Xin Sun, Tao Ge, Furu Wei, and Houfeng Wang. Instantaneous grammatical error correction with shallow aggressive decoding. In Proc. of ACL, 2021.","title":"Instantaneous grammatical error correction with shallow aggressive decoding","authors":"Xin Sun, Tao Ge, Furu Wei, Houfeng Wang","year":"2021","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2020a] Chencheng Wang, Liner Yang, Yingy- ing Wang, Yongping Du, and Erhong Yang. Chinese gram- matical error correction method based on transformer en- hanced architecture. Journal of Chinese Information Pro- cessing, 2020.","title":"Chinese grammatical error correction method based on transformer enhanced architecture","authors":"Wang","year":"2020","venue":"Journal of Chinese Information Processing","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2020b] Hongfei Wang, Michiki Kurosawa, Satoru Katsumata, and Mamoru Komachi. Chinese gram- matical correction using bert-based pre-trained model. arXiv preprint arXiv:2011.02093, 2020.","title":"Chinese grammatical correction using bert-based pre-trained model","authors":"Wang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2019] Yi Zhang, Tao Ge, Furu Wei, Ming Zhou, and Xu Sun. Sequence-to-sequence pre-training with data augmentation for sentence rewriting. arXiv preprint arXiv:1909.06002, 2019.","title":"Sequence-to-sequence pre-training with data augmentation for sentence rewriting","authors":"Yi Zhang, Tao Ge, Furu Wei, Ming Zhou, Xu Sun","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhao and Wang, 2020] Zewei Zhao and Houfeng Wang. Maskgec: Improving neural grammatical error correction via dynamic masking. In Proc. of AAAI, 2020.","title":"Maskgec: Improving neural grammatical error correction via dynamic masking","authors":"Wang ; ] Zewei Zhao, Houfeng Zhao, Wang","year":"2020","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhao et al., 2018] Yuanyuan Zhao, Nan Jiang, Weiwei Sun, and Xiaojun Wan. Overview of the nlpcc 2018 shared task: Grammatical error correction. In Proc. of NLPCC, 2018.","title":"Overview of the nlpcc 2018 shared task: Grammatical error correction","authors":"Yuanyuan Zhao, Nan Jiang, Weiwei Sun, Xiaojun Wan","year":"2018","venue":"Proc. of NLPCC","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhou et al., 2018] Junpei Zhou, Chen Li, Hengyou Liu, Zuyi Bao, Guangwei Xu, and Linlin Li. Chinese grammat- ical error correction using statistical and neural models. In Proc. of NLPCC, 2018.","title":"Chinese grammatical error correction using statistical and neural models","authors":"Junpei Zhou, Chen Li, Hengyou Liu, Zuyi Bao, Guangwei Xu, Linlin Li","year":"2018","venue":"Proc. of NLPCC","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhou et al., 2019] Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei, and Ming Zhou. Bert-based lexical substitution. In Proc. of ACL, 2019.","title":"Bert-based lexical substitution","authors":"Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei, Ming Zhou","year":"2019","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhou et al., 2020] Wangchunshu Zhou, Tao Ge, Chang Mu, Ke Xu, Furu Wei, and Ming Zhou. Improving grammatical error correction with machine translation pairs. In Proc. of EMNLP, 2020.","title":"Improving grammatical error correction with machine translation pairs","authors":"Wangchunshu Zhou, Tao Ge, Chang Mu, Ke Xu, Furu Wei, Ming Zhou","year":"2020","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhou et al., 2021] Wangchunshu Zhou, Tao Ge, Canwen Xu, Ke Xu, and Furu Wei. Improving sequence-to- sequence pre-training via sequence span rewriting. In Proc. of EMNLP, 2021.","title":"Improving sequence-tosequence pre-training via sequence span rewriting","authors":"Wangchunshu Zhou, Tao Ge, Canwen Xu, Ke Xu, Furu Wei","year":"2021","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}