{"type":"ScholarlyArticle","title":"AdMix: A Mixed Sample Data Augmentation Method for Neural Machine Translation","authors":[{"name":"Chang Jin","slug":"chang-jin","given":"Chang","family":"Jin","affiliation":"Institute of Aritificial Intelligence , School of Computer Science and Technology , Soochow University"},{"name":"Shigui Qiu","slug":"shigui-qiu","given":"Shigui","family":"Qiu","affiliation":"Institute of Aritificial Intelligence , School of Computer Science and Technology , Soochow University"},{"name":"Nini Xiao","slug":"nini-xiao","given":"Nini","family":"Xiao","affiliation":"Institute of Aritificial Intelligence , School of Computer Science and Technology , Soochow University"},{"name":"Hao Jia","slug":"hao-jia","given":"Hao","family":"Jia","affiliation":"Institute of Aritificial Intelligence , School of Computer Science and Technology , Soochow University"}],"abstract":"In Neural Machine Translation (NMT), data augmentation methods such as back-translation have proven their effectiveness in improving translation performance. In this paper, we propose a novel data augmentation approach for NMT, which is independent of any additional training data. Our approach, AdMix, consists of two parts: 1) introduce faint discrete noise (word replacement, word dropping, word swapping) into the original sentence pairs to form augmented samples; 2) generate new synthetic training data by softly mixing the augmented samples with their original samples in training corpus. Experiments on three translation datasets of different scales show that AdMix achieves significant improvements (1.0 to 2.7 BLEU points) over strong Transformer baseline. When combined with other data augmentation techniques (e.g., back-translation), our approach can obtain further improvements.","doi":"10.24963/ijcai.2022/579","url":"https://ijcai.org/proceedings/2022/579","pdf":"https://ijcai.org/proceedings/2022/0579.pdf","markdown":"https://ijcai.org/proceedings/2022/579.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/579","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Natural Language Processing","track":"Main Track","number":579,"pages":{"first":4171,"last":4177},"keywords":["Natural Language Processing: Machine Translation and Multilinguality"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986695","length_sec":90,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984838","length_sec":355,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Anaby-Tavor et al., 2020] Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. Do not have enough data? deep learning to the rescue! In AAAI, volume 34, pages 7383-7390, 2020.","title":"Do not have enough data? deep learning to the rescue!","authors":"Anaby-Tavor","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Artetxe et al., 2017] Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. Unsupervised neural ma- chine translation. arXiv preprint arXiv:1710.11041, 2017.","title":"Unsupervised neural machine translation","authors":"Mikel Artetxe, Gorka Labaka, Eneko Agirre, Kyunghyun Cho","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cheng et al., 2018] Yong Cheng, Zhaopeng Tu, Fandong Meng, Junjie Zhai, and Yang Liu. Towards robust neural machine translation. In ACL, pages 1756-1766, 2018.","title":"Towards robust neural machine translation","authors":"Yong Cheng, Zhaopeng Tu, Fandong Meng, Junjie Zhai, Yang Liu","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cheng et al., 2020] Yong Cheng, Lu Jiang, Wolfgang Macherey, and Jacob Eisenstein. AdvAug: Robust ad- versarial augmentation for neural machine translation. In ACL, pages 5961-5970, 2020.","title":"AdvAug: Robust adversarial augmentation for neural machine translation","authors":"Yong Cheng, Lu Jiang, Wolfgang Macherey, Jacob Eisenstein","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Clark et al., 2018] Kevin Clark, Minh-Thang Luong, Christopher D. Manning, and Quoc Le. Semi-supervised sequence modeling with cross-view training. In EMNLP, pages 1914-1925, 2018.","title":"Semi-supervised sequence modeling with cross-view training","authors":"Kevin Clark, Minh-Thang Luong, Christopher D Manning, Quoc Le","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[DeVries and Taylor, 2017] Terrance DeVries and Gra- ham W Taylor. Improved regularization of convolu- tional neural networks with cutout. arXiv preprint arXiv:1708.04552, 2017.","title":"Improved regularization of convolutional neural networks with cutout","authors":"Terrance Devries, Graham W Taylor","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Edunov et al., 2018] Sergey Edunov, Myle Ott, Michael Auli, David Grangier, and Marc'Aurelio Ranzato. Clas- sical structured prediction losses for sequence to sequence learning. In NAACL, pages 355-364, 2018.","title":"Classical structured prediction losses for sequence to sequence learning","authors":"Sergey Edunov, Myle Ott, Michael Auli, David Grangier, Marc'aurelio Ranzato","year":"2018","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gao et al., 2019] Fei Gao, Jinhua Zhu, Lijun Wu, Yingce Xia, Tao Qin, Xueqi Cheng, Wengang Zhou, and Tie-Yan Liu. Soft contextual data augmentation for neural machine translation. In ACL, pages 5539-5544, 2019.","title":"Soft contextual data augmentation for neural machine translation","authors":"Fei Gao, Jinhua Zhu, Lijun Wu, Yingce Xia, Tao Qin, Xueqi Cheng, Wengang Zhou, Tie-Yan Liu","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Guo et al., 2020] Demi Guo, Yoon Kim, and Alexander Rush. Sequence-level mixed sample data augmentation. In EMNLP, pages 5547-5552, 2020.","title":"Sequence-level mixed sample data augmentation","authors":"Demi Guo, Yoon Kim, Alexander Rush","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2020] Junxian He, Jiatao Gu, Jiajun Shen, and Marc'Aurelio Ranzato. Revisiting self-training for neural sequence generation. In ICLR, 2020.","title":"Revisiting self-training for neural sequence generation","authors":"He","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hendrycks et al., 2020] Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple method to improve robustness and uncertainty under data shift. In ICLR, 2020.","title":"Augmix: A simple method to improve robustness and uncertainty under data shift","authors":"Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, Balaji Lakshminarayanan","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kannan et al., 2018] Harini Kannan, Alexey Kurakin, and Ian Goodfellow. Adversarial logit pairing. arXiv preprint arXiv:1803.06373, 2018.","title":"Adversarial logit pairing","authors":"Alexey Harini Kannan, Ian Kurakin, Goodfellow","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural infor- mation processing systems, 25:1097-1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kumar et al., 2020] Varun Kumar, Ashutosh Choudhary, and Eunah Cho. Data augmentation using pre-trained transformer models. In AACL, pages 18-26, 2020.","title":"Data augmentation using pre-trained transformer models","authors":"Varun Kumar, Ashutosh Choudhary, Eunah Cho","year":"2020","venue":"AACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lample et al., 2018] Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. Unsuper- vised machine translation using monolingual corpora only. In ICLR, 2018.","title":"Unsupervised machine translation using monolingual corpora only","authors":"Guillaume Lample, Alexis Conneau, Ludovic Denoyer, Marc'aurelio Ranzato","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2021] Bohan Li, Yutai Hou, and Wanxiang Che. Data augmentation approaches in natural language pro- cessing: A survey. arXiv preprint arXiv:2110.01852, 2021.","title":"Data augmentation approaches in natural language processing: A survey","authors":"Bohan Li, Yutai Hou, Wanxiang Che","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Miyato et al., 2017] Takeru Miyato, Andrew M. Dai, and Ian J. Goodfellow. Adversarial training methods for semi- supervised text classification. In ICLR, 2017.","title":"Adversarial training methods for semisupervised text classification","authors":"Takeru Miyato, Andrew M Dai, Ian J Goodfellow","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for au- tomatic evaluation of machine translation. In ACL, pages 311-318, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sennrich et al., 2016a] Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. In ACL, pages 86-96, 2016.","title":"Improving neural machine translation models with monolingual data","authors":"Sennrich","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sennrich et al., 2016b] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL, pages 1715-1725, 2016.","title":"Neural machine translation of rare words with subword units","authors":"Sennrich","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2018] Xinyi Wang, Hieu Pham, Zihang Dai, and Graham Neubig. SwitchOut: an efficient data aug- mentation algorithm for neural machine translation. In EMNLP, pages 856-861, 2018.","title":"SwitchOut: an efficient data augmentation algorithm for neural machine translation","authors":"Xinyi Wang, Hieu Pham, Zihang Dai, Graham Neubig","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wei and Zou, 2019] Jason Wei and Kai Zou. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. In EMNLP, pages 6382-6388, 2019.","title":"Eda: Easy data augmentation techniques for boosting performance on text classification tasks","authors":"Zou ; ] Wei, Jason Wei, Kai Zou","year":"2019","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xie et al., 2017] Ziang Xie, Sida I Wang, Jiwei Li, Daniel Lévy, Aiming Nie, Dan Jurafsky, and Andrew Y Ng. Data noising as smoothing in neural network language models. arXiv preprint arXiv:1703.02573, 2017.","title":"Data noising as smoothing in neural network language models","authors":"Xie","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xie et al., 2020] Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V. Le. Unsupervised data augmentation for consistency training. In NeurIPS, 2020.","title":"Unsupervised data augmentation for consistency training","authors":"Xie","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2017] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Be- yond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017.","title":"mixup: Beyond empirical risk minimization","authors":"Hongyi Zhang, Moustapha Cisse, David Yann N Dauphin, Lopez-Paz","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}