{"type":"ScholarlyArticle","title":"Task-Level Curriculum Learning for Non-Autoregressive Neural Machine Translation","authors":[{"name":"Jinglin Liu","slug":"jinglin-liu","given":"Jinglin","family":"Liu","affiliation":"Zhejiang University"},{"name":"Yi Ren","slug":"yi-ren","given":"Yi","family":"Ren","affiliation":"Zhejiang University"},{"name":"Xu Tan","slug":"xu-tan","given":"Xu","family":"Tan","affiliation":"Microsoft Research Asia"},{"name":"Chen Zhang","slug":"chen-zhang","given":"Chen","family":"Zhang","affiliation":"Zhejiang University"},{"name":"Tao Qin","slug":"tao-qin","given":"Tao","family":"Qin","affiliation":"Microsoft Research Asia"},{"name":"Zhou Zhao","slug":"zhou-zhao","given":"Zhou","family":"Zhao","affiliation":"Zhejiang University"},{"name":"Tie-Yan Liu","slug":"tie-yan-liu","given":"Tie-Yan","family":"Liu","affiliation":"Microsoft Research Asia"}],"abstract":"Non-autoregressive translation (NAT) achieves faster inference speed but at the cost of worse accuracy compared with autoregressive translation (AT). Since AT and NAT can share model structure and AT is an easier task than NAT due to the explicit dependency on previous target-side tokens, a natural idea is to gradually shift the model training from the easier AT task to the harder NAT task. To smooth the shift from AT training to NAT training, in this paper, we introduce semi-autoregressive translation (SAT) as intermediate tasks. SAT contains a hyperparameter k, and each k value defines a SAT task with different degrees of parallelism. Specially, SAT covers AT and NAT as its special cases: it reduces to AT when k=1 and to NAT when k=N (N is the length of target sentence). We design curriculum schedules to gradually shift k from 1 to N, with different pacing functions and number of tasks trained at the same time. We called our method as task-level curriculum learning for NAT (TCL-NAT). Experiments on IWSLT14 De-En, IWSLT16 En-De, WMT14 En-De and De-En datasets show that TCL-NAT achieves significant accuracy improvements over previous NAT baselines and reduces the performance gap between NAT and AT models to 1-2 BLEU points, demonstrating the effectiveness of our proposed method.","doi":"10.24963/ijcai.2020/534","url":"https://ijcai.org/proceedings/2020/534","pdf":"https://ijcai.org/proceedings/2020/0534.pdf","markdown":"https://ijcai.org/proceedings/2020/534.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/534","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Natural Language Processing","track":"Main track","number":534,"pages":{"first":3861,"last":3867},"keywords":["Natural Language Processing: Machine Translation"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25148/file_25148.m3u8","length_sec":273,"provider":"hls"}],"references":[{"position":1,"raw":"[Anastasopoulos and Chiang, 2018] Antonios Anastasopou- los and David Chiang. Tied multitask learning for neural speech translation. In NAACL, pages 82-91, June 2018.","title":"Antonios Anastasopoulos and David Chiang. Tied multitask learning for neural speech translation","authors":"Chiang Anastasopoulos","year":"2018","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. ICLR, 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bengio et al., 2009] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICML, pages 41-48. ACM, 2009.","title":"Curriculum learning","authors":"Yoshua Bengio, Jérôme Louradour, Ronan Collobert, Jason Weston","year":"2009","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Caruana, 1997] Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.","title":"Multitask learning","authors":"Caruana","year":"1997","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In NAACL-HLT, 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dong et al., 2015] Daxiang Dong, Hua Wu, Wei He, Dian- hai Yu, and Haifeng Wang. Multi-task learning for mul- tiple language translation. In ACL-IJCNLP, pages 1723- 1732, 2015.","title":"Multi-task learning for multiple language translation","authors":"Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, Haifeng Wang","year":"2015","venue":"ACL-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Garg et al., 2019] Sarthak Garg, Stephan Peitz, Udhyaku- mar Nallasamy, and Matthias Paulik. Jointly learning to align and translate with transformer models. In EMNLP- IJCNLP, pages 4452-4461, November 2019.","title":"Jointly learning to align and translate with transformer models","authors":"Sarthak Garg, Stephan Peitz, Udhyakumar Nallasamy, Matthias Paulik","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gehring et al., 2017] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convo- lutional sequence to sequence learning. In ICML, pages 1243-1252, 2017.","title":"Convolutional sequence to sequence learning","authors":"Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N Dauphin","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ghazvininejad et al., 2019] Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. In EMNLP-IJCNLP, pages 6114-6123, 2019.","title":"Mask-predict: Parallel decoding of conditional masked language models","authors":"Marjan Ghazvininejad, Omer Levy, Yinhan Liu, Luke Zettlemoyer","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gu et al., 2018] Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. Non- autoregressive neural machine translation. In ICLR, 2018.","title":"Nonautoregressive neural machine translation","authors":"Gu","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gu et al., 2019] Jiatao Gu, Changhan Wang, and Junbo Zhao. Levenshtein transformer. In Advances in Neural In- formation Processing Systems, pages 11179-11189, 2019.","title":"Levenshtein transformer","authors":"Gu","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Guo et al., 2019a] Junliang Guo, Xu Tan, Di He, Tao Qin, Linli Xu, and Tie-Yan Liu. Non-autoregressive neural ma- chine translation with enhanced decoder input. In AAAI, volume 33, pages 3723-3730, 2019.","title":"Non-autoregressive neural machine translation with enhanced decoder input","authors":"Guo","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Guo et al., 2019b] Junliang Guo, Xu Tan, Linli Xu, Tao Qin, Enhong Chen, and Tie-Yan Liu. Fine-tuning by cur- riculum learning for non-autoregressive neural machine translation. arXiv preprint arXiv:1911.08717, 2019.","title":"Fine-tuning by curriculum learning for non-autoregressive neural machine translation","authors":"Guo","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kim and Rush, 2016] Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In EMNLP, pages 1317-1327, 2016.","title":"Sequence-level knowledge distillation","authors":"Yoon Kim, Alexander M Rush","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lee and Grauman, 2011] Yong Jae Lee and Kristen Grau- man. Learning the easy things first: Self-paced visual cat- egory discovery. In CVPR 2011, pages 1721-1728. IEEE, 2011.","title":"Learning the easy things first: Self-paced visual category discovery","authors":"Jae Yong, Kristen Lee, Grauman","year":"2011","venue":"CVPR 2011","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lee et al., 2018] Jason Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic non-autoregressive neural sequence modeling by iterative refinement. In EMNLP, pages 1173-1182, 2018.","title":"Deterministic non-autoregressive neural sequence modeling by iterative refinement","authors":"Lee","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2019] Zhuohan Li, Zi Lin, Di He, Fei Tian, QIN Tao, WANG Liwei, and Tie-Yan Liu. Hint-based training for non-autoregressive machine translation. In EMNLP- IJCNLP, pages 5712-5717, 2019.","title":"Hint-based training for non-autoregressive machine translation","authors":"Zhuohan Li, Zi Lin, Di He, Fei Tian, Tao, Tie-Yan Wang Liwei, Liu","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ma et al., 2019] Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig, and Eduard Hovy. Flowseq: Non- autoregressive conditional sequence generation with gen- erative flow. In EMNLP-IJCNLP, pages 4273-4283, 2019.","title":"Flowseq: Nonautoregressive conditional sequence generation with generative flow","authors":"Ma","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for au- tomatic evaluation of machine translation. In ACL, pages 311-318, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ren et al., 2019] Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech: Fast, robust and controllable text to speech. arXiv preprint arXiv:1905.09263, 2019.","title":"Fastspeech: Fast, robust and controllable text to speech","authors":"Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, Tie-Yan Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sachan and Xing, 2016] Mrinmaya Sachan and Eric Xing. Easy questions first? a case study on curriculum learning for question answering. In ACL, volume 1, pages 453-463, 2016.","title":"Easy questions first? a case study on curriculum learning for question answering","authors":"Xing Sachan, Mrinmaya Sachan, Eric Xing","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sarafianos et al., 2017] Nikolaos Sarafianos, Theodore Gi- annakopoulos, Christophoros Nikou, and Ioannis A Kaka- diaris. Curriculum learning for multi-task classification of visual attributes. In ICCV, pages 2608-2615, 2017.","title":"Curriculum learning for multi-task classification of visual attributes","authors":"Nikolaos Sarafianos, Theodore Giannakopoulos, Christophoros Nikou, Ioannis A Kakadiaris","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sennrich et al., 2016] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL, pages 1715-1725, 2016.","title":"Neural machine translation of rare words with subword units","authors":"Sennrich","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Song et al., 2019] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mass: Masked sequence to se- quence pre-training for language generation. In ICML, pages 5926-5936, 2019.","title":"Mass: Masked sequence to sequence pre-training for language generation","authors":"Song","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sutskever et al., 2014] I Sutskever, O Vinyals, and QV Le. Sequence to sequence learning with neural networks. Ad- vances in NIPS, 2014.","title":"QV Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"Advances in NIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 6000-6010, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Vaswani et al., 2018] Ashish Vaswani, Samy Bengio, Eu- gene Brevdo, Francois Chollet, Aidan Gomez, Stephan Gouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, et al. Tensor2tensor for neural machine trans- lation. In AMTA, pages 193-199, 2018.","title":"Tensor2tensor for neural machine translation","authors":"Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan Gomez, Stephan Gouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar","year":"2018","venue":"AMTA","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2018] Chunqi Wang, Ji Zhang, and Haiqing Chen. Semi-autoregressive neural machine translation. In EMNLP, pages 479-488, 2018.","title":"Semi-autoregressive neural machine translation","authors":"Wang","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2019] Yiren Wang, Fei Tian, Di He, Tao Qin, ChengXiang Zhai, and Tie-Yan Liu. Non-autoregressive machine translation with auxiliary regularization. In AAAI, 2019.","title":"Non-autoregressive machine translation with auxiliary regularization","authors":"Yiren Wang, Fei Tian, Di He, Tao Qin, Chengxiang Zhai, Tie-Yan Liu","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}