{"type":"ScholarlyArticle","title":"ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation","authors":[{"name":"Dongling Xiao","slug":"dongling-xiao","given":"Dongling","family":"Xiao","affiliation":"Baidu Inc. , China"},{"name":"Han Zhang","slug":"han-zhang","given":"Han","family":"Zhang","affiliation":"Baidu Inc. , China"},{"name":"Yukun Li","slug":"yukun-li","given":"Yukun","family":"Li","affiliation":"Baidu Inc. , China"},{"name":"Yu Sun","slug":"yu-sun","given":"Yu","family":"Sun","affiliation":"Baidu Inc. , China"},{"name":"Hao Tian","slug":"hao-tian","given":"Hao","family":"Tian","affiliation":"Baidu Inc. , China"},{"name":"Hua Wu","slug":"hua-wu","given":"Hua","family":"Wu","affiliation":"Baidu Inc. , China"},{"name":"Haifeng Wang","slug":"haifeng-wang","given":"Haifeng","family":"Wang","affiliation":"Baidu Inc. , China"}],"abstract":"Current pre-training works in natural language generation pay little attention to the problem of exposure bias on downstream tasks. To address this issue, we propose an enhanced multi-flow sequence to sequence pre-training and fine-tuning framework named ERNIE-GEN, which bridges the discrepancy between training and inference with an infilling generation mechanism and a noise-aware generation method. To make generation closer to human writing patterns, this framework introduces a span-by-span generation flow that trains the model to predict semantically-complete spans consecutively rather than predicting word by word. Unlike existing pre-training methods, ERNIE-GEN incorporates multi-granularity target sampling to construct pre-training data, which enhances the correlation between encoder and decoder. Experimental results demonstrate that ERNIE-GEN achieves state-of-the-art results with a much smaller amount of pre-training data and parameters on a range of language generation tasks, including abstractive summarization (Gigaword and CNN/DailyMail), question generation (SQuAD), dialogue generation (Persona-Chat) and generative question answering (CoQA). The source codes and pre-trained models have been released at https://github.com/PaddlePaddle/ERNIE/ernie-gen.","doi":"10.24963/ijcai.2020/553","url":"https://ijcai.org/proceedings/2020/553","pdf":"https://ijcai.org/proceedings/2020/0553.pdf","markdown":"https://ijcai.org/proceedings/2020/553.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/553","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Natural Language Processing","track":"Main track","number":553,"pages":{"first":3997,"last":4003},"keywords":["Natural Language Processing: Natural Language Processing","Natural Language Processing: Natural Language Generation","Natural Language Processing: Natural Language Summarization"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24829/file_24829.m3u8","length_sec":287,"provider":"hls"}],"references":[{"position":1,"raw":"[Bao et al., 2020] Siqi Bao, Huang He, Fan Wang, Hua Wu, and Haifeng Wang. Plato: Pre-trained dialogue generation model with discrete latent variable. In ACL, 2020.","title":"Plato: Pre-trained dialogue generation model with discrete latent variable","authors":"Siqi Bao, Huang He, Fan Wang, Hua Wu, Haifeng Wang","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In NAACL, pages 4171-4186, 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dong et al., 2019] Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiaowuen Hon. Unified language model pre-training for natural language understanding and generation. In NIPS, pages 13042-13054, 2019.","title":"Unified language model pre-training for natural language understanding and generation","authors":"Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, Hsiaowuen Hon","year":"2019","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Hermann et al., 2015] Karl Moritz Hermann, Tomas Ko- cisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In NIPS, pages 1693-1701, 2015.","title":"Teaching machines to read and comprehend","authors":"Hermann","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Joshi et al., 2019] Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. Span- bert: Improving pre-training by representing and predict- ing spans. arXiv preprint arXiv:1907.10529, 2019.","title":"Spanbert: Improving pre-training by representing and predicting spans","authors":"Mandar Joshi, Danqi Chen, Yinhan Liu, Luke Daniel S Weld, Omer Zettlemoyer, Levy","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Lewis et al., 2019] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: De- noising sequence-to-sequence pre-training for natural lan- guage generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019.","title":"Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension","authors":"Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, Luke Zettlemoyer","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Liu and Lapata, 2019] Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. In EMNLP- IJCNLP, pages 3721-3731, 2019.","title":"Text summarization with pretrained encoders","authors":"Yang Liu, Mirella Lapata","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Peters et al., 2018] Matthew E Peters, Mark Neumann, Mo- hit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word repre- sentations. In NAACL-HLT, pages 2227-2237, 2018.","title":"Deep contextualized word representations","authors":"Mark Matthew E Peters, Mohit Neumann, Matt Iyyer, Christopher Gardner, Kenton Clark, Luke Lee, Zettlemoyer","year":"2018","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Radford et al., 2018] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.","title":"Improving language understanding by generative pre-training","authors":"Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Raffel et al., 2019] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.","title":"Exploring the limits of transfer learning with a unified text-to-text transformer","authors":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Rajpurkar et al., 2016] Pranav Rajpurkar, Jian Zhang, Kon- stantin Lopyrev, and Percy Liang. Squad: 100,000+ ques- tions for machine comprehension of text. In EMNLP, pages 2383-2392, 2016.","title":"Squad: 100,000+ questions for machine comprehension of text","authors":"Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, Percy Liang","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ranzato et al., 2016] Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. In ICLR, 2016.","title":"Sequence level training with recurrent neural networks","authors":"Ranzato","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Reddy et al., 2019] Siva Reddy, Danqi Chen, and Christo- pher D Manning. Coqa: A conversational question an- swering challenge. In ACL, pages 249-266, 2019.","title":"Coqa: A conversational question answering challenge","authors":"Siva Reddy, Danqi Chen, Christopher D Manning","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Rothe et al., 2019] Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. Leveraging pre-trained check- points for sequence generation tasks. arXiv preprint arXiv:1907.12461, 2019.","title":"Leveraging pre-trained checkpoints for sequence generation tasks","authors":"Rothe","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Rush et al., 2015] Alexander M Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. In EMNLP, 2015.","title":"A neural attention model for abstractive sentence summarization","authors":"Sumit Alexander M Rush, Jason Chopra, Weston","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Song et al., 2019] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mass: Masked sequence to sequence pre-training for language generation. In ICML, 2019.","title":"Mass: Masked sequence to sequence pre-training for language generation","authors":"Song","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sun et al., 2019] Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxi- ang Zhu, Hao Tian, and Hua Wu. Ernie: Enhanced rep- resentation through knowledge integration. arXiv preprint arXiv:1904.09223, 2019.","title":"Ernie: Enhanced representation through knowledge integration","authors":"Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hua Hao Tian, Wu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sun et al., 2020] Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. Ernie 2.0: A continual pre-training framework for lan- guage understanding. In AAAI, 2020.","title":"Ernie 2.0: A continual pre-training framework for language understanding","authors":"Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hua Hao Tian, Haifeng Wu, Wang","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2018] Li Wang, Junlin Yao, Yunzhe Tao, Li Zhong, Wei Liu, and Qiang Du. A reinforced topic- aware convolutional sequence-to-sequence model for ab- stractive text summarization. In IJCAI, 2018.","title":"A reinforced topicaware convolutional sequence-to-sequence model for abstractive text summarization","authors":"Li Wang, Junlin Yao, Yunzhe Tao, Li Zhong, Wei Liu, Qiang Du","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2019] Wenlin Wang, Zhe Gan, Hongteng Xu, Ruiyi Zhang, Guoyin Wang, Dinghan Shen, Changyou Chen, and Lawrence Carin. Topic-guided variational auto- encoder for text generation. In NAACL-HLT, 2019.","title":"Topic-guided variational autoencoder for text generation","authors":"Wenlin Wang, Zhe Gan, Hongteng Xu, Ruiyi Zhang, Guoyin Wang, Dinghan Shen, Changyou Chen, Lawrence Carin","year":"2019","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yang et al., 2019a] Qian Yang, Dinghan Shen, Yong Cheng, Wenlin Wang, Guoyin Wang, Lawrence Carin, et al. An end-to-end generative architecture for paraphrase genera- tion. In EMNLP-IJCNLP, pages 3123-3133, 2019.","title":"An end-to-end generative architecture for paraphrase generation","authors":"Yang","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yang et al., 2019b] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for lan- guage understanding. In NIPS, pages 5754-5764, 2019.","title":"Xlnet: Generalized autoregressive pretraining for language understanding","authors":"Yang","year":"2019","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang and Bansal, 2019] Shiyue Zhang and Mohit Bansal. Addressing semantic drift in question generation for semi- supervised question answering. In EMNLP-IJCNLP, pages 2495-2509, 2019.","title":"Addressing semantic drift in question generation for semisupervised question answering","authors":"Bansal Zhang, Shiyue Zhang, Mohit Bansal","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2018] Saizheng Zhang, Emily Dinan, Jack Ur- banek, Arthur Szlam, Douwe Kiela, and Jason Weston. Personalizing dialogue agents: I have a dog, do you have pets too? In ACL, pages 2204-2213, 2018.","title":"Personalizing dialogue agents: I have a dog, do you have pets too? In ACL","authors":"Zhang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2019] Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J Liu. Pegasus: Pre-training with ex- tracted gap-sentences for abstractive summarization. arXiv preprint arXiv:1912.08777, 2019.","title":"Pegasus: Pre-training with extracted gap-sentences for abstractive summarization","authors":"Zhang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhao et al., 2018] Yao Zhao, Xiaochuan Ni, Yuanyuan Ding, and Qifa Ke. Paragraph-level neural question gen- eration with maxout pointer and gated self-attention net- works. In EMNLP, pages 3901-3910, 2018.","title":"Paragraph-level neural question generation with maxout pointer and gated self-attention networks","authors":"Yao Zhao, Xiaochuan Ni, Yuanyuan Ding, Qifa Ke","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}