{"type":"ScholarlyArticle","title":"Pretrained Language Model for Text Generation: A Survey","authors":[{"name":"Junyi Li","slug":"junyi-li","given":"Junyi","family":"Li","affiliation":""},{"name":"Tianyi Tang","slug":"tianyi-tang","given":"Tianyi","family":"Tang","affiliation":""},{"name":"Wayne Xin Zhao","slug":"wayne-xin-zhao","given":"Wayne Xin","family":"Zhao","affiliation":""},{"name":"Ji-Rong Wen","slug":"ji-rong-wen","given":"Ji-Rong","family":"Wen","affiliation":""}],"abstract":"Text generation has become one of the most important yet challenging tasks in natural language processing (NLP). The resurgence of deep learning\r\nhas greatly advanced this field by neural generation models, especially the paradigm of pretrained language models (PLMs). In this paper, we present\r\nan overview of the major advances achieved in the topic of PLMs for text generation. As the preliminaries, we present the general task definition\r\nand briefly describe the mainstream architectures of PLMs for text generation. As the core content, we discuss how to adapt existing PLMs to model\r\ndifferent input data and satisfy special properties in the generated text. We further summarize several important fine-tuning strategies for text generation. \r\nFinally, we present several future directions and conclude this paper. Our survey aims to provide text generation researchers a synthesis and pointer to related research.","doi":"10.24963/ijcai.2021/612","url":"https://ijcai.org/proceedings/2021/612","pdf":"https://ijcai.org/proceedings/2021/0612.pdf","markdown":"https://ijcai.org/proceedings/2021/612.md","bibtex":"https://ijcai.org/proceedings/2021/bibtex/612","volume":{"slug":"2021","title":"Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2021","url":"https://ijcai.org/proceedings/2021/","year":2021,"location":"Montreal","dates":"19-27 August 2021","editors":"Prof. Zhi-Hua Zhou","isbn":"978-0-9992411-9-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2021"},"section":"Survey Track","track":"Survey Track","number":612,"pages":{"first":4492,"last":4499},"keywords":["Natural language processing: General"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}