Pretrained Language Model for Text Generation: A Survey

Li, Junyi; Tang, Tianyi; Zhao, Wayne Xin; Wen, Ji-Rong

doi:10.24963/ijcai.2021/612

Pretrained Language Model for Text Generation: A Survey

Junyi Li, Tianyi Tang, Wayne Xin Zhao, Ji-Rong Wen

Download BibTeX Cite

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence. Survey Track. Pages 4492-4499.

https://doi.org/10.24963/ijcai.2021/612

Keywords

Natural language processing: General

Abstract

Text generation has become one of the most important yet challenging tasks in natural language processing (NLP). The resurgence of deep learning

has greatly advanced this field by neural generation models, especially the paradigm of pretrained language models (PLMs). In this paper, we present

an overview of the major advances achieved in the topic of PLMs for text generation. As the preliminaries, we present the general task definition

and briefly describe the mainstream architectures of PLMs for text generation. As the core content, we discuss how to adapt existing PLMs to model

different input data and satisfy special properties in the generated text. We further summarize several important fine-tuning strategies for text generation.

Finally, we present several future directions and conclude this paper. Our survey aims to provide text generation researchers a synthesis and pointer to related research.

Junyi Li, Tianyi Tang, Wayne Xin Zhao, and Ji-Rong Wen. 2021. Pretrained Language Model for Text Generation: A Survey. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21. IJCAI, 4492–4499. https://doi.org/10.24963/ijcai.2021/612

@inproceedings{ijcai2021p612,
  title     = {Pretrained Language Model for Text Generation: A Survey},
  author    = {Li, Junyi and Tang, Tianyi and Zhao, Wayne Xin and Wen, Ji-Rong},
  booktitle = {Proceedings of the Thirtieth International Joint Conference on
               Artificial Intelligence, {IJCAI-21}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Prof. Zhi-Hua Zhou},
  pages     = {4492--4499},
  year      = {2021},
  month     = {8},
  note      = {Survey Track},
  doi       = {10.24963/ijcai.2021/612},
  url       = {https://doi.org/10.24963/ijcai.2021/612},
}

TY  - CPAPER
TI  - Pretrained Language Model for Text Generation: A Survey
AU  - Li, Junyi
AU  - Tang, Tianyi
AU  - Zhao, Wayne Xin
AU  - Wen, Ji-Rong
T2  - Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2021
SP  - 4492
EP  - 4499
DO  - 10.24963/ijcai.2021/612
UR  - https://doi.org/10.24963/ijcai.2021/612
ER  -