{"type":"ScholarlyArticle","title":"Curriculum-Based Self-Training Makes Better Few-Shot Learners for Data-to-Text Generation","authors":[{"name":"Pei Ke","slug":"pei-ke","given":"Pei","family":"Ke","affiliation":"CoAI Group , DCST, IAI , BNRIST , Tsinghua University , Beijing , China"},{"name":"Haozhe Ji","slug":"haozhe-ji","given":"Haozhe","family":"Ji","affiliation":"CoAI Group , DCST, IAI , BNRIST , Tsinghua University , Beijing , China"},{"name":"Zhenyu Yang","slug":"zhenyu-yang","given":"Zhenyu","family":"Yang","affiliation":"OPPO Mobile Telecommunications Corp. , Ltd , China"},{"name":"Yi Huang","slug":"yi-huang","given":"Yi","family":"Huang","affiliation":"JIUTIAN Team , China Mobile Research Institute , Beijing 100053 , China; Tsinghua University-China Mobile Communications Group Co., Ltd. Joint Institute , Beijing , China"},{"name":"Junlan Feng","slug":"junlan-feng","given":"Junlan","family":"Feng","affiliation":"JIUTIAN Team , China Mobile Research Institute , Beijing 100053 , China; Tsinghua University-China Mobile Communications Group Co., Ltd. Joint Institute , Beijing , China"},{"name":"Xiaoyan Zhu","slug":"xiaoyan-zhu","given":"Xiaoyan","family":"Zhu","affiliation":"CoAI Group , DCST, IAI , BNRIST , Tsinghua University , Beijing , China"},{"name":"Minlie Huang","slug":"minlie-huang","given":"Minlie","family":"Huang","affiliation":"CoAI Group , DCST, IAI , BNRIST , Tsinghua University , Beijing , China; Tsinghua University-China Mobile Communications Group Co., Ltd. Joint Institute , Beijing , China"}],"abstract":"Despite the success of text-to-text pre-trained models in various natural language generation (NLG) tasks, the generation performance is largely restricted by the number of labeled data in downstream tasks, particularly in data-to-text generation tasks. Existing works mostly utilize abundant unlabeled structured data to conduct unsupervised pre-training for task adaption, which fail to model the complex relationship between source structured data and target texts. Thus, we introduce self-training as a better few-shot learner than task-adaptive pre-training, which explicitly captures this relationship via pseudo-labeled data generated by the pre-trained model. To alleviate the side-effect of low-quality pseudo-labeled data during self-training, we propose a novel method called Curriculum-Based Self-Training (CBST) to effectively leverage unlabeled data in a rearranged order determined by the difficulty of text generation. Experimental results show that our method can outperform fine-tuning and task-adaptive pre-training methods, and achieve state-of-the-art performance in the few-shot setting of data-to-text generation.","doi":"10.24963/ijcai.2022/580","url":"https://ijcai.org/proceedings/2022/580","pdf":"https://ijcai.org/proceedings/2022/0580.pdf","markdown":"https://ijcai.org/proceedings/2022/580.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/580","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Natural Language Processing","track":"Main Track","number":580,"pages":{"first":4178,"last":4184},"keywords":["Natural Language Processing: Language Generation","Natural Language Processing: Language Models"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986761","length_sec":76,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984904","length_sec":336,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Banerjee and Lavie, 2005] Satanjeev Banerjee and Alon Lavie. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In Sum- marization@ACL 2005, pages 65-72, 2005.","title":"METEOR: an automatic metric for MT evaluation with improved correlation with human judgments","authors":"Lavie Banerjee, Satanjeev Banerjee, Alon Lavie","year":"2005","venue":"Summarization@ACL 2005","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2009] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICML, volume 382, pages 41-48, 2009.","title":"Curriculum learning","authors":"Yoshua Bengio, Jérôme Louradour, Ronan Collobert, Jason Weston","year":"2009","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2020a] Wenhu Chen, Yu Su, Xifeng Yan, and William Yang Wang. KGPT: knowledge-grounded pre- training for data-to-text generation. In EMNLP, 2020.","title":"KGPT: knowledge-grounded pretraining for data-to-text generation","authors":"Chen","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2020b] Zhiyu Chen, Harini Eavani, Wenhu Chen, Yinyin Liu, and William Yang Wang. Few-shot NLG with pre-trained language model. In ACL, 2020.","title":"Few-shot NLG with pre-trained language model","authors":"Chen","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Du et al., 2021] Jingfei Du, Edouard Grave, Beliz Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Veselin Stoyanov, and Alexis Conneau. Self-training improves pre-training for natural language understanding. In NAACL-HLT, pages 5408-5418, 2021.","title":"Self-training improves pre-training for natural language understanding","authors":"Du","year":"2021","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fleiss, 1971] J. Fleiss. Measuring nominal scale agreement among many raters. Psychological Bulletin, 1971.","title":"Measuring nominal scale agreement among many raters","authors":"J Fleiss","year":"1971","venue":"Psychological Bulletin","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gehrmann et al., 2018] Sebastian Gehrmann, Falcon Z. Dai, Henry Elder, and Alexander M. Rush. End-to-end content and plan selection for data-to-text generation. In INLG, pages 46-56, 2018.","title":"End-to-end content and plan selection for data-to-text generation","authors":"Sebastian Gehrmann, Z Falcon, Henry Dai, Alexander M Elder, Rush","year":"2018","venue":"INLG","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gong et al., 2020] Heng Gong, Yawei Sun, Xiaocheng Feng, Bing Qin, Wei Bi, Xiaojiang Liu, and Ting Liu. Tablegpt: Few-shot table-to-text generation with table structure reconstruction and content matching. In COL- ING, pages 1978-1988, 2020.","title":"Tablegpt: Few-shot table-to-text generation with table structure reconstruction and content matching","authors":"Heng Gong, Yawei Sun, Xiaocheng Feng, Bing Qin, Wei Bi, Xiaojiang Liu, Ting Liu","year":"2020","venue":"COL-ING","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gururangan et al., 2020] Suchin Gururangan, Ana Maraso- vic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models to domains and tasks. In ACL, pages 8342-8360, 2020.","title":"Don't stop pretraining: Adapt language models to domains and tasks","authors":"Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, Noah A Smith","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2020] Junxian He, Jiatao Gu, Jiajun Shen, and Marc'Aurelio Ranzato. Revisiting self-training for neural sequence generation. In ICLR, 2020.","title":"Revisiting self-training for neural sequence generation","authors":"He","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hoyle et al., 2021] Alexander Miserlis Hoyle, Ana Maraso- vic, and Noah A. Smith. Promoting graph awareness in linearized graph-to-text generation. In Findings of ACL/IJCNLP, pages 944-956, 2021.","title":"Promoting graph awareness in linearized graph-to-text generation","authors":"Alexander Miserlis Hoyle, Ana Marasovic, Noah A Smith","year":"2021","venue":"Findings of ACL/IJCNLP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jia et al., 2019] Robin Jia, Aditi Raghunathan, Kerem Göksel, and Percy Liang. Certified robustness to adver- sarial word substitutions. In EMNLP-IJCNLP, 2019.","title":"Certified robustness to adversarial word substitutions","authors":"Robin Jia, Aditi Raghunathan, Kerem Göksel, Percy Liang","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jin et al., 2020] Zhijing Jin, Qipeng Guo, Xipeng Qiu, and Zheng Zhang. Genwiki: A dataset of 1.3 million content- sharing text and graphs for unsupervised graph-to-text generation. In COLING, pages 2398-2409, 2020.","title":"Genwiki: A dataset of 1.3 million contentsharing text and graphs for unsupervised graph-to-text generation","authors":"Jin","year":"2020","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kale and Rastogi, 2020] Mihir Kale and Abhinav Rastogi. Text-to-text pre-training for data-to-text tasks. In INLG, pages 97-102, 2020.","title":"Text-to-text pre-training for data-to-text tasks","authors":"Rastogi Kale, Mihir Kale, Abhinav Rastogi","year":"2020","venue":"INLG","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ke et al., 2021] Pei Ke, Haozhe Ji, Yu Ran, Xin Cui, Liwei Wang, Linfeng Song, Xiaoyan Zhu, and Minlie Huang. JointGT: Graph-text joint representation learning for text generation from knowledge graphs. In Findings of ACL- IJCNLP, pages 2526-2538, 2021.","title":"JointGT: Graph-text joint representation learning for text generation from knowledge graphs","authors":"Pei Ke, Haozhe Ji, Yu Ran, Xin Cui, Liwei Wang, Linfeng Song, Xiaoyan Zhu, Minlie Huang","year":"2021","venue":"Findings of ACL-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lebret et al., 2016] Rémi Lebret, David Grangier, and Michael Auli. Neural text generation from structured data with application to the biography domain. In EMNLP, pages 1203-1213, 2016.","title":"Neural text generation from structured data with application to the biography domain","authors":"Lebret","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lewis et al., 2020] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In ACL, pages 7871-7880, 2020.","title":"BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension","authors":"Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, Luke Zettlemoyer","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lin, 2004] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.","title":"Rouge: A package for automatic evaluation of summaries","authors":"Chin-Yew Lin","year":"2004","venue":"Text summarization branches out","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for au- tomatic evaluation of machine translation. In ACL, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Raffel et al., 2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-140:67, 2020.","title":"Exploring the limits of transfer learning with a unified text-to-text transformer","authors":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu","year":"2020","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ribeiro et al., 2020a] Leonardo F. R. Ribeiro, Martin Schmitt, Hinrich Schütze, and Iryna Gurevych. Investi- gating pretrained language models for graph-to-text gen- eration. CoRR, abs/2007.08426, 2020.","title":"Investigating pretrained language models for graph-to-text generation","authors":"Ribeiro","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ribeiro et al., 2020b] Leonardo F. R. Ribeiro, Yue Zhang, Claire Gardent, and Iryna Gurevych. Modeling global and local node contexts for text generation from knowledge graphs. TACL, 8:589-604, 2020.","title":"Modeling global and local node contexts for text generation from knowledge graphs","authors":"Ribeiro","year":"2020","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Scudder, 1965] H. J. Scudder. Probability of error of some adaptive pattern-recognition machines. IEEE Trans. Inf. Theory, 11(3):363-371, 1965.","title":"Probability of error of some adaptive pattern-recognition machines","authors":"H J Scudder","year":"1965","venue":"IEEE Trans. Inf. Theory","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shimorina and Gardent, 2018] Anastasia Shimorina and Claire Gardent. Handling rare items in data-to-text generation. In INLG, pages 360-370, 2018.","title":"Anastasia Shimorina and Claire Gardent. Handling rare items in data-to-text generation","authors":"Gardent Shimorina","year":"2018","venue":"INLG","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Trisedya et al., 2018] Bayu Distiawan Trisedya, Jianzhong Qi, Rui Zhang, and Wei Wang. GTR-LSTM: A triple en- coder for sentence generation from RDF data. In ACL, pages 1627-1637, 2018.","title":"GTR-LSTM: A triple encoder for sentence generation from RDF data","authors":"Jianzhong Bayu Distiawan Trisedya, Rui Qi, Wei Zhang, Wang","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xing and Wan, 2021] Xinyu Xing and Xiaojun Wan. Structure-aware pre-training for table-to-text generation. In Findings of ACL-IJCNLP, pages 2273-2278, 2021.","title":"Structure-aware pre-training for table-to-text generation","authors":"Xinyu Xing, Xiaojun Wan","year":"2021","venue":"Findings of ACL-IJCNLP","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}