{"type":"ScholarlyArticle","title":"CERT: Continual Pre-training on Sketches for Library-oriented Code Generation","authors":[{"name":"Daoguang Zan","slug":"daoguang-zan","given":"Daoguang","family":"Zan","affiliation":"Cooperative Innovation Center , Institute of Software , Chinese Academy of Sciences; University of Chinese Academy of Sciences"},{"name":"Bei Chen","slug":"bei-chen","given":"Bei","family":"Chen","affiliation":"Microsoft Research Asia"},{"name":"Dejian Yang","slug":"dejian-yang","given":"Dejian","family":"Yang","affiliation":"Microsoft Research Asia"},{"name":"Zeqi Lin","slug":"zeqi-lin","given":"Zeqi","family":"Lin","affiliation":"Microsoft Research Asia"},{"name":"Minsu Kim","slug":"minsu-kim","given":"Minsu","family":"Kim","affiliation":"Korea University"},{"name":"Bei Guan","slug":"bei-guan","given":"Bei","family":"Guan","affiliation":"University of Chinese Academy of Sciences; Integrative Innovation Center , Institute of Software , Chinese Academy of Sciences"},{"name":"Yongji Wang","slug":"yongji-wang","given":"Yongji","family":"Wang","affiliation":"University of Chinese Academy of Sciences; Integrative Innovation Center , Institute of Software , Chinese Academy of Sciences; State Key Laboratory of Computer Science , Institute of Software , Chinese Academy of Sciences"},{"name":"Weizhu Chen","slug":"weizhu-chen","given":"Weizhu","family":"Chen","affiliation":"Microsoft Azure AI"},{"name":"Jian-Guang Lou","slug":"jian-guang-lou","given":"Jian-Guang","family":"Lou","affiliation":"Microsoft Research Asia"}],"abstract":"Code generation is a longstanding challenge, aiming to generate a code snippet based on a natural language description. Usually, expensive text-code paired data is essential for training a code generation model. Recently, thanks to the success of pre-training techniques, large language models are trained on large unlabelled code corpora and perform well in generating code. In this paper, we investigate how to leverage an unlabelled code corpus to train a model for library-oriented code generation. Since it is a common practice for programmers to reuse third-party libraries, in which case the text-code paired data are harder to obtain due to the huge number of libraries. We observe that library-oriented code snippets are more likely to share similar code sketches. Hence, we present CERT with two steps: a sketcher generates the sketch, then a generator fills the details in the sketch. Both the sketcher and generator are continually pre-trained upon a base model using unlabelled data. Also, we carefully craft two benchmarks to evaluate library-oriented code generation named PandasEval and NumpyEval. Experimental results have shown the impressive performance of CERT. For example, it surpasses the base model by an absolute 15.67% improvement in terms of pass@1 on PandasEval. Our work is available at https://github.com/microsoft/PyCodeGPT.","doi":"10.24963/ijcai.2022/329","url":"https://ijcai.org/proceedings/2022/329","pdf":"https://ijcai.org/proceedings/2022/0329.pdf","markdown":"https://ijcai.org/proceedings/2022/329.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/329","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Data Mining","track":"Main Track","number":329,"pages":{"first":2369,"last":2375},"keywords":["Natural Language Processing: Language Models","Multidisciplinary Topics and Applications: Software Engineering","Natural Language Processing: Resources and Evaluation","Data Mining: Mining Codebase and Software Repository"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986491","length_sec":89,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984634","length_sec":352,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Ahmad et al., 2021] Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. Unified pre-training for program understanding and generation. In North Amer- ican Chapter of the Association for Computational Lin- guistics, pages 2655-2668, 2021.","title":"Unified pre-training for program understanding and generation","authors":"Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, Kai-Wei Chang","year":"2021","venue":"North American Chapter of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Black et al., 2021] Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Au- toregressive Language Modeling with Mesh-Tensorflow. In Zenodo, March 2021.","title":"GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow","authors":"Sid Black, Gao Leo, Phil Wang, Connor Leahy, Stella Biderman","year":"2021","venue":"Zenodo","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brown et al., 2021] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Neural Information Processing Systems, 2021.","title":"Language models are few-shot learners","authors":"Benjamin Tom B Brown, Nick Mann, Melanie Ryder, Jared Subbiah, Prafulla Kaplan, Arvind Dhariwal, Pranav Neelakantan, Girish Shyam, Amanda Sastry, Askell","year":"2021","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2021a] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.","title":"Evaluating large language models trained on code","authors":"Chen","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2021b] Xinyun Chen, Linyuan Gong, Alvin Cheung, and Dawn Song. Plotcoder: Hierarchical decod- ing for synthesizing visualization code in programmatic context. In Association for Computational Linguistics, pages 2169-2181, 2021.","title":"Plotcoder: Hierarchical decoding for synthesizing visualization code in programmatic context","authors":"Chen","year":"2021","venue":"Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dong and Lapata, 2018] Li Dong and Mirella Lapata. Coarse-to-fine decoding for neural semantic parsing. In Association for Computational Linguistics, pages 731- 742, 2018.","title":"Coarse-to-fine decoding for neural semantic parsing","authors":"Li Dong, Mirella Lapata","year":"2018","venue":"Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Feng et al., 2020] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. CodeBERT: A pre- trained model for programming and natural languages. In Findings of the Conference on Empirical Methods in Nat- ural Language Processing, pages 1536-1547, 2020.","title":"CodeBERT: A pretrained model for programming and natural languages","authors":"Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang","year":"2020","venue":"Findings of the Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Guo et al., 2020] Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, et al. GraphCodeBERT: Pre-training code representations with data flow. In International Con- ference on Learning Representations, 2020.","title":"GraphCodeBERT: Pre-training code representations with data flow","authors":"Shuo Daya Guo, Shuai Ren, Zhangyin Lu, Duyu Feng, Tang","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hendrycks et al., 2021] Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, et al. Measuring coding challenge competence with apps. In Neural Information Processing Systems Datasets and Benchmarks Track, 2021.","title":"Measuring coding challenge competence with apps","authors":"Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo","year":"2021","venue":"Neural Information Processing Systems Datasets and Benchmarks Track","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kanade et al., 2020] Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Learning and evaluating contextual embedding of source code. In International Conference on Machine Learning, pages 5110-5121, 2020.","title":"Learning and evaluating contextual embedding of source code","authors":"Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, Kensen Shi","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2022] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. arXiv preprint arXiv:2203.07814, 2022.","title":"Competition-level code generation with alphacode","authors":"Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lu et al., 2021] Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, et al. CodeXGLUE: A machine learning bench- mark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664, 2021.","title":"CodeXGLUE: A machine learning benchmark dataset for code understanding and generation","authors":"Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Murali et al., 2018] Vijayaraghavan Murali, Letao Qi, et al. Neural sketch learning for conditional program generation. In International Conference on Learning Representations, 2018.","title":"Neural sketch learning for conditional program generation","authors":"Vijayaraghavan Murali, Letao Qi","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Nijkamp et al., 2022] Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, et al. A conversational paradigm for program synthesis. arXiv preprint arXiv:2203.13474, 2022.","title":"A conversational paradigm for program synthesis","authors":"Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Niu et al., 2017] Haoran Niu, Iman Keivanloo, and Ying Zou. API usage pattern recommendation for software de- velopment. Journal of Systems and Software, 129:127- 139, 2017.","title":"API usage pattern recommendation for software development","authors":"Haoran Niu, Iman Keivanloo, Ying Zou","year":"2017","venue":"Journal of Systems and Software","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nye et al., 2019] Maxwell Nye, Luke Hewitt, Joshua Tenenbaum, and Armando Solar-Lezama. Learning to in- fer program sketches. In International Conference on Ma- chine Learning, pages 4861-4870, 2019.","title":"Learning to infer program sketches","authors":"Nye","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Paszke et al., 2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, et al. PyTorch: An imperative style, high-performance deep learning library. In Neural Infor- mation Processing Systems. 2019.","title":"PyTorch: An imperative style, high-performance deep learning library","authors":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer","year":"2019","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sun et al., 2019] Zeyu Sun, Qihao Zhu, Lili Mou, Yingfei Xiong, Ge Li, and Lu Zhang. A grammar-based structural cnn decoder for code generation. In Association for the Advancement of Artificial Intelligence, number 01, pages 7055-7062, 2019.","title":"A grammar-based structural cnn decoder for code generation","authors":"Sun","year":"2019","venue":"Association for the Advancement of Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2013] Jue Wang, Yingnong Dang, Hongyu Zhang, Kai Chen, Tao Xie, and Dongmei Zhang. Min- ing succinct and high-coverage API usage patterns from source code. In Mining Software Repositories, pages 319- 328, 2013.","title":"Mining succinct and high-coverage API usage patterns from source code","authors":"Jue Wang, Yingnong Dang, Hongyu Zhang, Kai Chen, Tao Xie, Dongmei Zhang","year":"2013","venue":"Mining Software Repositories","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2021] Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understand- ing and generation. In Empirical Methods in Natural Lan- guage Processing, pages 8696-8708, 2021.","title":"CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation","authors":"Yue Wang, Weishi Wang, Shafiq Joty, Steven Ch Hoi","year":"2021","venue":"Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wolf et al., 2019] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State- of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019.","title":"Huggingface's transformers: Stateof-the-art natural language processing","authors":"Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xu et al., 2022] Frank F Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. A systematic evaluation of large language models of code. In Deep Learning for Code Workshop, 2022.","title":"A systematic evaluation of large language models of code","authors":"Uri Frank F Xu, Graham Alon, Vincent Neubig, Josua Hellendoorn","year":"2022","venue":"Deep Learning for Code Workshop","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhong et al., 2009] Hao Zhong, Tao Xie, Lu Zhang, Jian Pei, and Hong Mei. MAPO: Mining and recommending API usage patterns. In European Conference on Object- Oriented Programming, pages 318-343. Springer, 2009.","title":"MAPO: Mining and recommending API usage patterns","authors":"Hao Zhong, Tao Xie, Lu Zhang, Jian Pei, Hong Mei","year":"2009","venue":"European Conference on Object-Oriented Programming","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}