{"type":"ScholarlyArticle","title":"Lyra: A Benchmark for Turducken-Style Code Generation","authors":[{"name":"Qingyuan Liang","slug":"qingyuan-liang","given":"Qingyuan","family":"Liang","affiliation":"Key Laboratory of High Confidence Software Technologies , Ministry of Education ( Peking University) .; School of Software & Microelectronics , Peking University. Beijing , PR China"},{"name":"Zeyu Sun","slug":"zeyu-sun","given":"Zeyu","family":"Sun","affiliation":"Key Laboratory of High Confidence Software Technologies , Ministry of Education ( Peking University) ."},{"name":"Qihao Zhu","slug":"qihao-zhu","given":"Qihao","family":"Zhu","affiliation":"Key Laboratory of High Confidence Software Technologies , Ministry of Education ( Peking University) ."},{"name":"Wenjie Zhang","slug":"wenjie-zhang","given":"Wenjie","family":"Zhang","affiliation":"Key Laboratory of High Confidence Software Technologies , Ministry of Education ( Peking University) .; School of Computer Science , Peking University. Beijing , PR China"},{"name":"Lian Yu","slug":"lian-yu","given":"Lian","family":"Yu","affiliation":"School of Software & Microelectronics , Peking University. Beijing , PR China"},{"name":"Yingfei Xiong","slug":"yingfei-xiong","given":"Yingfei","family":"Xiong","affiliation":"Key Laboratory of High Confidence Software Technologies , Ministry of Education ( Peking University) ."},{"name":"Lu Zhang","slug":"lu-zhang","given":"Lu","family":"Zhang","affiliation":"Key Laboratory of High Confidence Software Technologies , Ministry of Education ( Peking University) ."}],"abstract":"Recently, neural techniques have been used to generate source code automatically. While promising for declarative languages, these approaches achieve much poorer performance on datasets for imperative languages. Since a declarative language is typically embedded in an imperative language (i.e., the turducken-style programming) in real-world software development, the promising results on declarative languages can hardly lead to significant reduction of manual software development efforts.\r\n\r\nIn this paper, we define a new code generation task: given a natural language comment, this task aims to generate a program in a base imperative language with an embedded declarative language. To our knowledge, this is the first turducken-style code generation task. For this task, we present Lyra: a dataset in Python with embedded SQL. This dataset contains 2,000 carefully annotated database manipulation programs from real usage projects. Each program is paired with both a Chinese comment and an English comment. In our experiment, we adopted Transformer, BERT-style, and GPT-style models as baselines. In the best setting, GPT-style model can achieve 24% and 25.5% AST exact matching accuracy using Chinese and English comments, respectively. Therefore, we believe that Lyra provides a new challenge for code generation. Yet, overcoming this challenge may significantly boost the applicability of code generation techniques for real-world software development.","doi":"10.24963/ijcai.2022/588","url":"https://ijcai.org/proceedings/2022/588","pdf":"https://ijcai.org/proceedings/2022/0588.pdf","markdown":"https://ijcai.org/proceedings/2022/588.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/588","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Natural Language Processing","track":"Main Track","number":588,"pages":{"first":4238,"last":4244},"keywords":["Natural Language Processing: Applications","Natural Language Processing: Resources and Evaluation"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986595","length_sec":75,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984738","length_sec":284,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Chen et al., 2021] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.","title":"Evaluating large language models trained on code","authors":"Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dahl et al., 1994] Deborah A. Dahl, Madeleine Bates, Michael Brown, William M. Fisher, Kate Hunicke-Smith, David S. Pallett, Christine Pao, Alexander I. Rudnicky, and Elizabeth Shriberg. Expanding the scope of the ATIS task: The ATIS-3 corpus. In HLT. Morgan Kaufmann, 1994.","title":"Expanding the scope of the ATIS task: The ATIS-3 corpus","authors":"Deborah A Dahl, Madeleine Bates, Michael Brown, William M Fisher, Kate Hunicke-Smith, David S Pallett, Christine Pao, Alexander I Rudnicky, Elizabeth Shriberg","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Feng et al., 2020] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. Code- bert: A pre-trained model for programming and natural languages. In Trevor Cohn, Yulan He, and Yang Liu, edi- tors, EMNLP, pages 1536-1547, 2020.","title":"Codebert: A pre-trained model for programming and natural languages","authors":"Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, Ming Zhou","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Guo et al., 2020] Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. Graphcodebert: Pre-training code representations with data flow. arXiv preprint arXiv:2009.08366, 2020.","title":"Graphcodebert: Pre-training code representations with data flow","authors":"Shuo Daya Guo, Shuai Ren, Zhangyin Lu, Duyu Feng, Shujie Tang, Long Liu, Nan Zhou, Alexey Duan, Shengyu Svyatkovskiy, Fu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Husain et al., 2019] Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. Code- searchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436, 2019.","title":"Codesearchnet challenge: Evaluating the state of semantic code search","authors":"Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, Marc Brockschmidt","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Iyer et al., 2018] Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. Mapping language to code in programmatic context. In EMNLP, 2018.","title":"Mapping language to code in programmatic context","authors":"Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Luke Zettlemoyer","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Li et al., 2021] Yeting Li, Shuaimin Li, Zhiwu Xu, Jialun Cao, Zixuan Chen, Yun Hu, Haiming Chen, and Shing- Chi Cheung. Transregex: Multi-modal regular expression synthesis by generate-and-repair. In ICSE, 2021.","title":"Transregex: Multi-modal regular expression synthesis by generate-and-repair","authors":"Yeting Li, Shuaimin Li, Zhiwu Xu, Jialun Cao, Zixuan Chen, Yun Hu, Haiming Chen, Shing-Chi Cheung","year":"2021","venue":"ICSE","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Liang et al., 2022] Qingyuan Liang, Qihao Zhu, Zeyu Sun, Lu Zhang, Wenjie Zhang, Yingfei Xiong, Guangtai Liang, and Lian Yu. A survey of deep learning based text-to-sql generation (in chinese). Sci Sin Inform, page 1-30, 2022.","title":"A survey of deep learning based text-to-sql generation","authors":"Qingyuan Liang, Qihao Zhu, Zeyu Sun, Lu Zhang, Wenjie Zhang, Yingfei Xiong, Guangtai Liang, Lian Yu","year":"2022","venue":"Sci Sin Inform","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ling et al., 2016] Wang Ling, Phil Blunsom, Edward Grefenstette, Karl Moritz Hermann, Tomás Kociský, Fumin Wang, and Andrew W. Senior. Latent predictor networks for code generation. In ACL, 2016.","title":"Latent predictor networks for code generation","authors":"Wang Ling, Phil Blunsom, Edward Grefenstette, Karl Moritz Hermann, Tomás Kociský, Fumin Wang, Andrew W Senior","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2020] H. Liu, M. Shen, J. Zhu, N. Niu, G. Li, and L. Zhang. Deep learning based program generation from requirements text: Are we there yet? IEEE Transactions on Software Engineering, 2020.","title":"Deep learning based program generation from requirements text: Are we there yet","authors":"H Liu, M Shen, J Zhu, N Niu, G Li, L Zhang","year":"2020","venue":"IEEE Transactions on Software Engineering","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lu et al., 2021] Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664, 2021.","title":"Codexglue: A machine learning benchmark dataset for code understanding and generation","authors":"Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Oda et al., 2015] Yusuke Oda, Hiroyuki Fudaba, Graham Neubig, Hideaki Hata, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. Learning to generate pseudo-code from source code using statistical machine translation. In ASE, pages 574-584, 2015.","title":"Learning to generate pseudo-code from source code using statistical machine translation","authors":"Oda","year":"2015","venue":"ASE","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for au- tomatic evaluation of machine translation. In ACL, page 311-318, 2002.","title":"Bleu: A method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Price, 1990] Patti J. Price. Evaluation of spoken language systems: the ATIS domain. In HLT, 1990.","title":"Evaluation of spoken language systems: the ATIS domain","authors":"Patti J Price","year":"1990","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Radford et al., 2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Lan- guage models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.","title":"Language models are unsupervised multitask learners","authors":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever","year":"2019","venue":"OpenAI blog","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sennrich et al., 2016] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL, 2016.","title":"Neural machine translation of rare words with subword units","authors":"Sennrich","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sommerville, 1992] Ian Sommerville. Software engineer- ing, 4th Edition. International computer science series. Addison-Wesley, 1992.","title":"Software engineering","authors":"Ian Sommerville","year":"1992","venue":"International computer science series","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sun et al., 2020] Zeyu Sun, Qihao Zhu, Yingfei Xiong, Yi- can Sun, Lili Mou, and Lu Zhang. Treegen: A tree-based transformer architecture for code generation. In AAAI, pages 8984-8991, 2020.","title":"Treegen: A tree-based transformer architecture for code generation","authors":"Sun","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tao et al., 2020] Chuanqi Tao, Panpan Bao, and Zhiqiu Huang. Code line generation based on deep context- awareness of onsite programming. Sci. China Inf. Sci., 63(9):1-3, 2020.","title":"Code line generation based on deep contextawareness of onsite programming","authors":"Tao","year":"2020","venue":"Sci. China Inf. Sci","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xuan et al., 2021] Kuan Xuan, Yongbo Wang, Yongliang Wang, Zujie Wen, and Yang Dong. Sead: End-to-end text-to-sql generation with schema-aware denoising. arXiv preprint arXiv:2105.07911, 2021.","title":"Sead: End-to-end text-to-sql generation with schema-aware denoising","authors":"Yongbo Kuan Xuan, Yongliang Wang, Zujie Wang, Yang Wen, Dong","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yaghmazadeh et al., 2017] Navid Yaghmazadeh, Yuepeng Wang, Isil Dillig, and Thomas Dillig. Sqlizer: query syn- thesis from natural language. In OOPSLA, pages 1-26, 2017.","title":"Sqlizer: query synthesis from natural language","authors":"Navid Yaghmazadeh, Yuepeng Wang, Isil Dillig, Thomas Dillig","year":"2017","venue":"OOPSLA","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yin et al., 2018] Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. Learning to mine aligned code and natural language pairs from stack over- flow. In ICMSP, MSR '18, page 476-486, 2018.","title":"Learning to mine aligned code and natural language pairs from stack overflow","authors":"Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, Graham Neubig","year":"2018","venue":"ICMSP","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yu et al., 2018] Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir R. Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In EMNLP, pages 3911-3921, 2018.","title":"Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task","authors":"Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, Dragomir R Radev","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhong et al., 2017] Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1709.00103, 2017.","title":"Seq2sql: Generating structured queries from natural language using reinforcement learning","authors":"Victor Zhong, Caiming Xiong, Richard Socher","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}