{"type":"ScholarlyArticle","title":"Grape: Grammar-Preserving Rule Embedding","authors":[{"name":"Qihao Zhu","slug":"qihao-zhu","given":"Qihao","family":"Zhu","affiliation":"Key Laboratory of High Confidence Software Technologies , Ministry of Education( Peking University) ;"},{"name":"Zeyu Sun","slug":"zeyu-sun","given":"Zeyu","family":"Sun","affiliation":"Key Laboratory of High Confidence Software Technologies , Ministry of Education( Peking University) ;"},{"name":"Wenjie Zhang","slug":"wenjie-zhang","given":"Wenjie","family":"Zhang","affiliation":"Key Laboratory of High Confidence Software Technologies , Ministry of Education( Peking University) ;"},{"name":"Yingfei Xiong","slug":"yingfei-xiong","given":"Yingfei","family":"Xiong","affiliation":"School of Computer Science , Peking University , 100871 , P. R. China"},{"name":"Lu Zhang","slug":"lu-zhang","given":"Lu","family":"Zhang","affiliation":"School of Computer Science , Peking University , 100871 , P. R. China"}],"abstract":"Word embedding has been widely used in various areas to boost the performance of the neural models. However, when processing context-free languages, embedding grammar rules with word embedding loses two types of information. One is the structural relationship between the grammar rules, and the other one is the content information of the rule definition.\r\nIn this paper, we make the first attempt to learn a grammar-preserving rule embedding. We first introduce a novel graph structure to represent the context-free grammar. Then, we apply a Graph Neural Network (GNN) to extract the structural information and use a gating layer to integrate content information. \r\nWe conducted experiments on six widely-used benchmarks containing four context-free languages. The results show that our approach improves the accuracy of the base model by 0.8 to 6.4 percentage points. Furthermore, Grape also achieves 1.6 F1 score improvement on the method naming task which shows the generality of our approach.","doi":"10.24963/ijcai.2022/631","url":"https://ijcai.org/proceedings/2022/631","pdf":"https://ijcai.org/proceedings/2022/0631.pdf","markdown":"https://ijcai.org/proceedings/2022/631.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/631","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Natural Language Processing","track":"Main Track","number":631,"pages":{"first":4545,"last":4551},"keywords":["Natural Language Processing: Language Generation","Natural Language Processing: Applications","Natural Language Processing: Natural Language Semantics"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986614","length_sec":89,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984757","length_sec":354,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Alon et al., 2018] Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. code2seq: Generating sequences from struc- tured representations of code. In ICLR, 2018.","title":"code2seq: Generating sequences from structured representations of code","authors":"Alon","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alon et al., 2019] Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. Code2vec: Learning distributed repre- sentations of code. POPL, January 2019.","title":"Code2vec: Learning distributed representations of code","authors":"Uri Alon, Meital Zilberstein, Omer Levy, Eran Yahav","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dong and Lapata, 2016] Li Dong and Mirella Lapata. Lan- guage to Logical Form with Neural Attention. In ACL, pages 33-43, 2016.","title":"Language to Logical Form with Neural Attention","authors":"Li Dong, Mirella Lapata","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hellendoorn et al., 2020] Vincent J. Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, and David Bieber. Global relational models of source code. In ICLR, 2020.","title":"Global relational models of source code","authors":"J Vincent, Charles Hellendoorn, Rishabh Sutton, Petros Singh, David Maniatis, Bieber","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Iyer et al., 2018] Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. Mapping language to code in programmatic context. In EMNLP, pages 1643-1652, Brussels, Belgium, October-November 2018.","title":"Mapping language to code in programmatic context","authors":"Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Luke Zettlemoyer","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kipf and Welling, 2017] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.","title":"Semi-supervised classification with graph convolutional networks","authors":"Thomas N Kipf, Max Welling","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kwiatkowski et al., 2013] Tom Kwiatkowski, Eunsol Choi, Yoav Artzi, and Luke Zettlemoyer. Scaling semantic parsers with on-the-fly ontology matching. In EMNLP, pages 1545-1556, 2013.","title":"Scaling semantic parsers with on-the-fly ontology matching","authors":"Kwiatkowski","year":"2013","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2016] Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard S. Zemel. Gated graph sequence neural networks. In ICLR, 2016.","title":"Gated graph sequence neural networks","authors":"Yujia Li, Daniel Tarlow, Marc Brockschmidt, Richard S Zemel","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liang et al., 2022] Qingyuan Liang, Qihao Zhu, Zeyu Sun, Lu Zhang, Wenjie Zhang, Yingfei Xiong, Guangtai Liang, and Lian Yu. A survey of deep learning based text-to-sql generation (in chinese). SCIS, page 1-30, 2022.","title":"A survey of deep learning based text-to-sql generation","authors":"Qingyuan Liang, Qihao Zhu, Zeyu Sun, Lu Zhang, Wenjie Zhang, Yingfei Xiong, Guangtai Liang, Lian Yu","year":"2022","venue":"SCIS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ling et al., 2016] Wang Ling, Phil Blunsom, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiskỳ, Fumin Wang, and Andrew Senior. Latent Predictor Net- works for Code Generation. In ACL, pages 599-609, 2016.","title":"Latent Predictor Networks for Code Generation","authors":"Wang Ling, Phil Blunsom, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiskỳ, Fumin Wang, Andrew Senior","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lu et al., 2021] Shuai Lu, Daya Guo, et al. Codexglue: A machine learning benchmark dataset for code understand- ing and generation. In NeurIPS, 2021.","title":"Codexglue: A machine learning benchmark dataset for code understanding and generation","authors":"Shuai Lu, Daya Guo","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Kai Chen, Greg Cor- rado, and Jeffrey Dean. Efficient estimation of word rep- resentations in vector space. In ICLR, 2013.","title":"Efficient estimation of word representations in vector space","authors":"Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean","year":"2013","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mou et al., 2016] Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. Convolutional neural networks over tree struc- tures for programming language processing. In AAAI, pages 1287-1293, 2016.","title":"Convolutional neural networks over tree structures for programming language processing","authors":"Lili Mou, Ge Li, Lu Zhang, Tao Wang, Zhi Jin","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Park et al., 2019] Jun-U Park, Sang-Ki Ko, Marco Cognetta, and Yo-Sub Han. Softregex: Generating regex from natural language descriptions using softened regex equivalence. In EMNLP-IJCNLP, pages 6425-6431, 2019.","title":"Softregex: Generating regex from natural language descriptions using softened regex equivalence","authors":"Jun-U Park, Sang-Ki Ko, Marco Cognetta, Yo-Sub Han","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pennington et al., 2014] J. Pennington, R. Socher, and C. Manning. Glove: Global vectors for word represen- tation. In EMNLP, 2014.","title":"Glove: Global vectors for word representation","authors":"Pennington","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rabinovich et al., 2017] Maxim Rabinovich, Mitchell Stern, and Dan Klein. Abstract Syntax Networks for Code Generation and Semantic Parsing. In ACL, pages 1139-1149, 2017.","title":"Abstract Syntax Networks for Code Generation and Semantic Parsing","authors":"Rabinovich","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Radford et al., 2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.","title":"Language models are unsupervised multitask learners","authors":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever","year":"2019","venue":"OpenAI blog","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Scarselli et al., 2009] F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini. The graph neural network model. TNN, 20(1):61-80, 2009.","title":"The graph neural network model","authors":"F Scarselli, M Gori, A C Tsoi, M Hagenbuchner, G Monfardini","year":"2009","venue":"TNN","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shaw et al., 2019] Peter Shaw, Philip Massey, Angelica Chen, Francesco Piccinno, and Yasemin Altun. Gener- ating logical forms from graph representations of text and entities. In ACL, pages 95-106, July 2019.","title":"Generating logical forms from graph representations of text and entities","authors":"Peter Shaw, Philip Massey, Angelica Chen, Francesco Piccinno, Yasemin Altun","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sienčnik, 2015] Scharolta Katharina Sienčnik. Adapting word2vec to named entity recognition. In NODALIDA, pages 239-243, 2015.","title":"Scharolta Katharina Sienčnik. Adapting word2vec to named entity recognition","authors":"Sienčnik","year":"2015","venue":"NODALIDA","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sun et al., 2019] Zeyu Sun, Qihao Zhu, Lili Mou, Yingfei Xiong, Ge Li, and Lu Zhang. A grammar-based structural cnn decoder for code generation. In AAAI, volume 33, pages 7055-7062, 2019.","title":"A grammar-based structural cnn decoder for code generation","authors":"Sun","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sun et al., 2020] Zeyu Sun, Qihao Zhu, Yingfei Xiong, Yi- can Sun, Lili Mou, and Lu Zhang. Treegen: A tree-based transformer architecture for code generation. In AAAI, vol- ume 34, pages 8984-8991, 2020.","title":"Treegen: A tree-based transformer architecture for code generation","authors":"Sun","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 6000-6010, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2014] Adrienne Wang, Tom Kwiatkowski, and Luke Zettlemoyer. Morpho-syntactic lexical generaliza- tion for ccg semantic parsing. In EMNLP, pages 1284- 1295, 2014.","title":"Morpho-syntactic lexical generalization for ccg semantic parsing","authors":"Adrienne Wang, Tom Kwiatkowski, Luke Zettlemoyer","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wolf et al., 2014] Lior Wolf, Yair Hanani, Kfir Bar, and Nachum Dershowitz. Joint word2vec networks for bilin- gual semantic representations. IJCLA, 5(1):27-42, 2014.","title":"Joint word2vec networks for bilingual semantic representations","authors":"Lior Wolf, Yair Hanani, Kfir Bar, Nachum Dershowitz","year":"2014","venue":"IJCLA","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xiong et al., 2018] Yingfei Xiong, Bo Wang, Guirong Fu, and Linfei Zang. Learning to Synthesize. In GIW, 2018.","title":"Learning to Synthesize","authors":"Xiong","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Ye et al., 2020] Xi Ye, Qiaochu Chen, Isil Dillig, and Greg Durrett. Benchmarking multimodal regex synthesis with complex structures. In ACL, pages 6081-6094, 2020.","title":"Benchmarking multimodal regex synthesis with complex structures","authors":"Ye","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yin and Neubig, 2017] Pengcheng Yin and Graham Neu- big. A Syntactic Neural Model for General-Purpose Code Generation. In ACL, pages 440-450, 2017.","title":"Pengcheng Yin and Graham Neubig. A Syntactic Neural Model for General-Purpose Code Generation","authors":"Neubig Yin","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yin and Neubig, 2018] Pengcheng Yin and Graham Neu- big. Tranx: A transition-based neural abstract syntax parser for semantic parsing and code generation. 10 2018.","title":"Tranx: A transition-based neural abstract syntax parser for semantic parsing and code generation","authors":"Neubig Yin, Pengcheng Yin, Graham Neubig","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhong and Mei, 2020] Hao Zhong and Hong Mei. Learning a graph-based classifier for fault localization. SCIS, 63, 06 2020.","title":"Learning a graph-based classifier for fault localization","authors":"Hao Zhong, Hong Mei","year":"2020","venue":"SCIS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}