{"type":"ScholarlyArticle","title":"Code Completion with Neural Attention and Pointer Networks","authors":[{"name":"Jian Li","slug":"jian-li","given":"Jian","family":"Li","affiliation":"Department of Computer Science and Engineering , The Chinese University of Hong Kong , China; Shenzhen Research Institute , The Chinese University of Hong Kong , China"},{"name":"Yue Wang","slug":"yue-wang","given":"Yue","family":"Wang","affiliation":"Department of Computer Science and Engineering , The Chinese University of Hong Kong , China; Shenzhen Research Institute , The Chinese University of Hong Kong , China"},{"name":"Michael R. Lyu","slug":"michael-r-lyu","given":"Michael R.","family":"Lyu","affiliation":"Department of Computer Science and Engineering , The Chinese University of Hong Kong , China; Shenzhen Research Institute , The Chinese University of Hong Kong , China"},{"name":"Irwin King","slug":"irwin-king","given":"Irwin","family":"King","affiliation":"Department of Computer Science and Engineering , The Chinese University of Hong Kong , China; Shenzhen Research Institute , The Chinese University of Hong Kong , China"}],"abstract":"Intelligent code completion has become an essential research task to accelerate modern software development. To facilitate effective code completion for dynamically-typed programming languages, we apply neural language models by learning from large codebases, and develop a tailored attention mechanism for code completion. However, standard neural language models even with attention mechanism cannot correctly predict the out-of-vocabulary (OoV) words that restrict the code completion performance. In this paper, inspired by the prevalence of locally repeated terms in program source code, and the recently proposed pointer copy mechanism, we propose a pointer mixture network for better predicting OoV words in code completion. Based on the context, the pointer mixture network learns to either generate a within-vocabulary word through an RNN component, or regenerate an OoV word from local context through a pointer component. Experiments on two benchmarked datasets demonstrate the effectiveness of our attention mechanism and pointer mixture network on the code completion task.","doi":"10.24963/ijcai.2018/578","url":"https://ijcai.org/proceedings/2018/578","pdf":"https://ijcai.org/proceedings/2018/0578.pdf","markdown":"https://ijcai.org/proceedings/2018/578.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/578","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Natural Language Processing","track":"Main track","number":578,"pages":{"first":4159,"last":4165},"keywords":["Multidisciplinary Topics and Applications: Knowledge-based Software Engineering","Machine Learning: Deep Learning","Natural Language Processing: NLP Applications and Tools"],"videos":[],"references":[{"position":1,"raw":"[Allamanis and Sutton, 2014] Miltiadis Allamanis and Charles Sutton. Mining idioms from source code. In FSE'14: Proc. of the International Symposium on Foundations of Software Engineering, pages 472-483, 2014.","title":"Miltiadis Allamanis and Charles Sutton. Mining idioms from source code","authors":"Sutton Allamanis","year":"2014","venue":"FSE'14: Proc. of the International Symposium on Foundations of Software Engineering","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Allamanis et al., 2016] Miltiadis Allamanis, Hao Peng, and Charles Sutton. A convolutional attention network for ex- treme summarization of source code. In ICML'16: Proc. of the International Conference on Machine Learning, pages 2091-2100, 2016.","title":"A convolutional attention network for extreme summarization of source code","authors":"Miltiadis Allamanis, Hao Peng, Charles Sutton","year":"2016","venue":"ICML'16: Proc. of the International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bhoopchand et al., 2016] Avishkar Bhoopchand, Tim Rocktäschel, Earl Barr, and Sebastian Riedel. Learning python code suggestion with a sparse pointer network. arXiv preprint arXiv:1611.08307, 2016.","title":"Learning python code suggestion with a sparse pointer network","authors":"Avishkar Bhoopchand, Tim Rocktäschel, Earl Barr, Sebastian Riedel","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bielik et al., 2016] Pavol Bielik, Veselin Raychev, and Mar- tin Vechev. Phog: probabilistic model for code. In ICM- L'16: Proc. of the International Conference on Machine Learning, pages 2933-2942, 2016.","title":"Pavol Bielik, Veselin Raychev, and Martin Vechev. Phog: probabilistic model for code","authors":"Bielik","year":"2016","venue":"ICM-L'16: Proc. of the International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cheng et al., 2016] Jianpeng Cheng, Li Dong, and Mirella Lapata. Long short-term memory-networks for machine reading. arXiv preprint arXiv:1601.06733, 2016.","title":"Long short-term memory-networks for machine reading","authors":"Cheng","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gu et al., 2016] Jiatao Gu, Zhengdong Lu, Hang Li, and Victor OK Li. Incorporating copying mechanism in sequence-to-sequence learning. arXiv preprint arX- iv:1603.06393, 2016.","title":"Incorporating copying mechanism in sequence-to-sequence learning","authors":"Gu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gulcehre et al., 2016] Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Ben- gio. Pointing the unknown words. arXiv preprint arXiv:1603.08148, 2016.","title":"Pointing the unknown words","authors":"Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, Yoshua Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hindle et al., 2012] Abram Hindle, Earl T Barr, Zhendong Su, Mark Gabel, and Premkumar Devanbu. On the natu- ralness of software. In ICSE'12: Proc. of the Internation- al Conference on Software Engineering, pages 837-847, 2012.","title":"On the naturalness of software","authors":"Abram Hindle, Earl T Barr, Zhendong Su, Mark Gabel, Premkumar Devanbu","year":"2012","venue":"ICSE'12: Proc. of the International Conference on Software Engineering","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2017] Jian Li, Pinjia He, Jieming Zhu, and Michael R Lyu. Software defect prediction via convolu- tional neural network. In QRS'17: Proc. of the Interna- tional Conference on Software Quality, Reliability and Se- curity, pages 318-328, 2017.","title":"Software defect prediction via convolutional neural network","authors":"Li","year":"2017","venue":"QRS'17: Proc. of the International Conference on Software Quality, Reliability and Security","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ling et al., 2016] Wang Ling, Edward Grefenstette, Kar- l Moritz Hermann, Tomáš Kočiskỳ, Andrew Senior, Fumin Wang, and Phil Blunsom. Latent predictor networks for code generation. arXiv preprint arXiv:1603.06744, 2016.","title":"Latent predictor networks for code generation","authors":"Wang Ling, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiskỳ, Andrew Senior, Fumin Wang, Phil Blunsom","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2016] Chang Liu, Xin Wang, Richard Shin, Joseph E Gonzalez, and Dawn Song. Neural code com- pletion. 2016.","title":"Neural code completion","authors":"Chang Liu, Xin Wang, Richard Shin, Joseph E Gonzalez, Dawn Song","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Luong et al., 2014] Minh-Thang Luong, Ilya Sutskever, Quoc V Le, Oriol Vinyals, and Wojciech Zaremba. Ad- dressing the rare word problem in neural machine transla- tion. arXiv preprint arXiv:1410.8206, 2014.","title":"Addressing the rare word problem in neural machine translation","authors":"Luong","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Merity et al., 2016] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016.","title":"Pointer sentinel mixture models","authors":"Stephen Merity, Caiming Xiong, James Bradbury, Richard Socher","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mikolov et al., 2010] Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. Re- current neural network based language model. In INTER- SPEECH, 2010.","title":"Recurrent neural network based language model","authors":"Mikolov","year":"2010","venue":"INTER-SPEECH","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mou et al., 2016] Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. Convolutional neural networks over tree struc- tures for programming language processing. In AAAI, pages 1287-1293, 2016.","title":"Convolutional neural networks over tree structures for programming language processing","authors":"Lili Mou, Ge Li, Lu Zhang, Tao Wang, Zhi Jin","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Raychev et al., 2014] Veselin Raychev, Martin Vechev, and Eran Yahav. Code completion with statistical language models. In ACM SIGPLAN Notices, volume 49, pages 419-428, 2014.","title":"Code completion with statistical language models","authors":"Veselin Raychev, Martin Vechev, Eran Yahav","year":"2014","venue":"ACM SIGPLAN Notices","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Raychev et al., 2016] Veselin Raychev, Pavol Bielik, and Martin Vechev. Probabilistic model for code with decision trees. In OOPSLA'16: Proc. of the International Con- ference on Object-Oriented Programming, Systems, Lan- guages, and Applications, pages 731-747, 2016.","title":"Probabilistic model for code with decision trees","authors":"Veselin Raychev, Pavol Bielik, Martin Vechev","year":"2016","venue":"OOPSLA'16: Proc. of the International Conference on Object-Oriented Programming, Systems, Languages, and Applications","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tran et al., 2016] Ke Tran, Arianna Bisazza, and Christof Monz. Recurrent memory networks for language model- ing. arXiv preprint arXiv:1601.01272, 2016.","title":"Recurrent memory networks for language modeling","authors":"Ke Tran, Arianna Bisazza, Christof Monz","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tu et al., 2014] Zhaopeng Tu, Zhendong Su, and Premku- mar Devanbu. On the localness of software. In FSE'14: Proc. of the International Symposium on Foundations of Software Engineering, pages 269-280, 2014.","title":"Zhaopeng Tu, Zhendong Su, and Premkumar Devanbu. On the localness of software","authors":"Tu","year":"2014","venue":"FSE'14: Proc. of the International Symposium on Foundations of Software Engineering","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vinyals et al., 2015] Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In NIPS'15: Advances in Neural Information Processing Systems, pages 2692- 2700, 2015.","title":"Pointer networks","authors":"Oriol Vinyals, Meire Fortunato, Navdeep Jaitly","year":"2015","venue":"NIPS'15: Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[White et al., 2015] Martin White, Christopher Vendome, Mario Linares-Vásquez, and Denys Poshyvanyk. Toward deep learning software repositories. In MSR'15: Proc. of the Working Conference on Mining Software Repositories, pages 334-345, 2015.","title":"Toward deep learning software repositories","authors":"Martin White, Christopher Vendome, Mario Linares-Vásquez, Denys Poshyvanyk","year":"2015","venue":"MSR'15: Proc. of the Working Conference on Mining Software Repositories","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zaremba et al., 2014] Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regulariza- tion. arXiv preprint arXiv:1409.2329, 2014.","title":"Recurrent neural network regularization","authors":"Wojciech Zaremba, Ilya Sutskever, Oriol Vinyals","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}