{"type":"ScholarlyArticle","title":"Revisiting the Evaluation of Deep Learning-Based Compiler Testing","authors":[{"name":"Yongqiang Tian","slug":"yongqiang-tian","given":"Yongqiang","family":"Tian","affiliation":"University of Waterloo; The Hong Kong University of Science and Technology"},{"name":"Zhenyang Xu","slug":"zhenyang-xu","given":"Zhenyang","family":"Xu","affiliation":"University of Waterloo"},{"name":"Yiwen Dong","slug":"yiwen-dong","given":"Yiwen","family":"Dong","affiliation":"University of Waterloo"},{"name":"Chengnian Sun","slug":"chengnian-sun","given":"Chengnian","family":"Sun","affiliation":"University of Waterloo"},{"name":"Shing-Chi Cheung","slug":"shing-chi-cheung","given":"Shing-Chi","family":"Cheung","affiliation":"The Hong Kong University of Science and Technology"}],"abstract":"A high-quality program generator is essential to effective automated compiler testing. Engineering such a program generator is difficult, time-consuming, and specific to the language under testing, thus requiring tremendous efforts from human experts with language-specific domain knowledge. To avoid repeatedly writing program generators for different languages, researchers recently proposed a language-agnostic approach based on deep learning techniques to automatically learn a program generator (referred to as DLG) from existing programs. Evaluations show that DLGs outperform Language-Specific Program Generators (LSGs) in testing compilers.\r\nHowever, we argue that it is unfair to use LSGs as baselines to evaluate DLGs. LSGs aim to validate compiler optimizations by only generating compilable, well-defined test programs; this restriction inevitably impairs the diversity of the language features used in the generated programs. In contrast, DLGs do not aim to validate the correctness of compiler optimizations, and its generated programs are not guaranteed to be well-defined or even compilable. Therefore, it is not surprising that DLG-generated programs are more diverse in terms of used language features than LSG-generated ones. \r\n\r\nThis study revisits the evaluation of DLGs, and proposes a new, fair, simple yet strong baseline named Kitten for evaluating DLGs. Given a dataset consisting of human-written programs, instead of using deep learning techniques to learn a program generator, Kitten directly derives new programs by mutating the programs in the dataset. Extensive experiments with more than 1,500 CPU-hours demonstrate that the state-of-the-art DLGs fail to compete against such a simple baseline: 3 v.s. 1,750 hang bugs, 1 v.s. 34 distinct compiler crashes. We believe that DLGs still have a large room for improvement.","doi":"10.24963/ijcai.2023/542","url":"https://ijcai.org/proceedings/2023/542","pdf":"https://ijcai.org/proceedings/2023/0542.pdf","markdown":"https://ijcai.org/proceedings/2023/542.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/542","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Multidisciplinary Topics and Applications","track":"Main Track","number":542,"pages":{"first":4873,"last":4882},"keywords":["Multidisciplinary Topics and Applications: MDA: Software engineering"],"videos":[],"references":[{"position":1,"raw":"[Aho et al., 2006] Alfred V. Aho, Monica S. Lam, Ravi Sethi, and Jeffrey D. Ullman. Compilers: Principles, Tech- niques, and Tools (2nd Edition), chapter 2.2.3. ParseTree. Addison Wesley, August 2006.","title":"Compilers: Principles, Techniques, and Tools","authors":"Alfred V Aho, Monica S Lam, Ravi Sethi, Jeffrey D Ullman","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Antlr, 2022a] Antlr. Antlr. https://www.antlr.org/, 2022. Online; accessed 26-July-2022.","title":"Antlr. Antlr","authors":"Antlr","year":"2022","venue":null,"doi":null,"url":"https://www.antlr.org/,2022","internal_url":null},{"position":3,"raw":"2022b] Antlr. Antlr grammar. https://github.com/ antlr/grammars-v4, 2022. Online; accessed 26-July-2022.","title":"Antlr. Antlr grammar","authors":null,"year":"2022","venue":null,"doi":null,"url":"https://github.com/antlr/grammars-v4","internal_url":null},{"position":4,"raw":"[Aschermann et al., 2019] Cornelius Aschermann, Tom- maso Frassetto, Thorsten Holz, Patrick Jauernig, Ahmad-Reza Sadeghi, and Daniel Teuchert. Nautilus: Fishing for deep bugs with grammars. In NDSS, 2019.","title":"Nautilus: Fishing for deep bugs with grammars","authors":"Cornelius Aschermann, Tommaso Frassetto, Thorsten Holz, Patrick Jauernig, Ahmad-Reza Sadeghi, Daniel Teuchert","year":"2019","venue":"NDSS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bison, 2022] Bison. Bison grammar files. http://web.mit. edu/gnu/doc/html/bison 6.html#SEC34, 2022. Online; ac- cessed 26-July-2022.","title":"Bison. Bison grammar files","authors":"Bison","year":"2022","venue":null,"doi":null,"url":"http://web.mit.edu/gnu/doc/html/bison6.html#SEC34,2022.Online;ac-cessed26-","internal_url":null},{"position":6,"raw":"[Cummins et al., 2018] Chris Cummins, Pavlos Petoumenos, Alastair Murray, and Hugh Leather. Compiler fuzzing through deep learning. In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2018, page 95-105, New York, NY, USA, 2018. Association for Computing Machinery.","title":"Compiler fuzzing through deep learning","authors":"Chris Cummins, Pavlos Petoumenos, Alastair Murray, Hugh Leather","year":"2018","venue":"Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2018","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Developers, 2022] Developers. Github issue for deepfuzz. https://github.com/s3team/DeepFuzz/issues/1, 2022. On- line; accessed 26-July-2022.","title":"Developers. Github issue for deepfuzz","authors":null,"year":"2022","venue":null,"doi":null,"url":"https://github.com/s3team/DeepFuzz/issues/1,2022.On-line","internal_url":null},{"position":8,"raw":"[Dong et al., 2022] Yiwen Dong, Zheyang Li, Yongqiang Tian, Chengnian Sun, Michael W. Godfrey, and Meiyap- pan Nagappan. Bash in the wild: Language usage, code smells, and bugs. ACM Trans. Softw. Eng. Methodol., apr 2022. Just Accepted.","title":"Bash in the wild: Language usage, code smells, and bugs","authors":"Yiwen Dong, Zheyang Li, Yongqiang Tian, Chengnian Sun, Michael W Godfrey, Meiyappan Nagappan","year":"2022","venue":"ACM Trans. Softw. Eng. Methodol","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[IOS, 2005] IOS. ISO/IEC 9899:TC2: Programming Lan- guages-C. https://www.open-std.org/jtc1/sc22/wg14/ www/docs/n1124.pdf, 2005.","title":"Programming Languages-C","authors":null,"year":"2005","venue":null,"doi":null,"url":"https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1124.pdf","internal_url":null},{"position":10,"raw":"[Jiang et al., 2020] Bo Jiang, Xiaoyan Wang, W. K. Chan, T. H. Tse, Na Li, Yongfeng Yin, and Zhenyu Zhang. Cuda- smith: A fuzzer for cuda compilers. In 2020 IEEE 44th An- nual Computers, Software, and Applications Conference (COMPSAC), pages 861-871, 2020.","title":"Cudasmith: A fuzzer for cuda compilers","authors":"Bo Jiang, Xiaoyan Wang, W K Chan, T H Tse, Na Li, Yongfeng Yin, Zhenyu Zhang","year":"2020","venue":"IEEE 44th Annual Computers, Software, and Applications Conference (COMPSAC)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Klees et al., 2018] George Klees, Andrew Ruef, Benji Cooper, Shiyi Wei, and Michael Hicks. Evaluating fuzz testing. In Proceedings of the 2018 ACM SIGSAC Con- ference on Computer and Communications Security, CCS '18, page 2123-2138, New York, NY, USA, 2018. Associ- ation for Computing Machinery.","title":"Evaluating fuzz testing","authors":"George Klees, Andrew Ruef, Benji Cooper, Shiyi Wei, Michael Hicks","year":"2018","venue":"Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, CCS '18","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Le et al., 2014] Vu Le, Mehrdad Afshari, and Zhendong Su. Compiler validation via equivalence modulo inputs. In Michael F. P. O'Boyle and Keshav Pingali, editors, ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI '14, Edinburgh, United King- dom -June 09 -11, 2014, pages 216-226. ACM, 2014.","title":"Compiler validation via equivalence modulo inputs","authors":"Mehrdad Vu Le, Zhendong Afshari, Su","year":"2014","venue":"ACM SIGPLAN Conference on Programming Language Design and Implementation","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Le et al., 2015] Vu Le, Chengnian Sun, and Zhendong Su. Finding deep compiler bugs via guided stochastic program mutation. In Jonathan Aldrich and Patrick Eugster, ed- itors, Proceedings of the 2015 ACM SIGPLAN Interna- tional Conference on Object-Oriented Programming, Sys- tems, Languages, and Applications, OOPSLA 2015, part of SPLASH 2015, Pittsburgh, PA, USA, October 25-30, 2015, pages 386-399. ACM, 2015.","title":"Finding deep compiler bugs via guided stochastic program mutation","authors":"Chengnian Vu Le, Zhendong Sun, Su","year":"2015","venue":"Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2015, part of SPLASH 2015","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lidbury et al., 2015] Christopher Lidbury, Andrei Lascu, Nathan Chong, and Alastair F. Donaldson. Many-core compiler fuzzing. SIGPLAN Not., 50(6):65-76, jun 2015.","title":"Many-core compiler fuzzing","authors":"Christopher Lidbury, Andrei Lascu, Nathan Chong, Alastair F Donaldson","year":"2015","venue":"SIGPLAN Not","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2019] Xiao Liu, Xiaoting Li, Rupesh Prajapati, and Dinghao Wu. Deepfuzz: Automatic generation of syn- tax valid c programs for fuzz testing. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):1044- 1051, Jul. 2019.","title":"Deepfuzz: Automatic generation of syntax valid c programs for fuzz testing","authors":"Xiao Liu, Xiaoting Li, Rupesh Prajapati, Dinghao Wu","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Livinskii et al., 2020] Vsevolod Livinskii, Dmitry Babokin, and John Regehr. Random testing for c and c++ compilers with yarpgen. Proc. ACM Program. Lang., 4(OOPSLA), nov 2020.","title":"Random testing for c and c++ compilers with yarpgen","authors":"Vsevolod Livinskii, Dmitry Babokin, John Regehr","year":"2020","venue":"Proc. ACM Program. Lang","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sun et al., 2016a] Chengnian Sun, Vu Le, and Zhendong Su. Finding compiler bugs via live code mutation. In Pro- ceedings of the 2016 ACM SIGPLAN International Con- ference on Object-Oriented Programming, Systems, Lan- guages, and Applications, OOPSLA 2016, page 849-863, New York, NY, USA, 2016. Association for Computing Machinery.","title":"Finding compiler bugs via live code mutation","authors":"Sun","year":"2016","venue":"Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2016","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sun et al., 2016b] Chengnian Sun, Vu Le, Qirun Zhang, and Zhendong Su. Toward understanding compiler bugs in gcc and llvm. In Proceedings of the 25th International Sym- posium on Software Testing and Analysis, ISSTA 2016, page 294-305, New York, NY, USA, 2016. Association for Computing Machinery.","title":"Toward understanding compiler bugs in gcc and llvm","authors":"Sun","year":"2016","venue":"Proceedings of the 25th International Symposium on Software Testing and Analysis, ISSTA 2016","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sun et al., 2018] Chengnian Sun, Yuanbo Li, Qirun Zhang, Tianxiao Gu, and Zhendong Su. Perses: Syntax-guided program reduction. In Proceedings of the 40th In- ternational Conference on Software Engineering, page 361-371. Association for Computing Machinery, 2018.","title":"Perses: Syntax-guided program reduction","authors":"Sun","year":"2018","venue":"Proceedings of the 40th International Conference on Software Engineering","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2019] Junjie Wang, Bihuan Chen, Lei Wei, and Yang Liu. Superion: Grammar-aware greybox fuzzing. In 2019 IEEE/ACM 41st International Conference on Soft- ware Engineering (ICSE), pages 724-735. IEEE, 2019.","title":"Superion: Grammar-aware greybox fuzzing","authors":"Junjie Wang, Bihuan Chen, Lei Wei, Yang Liu","year":"2019","venue":"IEEE/ACM 41st International Conference on Software Engineering (ICSE)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Xu et al., 2020a] Haoran Xu, Shuhui Fan, Yongjun Wang, Zhijian Huang, Hongzuo Xu, and Peidai Xie. Tree2tree structural language modeling for compiler fuzzing. In Meikang Qiu, editor, Algorithms and Architectures for Parallel Processing, pages 563-578, Cham, 2020. Springer International Publishing.","title":"Tree2tree structural language modeling for compiler fuzzing","authors":"Xu","year":"2020","venue":"Algorithms and Architectures for Parallel Processing","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xu et al., 2020b] Haoran Xu, Yongjun Wang, Shuhui Fan, Peidai Xie, and Aizhi Liu. Dsmith: Compiler fuzzing through generative deep learning model with attention. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1-9, 2020.","title":"Dsmith: Compiler fuzzing through generative deep learning model with attention","authors":"Xu","year":"2020","venue":"2020 International Joint Conference on Neural Networks (IJCNN)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yang et al., 2011] Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. Finding and understanding bugs in c compil- ers. In Proceedings of the 32nd ACM SIGPLAN Confer- ence on Programming Language Design and Implementa- tion, PLDI '11, page 283-294, New York, NY, USA, 2011. Association for Computing Machinery.","title":"Finding and understanding bugs in c compilers","authors":"Xuejun Yang, Yang Chen, Eric Eide, John Regehr","year":"2011","venue":"Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI '11","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2022] Zenong Zhang, Zach Patterson, Michael Hicks, and Shiyi Wei. FIXREVERTER: A realistic bug injection methodology for benchmarking fuzz testing. In 31st USENIX Security Symposium (USENIX Security 22), pages 3699-3715, Boston, MA, August 2022. USENIX Association.","title":"FIXREVERTER: A realistic bug injection methodology for benchmarking fuzz testing","authors":"Zhang","year":"2022","venue":"31st USENIX Security Symposium (USENIX Security 22)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}