{"type":"ScholarlyArticle","title":"Purpose Enhanced Reasoning through Iterative Prompting: Uncover Latent Robustness of ChatGPT on Code Comprehension","authors":[{"name":"Yi Wang","slug":"yi-wang","given":"Yi","family":"Wang","affiliation":"North Carolina State University"},{"name":"Qidong Zhao","slug":"qidong-zhao","given":"Qidong","family":"Zhao","affiliation":"North Carolina State University"},{"name":"Dongkuan Xu","slug":"dongkuan-xu","given":"Dongkuan","family":"Xu","affiliation":"North Carolina State University"},{"name":"Xu Liu","slug":"xu-liu","given":"Xu","family":"Liu","affiliation":"North Carolina State University"}],"abstract":"Code comments are crucial for gaining in-depth insights to facilitate code comprehension. The key to obtaining these insights lies in precisely summarizing the main purpose of the code. Recent approaches on code comment generation lie in prompting large language models (LLMs) such as ChatGPT, instead of training/fine-tuning specific models. Although ChatGPT demonstrates an impressive performance in code comprehension, it still suffers from robustness challenges in consistently producing high-quality code comments. This is because ChatGPT prioritizes the semantics of code tokens, which makes it vulnerable to commonly encountered benign perturbations such as variable name replacements. This study proposes a modular prompting paradigm Perthept to effectively mitigate the negative effects caused by such minor perturbations. Perthept iteratively enhances the reasoning depth to reach the main purpose of the code. Perthept demonstrates robustness under the scenario where there is stochasticity or unreliability in ChatGPT's responses. We give a comprehensive evaluation across four public datasets to show the consistent robustness improvement with our proposed methodology over other models.","doi":"10.24963/ijcai.2024/720","url":"https://ijcai.org/proceedings/2024/720","pdf":"https://ijcai.org/proceedings/2024/0720.pdf","markdown":"https://ijcai.org/proceedings/2024/720.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/720","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Natural Language Processing","track":"Main Track","number":720,"pages":{"first":6513,"last":6521},"keywords":["Natural Language Processing: NLP: Summarization","Natural Language Processing: NLP: Tools"],"videos":[],"references":[{"position":1,"raw":"[Ahmad et al., 2020] Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. A transformer-based approach for source code summariza- tion. arXiv preprint arXiv:2005.00653, 2020.","title":"A transformer-based approach for source code summarization","authors":"Ahmad Wasi Uddin, Saikat Chakraborty, Baishakhi Ray, Kai-Wei Chang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ahmad et al., 2021] Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. Unified pre-training for program understanding and generation. arXiv preprint arXiv:2103.06333, 2021.","title":"Unified pre-training for program understanding and generation","authors":"Ahmad Wasi Uddin, Saikat Chakraborty, Baishakhi Ray, Kai-Wei Chang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[de Souza et al., 2005] Sergio Cozzetti B de Souza, Nicolas Anquetil, and Káthia M de Oliveira. A study of the docu- mentation essential to software maintenance. In Proceed- ings of the 23rd annual international conference on Design of communication: documenting & designing for perva- sive information, pages 68-75, 2005.","title":"A study of the documentation essential to software maintenance","authors":"Sergio Cozzetti, B De Souza, Nicolas Anquetil, Káthia M De Oliveira","year":"2005","venue":"Proceedings of the 23rd annual international conference on Design of communication: documenting & designing for pervasive information","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Feng et al., 2020] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155, 2020.","title":"Codebert: A pre-trained model for programming and natural languages","authors":"Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Goodfellow et al., 2014] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.","title":"Explaining and harnessing adversarial examples","authors":"Ian J Goodfellow, Jonathon Shlens, Christian Szegedy","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Guo et al., 2022] Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. Unixcoder: Unified cross-modal pre-training for code representation. arXiv preprint arXiv:2203.03850, 2022.","title":"Unixcoder: Unified cross-modal pre-training for code representation","authors":"Shuai Daya Guo, Nan Lu, Yanlin Duan, Ming Wang, Jian Zhou, Yin","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He, 2019] Hao He. Understanding source code comments at large-scale. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineer- ing, pages 1217-1219, 2019.","title":"Understanding source code comments at large-scale","authors":"Hao He","year":"2019","venue":"Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu et al., 2018a] Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. Deep code comment generation. In Proceedings of the 26th conference on program comprehension, pages 200-210, 2018.","title":"Deep code comment generation","authors":"Hu","year":"2018","venue":"Proceedings of the 26th conference on program comprehension","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hu et al., 2018b] Xing Hu, Ge Li, Xin Xia, David Lo, Shuai Lu, and Zhi Jin. Summarizing source code with transferred api knowledge. 2018.","title":"Summarizing source code with transferred api knowledge","authors":"Hu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2018/314"},{"position":10,"raw":"[Husain et al., 2019] Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. Codesearchnet challenge: Evaluating the state of seman- tic code search. arXiv preprint arXiv:1909.09436, 2019.","title":"Codesearchnet challenge: Evaluating the state of semantic code search","authors":"Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, Marc Brockschmidt","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jha and Reddy, 2022] Akshita Jha and Chandan K Reddy. Codeattack: Code-based adversarial attacks for pre- trained programming language models. arXiv preprint arXiv:2206.00052, 2022.","title":"Codeattack: Code-based adversarial attacks for pretrained programming language models","authors":"Akshita Jha, K Chandan, Reddy","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kojima et al., 2022] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916, 2022.","title":"Large language models are zero-shot reasoners","authors":"Kojima","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lester et al., 2021] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021.","title":"The power of scale for parameter-efficient prompt tuning","authors":"Brian Lester, Rami Al-Rfou, Noah Constant","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li and Liang, 2021] Xiang Lisa Li and Percy Liang. Prefix- tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021.","title":"Prefixtuning: Optimizing continuous prompts for generation","authors":"Lisa Xiang, Percy Li, Liang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Madaan et al., 2023] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023.","title":"Self-refine: Iterative refinement with self-feedback","authors":"Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Panichella et al., 2016] Sebastiano Panichella, Annibale Panichella, Moritz Beller, Andy Zaidman, and Harald C Gall. The impact of test case summaries on bug fixing per- formance: An empirical investigation. In Proceedings of the 38th international conference on software engineering, pages 547-558, 2016.","title":"The impact of test case summaries on bug fixing performance: An empirical investigation","authors":"Sebastiano Panichella, Annibale Panichella, Moritz Beller, Andy Zaidman, Harald C Gall","year":"2016","venue":"Proceedings of the 38th international conference on software engineering","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Reif et al., 2021] Emily Reif, Daphne Ippolito, Ann Yuan, Andy Coenen, Chris Callison-Burch, and Jason Wei. A recipe for arbitrary text style transfer with large language models. arXiv preprint arXiv:2109.03910, 2021.","title":"A recipe for arbitrary text style transfer with large language models","authors":"Emily Reif, Daphne Ippolito, Ann Yuan, Andy Coenen, Chris Callison-Burch, Jason Wei","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sontakke et al., 2022] Ankita Nandkishor Sontakke, Man- asi Patwardhan, Lovekesh Vig, Raveendra Kumar Medicherla, Ravindra Naik, and Gautam Shroff. Code summarization: Do transformers really understand code? In Deep Learning for Code Workshop, 2022.","title":"Code summarization: Do transformers really understand code?","authors":"Ankita Nandkishor Sontakke, Manasi Patwardhan, Lovekesh Vig, Raveendra Kumar Medicherla, Ravindra Naik, Gautam Shroff","year":"2022","venue":"Deep Learning for Code Workshop","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sridhara et al., 2010] Giriprasad Sridhara, Emily Hill, Di- vya Muppaneni, Lori Pollock, and K Vijay-Shanker. To- wards automatically generating summary comments for java methods. In Proceedings of the IEEE/ACM inter- national conference on Automated software engineering, pages 43-52, 2010.","title":"Towards automatically generating summary comments for java methods","authors":"Sridhara","year":"2010","venue":"Proceedings of the IEEE/ACM international conference on Automated software engineering","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wan et al., 2018] Yao Wan, Zhou Zhao, Min Yang, Guan- dong Xu, Haochao Ying, Jian Wu, and Philip S Yu. Im- proving automatic source code summarization via deep reinforcement learning. In Proceedings of the 33rd ACM/IEEE international conference on automated soft- ware engineering, pages 397-407, 2018.","title":"Improving automatic source code summarization via deep reinforcement learning","authors":"Zhou Yao Wan, Min Zhao, Guandong Yang, Haochao Xu, Jian Ying, Philip S Wu, Yu","year":"2018","venue":"Proceedings of the 33rd ACM/IEEE international conference on automated software engineering","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2023] Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Haojun Huang, Wei Ye, Xiubo Geng, et al. On the ro- bustness of chatgpt: An adversarial and out-of-distribution perspective. arXiv preprint arXiv:2302.12095, 2023.","title":"On the robustness of chatgpt: An adversarial and out-of-distribution perspective","authors":"Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Haojun Huang, Wei Ye, Xiubo Geng","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wei et al., 2019] Bolin Wei, Ge Li, Xin Xia, Zhiyi Fu, and Zhi Jin. Code generation as a dual task of code summariza- tion. Advances in neural information processing systems, 32, 2019.","title":"Code generation as a dual task of code summarization","authors":"Bolin Wei, Ge Li, Xin Xia, Zhiyi Fu, Zhi Jin","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wei et al., 2022] Jason Wei, Xuezhi Wang, Dale Schuur- mans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large lan- guage models. arXiv preprint arXiv:2201.11903, 2022.","title":"Chain of thought prompting elicits reasoning in large language models","authors":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, Denny Zhou","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu et al., 2020] Hongqiu Wu, Hai Zhao, and Min Zhang. Code summarization with structure-induced transformer. arXiv preprint arXiv:2012.14710, 2020.","title":"Code summarization with structure-induced transformer","authors":"Wu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhou et al., 2022] Yu Zhou, Xiaoqing Zhang, Juanjuan Shen, Tingting Han, Taolue Chen, and Harald Gall. Adver- sarial robustness of deep code comment generation. ACM Transactions on Software Engineering and Methodology (TOSEM), 31(4):1-30, 2022.","title":"Adversarial robustness of deep code comment generation","authors":"Yu Zhou, Xiaoqing Zhang, Juanjuan Shen, Tingting Han, Taolue Chen, Harald Gall","year":"2022","venue":"ACM Transactions on Software Engineering and Methodology (TOSEM)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}