{"type":"ScholarlyArticle","title":"Search-Based Testing of Reinforcement Learning","authors":[{"name":"Martin Tappler","slug":"martin-tappler","given":"Martin","family":"Tappler","affiliation":"Institute of Software Technology , Graz University of Technology; TU Graz-SAL DES Lab , Silicon Austria Labs , Graz , Austria"},{"name":"Filip Cano Córdoba","slug":"filip-cano-cordoba","given":"Filip","family":"Cano Córdoba","affiliation":"Institute of Applied Information Processing and Communications , Graz University of Technology"},{"name":"Bernhard K. Aichernig","slug":"bernhard-k-aichernig","given":"Bernhard K.","family":"Aichernig","affiliation":"Institute of Software Technology , Graz University of Technology; TU Graz-SAL DES Lab , Silicon Austria Labs , Graz , Austria"},{"name":"Bettina Könighofer","slug":"bettina-konighofer","given":"Bettina","family":"Könighofer","affiliation":"Institute of Applied Information Processing and Communications , Graz University of Technology; Lamarr Security Research"}],"abstract":"Evaluation of deep reinforcement learning (RL) is inherently challenging. Especially the opaqueness of learned policies and the stochastic nature of both agents and environments make testing the behavior of deep RL agents difficult. We present a search-based testing framework that enables a wide range of novel analysis capabilities for evaluating the safety and performance of deep RL agents. For safety testing, our framework utilizes a search algorithm that searches for a reference trace that solves the RL task. The backtracking states of the search, called boundary states, pose safety-critical situations. We create safety test-suites that evaluate how well the RL agent escapes safety-critical situations near these boundary states. For robust performance testing, we create a diverse set of traces via fuzz testing. These fuzz traces are used to bring the agent into a wide variety of potentially unknown states from which the average performance of the agent is compared to the average performance of the fuzz traces. We apply our search-based testing approach on RL for Nintendo's Super Mario Bros.","doi":"10.24963/ijcai.2022/72","url":"https://ijcai.org/proceedings/2022/72","pdf":"https://ijcai.org/proceedings/2022/0072.pdf","markdown":"https://ijcai.org/proceedings/2022/72.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/72","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":72,"pages":{"first":503,"last":510},"keywords":["AI Ethics, Trust, Fairness: Safety & Robustness","Search: Search and Machine Learning","Agent-based and Multi-agent Systems: Formal Verification, Validation and Synthesis","Machine Learning: Deep Reinforcement Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986905","length_sec":85,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985048","length_sec":356,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Achiam and Amodei, 2019] Joshua Achiam and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. Preprint. Under review, 2019.","title":"Joshua Achiam and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning","authors":"Amodei Achiam","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Agarwal et al., 2021] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Belle- mare. Deep reinforcement learning at the edge of the sta- tistical precipice. In NeurIPS, volume 34, pages 29304- 29320, 2021.","title":"Deep reinforcement learning at the edge of the statistical precipice","authors":"Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, Marc Bellemare","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Alshiekh et al., 2018] Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. Safe reinforcement learning via shielding. In AAAI-18, pages 2669-2678, 2018.","title":"Safe reinforcement learning via shielding","authors":"Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, Ufuk Topcu","year":"2018","venue":"AAAI-18","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Aschermann et al., 2020] Cornelius Aschermann, Sergej Schumilo, Ali Abbasi, and Thorsten Holz. Ijon: Explor- ing deep state spaces via fuzzing. In IEEE SP 2020, pages 1597-1612, 2020.","title":"Ijon: Exploring deep state spaces via fuzzing","authors":"Cornelius Aschermann, Sergej Schumilo, Ali Abbasi, Thorsten Holz","year":"2020","venue":"IEEE SP 2020","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bellemare et al., 2013] Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. J. Artif. Intell. Res., 47:253-279, 2013.","title":"The arcade learning environment: An evaluation platform for general agents","authors":"G Marc, Yavar Bellemare, Joel Naddaf, Michael Veness, Bowling","year":"2013","venue":"J. Artif. Intell. Res","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Böttinger et al., 2018] Konstantin Böttinger, Patrice Gode- froid, and Rishabh Singh. Deep reinforcement fuzzing. In IEEE SP Workshops 2018, pages 116-122, 2018.","title":"Deep reinforcement fuzzing","authors":"Konstantin Böttinger, Patrice Godefroid, Rishabh Singh","year":"2018","venue":"IEEE SP Workshops 2018","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI gym. arXiv preprint arXiv:1606.01540, 2016.","title":"OpenAI gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Corsi et al., 2021] Davide Corsi, Enrico Marchesini, and Alessandro Farinelli. Formal verification of neural net- works for safety-critical tasks in deep reinforcement learn- ing. In UAI, volume 161 of Proceedings of Machine Learning Research, pages 333-343, 2021.","title":"Formal verification of neural networks for safety-critical tasks in deep reinforcement learning","authors":"Davide Corsi, Enrico Marchesini, Alessandro Farinelli","year":"2021","venue":"Proceedings of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Drozd and Wagner, 2018] William Drozd and Michael D. Wagner. FuzzerGym: A competitive framework for fuzzing and learning. CoRR, abs/1807.07490, 2018.","title":"FuzzerGym: A competitive framework for fuzzing and learning","authors":"William Drozd, Michael D Wagner","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ehlers, 2017] Rüdiger Ehlers. Formal verification of piece- wise linear feed-forward neural networks. In ATVA 2017, volume 10482 of LNCS, pages 269-286, 2017.","title":"Formal verification of piecewise linear feed-forward neural networks","authors":"Rüdiger Ehlers","year":"2017","venue":"ATVA 2017","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Feng et al., 2020] Yuansong Feng, Suraj Subramanian, Howard Wang, and Steven Guo. Train a mario-playing RL agent. https://pytorch.org/tutorials/intermediate/mario rl tutorial.html, 2020. A PyTorch tutorial [online], accessed: 2022, January 07.","title":"Train a mario-playing RL agent","authors":"Feng","year":"2020","venue":null,"doi":null,"url":"https://pytorch.org/tutorials/intermediate/mariorltutorial.html,2020.APyTorchtutorial","internal_url":null},{"position":12,"raw":"[Garcıa and Fernández, 2015] Javier Garcıa and Fernando Fernández. A comprehensive survey on safe reinforce- ment learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.","title":"Javier Garcıa and Fernando Fernández. A comprehensive survey on safe reinforcement learning","authors":"Fernández Garcıa","year":"2015","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Harel-Canada et al., 2020] Fabrice Harel-Canada, Lingx- iao Wang, Muhammad Ali Gulzar, Quanquan Gu, and Miryung Kim. Is neuron coverage a meaningful measure for testing deep neural networks? In ESEC/FSE, pages 851-862. ACM, 2020.","title":"Is neuron coverage a meaningful measure for testing deep neural networks?","authors":"Fabrice Harel-Canada, Lingxiao Wang, Muhammad Ali Gulzar, Quanquan Gu, Miryung Kim","year":"2020","venue":"ESEC/FSE","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Karakovskiy and Togelius, 2012] Sergey Karakovskiy and Julian Togelius. The mario AI benchmark and competi- tions. IEEE T-CIAIG, 4(1):55-67, 2012.","title":"The mario AI benchmark and competitions","authors":"Sergey Karakovskiy, Julian Togelius","year":"2012","venue":"IEEE T-CIAIG","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Khalili and Tacchella, 2014] Ali Khalili and Armando Tac- chella. Learning nondeterministic Mealy machines. In ICGI 2014, volume 34 of JMLR Workshop and Conference Proceedings, pages 109-123, 2014.","title":"Learning nondeterministic Mealy machines","authors":"Ali Khalili, Armando Tacchella","year":"2014","venue":"JMLR Workshop and Conference Proceedings","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kiran et al., 2022] B Ravi Kiran, Ibrahim Sobh, Victor Tal- paert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yoga- mani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on In- telligent Transportation Systems, 23(6):4909-4926, 2022.","title":"Deep reinforcement learning for autonomous driving: A survey","authors":"Ibrahim Ravi Kiran, Victor Sobh, Patrick Talpaert, Ahmad A Al Mannion, Senthil Sallab, Patrick Yogamani, Pérez","year":"2022","venue":"IEEE Transactions on Intelligent Transportation Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Küttler et al., 2020] Heinrich Küttler, Nantas Nardelli, Alexander H. Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The NetHack Learning Environment. In NeurIPS, pages 7671-7684, 2020.","title":"The NetHack Learning Environment","authors":"Heinrich Küttler, Nantas Nardelli, Alexander H Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, Tim Rocktäschel","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ma et al., 2018] Lei Ma, Felix Juefei-Xu, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Chunyang Chen, Ting Su, Li Li, Yang Liu, Jianjun Zhao, and Yadong Wang. Deep- Gauge: multi-granularity testing criteria for deep learning systems. In ASE, pages 120-131. ACM, 2018.","title":"Deep-Gauge: multi-granularity testing criteria for deep learning systems","authors":"Lei Ma, Felix Juefei-Xu, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Chunyang Chen, Ting Su, Li Li, Yang Liu, Jianjun Zhao, Yadong Wang","year":"2018","venue":"ASE","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pathak et al., 2017] Shashank Pathak, Luca Pulina, and Ar- mando Tacchella. Verification and repair of control poli- cies for safe reinforcement learning. Applied Intelligence, 48:886-908, 2017.","title":"Verification and repair of control policies for safe reinforcement learning","authors":"Shashank Pathak, Luca Pulina, Armando Tacchella","year":"2017","venue":"Applied Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pei et al., 2019] Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. DeepXplore: automated whitebox testing of deep learning systems. Commun. ACM, 62(11):137-145, 2019.","title":"DeepXplore: automated whitebox testing of deep learning systems","authors":"Kexin Pei, Yinzhi Cao, Junfeng Yang, Suman Jana","year":"2019","venue":"Commun. ACM","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pezzè and Young, 2007] Mauro Pezzè and Michal Young. Software testing and analysis -process, principles and techniques. Wiley, 2007.","title":"Software testing and analysis -process, principles and techniques","authors":"Mauro Pezzè, Michal Young","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Pranger et al., 2021] Stefan Pranger, Bettina Könighofer, Martin Tappler, Martin Deixelberger, Nils Jansen, and Roderick Bloem. Adaptive shielding under uncertainty. In ACC, pages 3467-3474, 2021.","title":"Adaptive shielding under uncertainty","authors":"Stefan Pranger, Bettina Könighofer, Martin Tappler, Martin Deixelberger, Nils Jansen, Roderick Bloem","year":"2021","venue":"ACC","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Schrittwieser et al., 2020] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604-609, 2020.","title":"Mastering atari, go, chess and shogi by planning with a learned model","authors":"Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel","year":"2020","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Schumilo et al., 2022] Sergej Schumilo, Cornelius Ascher- mann, Andrea Jemmett, Ali Abbasi, and Thorsten Holz. Nyx-net: network fuzzing with incremental snapshots. In EuroSys, pages 166-180, 2022.","title":"Nyx-net: network fuzzing with incremental snapshots","authors":"Sergej Schumilo, Cornelius Aschermann, Andrea Jemmett, Ali Abbasi, Thorsten Holz","year":"2022","venue":"EuroSys","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Scott et al., 2021] Joseph Scott, Trishal Sudula, Hammad Rehman, Federico Mora, and Vijay Ganesh. Bandit- fuzz: Fuzzing SMT solvers with multi-agent reinforce- ment learning. In FM, volume 13047 of LNCS, pages 103- 121, 2021.","title":"Banditfuzz: Fuzzing SMT solvers with multi-agent reinforcement learning","authors":"Joseph Scott, Trishal Sudula, Hammad Rehman, Federico Mora, Vijay Ganesh","year":"2021","venue":"FM","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484-489, 2016.","title":"Mastering the game of Go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Marc Driessche, Lanctot","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sutton and Barto, 1998] Richard S. Sutton and Andrew G. Barto. Reinforcement learning -an introduction. Adaptive computation and machine learning. MIT Press, 1998.","title":"Reinforcement learning -an introduction. Adaptive computation and machine learning","authors":"Richard S Sutton, Andrew G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tassa et al., 2018] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. DeepMind control suite. arXiv preprint arXiv:1801.00690, 2018.","title":"DeepMind control suite","authors":"Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego De Las, David Casas, Abbas Budden, Josh Abdolmaleki, Andrew Merel, Lefrancq","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tian et al., 2018] Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. DeepTest: automated testing of deep- neural-network-driven autonomous cars. In ICSE, pages 303-314. ACM, 2018.","title":"DeepTest: automated testing of deepneural-network-driven autonomous cars","authors":"Yuchi Tian, Kexin Pei, Suman Jana, Baishakhi Ray","year":"2018","venue":"ICSE","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Trujillo et al., 2020] Miller Trujillo, Mario Linares- Vásquez, Camilo Escobar-Velásquez, Ivana Dusparic, and Nicolás Cardozo. Does neuron coverage matter for deep reinforcement learning?: A preliminary study. In ICSE '20 Workshops, pages 215-220, 2020.","title":"Does neuron coverage matter for deep reinforcement learning?: A preliminary study","authors":"Miller Trujillo, Mario Linares-Vásquez, Camilo Escobar-Velásquez, Ivana Dusparic, Nicolás Cardozo","year":"2020","venue":"ICSE '20 Workshops","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang et al., 2021] Daimeng Wang, Zheng Zhang, Hang Zhang, Zhiyun Qian, Srikanth V. Krishnamurthy, and Nael B. Abu-Ghazaleh. Syzvegas: Beating kernel fuzzing odds with reinforcement learning. In USENIX Security, pages 2741-2758, 2021.","title":"Syzvegas: Beating kernel fuzzing odds with reinforcement learning","authors":"Daimeng Wang, Zheng Zhang, Hang Zhang, Zhiyun Qian, V Srikanth, Krishnamurthy, B Nael, Abu-Ghazaleh","year":"2021","venue":"USENIX Security","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zeller et al., 2021] Andreas Zeller, Rahul Gopinath, Mar- cel Böhme, Gordon Fraser, and Christian Holler. The fuzzing book. https://www.fuzzingbook.org/, 2021. ac- cessed: 2022, January 07.","title":"The fuzzing book","authors":"Andreas Zeller, Rahul Gopinath, Marcel Böhme, Gordon Fraser, Christian Holler","year":null,"venue":null,"doi":null,"url":"https://www.fuzzingbook.org/,2021.ac-cessed:2022","internal_url":null},{"position":33,"raw":"[Zhang et al., 2018] Mengshi Zhang, Yuqun Zhang, Ling- ming Zhang, Cong Liu, and Sarfraz Khurshid. Deep- Road: GAN-based metamorphic testing and input valida- tion framework for autonomous driving systems. In ASE, pages 132-142. ACM, 2018.","title":"Deep-Road: GAN-based metamorphic testing and input validation framework for autonomous driving systems","authors":"Zhang","year":"2018","venue":"ASE","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}