{"type":"ScholarlyArticle","title":"Verifying Reinforcement Learning up to Infinity","authors":[{"name":"Edoardo Bacci","slug":"edoardo-bacci","given":"Edoardo","family":"Bacci","affiliation":"University of Birmingham"},{"name":"Mirco Giacobbe","slug":"mirco-giacobbe","given":"Mirco","family":"Giacobbe","affiliation":"University of Oxford"},{"name":"David Parker","slug":"david-parker","given":"David","family":"Parker","affiliation":"University of Birmingham"}],"abstract":"Formally verifying that reinforcement learning systems act\r\nsafely is increasingly important, but existing methods\r\nonly verify over finite time.\r\nThis is of limited use for dynamical systems that run indefinitely. \r\nWe introduce the first method for verifying the time-unbounded\r\nsafety of neural networks controlling dynamical systems.\r\nWe develop a novel abstract interpretation method which,\r\nby constructing adaptable template-based polyhedra using MILP and interval\r\narithmetic, yields sound---safe and invariant---overapproximations\r\nof the reach set.\r\nThis provides stronger safety guarantees\r\nthan previous time-bounded methods and shows whether\r\nthe agent has generalised beyond the length of its training episodes.\r\nOur method supports ReLU activation functions\r\nand systems with linear, piecewise linear and non-linear dynamics\r\ndefined with polynomial and transcendental functions.\r\nWe demonstrate its efficacy on a range of benchmark control problems.","doi":"10.24963/ijcai.2021/297","url":"https://ijcai.org/proceedings/2021/297","pdf":"https://ijcai.org/proceedings/2021/0297.pdf","markdown":"https://ijcai.org/proceedings/2021/297.md","bibtex":"https://ijcai.org/proceedings/2021/bibtex/297","volume":{"slug":"2021","title":"Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2021","url":"https://ijcai.org/proceedings/2021/","year":2021,"location":"Montreal","dates":"19-27 August 2021","editors":"Prof. Zhi-Hua Zhou","isbn":"978-0-9992411-9-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2021"},"section":"Machine Learning","track":"Main Track","number":297,"pages":{"first":2154,"last":2160},"keywords":["Machine Learning: Deep Reinforcement Learning","Robotics: Learning in Robotics","Multidisciplinary Topics and Applications: Validation and Verification"],"videos":[],"references":[{"position":1,"raw":"[Alshiekh et al., 2018] Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. Safe reinforcement learning via shielding. In AAAI, pages 2669-2678. AAAI Press, 2018.","title":"Safe reinforcement learning via shielding","authors":"Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, Ufuk Topcu","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bacci and Parker, 2020] Edoardo Bacci and David Parker. Probabilistic guarantees for safe deep reinforcement learn- ing. In FORMATS, pages 231-248. Springer, 2020.","title":"Probabilistic guarantees for safe deep reinforcement learning","authors":"Edoardo Bacci, David Parker","year":"2020","venue":"FORMATS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bogomolov et al., 2017] Sergiy Bogomolov, Goran Frehse, Mirco Giacobbe, and Thomas A. Henzinger. Counterex- ample-guided refinement of template polyhedra. In TACAS (1), pages 589-606, 2017.","title":"Counterexample-guided refinement of template polyhedra","authors":"Sergiy Bogomolov, Goran Frehse, Mirco Giacobbe, Thomas A Henzinger","year":"2017","venue":"TACAS (1)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Lud- wig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI gym. arXiv:1606.01540, 2016.","title":"OpenAI gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bunel et al., 2018] Rudy Bunel, Ilker Turkaslan, Philip H. S. Torr, Pushmeet Kohli, and Pawan Kumar Mudigonda. A unified view of piecewise linear neural network verification. In NeurIPS, pages 4795-4804, 2018.","title":"A unified view of piecewise linear neural network verification","authors":"Rudy Bunel, Ilker Turkaslan, H S Philip, Pushmeet Torr, Pawan Kohli, Mudigonda Kumar","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cheng et al., 2019] Richard Cheng, Gábor Orosz, Richard M. Murray, and Joel W. Burdick. End-to- end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In AAAI, pages 3387-3395. AAAI Press, 2019.","title":"End-toend safe reinforcement learning through barrier functions for safety-critical continuous control tasks","authors":"Richard Cheng, Gábor Orosz, Richard M Murray, Joel W Burdick","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Desjardins and Chaib-draa, 2011] Charles Desjardins and Brahim Chaib-draa. Cooperative adaptive cruise control: A reinforcement learning approach. IEEE Trans. Intell. Transp. Syst., 12(4):1248-1260, 2011.","title":"Cooperative adaptive cruise control: A reinforcement learning approach","authors":"Chaib-Draa Desjardins","year":"2011","venue":"IEEE Trans. Intell. Transp. Syst","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dutta et al., 2018] Souradeep Dutta, Susmit Jha, Sriram Sankaranarayanan, and Ashish Tiwari. Output range analy- sis for deep feedforward neural networks. In NFM, pages 121-138. Springer, 2018.","title":"Output range analysis for deep feedforward neural networks","authors":"Dutta","year":"2018","venue":"NFM","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ehlers, 2017] Rüdiger Ehlers. Formal verification of piece- wise linear feed-forward neural networks. In ATVA, pages 269-286. Springer, 2017.","title":"Formal verification of piecewise linear feed-forward neural networks","authors":"Rüdiger Ehlers","year":"2017","venue":"ATVA","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Frehse et al., 2018] Goran Frehse, Mirco Giacobbe, and Thomas A. Henzinger. Space-time interpolants. In CAV (1), pages 468-486. Springer, 2018.","title":"Space-time interpolants","authors":"Goran Frehse, Mirco Giacobbe, Thomas A Henzinger","year":"2018","venue":"CAV (1)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Fulton and Platzer, 2018] Nathan Fulton and André Platzer. Safe reinforcement learning via formal methods: Toward safe control through proof and learning. In AAAI, pages 6485-6492. AAAI Press, 2018.","title":"Safe reinforcement learning via formal methods: Toward safe control through proof and learning","authors":"Fulton, Platzer, Nathan Fulton, André Platzer","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gehr et al., 2018] Timon Gehr, Matthew Mirman, Dana Drachsler-Cohen, Petar Tsankov, Swarat Chaudhuri, and Martin T. Vechev. AI2: safety and robustness certification of neural networks with abstract interpretation. In IEEE SP, pages 3-18. IEEE Computer Society, 2018.","title":"Vechev. AI2: safety and robustness certification of neural networks with abstract interpretation","authors":"Timon Gehr, Matthew Mirman, Dana Drachsler-Cohen, Petar Tsankov, Swarat Chaudhuri, Martin T","year":"2018","venue":"IEEE SP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hasanbeig et al., 2019] Mohammadhosein Hasanbeig, Alessandro Abate, and Daniel Kroening. Logically- constrained neural fitted q-iteration. In AAMAS, pages 2012-2014.","title":"Logicallyconstrained neural fitted q-iteration","authors":"Hasanbeig","year":"2019","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"IFAAMAS, 2019.","title":null,"authors":"Ifaamas","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Huang et al., 2017] Xiaowei Huang, Marta Kwiatkowska, Sen Wang, and Min Wu. Safety verification of deep neural networks. In CAV (1), pages 3-29. Springer, 2017.","title":"Safety verification of deep neural networks","authors":"Xiaowei Huang, Marta Kwiatkowska, Sen Wang, Min Wu","year":"2017","venue":"CAV (1)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Jaeger et al., 2019] Manfred Jaeger, Peter Gjøl Jensen, Kim Guldstrand Larsen, Axel Legay, Sean Sedwards, and Jakob Haahr Taankvist. Teaching stratego to play ball: Op- timal synthesis for continuous space mdps. In ATVA, pages 81-97. Springer, 2019.","title":"Teaching stratego to play ball: Optimal synthesis for continuous space mdps","authors":"Jaeger","year":"2019","venue":"ATVA","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Katz et al., 2019] Guy Katz, Derek A. Huang, Duligur Ibel- ing, Kyle Julian, Christopher Lazarus, Rachel Lim, Parth Shah, Shantanu Thakoor, Haoze Wu, Aleksandar Zeljic, David L. Dill, Mykel J. Kochenderfer, and Clark W. Barrett. The marabou framework for verification and analysis of deep neural networks. In CAV (1), pages 443-452. Springer, 2019.","title":"The marabou framework for verification and analysis of deep neural networks","authors":"Guy Katz, Derek A Huang, Duligur Ibeling, Kyle Julian, Christopher Lazarus, Rachel Lim, Parth Shah, Shantanu Thakoor, Haoze Wu, Aleksandar Zeljic, David L Dill, J Mykel, Clark W Kochenderfer, Barrett","year":"2019","venue":"CAV (1)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kendall et al., 2019] Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, and Amar Shah. Learning to drive in a day. In ICRA, pages 8248-8254. IEEE, 2019.","title":"Learning to drive in a day","authors":"Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, Amar Shah","year":"2019","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li and Bastani, 2020] Shuo Li and Osbert Bastani. Robust model predictive shielding for safe reinforcement learn- ing with stochastic dynamics. In ICRA, pages 7166-7172. IEEE, 2020.","title":"Robust model predictive shielding for safe reinforcement learning with stochastic dynamics","authors":"Bastani ; ] Shuo Li, Osbert Li, Bastani","year":"2020","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Luckcuck et al., 2019] Matt Luckcuck, Marie Farrell, Louise A. Dennis, Clare Dixon, and Michael Fisher. Formal specification and verification of autonomous robotic systems: A survey. ACM Comput. Surv., 52(5):100:1-100:41, 2019.","title":"Formal specification and verification of autonomous robotic systems: A survey","authors":"Matt Luckcuck, Marie Farrell, Louise A Dennis, Clare Dixon, Michael Fisher","year":"2019","venue":"ACM Comput. Surv","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Moosavi-Dezfooli et al., 2016] Seyed-Mohsen Moosavi- Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: A simple and accurate method to fool deep neural networks. In CVPR, pages 2574-2582. IEEE Computer Society, 2016.","title":"Deepfool: A simple and accurate method to fool deep neural networks","authors":"Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Pascal Frossard","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sankaranarayanan et al., 2005] Sriram Sankaranarayanan, Henny B. Sipma, and Zohar Manna. Scalable analysis of linear systems using mathematical programming. In VMCAI, pages 25-41. Springer, 2005.","title":"Scalable analysis of linear systems using mathematical programming","authors":"Sriram Sankaranarayanan, B Henny, Zohar Sipma, Manna","year":"2005","venue":"VMCAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Singh et al., 2019] Gagandeep Singh, Timon Gehr, Markus Püschel, and Martin T. Vechev. An abstract domain for certifying neural networks. Proc. ACM Program. Lang., 3(POPL):41:1-41:30, 2019.","title":"An abstract domain for certifying neural networks","authors":"Gagandeep Singh, Timon Gehr, Markus Püschel, Martin T Vechev","year":"2019","venue":"Proc. ACM Program. Lang","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tjeng et al., 2019] Vincent Tjeng, Kai Y. Xiao, and Russ Tedrake. Evaluating robustness of neural networks with mixed integer programming. In ICLR (Poster). OpenRe- view.net, 2019.","title":"Evaluating robustness of neural networks with mixed integer programming","authors":"Vincent Tjeng, Kai Y Xiao, Russ Tedrake","year":"2019","venue":"ICLR (Poster)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tran et al., 2020] Hoang-Dung Tran, Xiaodong Yang, Die- go Manzanas Lopez, Patrick Musau, Luan Viet Nguyen, Weiming Xiang, Stanley Bak, and Taylor T. Johnson. NNV: the neural network verification tool for deep neural net- works and learning-enabled cyber-physical systems. In CAV (1), pages 3-17. Springer, 2020.","title":"NNV: the neural network verification tool for deep neural networks and learning-enabled cyber-physical systems","authors":"Hoang-Dung Tran, Xiaodong Yang, Diego Manzanas Lopez, Patrick Musau, Luan Viet Nguyen, Weiming Xiang, Stanley Bak, Taylor T Johnson","year":"2020","venue":"CAV (1)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xiang et al., 2018] Weiming Xiang, Hoang-Dung Tran, and Taylor T. Johnson. Output reachable set estimation and verification for multilayer neural networks. IEEE Trans. Neural Networks Learn. Syst., 29(11):5777-5783, 2018.","title":"Output reachable set estimation and verification for multilayer neural networks","authors":"Weiming Xiang, Hoang-Dung Tran, Taylor T Johnson","year":"2018","venue":"IEEE Trans. Neural Networks Learn. Syst","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}