{"type":"ScholarlyArticle","title":"Verifiable RNN-Based Policies for POMDPs Under Temporal Logic Constraints","authors":[{"name":"Steven Carr","slug":"steven-carr","given":"Steven","family":"Carr","affiliation":"The University of Texas at Austin"},{"name":"Nils Jansen","slug":"nils-jansen","given":"Nils","family":"Jansen","affiliation":"Radboud University , Nijmegen , The Netherlands"},{"name":"Ufuk Topcu","slug":"ufuk-topcu","given":"Ufuk","family":"Topcu","affiliation":"The University of Texas at Austin"}],"abstract":"Recurrent neural networks (RNNs) have emerged as an effective representation of control policies in sequential decision-making problems.\r\nHowever, a major drawback in the application of RNN-based policies is the difficulty in providing formal guarantees on the satisfaction of behavioral specifications, e.g. safety and/or reachability. \r\nBy integrating techniques from formal methods and machine learning, we propose an approach to automatically extract a finite-state controller (FSC) from an RNN, which, when composed with a finite-state system model, is amenable to existing formal verification tools.\r\nSpecifically, we introduce an iterative modification to the so-called quantized bottleneck insertion technique to create an FSC as a randomized policy with memory.\r\nFor the cases in which the resulting FSC fails to satisfy the specification, verification generates diagnostic information.\r\nWe utilize this information to either adjust the amount of memory in the extracted FSC or perform focused retraining of the RNN.\r\nWhile generally applicable, we detail the resulting iterative procedure in the context of policy synthesis for partially observable Markov decision processes (POMDPs), which is known to be notoriously hard.\r\nThe numerical experiments show that the proposed approach outperforms traditional POMDP synthesis methods by 3 orders of magnitude within 2% of optimal benchmark values.","doi":"10.24963/ijcai.2020/570","url":"https://ijcai.org/proceedings/2020/570","pdf":"https://ijcai.org/proceedings/2020/0570.pdf","markdown":"https://ijcai.org/proceedings/2020/570.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/570","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Planning and Scheduling","track":"Main track","number":570,"pages":{"first":4121,"last":4127},"keywords":["Planning and Scheduling: POMDPs","Planning and Scheduling: Markov Decisions Processes","Planning and Scheduling: Planning with Incomplete information"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25521/file_25521.m3u8","length_sec":300,"provider":"hls"}],"references":[{"position":1,"raw":"[Akintunde et al., 2019] Michael E. Akintunde, Andreea Kevorchian, Alessio Lomuscio, and Edoardo Pirovano. Verification of RNN-based neural agent-environment sys- tems. In AAAI, pages 6006-6013. AAAI Press, 2019.","title":"Verification of RNN-based neural agent-environment systems","authors":"Michael E Akintunde, Andreea Kevorchian, Alessio Lomuscio, Edoardo Pirovano","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baier and Katoen, 2008] Christel Baier and Joost-Pieter Katoen. Principles of Model Checking. MIT Press, 2008.","title":"Principles of Model Checking","authors":"Christel Baier, Joost-Pieter Katoen","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bakker, 2001] Bram Bakker. Reinforcement learning with long short-term memory. In NIPS, pages 1475-1482. MIT Press, 2001.","title":"Reinforcement learning with long short-term memory","authors":"Bram Bakker, Bakker","year":"2001","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Baum and Wilczek, 1987] Eric B. Baum and Frank Wilczek. Supervised learning of probability distributions by neural networks. In NIPS, pages 52-61. American Institue of Physics, 1987.","title":"Supervised learning of probability distributions by neural networks","authors":"Eric B Baum, Frank Wilczek","year":"1987","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Carr et al., 2019] Steven Carr, Nils Jansen, Ralf Wimmer, Alexandru Constantin Serban, Bernd Becker, and Ufuk Topcu. Counterexample-guided strategy improvement for POMDPs using recurrent neural networks. In IJCAI, pages 5532-5539, 2019.","title":"Counterexample-guided strategy improvement for POMDPs using recurrent neural networks","authors":"Steven Carr, Nils Jansen, Ralf Wimmer, Alexandru Constantin Serban, Bernd Becker, Ufuk Topcu","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/768"},{"position":6,"raw":"[Cassandra, 1998] Anthony R Cassandra. A survey of POMDP applications. In AAAI, volume 1724, 1998.","title":"A survey of POMDP applications","authors":"Cassandra Anthony","year":"1998","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cover and Thomas, 2012] Thomas M Cover and Joy A Thomas. Elements of information theory. John Wiley & Sons, 2012.","title":"Elements of information theory","authors":"M Thomas, Joy A Cover, Thomas","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dehnert et al., 2017] Christian Dehnert, Sebastian Junges, Joost-Pieter Katoen, and Matthias Volk. A storm is com- ing: A modern probabilistic model checker. In CAV (2), volume 10427 of LNCS, pages 592-600. Springer, 2017.","title":"A storm is coming: A modern probabilistic model checker","authors":"Dehnert","year":"2017","venue":"CAV (2)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Goodfellow and Vinyals, 2015] Ian J. Goodfellow and Oriol Vinyals. Qualitatively characterizing neural network opti- mization problems. In ICLR, 2015.","title":"Qualitatively characterizing neural network optimization problems","authors":"Ian J Goodfellow, Oriol Vinyals","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Goodfellow et al., 2016] Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, vol- ume 1. MIT Press, 2016.","title":"Deep learning","authors":"Ian Goodfellow, Yoshua Bengio, Aaron Courville, Yoshua Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hausknecht and Stone, 2015] Matthew J. Hausknecht and Peter Stone. Deep recurrent Q-learning for partially ob- servable MDPs. In AAAI, pages 29-37. AAAI Press, 2015.","title":"Deep recurrent Q-learning for partially observable MDPs","authors":"Matthew J Hausknecht, Peter Stone","year":"2015","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Huang et al., 2017] Xiaowei Huang, Marta Kwiatkowska, Sen Wang, and Min Wu. Safety verification of deep neu- ral networks. In CAV (1), volume 10426 of LNCS, pages 3-29. Springer, 2017.","title":"Safety verification of deep neural networks","authors":"Xiaowei Huang, Marta Kwiatkowska, Sen Wang, Min Wu","year":"2017","venue":"CAV (1)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Junges et al., 2018] Sebastian Junges, Nils Jansen, Ralf Wimmer, Tim Quatmann, Leonore Winterer, Joost-Pieter Katoen, and Bernd Becker. Finite-state controllers of POMDPs via parameter synthesis. In UAI, 2018.","title":"Finite-state controllers of POMDPs via parameter synthesis","authors":"Sebastian Junges, Nils Jansen, Ralf Wimmer, Tim Quatmann, Leonore Winterer, Joost-Pieter Katoen, Bernd Becker","year":"2018","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Katz et al., 2017] Guy Katz, Clark Barrett, David L. Dill, Kyle Julian, and Mykel J. Kochenderfer. Reluplex: An efficient SMT solver for verifying deep neural networks. In CAV, pages 97-117. Springer, 2017.","title":"Reluplex: An efficient SMT solver for verifying deep neural networks","authors":"Guy Katz, Clark Barrett, David L Dill, Kyle Julian, Mykel J Kochenderfer","year":"2017","venue":"CAV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Koul et al., 2019] Anurag Koul, Alan Fern, and Sam Grey- danus. Learning finite state representations of recurrent policy networks. In ICLR, 2019.","title":"Learning finite state representations of recurrent policy networks","authors":"Koul","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Littman et al., 1995] Michael L. Littman, Anthony R. Cas- sandra, and Leslie Pack Kaelbling. Learning policies for partially observable environments: Scaling up. In ICML, pages 362-370.","title":"Learning policies for partially observable environments: Scaling up","authors":"L Michael, Anthony R Littman, Leslie Cassandra, Kaelbling Pack","year":null,"venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"Morgan Kaufmann, 1995.","title":null,"authors":"Morgan Kaufmann","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Meuleau et al., 1999] Nicolas Meuleau, Kee-Eung Kim, Leslie Pack Kaelbling, and Anthony R. Cassandra. Solv- ing POMDPs by searching the space of finite policies. In UAI, pages 417-426.","title":"Solving POMDPs by searching the space of finite policies","authors":"Nicolas Meuleau, Kee-Eung Kim, Leslie Pack Kaelbling, Anthony R Cassandra","year":null,"venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"Morgan Kaufmann, 1999.","title":null,"authors":"Morgan Kaufmann","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Michalenko et al., 2019] Joshua J. Michalenko, Ameesh Shah, Abhinav Verma, Richard G. Baraniuk, Swarat Chaudhuri, and Ankit B. Patel. Representing formal lan- guages: A comparison between finite automata and recur- rent neural networks. In ICLR, 2019.","title":"Representing formal languages: A comparison between finite automata and recurrent neural networks","authors":"Joshua J Michalenko, Ameesh Shah, Abhinav Verma, Richard G Baraniuk, Swarat Chaudhuri, Ankit B Patel","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Mulder et al., 2015] Wim De Mulder, Steven Bethard, and Marie-Francine Moens. A survey on the application of recurrent neural networks to statistical language modeling. Computer Speech & Language, 30(1):61-98, 2015.","title":"A survey on the application of recurrent neural networks to statistical language modeling","authors":"Mulder","year":"2015","venue":"Computer Speech & Language","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Norman et al., 2017] Gethin Norman, David Parker, and Xueyi Zou. Verification and control of partially observable probabilistic systems. Real-Time Systems, 53(3):354-402, 2017.","title":"Verification and control of partially observable probabilistic systems","authors":"Gethin Norman, David Parker, Xueyi Zou","year":"2017","venue":"Real-Time Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Pascanu et al., 2014] Razvan Pascanu, C ¸aglar Gülc ¸ehre, Kyunghyun Cho, and Yoshua Bengio. How to construct deep recurrent neural networks. In ICLR, 2014.","title":"How to construct deep recurrent neural networks","authors":"Razvan Pascanu, C ¸aglar Gülc ¸ehre, Kyunghyun Cho, Yoshua Bengio","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Pnueli, 1977] Amir Pnueli. The temporal logic of programs. In FOCS, pages 46-57. IEEE Computer Society, 1977.","title":"The temporal logic of programs","authors":"Amir Pnueli, Pnueli","year":"1977","venue":"FOCS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Poupart and Boutilier, 2003] Pascal Poupart and Craig Boutilier. Bounded finite state controllers. In NIPS, pages 823-830. MIT Press, 2003.","title":"Bounded finite state controllers","authors":"Boutilier Poupart, Pascal Poupart, Craig Boutilier","year":"2003","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tsoi and Back, 1997] Ah Chung Tsoi and Andrew D. Back. Discrete time recurrent neural network architectures: A unifying review. Neurocomputing, 15:183-223, 1997.","title":"Discrete time recurrent neural network architectures: A unifying review","authors":"Chung Ah, Andrew D Tsoi, Back","year":"1997","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Walraven and Spaan, 2017] Erwin Walraven and Matthijs T. J. Spaan. Accelerated vector pruning for optimal pomdp solvers. In AAAI, pages 3672-3678, 2017.","title":"Accelerated vector pruning for optimal pomdp solvers","authors":"Erwin Walraven, T J Matthijs, Spaan","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2018] Qinglong Wang, Kaixuan Zhang, Xue Liu, and C. Lee Giles. Verification of recurrent neural networks through rule extraction. CoRR, abs/1811.06029, 2018.","title":"Verification of recurrent neural networks through rule extraction","authors":"Qinglong Wang, Kaixuan Zhang, Xue Liu, C Lee Giles","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Weiss et al., 2018] Gail Weiss, Yoav Goldberg, and Eran Yahav. Extracting automata from recurrent neural net- works using queries and counterexamples. In ICML, vol- ume 80 of Proceedings of Machine Learning Research, pages 5244-5253. PMLR, 2018.","title":"Extracting automata from recurrent neural networks using queries and counterexamples","authors":"Gail Weiss, Yoav Goldberg, Eran Yahav","year":"2018","venue":"Proceedings of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wierstra et al., 2007] Daan Wierstra, Alexander Förster, Jan Peters, and Jürgen Schmidhuber. Solving deep mem- ory POMDPs with recurrent policy gradients. In ICANN, pages 697-706. Springer, 2007.","title":"Solving deep memory POMDPs with recurrent policy gradients","authors":"Daan Wierstra, Alexander Förster, Jan Peters, Jürgen Schmidhuber","year":"2007","venue":"ICANN","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zeng et al., 1993] Zheng Zeng, Rodney M. Goodman, and Padhraic Smyth. Learning finite machines with self- clustering recurrent networks. Neural Comput., 5(6):976- 990, November 1993.","title":"Learning finite machines with selfclustering recurrent networks","authors":"Zheng Zeng, Rodney M Goodman, Padhraic Smyth","year":"1993","venue":"Neural Comput","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}