{"type":"ScholarlyArticle","title":"Counterexample-Guided Strategy Improvement for POMDPs Using Recurrent Neural Networks","authors":[{"name":"Steven Carr","slug":"steven-carr","given":"Steven","family":"Carr","affiliation":"The University of Texas at Austin"},{"name":"Nils Jansen","slug":"nils-jansen","given":"Nils","family":"Jansen","affiliation":"Radboud University , Nijmegen , The Netherlands"},{"name":"Ralf Wimmer","slug":"ralf-wimmer","given":"Ralf","family":"Wimmer","affiliation":"Albert-Ludwigs-Universität Freiburg , Freiburg im Breisgau , Germany; Concept Engineering GmbH , Freiburg im Breisgau , Germany"},{"name":"Alexandru Serban","slug":"alexandru-serban","given":"Alexandru","family":"Serban","affiliation":"Radboud University , Nijmegen , The Netherlands"},{"name":"Bernd Becker","slug":"bernd-becker","given":"Bernd","family":"Becker","affiliation":"Albert-Ludwigs-Universität Freiburg , Freiburg im Breisgau , Germany"},{"name":"Ufuk Topcu","slug":"ufuk-topcu","given":"Ufuk","family":"Topcu","affiliation":"The University of Texas at Austin"}],"abstract":"We study strategy synthesis for partially observable Markov decision processes (POMDPs). The particular problem is to determine strategies that provably adhere to (probabilistic) temporal logic constraints. This problem is computationally intractable and theoretically hard. We propose a novel method that combines techniques from machine learning and formal verification. First, we train a recurrent neural network (RNN) to encode POMDP strategies. The RNN accounts for memory-based decisions without the need to expand the full belief space of a POMDP. Secondly, we restrict the RNN-based strategy to represent a finite-memory strategy and implement it on a specific POMDP. For the resulting finite Markov chain, efficient formal verification techniques provide provable guarantees against temporal logic specifications. If the specification is not satisfied, counterexamples supply diagnostic information. We use this information to improve the strategy by iteratively training the RNN. Numerical experiments show that the proposed method elevates the state of the art in POMDP solving by up to three orders of magnitude in terms of solving times and model sizes.","doi":"10.24963/ijcai.2019/768","url":"https://ijcai.org/proceedings/2019/768","pdf":"https://ijcai.org/proceedings/2019/0768.pdf","markdown":"https://ijcai.org/proceedings/2019/768.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/768","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Planning and Scheduling","track":"Main track","number":768,"pages":{"first":5532,"last":5539},"keywords":["Planning and Scheduling: POMDPs","Planning and Scheduling: Markov Decisions Processes","Agent-based and Multi-agent Systems: Formal Verification, Validation and Synthesis"],"videos":[],"references":[{"position":1,"raw":"[Amato et al., 2010] Christopher Amato, Daniel S Bernstein, and Shlomo Zilberstein. Optimizing fixed-size stochas- tic controllers for POMDPs and decentralized POMDPs. AAMAS, 21(3):293-320, 2010.","title":"Optimizing fixed-size stochastic controllers for POMDPs and decentralized POMDPs","authors":"Christopher Amato, Daniel S Bernstein, Shlomo Zilberstein","year":"2010","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baier and Katoen, 2008] Christel Baier and Joost-Pieter Ka- toen. Principles of Model Checking. MIT Press, 2008.","title":"Christel Baier and Joost-Pieter Katoen","authors":"Katoen Baier","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chatterjee et al., 2015] Krishnendu Chatterjee, Martin Chmelík, Raghav Gupta, and Ayush Kanodia. Qualitative analysis of POMDPs with temporal logic specifications for robotics applications. In ICRA, pages 325-330, 2015.","title":"Qualitative analysis of POMDPs with temporal logic specifications for robotics applications","authors":"Krishnendu Chatterjee, Martin Chmelík, Raghav Gupta, Ayush Kanodia","year":"2015","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chatterjee et al., 2016] Krishnendu Chatterjee, Martin Chmelík, and Mathieu Tracol. What is decidable about partially observable Markov decision processes with ω-regular objectives. Journal of Computer and System Sciences, 82(5):878-911, 2016.","title":"What is decidable about partially observable Markov decision processes with ω-regular objectives","authors":"Krishnendu Chatterjee, Martin Chmelík, Mathieu Tracol","year":"2016","venue":"Journal of Computer and System Sciences","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chollet and others, 2015] Franc ¸ois Chollet et al. Keras. https://github.com/fchollet/keras, 2015. Accessed on 02.13.19.","title":"Franc ¸ois Chollet","authors":"Chollet","year":"2015","venue":null,"doi":null,"url":"https://github.com/fchollet/keras,2015","internal_url":null},{"position":6,"raw":"[Cubuktepe et al., 2017] Murat Cubuktepe, Nils Jansen, Se- bastian Junges, Joost-Pieter Katoen, Ivan Papusha, Hasan A. Poonawala, and Ufuk Topcu. Sequential convex program- ming for the efficient verification of parametric MDPs. In TACAS (2), volume 10206 of LNCS, pages 133-150, 2017.","title":"Sequential convex programming for the efficient verification of parametric MDPs","authors":"Nils Murat Cubuktepe, Sebastian Jansen, Joost-Pieter Junges, Ivan Katoen, Papusha, A Hasan, Ufuk Poonawala, Topcu","year":"2017","venue":"TACAS (2)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cubuktepe et al., 2018] Murat Cubuktepe, Nils Jansen, Se- bastian Junges, Joost-Pieter Katoen, and Ufuk Topcu. Syn- thesis in pmdps: A tale of 1001 parameters. In ATVA, vol- ume 11138 of Lecture Notes in Computer Science, pages 160-176. Springer, 2018.","title":"Synthesis in pmdps: A tale of 1001 parameters","authors":"Nils Murat Cubuktepe, Sebastian Jansen, Joost-Pieter Junges, Ufuk Katoen, Topcu","year":"2018","venue":"ATVA","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dehnert et al., 2017] Christian Dehnert, Sebastian Junges, Joost-Pieter Katoen, and Matthias Volk. A storm is coming: A modern probabilistic model checker. In CAV (2), volume 10427 of LNCS, pages 592-600. Springer, 2017.","title":"A storm is coming: A modern probabilistic model checker","authors":"Dehnert","year":"2017","venue":"CAV (2)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Haesaert et al., 2018] Sofie Haesaert, Petter Nilsson, Cris- tian Ioan Vasile, Rohan Thakker, Ali-akbar Agha- mohammadi, Aaron D. Ames, and Richard M. Murray. Temporal logic control of POMDPs via label-based stochas- tic simulation relations. In ADHS, volume 51(16) of IFAC- PapersOnLine, pages 271-276. Elsevier, 2018.","title":"Temporal logic control of POMDPs via label-based stochastic simulation relations","authors":"Sofie Haesaert, Petter Nilsson, Cristian Ioan Vasile, Rohan Thakker, Ali-Akbar Aghamohammadi, Aaron D Ames, Richard M Murray","year":"2018","venue":"IFAC-PapersOnLine","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hahn et al., 2010] Ernst Moritz Hahn, Holger Hermanns, and Lijun Zhang. Probabilistic reachability for parametric Markov models. Software Tools for Technology Transfer, 13(1):3-19, 2010.","title":"Probabilistic reachability for parametric Markov models","authors":"Hahn","year":"2010","venue":"Software Tools for Technology Transfer","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hausknecht and Stone, 2015] Matthew Hausknecht and Pe- ter Stone. Deep recurrent q-learning for partially observable MDPs. CoRR, abs/1507.06527, 7(1), 2015.","title":"Deep recurrent q-learning for partially observable MDPs","authors":"Matthew Hausknecht, Peter Stone","year":"2015","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hauskrecht, 2000] Milos Hauskrecht. Value-function ap- proximations for partially observable Markov decision pro- cesses. J. Artif. Intell. Res., 13:33-94, 2000.","title":"Milos Hauskrecht. Value-function approximations for partially observable Markov decision processes","authors":"Hauskrecht","year":"2000","venue":"J. Artif. Intell. Res","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Junges et al., 2018] Sebastian Junges, Nils Jansen, Ralf Wimmer, Tim Quatmann, Leonore Winterer, Joost-Pieter Katoen, and Bernd Becker. Finite-state controllers of POMDPs via parameter synthesis. In UAI, 2018.","title":"Finite-state controllers of POMDPs via parameter synthesis","authors":"Sebastian Junges, Nils Jansen, Ralf Wimmer, Tim Quatmann, Leonore Winterer, Joost-Pieter Katoen, Bernd Becker","year":"2018","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Junges et al., 2019] Sebastian Junges, Erika Ábrahám, Chris- tian Hensel, Nils Jansen, Joost-Pieter Katoen, Tim Quat- mann, and Matthias Volk. Parameter synthesis for markov models. CoRR, abs/1903.07993, 2019.","title":"Parameter synthesis for markov models","authors":"Sebastian Junges, Erika Ábrahám, Christian Hensel, Nils Jansen, Joost-Pieter Katoen, Tim Quatmann, Matthias Volk","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kaelbling et al., 1998] Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and act- ing in partially observable stochastic domains. Artif. Intell., 101(1):99-134, 1998.","title":"Planning and acting in partially observable stochastic domains","authors":"Leslie Pack Kaelbling, Michael L Littman, Anthony R Cassandra","year":"1998","venue":"Artif. Intell","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kearns et al., 2000] Michael J Kearns, Yishay Mansour, and Andrew Y Ng. Approximate planning in large POMDPs via reusable trajectories. In NIPS, pages 1001-1007, 2000.","title":"Approximate planning in large POMDPs via reusable trajectories","authors":"J Michael, Yishay Kearns, Andrew Y Mansour, Ng","year":"2000","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kingma and Ba, 2014] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kwiatkowska et al., 2011] Marta Kwiatkowska, Gethin Nor- man, and David Parker. PRISM 4.0: Verification of proba- bilistic real-time systems. In CAV, volume 6806 of LNCS, pages 585-591. Springer, 2011.","title":"PRISM 4.0: Verification of probabilistic real-time systems","authors":"Marta Kwiatkowska, Gethin Norman, David Parker","year":"2011","venue":"CAV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Littman et al., 2017] Michael L. Littman, Ufuk Topcu, Jie Fu, Charles Isbell, Min Wen, and James MacGlashan. Environment-independent task specifications via GLTL. arXiv preprint 1704.04341, 2017.","title":"Environment-independent task specifications via GLTL","authors":"L Michael, Ufuk Littman, Jie Topcu, Charles Fu, Min Isbell, James Wen, Macglashan","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Madani et al., 1999] Omid Madani, Steve Hanks, and Anne Condon. On the undecidability of probabilistic planning and infinite-horizon partially observable Markov decision problems. In AAAI, pages 541-548. AAAI Press, 1999.","title":"On the undecidability of probabilistic planning and infinite-horizon partially observable Markov decision problems","authors":"Omid Madani, Steve Hanks, Anne Condon","year":"1999","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Meuleau et al., 1999] Nicolas Meuleau, Leonid Peshkin, Kee-Eung Kim, and Leslie Pack Kaelbling. Learning finite- state controllers for partially observable environments. In UAI, pages 427-436.","title":"Learning finitestate controllers for partially observable environments","authors":"Nicolas Meuleau, Leonid Peshkin, Kee-Eung Kim, Leslie Pack, Kaelbling","year":null,"venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"Morgan Kaufmann, 1999.","title":null,"authors":"Morgan Kaufmann","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Norman et al., 2017] Gethin Norman, David Parker, and Xueyi Zou. Verification and control of partially observable probabilistic systems. Real-Time Systems, 53(3):354-402, 2017.","title":"Verification and control of partially observable probabilistic systems","authors":"Gethin Norman, David Parker, Xueyi Zou","year":"2017","venue":"Real-Time Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Papadimitriou and Tsitsiklis, 1987] Christos H. Papadim- itriou and John N. Tsitsiklis. The complexity of Markov decision processes. Mathematics of Operations Research, 12(3):441-450, 1987.","title":"The complexity of Markov decision processes","authors":"Christos H Papadimitriou, John N Tsitsiklis","year":"1987","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Parisotto and Salakhutdinov, 2018] Emilio Parisotto and Ruslan Salakhutdinov. Neural map: Structured memory for deep reinforcement learning. In ICLR. OpenReview.net, 2018.","title":"Emilio Parisotto and Ruslan Salakhutdinov. Neural map: Structured memory for deep reinforcement learning","authors":"Salakhutdinov Parisotto","year":"2018","venue":"ICLR. OpenReview.net","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Pascanu et al., 2013] Razvan Pascanu, C ¸aglar Gülc ¸ehre, Kyunghyun Cho, and Yoshua Bengio. How to construct deep recurrent neural networks. CoRR, abs/1312.6026, 2013.","title":"How to construct deep recurrent neural networks","authors":"Razvan Pascanu, C ¸aglar Gülc ¸ehre, Kyunghyun Cho, Yoshua Bengio","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Pineau et al., 2003] Joelle Pineau, Geoff Gordon, and Sebas- tian Thrun. Point-based value iteration: An anytime algo- rithm for POMDPs. In IJCAI, pages 1025-1032.","title":"Point-based value iteration: An anytime algorithm for POMDPs","authors":"Joelle Pineau, Geoff Gordon, Sebastian Thrun","year":null,"venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2003/146"},{"position":30,"raw":"Morgan Kaufmann, 2003.","title":null,"authors":"Morgan Kaufmann","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Pnueli, 1977] Amir Pnueli. The temporal logic of programs. In FOCS, pages 46-57. IEEE Computer Society, 1977.","title":"The temporal logic of programs","authors":"Amir Pnueli, Pnueli","year":"1977","venue":"FOCS","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Pritzel et al., 2017] Alexander Pritzel, Benigno Uria, Sri- ram Srinivasan, Adrià Puigdomènech Badia, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell. Neu- ral episodic control. In ICML, volume 70 of Proc. of Ma- chine Learning Research, pages 2827-2836. PMLR, 2017.","title":"Neural episodic control","authors":"Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adrià Puigdomènech Badia, Oriol Vinyals, Demis Hassabis, Daan Wierstra, Charles Blundell","year":"2017","venue":"Proc. of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Santoro et al., 2018] Adam Santoro, Ryan Faulkner, David Raposo, Jack W. Rae, Mike Chrzanowski, Theophane We- ber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Timothy P. Lillicrap. Relational recurrent neural networks. In NeurIPS, pages 7310-7321, 2018.","title":"Relational recurrent neural networks","authors":"Adam Santoro, Ryan Faulkner, David Raposo, Jack W Rae, Mike Chrzanowski, Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, Timothy P Lillicrap","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Silver and Veness, 2010] David Silver and Joel Veness. Monte-carlo planning in large POMDPs. In NIPS, pages 2164-2172, 2010.","title":"Monte-carlo planning in large POMDPs","authors":"David Silver, Joel Veness","year":"2010","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Sutton et al., 2000] Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approxi- mation. In NIPS, pages 1057-1063, 2000.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"Richard S Sutton, David A Mcallester, P Satinder, Yishay Singh, Mansour","year":"2000","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Vlassis et al., 2012] Nikos Vlassis, Michael L. Littman, and David Barber. On the computational complexity of stochas- tic controller optimization in POMDPs. ACM Trans. on Computation Theory, 4(4):12:1-12:8, 2012.","title":"On the computational complexity of stochastic controller optimization in POMDPs","authors":"Nikos Vlassis, Michael L Littman, David Barber","year":"2012","venue":"ACM Trans. on Computation Theory","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Walraven and Spaan, 2017] Erwin Walraven and Matthijs Spaan. Accelerated vector pruning for optimal POMDP solvers. In AAAI, pages 3672-3678. AAAI Press, 2017.","title":"Accelerated vector pruning for optimal POMDP solvers","authors":"Spaan Walraven, Erwin Walraven, Matthijs Spaan","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Wang et al., 2018] Yue Wang, Swarat Chaudhuri, and Ly- dia E. Kavraki. Bounded policy synthesis for pomdps with safe-reachability objectives. In AAMAS, pages 238-246. Int'l Foundation for Autonomous Agents and Multiagent Systems Richland, SC, USA / ACM, 2018.","title":"Bounded policy synthesis for pomdps with safe-reachability objectives","authors":"Yue Wang, Swarat Chaudhuri, Lydia E Kavraki","year":"2018","venue":"Int'l Foundation for Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Wierstra et al., 2007] Daan Wierstra, Alexander Förster, Jan Peters, and Jürgen Schmidhuber. Solving deep memory POMDPs with recurrent policy gradients. In ICANN, pages 697-706. Springer, 2007.","title":"Solving deep memory POMDPs with recurrent policy gradients","authors":"Daan Wierstra, Alexander Förster, Jan Peters, Jürgen Schmidhuber","year":"2007","venue":"ICANN","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Wimmer et al., 2014] Ralf Wimmer, Nils Jansen, Erika Ábrahám, Joost-Pieter Katoen, and Bernd Becker. Minimal counterexamples for linear-time probabilistic verification. Theoretical Computer Science, 549:61-100, 2014.","title":"Minimal counterexamples for linear-time probabilistic verification","authors":"Ralf Wimmer, Nils Jansen, Erika Ábrahám, Joost-Pieter Katoen, Bernd Becker","year":"2014","venue":"Theoretical Computer Science","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}