{"type":"ScholarlyArticle","title":"Online Planning in POMDPs with Self-Improving Simulators","authors":[{"name":"Jinke He","slug":"jinke-he","given":"Jinke","family":"He","affiliation":"Delft University of Technology , The Netherlands"},{"name":"Miguel Suau","slug":"miguel-suau","given":"Miguel","family":"Suau","affiliation":"Delft University of Technology , The Netherlands"},{"name":"Hendrik Baier","slug":"hendrik-baier","given":"Hendrik","family":"Baier","affiliation":"Centrum Wiskunde & Informatica , The Netherlands"},{"name":"Michael Kaisers","slug":"michael-kaisers","given":"Michael","family":"Kaisers","affiliation":"Centrum Wiskunde & Informatica , The Netherlands"},{"name":"Frans A. Oliehoek","slug":"frans-a-oliehoek","given":"Frans A.","family":"Oliehoek","affiliation":"Delft University of Technology , The Netherlands"}],"abstract":"How  can  we  plan  efficiently  in  a  large  and  complex environment when the time budget is limited? Given  the  original  simulator  of  the  environment, which may be computationally very demanding, we propose to learn online an approximate but much faster simulator that improves over time.  To plan reliably and efficiently while the approximate simulator is learning, we develop a method that adaptively decides which simulator to use for every simulation, based on a statistic that measures the accuracy of the approximate simulator. This allows us to use the approximate simulator to replace the original simulator for faster simulations when it is accurate enough under the current context, thus trading off  simulation speed  and  accuracy.   Experimental results in two large domains show that when integrated with POMCP, our approach allows to plan with improving efficiency over time.","doi":"10.24963/ijcai.2022/642","url":"https://ijcai.org/proceedings/2022/642","pdf":"https://ijcai.org/proceedings/2022/0642.pdf","markdown":"https://ijcai.org/proceedings/2022/642.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/642","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Planning and Scheduling","track":"Main Track","number":642,"pages":{"first":4628,"last":4634},"keywords":["Planning and Scheduling: POMDPs","Planning and Scheduling: Planning Algorithms","Planning and Scheduling: Real-time Planning","Planning and Scheduling: Planning under Uncertainty"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986775","length_sec":90,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984918","length_sec":350,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Anand et al., 2015a] Ankit Anand, Aditya Grover, and Parag Singla. A novel abstraction framework for online planning. In AAMAS, 2015.","title":"A novel abstraction framework for online planning","authors":"Anand","year":"2015","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anand et al., 2015b] Ankit Anand, Aditya Grover, Parag Singla, et al. ASAP-UCT: Abstraction of state-action pairs in UCT. In IJCAI, 2015.","title":"ASAP-UCT: Abstraction of state-action pairs in UCT","authors":"Anand","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/216"},{"position":3,"raw":"[Anand et al., 2016] Ankit Anand, Ritesh Noothigattu, Parag Singla, et al. OGA-UCT: On-the-go abstractions in UCT. In ICAPS, 2016.","title":"OGA-UCT: On-the-go abstractions in UCT","authors":"Ankit Anand, Ritesh Noothigattu, Parag Singla","year":"2016","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Audet et al., 2000] Charles Audet, J Denni, Douglas Moore, Andrew Booker, and Paul Frank. A surrogate- model-based method for constrained optimization. In 8th symposium on multidisciplinary analysis and optimiza- tion, page 4891, 2000.","title":"A surrogatemodel-based method for constrained optimization","authors":"Charles Audet, J Denni, Douglas Moore, Andrew Booker, Paul Frank","year":"2000","venue":"8th symposium on multidisciplinary analysis and optimization","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Auer et al., 2002] Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47(2-3):235-256, may 2002.","title":"Finite-time analysis of the multiarmed bandit problem","authors":"Auer","year":"2002","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bai et al., 2016] Aijun Bai, Siddharth Srivastava, and Stu- art J Russell. Markovian state and action abstractions for MDPs via hierarchical MCTS. In IJCAI, 2016.","title":"Markovian state and action abstractions for MDPs via hierarchical MCTS","authors":"Aijun Bai, Siddharth Srivastava, Stuart J Russell","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/430"},{"position":7,"raw":"[Boutilier et al., 1999] Craig Boutilier, Thomas Dean, and Steve Hanks. Decision-theoretic planning: Structural as- sumptions and computational leverage. JAIR, 11, 1999.","title":"Decision-theoretic planning: Structural assumptions and computational leverage","authors":"Craig Boutilier, Thomas Dean, Steve Hanks","year":"1999","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Browne et al., 2012] Cameron B Browne, Edward Pow- ley, Daniel Whitehouse, Simon M Lucas, Peter I Cowl- ing, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in games, 4(1):1-43, 2012.","title":"A survey of monte carlo tree search methods","authors":"Edward Cameron B Browne, Daniel Powley, Whitehouse, M Simon, Peter I Lucas, Philipp Cowling, Stephen Rohlfshagen, Diego Tavener, Spyridon Perez, Simon Samothrakis, Colton","year":"2012","venue":"IEEE Transactions on Computational Intelligence and AI in games","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Buesing et al., 2018] Lars Buesing, Theophane Weber, Sébastien Racaniere, SM Eslami, Danilo Rezende, David P Reichert, Fabio Viola, Frederic Besse, Karol Gregor, Demis Hassabis, et al. Learning and querying fast generative models for reinforcement learning. arXiv preprint arXiv:1802.03006, 2018.","title":"Learning and querying fast generative models for reinforcement learning","authors":"Buesing","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chen et al., 2020] Qifeng Chen, Caisheng Wei, Yongfeng Shi, and Yunhe Meng. Satellite swarm reconfiguration planning based on surrogate models. Journal of Guidance, Control, and Dynamics, 43(9):1750-1756, 2020.","title":"Satellite swarm reconfiguration planning based on surrogate models","authors":"Qifeng Chen, Caisheng Wei, Yongfeng Shi, Yunhe Meng","year":"2020","venue":"Journal of Guidance, Control, and Dynamics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Chitnis and Lozano-Pérez, 2020] Rohan Chitnis and Tomás Lozano-Pérez. Learning compact models for planning with exogenous processes. In CoRL, 2020.","title":"Learning compact models for planning with exogenous processes","authors":"Lozano-Pérez ; Rohan Chitnis, Tomás Chitnis, Lozano-Pérez","year":"2020","venue":"CoRL","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP, 2014.","title":"Learning phrase representations using RNN encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Congeduti et al., 2021] Elena Congeduti, Alexander Mey, and Frans A Oliehoek. Loss bounds for approximate influence-based abstraction. In AAMAS, 2021. [Grzeszczuk et al., 1998] Radek Grzeszczuk, Demetri Ter- zopoulos, and Geoffrey Hinton. Neuroanimator: Fast neu- ral network emulation and control of physics-based mod- els. In Proceedings of the 25th annual conference on Com- puter graphics and interactive techniques, 1998.","title":"Neuroanimator: Fast neural network emulation and control of physics-based models","authors":"Elena Congeduti, Alexander Mey, Frans A Oliehoek","year":"1998","venue":"Proceedings of the 25th annual conference on Computer graphics and interactive techniques","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hansen and Feng, 2000] Eric A Hansen and Zhengzhu Feng. Dynamic programming for POMDPs using a fac- tored state representation. In AIPS, 2000.","title":"Dynamic programming for POMDPs using a factored state representation","authors":"Feng Hansen, Eric A Hansen, Zhengzhu Feng","year":"2000","venue":"AIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[He et al., 2020] Jinke He, Miguel Suau de Castro, and Frans Oliehoek. Influence-augmented online planning for com- plex environments. In NeurIPS, 2020.","title":"Influence-augmented online planning for complex environments","authors":"Jinke He, Miguel Suau De Castro, Frans Oliehoek","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hostetler et al., 2014] Jesse Hostetler, Alan Fern, and Tom Dietterich. State aggregation in Monte Carlo tree search. In AAAI, 2014.","title":"State aggregation in Monte Carlo tree search","authors":"Jesse Hostetler, Alan Fern, Tom Dietterich","year":"2014","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Jiang et al., 2014] Nan Jiang, Satinder Singh, and Richard Lewis. Improving UCT planning via approximate homo- morphisms. In AAMAS, 2014.","title":"Improving UCT planning via approximate homomorphisms","authors":"Nan Jiang, Satinder Singh, Richard Lewis","year":"2014","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kaelbling et al., 1998] Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and act- ing in partially observable stochastic domains. Artificial Intelligence, 101(1-2):99-134, may 1998.","title":"Planning and acting in partially observable stochastic domains","authors":"Leslie Pack Kaelbling, Michael L Littman, Anthony R Cassandra","year":"1998","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kennedy and O'Hagan, 2000] Marc C Kennedy and An- thony O'Hagan. Predicting the output from a complex computer code when fast approximations are available. Biometrika, 87(1):1-13, 2000.","title":"Predicting the output from a complex computer code when fast approximations are available","authors":"O'hagan ; Kennedy, C Marc, Anthony O' Kennedy, Hagan","year":"2000","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Oliehoek et al., 2021] Frans Oliehoek, Stefan Witwicki, and Leslie Kaelbling. A sufficient statistic for influence in structured multiagent environments. JAIR, 70, 2021.","title":"A sufficient statistic for influence in structured multiagent environments","authors":"Oliehoek","year":"2021","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Paninski, 2003] Liam Paninski. Estimation of entropy and mutual information. Neural computation, 2003.","title":"Estimation of entropy and mutual information","authors":"Liam Paninski","year":"2003","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Peherstorfer et al., 2018] Benjamin Peherstorfer, Karen Willcox, and Max Gunzburger. Survey of multifidelity methods in uncertainty propagation, inference, and optimization. Siam Review, 60(3):550-591, 2018.","title":"Survey of multifidelity methods in uncertainty propagation, inference, and optimization","authors":"Benjamin Peherstorfer, Karen Willcox, Max Gunzburger","year":"2018","venue":"Siam Review","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pehlivanoglu and Yagiz, 2012] Y Volkan Pehlivanoglu and Bedri Yagiz. Aerodynamic design prediction using surrogate-based modeling in genetic algorithm architec- ture. Aerospace Science and Technology, 2012.","title":"Aerodynamic design prediction using surrogate-based modeling in genetic algorithm architecture","authors":"Pehlivanoglu Volkan, Bedri Yagiz","year":"2012","venue":"Aerospace Science and Technology","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Peng and Wang, 2016] Haijun Peng and Wei Wang. Adap- tive surrogate model-based fast path planning for space- craft formation reconfiguration on libration point orbits. Aerospace Science and Technology, 54:151-163, 2016.","title":"Adaptive surrogate model-based fast path planning for spacecraft formation reconfiguration on libration point orbits","authors":"Wang ; ] Haijun Peng, Wei Peng, Wang","year":"2016","venue":"Aerospace Science and Technology","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ruder, 2016] Sebastian Ruder. An overview of gradi- ent descent optimization algorithms. arXiv preprint arXiv:1609.04747, sep 2016.","title":"An overview of gradient descent optimization algorithms","authors":"Sebastian Ruder","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Silver and Veness, 2010] David Silver and Joel Veness. Monte-Carlo planning in large POMDPs. In NeurIPS, 2010.","title":"Monte-Carlo planning in large POMDPs","authors":"Veness Silver, David Silver, Joel Veness","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Somani et al., 2013] Adhiraj Somani, Nan Ye, David Hsu, and Wee Sun Lee. Despot: Online pomdp planning with regularization. In NeurIPS, 2013.","title":"Despot: Online pomdp planning with regularization","authors":"Adhiraj Somani, Nan Ye, David Hsu, Wee Sun, Lee","year":"2013","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Spaan, 2012] Matthijs TJ Spaan. Partially observable markov decision processes. In Reinforcement Learning, pages 387-414. Springer, 2012.","title":"Partially observable markov decision processes","authors":"; Spaan, T J Matthijs, Spaan","year":"2012","venue":"Reinforcement Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}