{"type":"ScholarlyArticle","title":"Open Loop Execution of Tree-Search Algorithms","authors":[{"name":"Erwan Lecarpentier","slug":"erwan-lecarpentier","given":"Erwan","family":"Lecarpentier","affiliation":"ONERA -The French Aerospace Lab , Toulouse , France; ISAE -SUPAERO , University of Toulouse , France"},{"name":"Guillaume Infantes","slug":"guillaume-infantes","given":"Guillaume","family":"Infantes","affiliation":"ONERA -The French Aerospace Lab , Toulouse , France"},{"name":"Charles Lesire","slug":"charles-lesire","given":"Charles","family":"Lesire","affiliation":"ONERA -The French Aerospace Lab , Toulouse , France"},{"name":"Emmanuel Rachelson","slug":"emmanuel-rachelson","given":"Emmanuel","family":"Rachelson","affiliation":"ISAE -SUPAERO , University of Toulouse , France"}],"abstract":"In the context of tree-search stochastic planning algorithms where a generative model is available, we consider on-line planning algorithms building trees in order to recommend an action. We investigate the question of avoiding re-planning in subsequent decision steps by directly using sub-trees as action recommender. Firstly, we propose a method for open loop control via a new algorithm taking the decision of re-planning or not at each time step based on an analysis of the statistics of the sub-tree. Secondly, we show that the probability of selecting a suboptimal action at any depth of the tree can be upper bounded and converges towards zero. Moreover, this upper bound decays in a logarithmic way between subsequent depths. This leads to a distinction between node-wise optimality and state-wise optimality. Finally, we empirically demonstrate that our method achieves a compromise between loss of performance and computational gain.","doi":"10.24963/ijcai.2018/327","url":"https://ijcai.org/proceedings/2018/327","pdf":"https://ijcai.org/proceedings/2018/0327.pdf","markdown":"https://ijcai.org/proceedings/2018/327.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/327","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":327,"pages":{"first":2362,"last":2368},"keywords":["Machine Learning: Reinforcement Learning","Uncertainty in AI: Sequential Decision Making","Planning and Scheduling: Real-time Planning","Uncertainty in AI: Markov Decision Processes","Planning and Scheduling: Planning Algorithms"],"videos":[],"references":[{"position":1,"raw":"[Auer et al., 2002] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2-3):235-256, 2002.","title":"Finite-time analysis of the multiarmed bandit problem","authors":"Auer","year":"2002","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bellemare et al., 2017] Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on re- inforcement learning. arXiv preprint arXiv:1707.06887, 2017.","title":"A distributional perspective on reinforcement learning","authors":"G Marc, Will Bellemare, Rémi Dabney, Munos","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bubeck and Munos, 2010] Sébastien Bubeck and Rémi Munos. Open loop optimistic planning. In COLT, 2010.","title":"Open loop optimistic planning","authors":"Munos Bubeck, Sébastien Bubeck, Rémi Munos","year":"2010","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[De Maesschalck et al., 2000] Roy De Maesschalck, Del- phine Jouan-Rimbaud, and Désiré L. Massart. The Maha- lanobis distance. Chemometrics and intelligent laboratory systems, 50(1):1-18, 2000.","title":"The Mahalanobis distance","authors":"Roy De Maesschalck, Delphine Jouan-Rimbaud, Désiré L Massart","year":"2000","venue":"Chemometrics and intelligent laboratory systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Enzenberger et al., 2010] Markus Enzenberger, Martin Muller, Broderick Arneson, and Richard Segal. Fuego -an open-source framework for board games and Go engine based on Monte Carlo tree search. IEEE Trans- actions on Computational Intelligence and AI in Games, 2(4):259-270, 2010.","title":"Fuego -an open-source framework for board games and Go engine based on Monte Carlo tree search","authors":"Markus Enzenberger, Martin Muller, Broderick Arneson, Richard Segal","year":"2010","venue":"IEEE Transactions on Computational Intelligence and AI in Games","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Heusner, 2011] Manuel Heusner. UCT for pac-man. Bach- elor thesis, Univ. of Basel, 2011.","title":"UCT for pac-man","authors":"Manuel Heusner","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kaelbling et al., 1998] Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and act- ing in partially observable stochastic domains. Artificial intelligence, 101(1):99-134, 1998.","title":"Planning and acting in partially observable stochastic domains","authors":"Leslie Pack Kaelbling, Michael L Littman, Anthony R Cassandra","year":"1998","venue":"Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Keller and Helmert, 2013] Thomas Keller and Malte Helmert. Trial-based heuristic tree search for finite horizon MDPs. In ICAPS, 2013. [Kocsis and Szepesvári, 2006] Levente Kocsis and Csaba Szepesvári. Bandit based Monte Carlo planning. In ECML, volume 6, pages 282-293. Springer, 2006.","title":"Trial-based heuristic tree search for finite horizon MDPs","authors":"Helmert Keller, Thomas Keller, Malte Helmert, Levente Kocsis, Csaba Szepesvári","year":"2006","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Perez et al., 2012a] Diego Perez, Philipp Rohlfshagen, and Simon M. Lucas. Monte Carlo tree search: Long-term versus short-term planning. In Computational Intelligence and Games (CIG), 2012 IEEE Conference on, pages 219- 226. IEEE, 2012.","title":"Monte Carlo tree search: Long-term versus short-term planning","authors":"Perez","year":"2012","venue":"Computational Intelligence and Games (CIG), 2012 IEEE Conference on","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Perez et al., 2012b] Diego Perez, Philipp Rohlfshagen, and Simon M. Lucas. The physical travelling salesman prob- lem: WCCI 2012 competition. In Evolutionary Compu- tation (CEC), 2012 IEEE Congress on, pages 1-8. IEEE, 2012.","title":"The physical travelling salesman problem: WCCI 2012 competition","authors":"Perez","year":"2012","venue":"Evolutionary Computation (CEC)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Puterman, 2014] Martin L. Puterman. Markov decision pro- cesses: discrete stochastic dynamic programming. John Wiley & Sons, 2014.","title":"Markov decision processes: discrete stochastic dynamic programming","authors":"Martin L Puterman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Rachelson and Lagoudakis, 2010] Emmanuel Rachelson and Michail G. Lagoudakis. On the locality of action domination in sequential decision making. In ISAIM, 2010.","title":"On the locality of action domination in sequential decision making","authors":"Emmanuel Rachelson, Michail G Lagoudakis","year":"2010","venue":"ISAIM","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Silver and Veness, 2010] David Silver and Joel Veness. Monte Carlo planning in large POMDPs. In Advances in neural information processing systems, pages 2164-2172, 2010.","title":"Monte Carlo planning in large POMDPs","authors":"Veness Silver, David Silver, Joel Veness","year":"2010","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Silver et al., 2008] David Silver, Richard S. Sutton, and Martin Müller. Sample-based learning and search with permanent and transient memories. In Proceedings of the 25th international conference on Machine Learning, pages 968-975. ACM, 2008.","title":"Sample-based learning and search with permanent and transient memories","authors":"David Silver, Richard S Sutton, Martin Müller","year":"2008","venue":"Proceedings of the 25th international conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484-489, 2016.","title":"Mastering the game of Go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Julian Driessche, Ioannis Schrittwieser, Veda Antonoglou, Marc Panneershelvam, Lanctot","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sutton and Barto, 1998] Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. MIT press Cambridge, 1998.","title":"Reinforcement learning: An introduction","authors":"Richard S Sutton, Andrew G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sutton, 1991] Richard S. Sutton. Dyna, an integrated archi- tecture for learning, planning, and reacting. ACM SIGART Bulletin, 2(4):160-163, 1991.","title":"Dyna, an integrated architecture for learning, planning, and reacting","authors":"Richard S Sutton","year":"1991","venue":"ACM SIGART Bulletin","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Weinstein and Littman, 2012] Ari Weinstein and Michael L. Littman. Bandit-based planning and learning in continuous-action markov decision processes. In ICAPS, 2012.","title":"Bandit-based planning and learning in continuous-action markov decision processes","authors":"Littman Weinstein, Ari Weinstein, Michael L Littman","year":"2012","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wilson et al., 2014] Mark A. Wilson, James McMahon, and David W. Aha. Bounded expectations for discrepancy de- tection in goal-driven autonomy. In AI and Robotics: Pa- pers from the AAAI Workshop, 2014.","title":"Bounded expectations for discrepancy detection in goal-driven autonomy","authors":"Mark A Wilson, James Mcmahon, David W Aha","year":"2014","venue":"AI and Robotics: Papers from the AAAI Workshop","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wilson et al., 2016] Mark A. Wilson, James McMahon, Ar- tur Wolek, David W. Aha, and Brian H. Houston. Toward goal reasoning for autonomous underwater vehicles: Re- sponding to unexpected agents. In Goal Reasoning: Pa- pers from the IJCAI Workshop, 2016.","title":"Toward goal reasoning for autonomous underwater vehicles: Responding to unexpected agents","authors":"Mark A Wilson, James Mcmahon, Artur Wolek, David W Aha, Brian H Houston","year":"2016","venue":"Goal Reasoning: Papers from the IJCAI Workshop","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}