{"type":"ScholarlyArticle","title":"Boundary Extension Features for Width-Based Planning with Simulators on Continuous-State Domains","authors":[{"name":"Florent Teichteil-Königsbuch","slug":"florent-teichteil-konigsbuch","given":"Florent","family":"Teichteil-Königsbuch","affiliation":"Airbus , Artificial Intelligence Research"},{"name":"Miquel Ramirez","slug":"miquel-ramirez","given":"Miquel","family":"Ramírez","affiliation":"Dept. Electrical and Electronic Engineering , The University of Melbourne"},{"name":"Nir Lipovetzky","slug":"nir-lipovetzky","given":"Nir","family":"Lipovetzky","affiliation":"School of Computing and Information Sciences , The University of Melbourne"}],"abstract":"Width-based planning algorithms have been demonstrated to be competitive with state-of-the-art heuristic search and SAT-based approaches, without requiring access to a model of action effects and preconditions, just access to a black-box simulator. Width-based planners search is guided by a measure of the novelty of states, that requires observations on simulator states to be given as a set of features. This paper proposes agnostic feature mapping mechanisms that define the features online, as exploration progresses and the domain of continuous state variables is revealed. We demonstrate the effectiveness of these features on the OpenAI gym \"classical control\" suite of benchmarks. We compare our online planners with state-of-the-art deep reinforcement learning algorithms, and show that width-based planners using our features can find policies of the same quality with significantly less computational resources.","doi":"10.24963/ijcai.2020/578","url":"https://ijcai.org/proceedings/2020/578","pdf":"https://ijcai.org/proceedings/2020/0578.pdf","markdown":"https://ijcai.org/proceedings/2020/578.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/578","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Planning and Scheduling","track":"Main track","number":578,"pages":{"first":4183,"last":4189},"keywords":["Planning and Scheduling: Planning Algorithms","Machine Learning: Reinforcement Learning","Planning and Scheduling: Real-time Planning","Planning and Scheduling: Search in Planning and Scheduling"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25900/file_25900.m3u8","length_sec":891,"provider":"hls"}],"references":[{"position":1,"raw":"[AIRBUS -AI Research, 2020] AIRBUS - AI Re- search. scikit-decide library. https://github.com/ airbus/scikit-decide, 2020.","title":"AIRBUS -AI Research. scikit-decide library","authors":null,"year":"2020","venue":null,"doi":null,"url":"https://github.com/airbus/scikit-decide,2020","internal_url":null},{"position":2,"raw":"[Bandres et al., 2018] Wilmer Bandres, Blai Bonet, and Hector Geffner. Planning with pixels in (almost) real time. In AAAI, 2018.","title":"Planning with pixels in (almost) real time","authors":"Wilmer Bandres, Blai Bonet, Hector Geffner","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bertsekas, 2017] Dmitri P. Bertsekas. Dynamic Program- ming and Optimal Control. Athena Scientific, 4th edition, 2017.","title":"Dynamic Programming and Optimal Control","authors":"Dmitri P Bertsekas","year":"2017","venue":"Athena Scientific","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bonet and Geffner, 2013] Blai Bonet and Hector Geffner. A Concise Introduction to the Models and Methods for Auto- mated Planning. Morgan & Claypool, 2013.","title":"A Concise Introduction to the Models and Methods for Automated Planning","authors":"Geffner Bonet, Blai Bonet, Hector Geffner","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Boyan and Moore, 1995] Justin A. Boyan and Andrew W. Moore. Generalization in reinforcement learning: Safely approximating the value function. In NIPS, 1995.","title":"Generalization in reinforcement learning: Safely approximating the value function","authors":"Justin A Boyan, Andrew W Moore","year":"1995","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Frances et al., 2017] Guillem Frances, Miquel Ramírez Jávega, Nir Lipovetzky, and Hector Geffner. Purely declarative action descriptions are overrated: Classical planning with simulators. In IJCAI, 2017.","title":"Purely declarative action descriptions are overrated: Classical planning with simulators","authors":"Guillem Frances, Miquel Ramírez Jávega, Nir Lipovetzky, Hector Geffner","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/600"},{"position":7,"raw":"[Geffner and Geffner, 2015] Tomás Geffner and Hector Geffner. Width-based planning for general video-game playing. In AIIDE, pages 23-29, 2015.","title":"Width-based planning for general video-game playing","authors":"Geffner Geffner, Tomás Geffner, Hector Geffner","year":"2015","venue":"AIIDE","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gerevini et al., 2019] Alfonso E. Gerevini, Nir Lipovetzky, Francesco Percassi, Alessandro Saetti, and Ivan Serina. Best-first search for multi agent privacy-preserving plan- ning. In ICAPS, 2019.","title":"Best-first search for multi agent privacy-preserving planning","authors":"Alfonso E Gerevini, Nir Lipovetzky, Francesco Percassi, Alessandro Saetti, Ivan Serina","year":"2019","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Helmert et al., 2008] Malte Helmert, Minh Do, and Ioannis Refanidis. Ipc 2008 deterministic competition. In 6th Int'l Planning Competition Booklet (ICAPS-08), 2008.","title":"Ipc 2008 deterministic competition","authors":"Malte Helmert, Minh Do, Ioannis Refanidis","year":"2008","venue":"6th Int'l Planning Competition Booklet (ICAPS-08)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Helmert, 2002] Malte Helmert. Decidability and undecid- ability results for planning with numerical state variables. In AIPS, 2002.","title":"Decidability and undecidability results for planning with numerical state variables","authors":"Malte Helmert, Helmert","year":"2002","venue":"AIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jinnai and Fukunaga, 2017] Yuu Jinnai and Alex Fukunaga. Learning to prune dominated action sequences in online black-box planning. In AAAI, pages 839-845, 2017.","title":"Learning to prune dominated action sequences in online black-box planning","authors":"Fukunaga Jinnai, Yuu Jinnai, Alex Fukunaga","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Keyder and Geffner, 2009] Emil Keyder and Hector Geffner. Soft goals can be compiled away. Journal of Artificial Intelligence Research, 36:189-211, 2009.","title":"Emil Keyder and Hector Geffner. Soft goals can be compiled away","authors":"Geffner Keyder","year":"2009","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lin and Wright, 2010] Stephen Lin and Robert Wright. Evolutionary tile coding: An automated state abstraction algorithm for reinforcement learning. In AAAI, 2010.","title":"Evolutionary tile coding: An automated state abstraction algorithm for reinforcement learning","authors":"Stephen Lin, Robert Wright","year":"2010","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lipovetzky and Geffner, 2012] Nir Lipovetzky and Hector Geffner. Width and serialization of classical planning problems. In ECAI, pages 540-545, 2012.","title":"Width and serialization of classical planning problems","authors":"Geffner Lipovetzky, Nir Lipovetzky, Hector Geffner","year":"2012","venue":"ECAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lipovetzky and Geffner, 2017a] Nir Lipovetzky and Hector Geffner. Best-first width search: Exploration and exploita- tion in classical planning. In AAAI, 2017.","title":"Best-first width search: Exploration and exploitation in classical planning","authors":"Geffner Lipovetzky, Nir Lipovetzky, Hector Geffner","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lipovetzky and Geffner, 2017b] Nir Lipovetzky and Hector Geffner. A polynomial planning algorithm that beats lama and ff. In ICAPS, 2017.","title":"A polynomial planning algorithm that beats lama and ff","authors":"Geffner Lipovetzky, Nir Lipovetzky, Hector Geffner","year":"2017","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lipovetzky et al., 2015] Nir Lipovetzky, Miquel Ramirez, and Hector Geffner. Classical planning with simulators: results on the atari video games. In IJCAI, 2015.","title":"Classical planning with simulators: results on the atari video games","authors":"Nir Lipovetzky, Miquel Ramirez, Hector Geffner","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/230"},{"position":18,"raw":"[Mitrovic et al., 2010] Djordje Mitrovic, Stefan Klanke, and Sethu Vijayakumar. Adaptive optimal feedback control with learned internal dynamics models. In Olivier Sigaud and Jan Peters, editors, From Motor Learning to Interac- tion Learning in Robots. Springer, 2010.","title":"Adaptive optimal feedback control with learned internal dynamics models","authors":"Djordje Mitrovic, Stefan Klanke, Sethu Vijayakumar","year":"2010","venue":"From Motor Learning to Interaction Learning in Robots","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Moore, 1990] A. Moore. Efficient Memory-Based Learning for Robot Control. PhD thesis, University of Cambridge, 1990.","title":"Efficient Memory-Based Learning for Robot Control","authors":"A Moore","year":"1990","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nocedal and Wright, 2006] Jorge Nocedal and Stephen J. Wright. Numerical Optimization. Springer, 2nd edition, 2006.","title":"Numerical Optimization","authors":"Jorge Nocedal, Stephen J Wright","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[O'Toole et al., 2019] Stefan O'Toole, Miquel Ramirez, Nir Lipovetzky, and Adrian Pearce. Width-based lookaheads augmented with base policies for stochastic shortest paths. In ICAPS Workshop on Heuristics and Domain Indepen- dent Planning, 2019.","title":"Width-based lookaheads augmented with base policies for stochastic shortest paths","authors":"O'toole","year":"2019","venue":"ICAPS Workshop on Heuristics and Domain Independent Planning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Perez et al., 2012] Alejandro Perez, Robert Platt Jr., George Konidaris, Leslie Kaebling, and Tomas Lozano-Perez. Lqr-rrt*: Optimal sampling-based motion planning with automatically derived extension heuristics. In ICRA, 2012.","title":"Lqr-rrt*: Optimal sampling-based motion planning with automatically derived extension heuristics","authors":"Alejandro Perez, Robert Platt, George Konidaris, Leslie Kaebling, Tomas Lozano-Perez","year":"2012","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ramirez et al., 2018] Miquel Ramirez, Michael Pa- pasimeon, Nir Lipovetzky, Lyndon Benke, Tim Miller, Adrian R Pearce, Enrico Scala, and Mohammad Zamani. Integrated hybrid planning and programmed control for real time uav maneuvering. In AAMAS, 2018.","title":"Integrated hybrid planning and programmed control for real time uav maneuvering","authors":"Miquel Ramirez, Michael Papasimeon, Nir Lipovetzky, Lyndon Benke, Tim Miller, Adrian R Pearce, Enrico Scala, Mohammad Zamani","year":"2018","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Shleyfman et al., 2016] Alexander Shleyfman, Alexander Tuisov, and Carmel Domshlak. Blind search for atari- like online planning revisited. In IJCAI, pages 3251-3257, 2016.","title":"Blind search for atarilike online planning revisited","authors":"Alexander Shleyfman, Alexander Tuisov, Carmel Domshlak","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Stepanov and McJones, 2009] Alexander Stepanov and Paul McJones. Elements of Programming. Addison- Wesley, 2009.","title":"Elements of Programming","authors":"Alexander Stepanov, Paul Mcjones","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Stevens et al., 2015] Brian L. Stevens, Frank L. Lewis, and Eric N. Johnson. Aircraft Control and Simulation: Dy- namics, Controls Design and Autonomous Systems. John Wiley & Sons, 2015.","title":"Aircraft Control and Simulation: Dynamics, Controls Design and Autonomous Systems","authors":"Brian L Stevens, Frank L Lewis, Eric N Johnson","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2nd edition, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sutton, 1996] Richard S. Sutton. Generalization in rein- forcement learning: Successful examples using sparse coarse coding. In NIPS, 1996.","title":"Generalization in reinforcement learning: Successful examples using sparse coarse coding","authors":"Richard S Sutton","year":"1996","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Tedrake and others, 2019] Russ Tedrake et al. Drake: Model-based design and verification for robotics, 2019.","title":"Drake: Model-based design and verification for robotics","authors":"Russ Tedrake, Tedrake","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Whiteson et al., 2007] Shimon Whiteson, Matthew E. Tay- lor, and Peter Stone. Adaptive tile coding for value func- tion approximation. Technical report, Computer Science Department, University of Texas at Austin, 2007.","title":"Adaptive tile coding for value function approximation","authors":"Shimon Whiteson, Matthew E Taylor, Peter Stone","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Willems, 2007] Jan C. Willems. The behavioral approach to open and interconnected systems. IEEE Control Systems Magazine, 27(6):46-99, 2007.","title":"The behavioral approach to open and interconnected systems","authors":"Jan C Willems","year":"2007","venue":"IEEE Control Systems Magazine","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}