{"type":"ScholarlyArticle","title":"Real-Time Navigation in Classical Platform Games via Skill Reuse","authors":[{"name":"Michael Dann","slug":"michael-dann","given":"Michael","family":"Dann","affiliation":"School of Science , RMIT University , Australia"},{"name":"Fabio Zambetta","slug":"fabio-zambetta","given":"Fabio","family":"Zambetta","affiliation":"School of Science , RMIT University , Australia"},{"name":"John Thangarajah","slug":"john-thangarajah","given":"John","family":"Thangarajah","affiliation":"School of Science , RMIT University , Australia"}],"abstract":"In platform videogames, players are frequently tasked with solving medium-term navigation problems in order to gather items or powerups. Artificial agents must generally obtain some form of direct experience before they can solve such tasks. Experience is gained either through training runs, or by exploiting knowledge of the game's physics to generate detailed simulations. Human players, on the other hand, seem to look ahead in high-level, abstract steps. Motivated by human play, we introduce an approach that leverages not only abstract \"skills\", but also knowledge of what those skills can and cannot achieve. We apply this approach to Infinite Mario, where despite facing randomly generated, maze-like levels, our agent is capable of deriving complex plans in real-time, without relying on perfect knowledge of the game's physics.","doi":"10.24963/ijcai.2017/219","url":"https://ijcai.org/proceedings/2017/219","pdf":"https://ijcai.org/proceedings/2017/0219.pdf","markdown":"https://ijcai.org/proceedings/2017/219.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/219","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":219,"pages":{"first":1582,"last":1588},"keywords":["Multidisciplinary Topics and Applications: Computer Games","Machine Learning: Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Bellemare et al., 2016] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying Count-Based Exploration and Intrinsic Motivation. In Advances in Neural Information Processing Systems, pages 1471-1479, 2016.","title":"Unifying Count-Based Exploration and Intrinsic Motivation","authors":"Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, Remi Munos","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dietterich, 2000] Thomas G Dietterich. Hierarchical Rein- forcement Learning with the MAXQ Value Function De- composition. Journal of Artificial Intelligence Research (JAIR), 13:227-303, 2000.","title":"Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition","authors":"G Thomas, Dietterich","year":"2000","venue":"Journal of Artificial Intelligence Research (JAIR)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Digney, 1998] Bruce L Digney. Learning Hierarchical Con- trol Structures for Multiple Tasks and Changing Environ- ments. In Proceedings of the 5th International Conference on Simulation of Adaptive Behavior, volume 5, pages 321- 330, 1998.","title":"Learning Hierarchical Control Structures for Multiple Tasks and Changing Environments","authors":"L Bruce, Digney","year":"1998","venue":"Proceedings of the 5th International Conference on Simulation of Adaptive Behavior","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Jacobsen et al., 2014] Emil Juul Jacobsen, Rasmus Greve, and Julian Togelius. Monte Mario: Platforming with MCTS. In Proceedings of the 2014 Conference on Genetic and Evolutionary Computation, pages 293-300. ACM, 2014.","title":"Monte Mario: Platforming with MCTS","authors":"Jacobsen","year":"2014","venue":"Proceedings of the 2014 Conference on Genetic and Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Konidaris and Barto, 2007] George Konidaris and Andrew Barto. Building Portable Options: Skill Transfer in Rein- forcement Learning. In Proceedings of the 20th Interna- tional Joint Conference on Artificial Intelligence (IJCAI), volume 7, pages 895-900, 2007.","title":"Building Portable Options: Skill Transfer in Reinforcement Learning","authors":"Barto Konidaris, George Konidaris, Andrew Barto","year":"2007","venue":"Proceedings of the 20th International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2007/144"},{"position":6,"raw":"[Konidaris and Barto, 2009] George Konidaris and Andrew Barto. Skill Discovery in Continuous Reinforcement Learning Domains Using Skill Chaining. In Advances in Neural Information Processing Systems (NIPS), pages 1015-1023, 2009.","title":"Skill Discovery in Continuous Reinforcement Learning Domains Using Skill Chaining","authors":"Barto Konidaris, George Konidaris, Andrew Barto","year":"2009","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Lin, 1993] Long-Ji Lin. Reinforcement Learning for Robots Using Neural Networks. PhD thesis, Carnegie Mellon Uni- versity, 1993. Technical Report CMU-CS-93-103.","title":"Reinforcement Learning for Robots Using Neural Networks","authors":"Long-Ji Lin","year":"1993","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Mannor et al., 2004] Shie Mannor, Ishai Menache, Amit Hoze, and Uri Klein. Dynamic Abstraction in Reinforce- ment Learning via Clustering. In Proceedings of the 21st International Conference on Machine Learning, page 71. ACM, 2004.","title":"Dynamic Abstraction in Reinforcement Learning via Clustering","authors":"Shie Mannor, Ishai Menache, Amit Hoze, Uri Klein","year":"2004","venue":"Proceedings of the 21st International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Menache et al., 2002] Ishai Menache, Shie Mannor, and Nahum Shimkin. Q-Cut -Dynamic Discovery of Sub- Goals in Reinforcement Learning. In Machine Learning: ECML 2002, pages 295-306. Springer, 2002.","title":"Q-Cut -Dynamic Discovery of Sub-Goals in Reinforcement Learning","authors":"Ishai Menache, Shie Mannor, Nahum Shimkin","year":"2002","venue":"Machine Learning: ECML 2002","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-Level Con- trol Through Deep Reinforcement Learning. Nature, 518(7540):529-533, 2015.","title":"Human-Level Control Through Deep Reinforcement Learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Parr and Russell, 1998] Ronald Parr and Stuart Russell. Re- inforcement Learning with Hierarchies of Machines. Ad- vances in Neural Information Processing Systems (NIPS), pages 1043-1049, 1998.","title":"Reinforcement Learning with Hierarchies of Machines","authors":"Russell ; Parr, Ronald Parr, Stuart Russell","year":"1998","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Pickett and Barto, 2002] Marc Pickett and Andrew G Barto. Policyblocks: An Algorithm for Creating Useful Macro- Actions in Reinforcement Learning. In Proceedings of the 19th International Conference on Machine Learning, vol- ume 2, pages 506-513, 2002.","title":"Policyblocks: An Algorithm for Creating Useful Macro-Actions in Reinforcement Learning","authors":"Barto Pickett, Marc Pickett, Andrew G Barto","year":"2002","venue":"Proceedings of the 19th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[S ¸ims ¸ek and Barto, 2004] Özgür S ¸ims ¸ek and Andrew G Barto. Using Relative Novelty to Identify Useful Temporal Abstractions in Reinforcement Learning. In Proceedings of the 21st International Conference on Machine Learn- ing, page 95. ACM, 2004.","title":"Using Relative Novelty to Identify Useful Temporal Abstractions in Reinforcement Learning","authors":"; ] Barto, S Özgür, Andrew G Barto","year":"2004","venue":"Proceedings of the 21st International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"Özgür S ¸ims ¸ek, Alicia P Wolfe, and An- drew G Barto. Identifying Useful Subgoals in Reinforce- ment Learning by Local Graph Partitioning. In Proceed- ings of the 22nd International Conference on Machine Learning, pages 816-823. ACM, 2005.","title":"Identifying Useful Subgoals in Reinforcement Learning by Local Graph Partitioning","authors":"Alicia P Özgür S ¸ims ¸ek, Andrew G Wolfe, Barto","year":"2005","venue":"Proceedings of the 22nd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sutton and Barto, 1998] Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction, vol- ume 1. Cambridge Univ Press, 1998.","title":"Reinforcement Learning: An Introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sutton et al., 1999] Richard S Sutton, Doina Precup, and Satinder Singh. Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning. Artificial Intelligence, 112(1):181-211, 1999.","title":"Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning","authors":"Doina Richard S Sutton, Satinder Precup, Singh","year":"1999","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Thrun et al., 1995] Sebastian Thrun, Anton Schwartz, et al. Finding Structure in Reinforcement Learning. Advances in Neural Information Processing Systems (NIPS), pages 385-392, 1995.","title":"Finding Structure in Reinforcement Learning","authors":"Sebastian Thrun, Anton Schwartz","year":"1995","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Togelius et al., 2009] Julian Togelius, Sergey Karakovskiy, Jan Koutník, and Jürgen Schmidhuber. Super Mario evo- lution. In Computational Intelligence and Games, 2009. CIG 2009. IEEE Symposium on, pages 156-161. IEEE, 2009.","title":"Super Mario evolution","authors":"Julian Togelius, Sergey Karakovskiy, Jan Koutník, Jürgen Schmidhuber","year":"2009","venue":"Computational Intelligence and Games","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Togelius et al., 2010] Julian Togelius, Sergey Karakovskiy, and Robin Baumgarten. The 2009 Mario AI Competition. In Evolutionary Computation (CEC), 2010 IEEE Congress on, pages 1-8. IEEE, 2010.","title":"The 2009 Mario AI Competition","authors":"Julian Togelius, Sergey Karakovskiy, Robin Baumgarten","year":"2010","venue":"Evolutionary Computation (CEC)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Vezhnevets et al., 2016] Alexander Vezhnevets, Volodymyr Mnih, Simon Osindero, Alex Graves, Oriol Vinyals, John Agapiou, et al. Strategic Attentive Writer for Learning Macro-Actions. In Advances in Neural Information Pro- cessing Systems (NIPS 2016), pages 3486-3494, 2016.","title":"Strategic Attentive Writer for Learning Macro-Actions","authors":"Alexander Vezhnevets, Volodymyr Mnih, Simon Osindero, Alex Graves, Oriol Vinyals, John Agapiou","year":"2016","venue":"Advances in Neural Information Processing Systems (NIPS 2016)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vien and Toussaint, 2015] Ngo Anh Vien and Marc Tous- saint. Hierarchical Monte-Carlo Planning. In Proceed- ings of the 29th AAAI Conference on Artificial Intelligence, pages 3613-3619, 2015.","title":"Hierarchical Monte-Carlo Planning","authors":"Anh Ngo, Marc Vien, Toussaint","year":"2015","venue":"Proceedings of the 29th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}