{"type":"ScholarlyArticle","title":"Efficient Reinforcement Learning with Hierarchies of Machines by Leveraging Internal Transitions","authors":[{"name":"Aijun Bai","slug":"aijun-bai","given":"Aijun","family":"Bai","affiliation":"UC Berkeley"},{"name":"Stuart Russell","slug":"stuart-russell","given":"Stuart","family":"Russell","affiliation":"UC Berkeley"}],"abstract":"In the context of hierarchical reinforcement learning, the idea of hierarchies of abstract machines (HAMs) is to write a partial policy as a set of hierarchical finite state machines with unspecified choice states, and use reinforcement learning to learn an optimal completion of this partial policy. Given a HAM with potentially deep hierarchical structure, there often exist many internal transitions where a machine calls another machine with the environment state unchanged. In this paper, we propose a new hierarchical reinforcement learning algorithm that discovers such internal transitions automatically, and shortcircuits them recursively in computation of Q values. The resulting HAMQ-INT algorithm outperforms the state of the art significantly on the benchmark Taxi domain and a much more complex RoboCup Keepaway domain.","doi":"10.24963/ijcai.2017/196","url":"https://ijcai.org/proceedings/2017/196","pdf":"https://ijcai.org/proceedings/2017/0196.pdf","markdown":"https://ijcai.org/proceedings/2017/196.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/196","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":196,"pages":{"first":1418,"last":1424},"keywords":["Robotics and Vision: Robotics","Planning and Scheduling: Markov Decisions Processes","Robotics and Vision: Multi-Robot Systems","Machine Learning: Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Andre and Russell, 2001] David Andre and Stuart J Russell. Programmable reinforcement learning agents. Advances in neural information processing systems, pages 1019-1025, 2001.","title":"Programmable reinforcement learning agents","authors":"David Andre, Stuart J Russell","year":"2001","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Andre and Russell, 2002] David Andre and Stuart J. Rus- sell. State abstraction for programmable reinforcement learning agents. In Proceedings of the 8th National Con- ference on Artificial Intelligence and 14th Conference on Innovative Applications of Artificial Intelligence, pages 119-125, 2002.","title":"State abstraction for programmable reinforcement learning agents","authors":"David Andre, Stuart J Russell","year":"2002","venue":"Proceedings of the 8th National Conference on Artificial Intelligence and 14th Conference on Innovative Applications of Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Barto and Mahadevan, 2003] A.G. Barto and S. Mahade- van. Recent advances in hierarchical reinforcement learn- ing. Discrete Event Dynamic Systems, 13:341-379, 2003.","title":"Recent advances in hierarchical reinforcement learning","authors":"A G Barto, S Mahadevan","year":"2003","venue":"Discrete Event Dynamic Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bellman, 1957] Richard Bellman. Dynamic Programming. Princeton University Press, Princeton, NJ, USA, 1957.","title":"Dynamic Programming","authors":"Richard Bellman","year":"1957","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dietterich, 1999] Thomas G Dietterich. Hierarchical rein- forcement learning with the MAXQ value function de- composition. Journal of Machine Learning Research, 13(1):63, May 1999.","title":"Hierarchical reinforcement learning with the MAXQ value function decomposition","authors":"G Thomas, Dietterich","year":"1999","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kober and Peters, 2012] Jens Kober and Jan Peters. Rein- forcement learning in robotics: A survey. In Reinforcement Learning, pages 579-610. Springer, 2012.","title":"Reinforcement learning in robotics: A survey","authors":"Peters Kober, Jens Kober, Jan Peters","year":"2012","venue":"Reinforcement Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Marthi et al., 2005] Bhaskara Marthi, Stuart J Russell, David Latham, and Carlos Guestrin. Concurrent hierar- chical reinforcement learning. In IJCAI, pages 779-785, 2005.","title":"Concurrent hierarchical reinforcement learning","authors":"Bhaskara Marthi, J Stuart, David Russell, Carlos Latham, Guestrin","year":"2005","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2005/205"},{"position":8,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Moore, 1956] Edward F Moore. Gedanken-experiments on sequential machines. Automata studies, 34:129-153, 1956.","title":"Gedanken-experiments on sequential machines","authors":"Moore Edward","year":"1956","venue":"Automata studies","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Parr and Russell, 1998] Ronald Parr and Stuart Russell. Re- inforcement learning with hierarchies of machines. In Advances in Neural Information Processing Systems, vol- ume 10, 1998.","title":"Reinforcement learning with hierarchies of machines","authors":"Russell ; Parr, Ronald Parr, Stuart Russell","year":"1998","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484-489, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Marc Driessche, Lanctot","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Stone et al., 2005] P. Stone, R.S. Sutton, and G. Kuhlmann. Reinforcement learning for robocup soccer keepaway. Adaptive Behavior, 13(3):165-188, 2005.","title":"Reinforcement learning for robocup soccer keepaway","authors":"P Stone, R S Sutton, G Kuhlmann","year":"2005","venue":"Adaptive Behavior","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Sutton and Barto, 1998] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Sutton et al., 1999] R.S. Sutton, D. Precup, and S. Singh. Between MDPs and semi-MDPs: A framework for tem- poral abstraction in reinforcement learning. Artificial In- telligence, 112(1):181-211, 1999.","title":"Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning","authors":"R S Sutton, D Precup, S Singh","year":"1999","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}