{"type":"ScholarlyArticle","title":"Hybrid Actor-Critic Reinforcement Learning in Parameterized Action Space","authors":[{"name":"Zhou Fan","slug":"zhou-fan","given":"Zhou","family":"Fan","affiliation":"University"},{"name":"Rui Su","slug":"rui-su","given":"Rui","family":"Su","affiliation":"University"},{"name":"Weinan Zhang","slug":"weinan-zhang","given":"Weinan","family":"Zhang","affiliation":"University"},{"name":"Yong Yu","slug":"yong-yu","given":"Yong","family":"Yu","affiliation":""}],"abstract":"In this paper we propose a hybrid architecture of actor-critic algorithms for reinforcement learning in parameterized action space, which consists of multiple parallel sub-actor networks to decompose the structured action space into simpler action spaces along with a critic network to guide the training of all sub-actor networks. While this paper is mainly focused on parameterized action space, the proposed architecture, which we call hybrid actor-critic, can be extended for more general action spaces which has a hierarchical structure. We present an instance of the hybrid actor-critic architecture based on proximal policy optimization (PPO), which we refer to as hybrid proximal policy optimization (H-PPO). Our experiments test H-PPO on a collection of tasks with parameterized action space, where H-PPO demonstrates superior performance over previous methods of parameterized action reinforcement learning.","doi":"10.24963/ijcai.2019/316","url":"https://ijcai.org/proceedings/2019/316","pdf":"https://ijcai.org/proceedings/2019/0316.pdf","markdown":"https://ijcai.org/proceedings/2019/316.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/316","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":316,"pages":{"first":2279,"last":2285},"keywords":["Machine Learning: Reinforcement Learning","Machine Learning Applications: Applications of Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Hasselt et al., 2016] Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'16. AAAI Press, 2016.","title":"Deep reinforcement learning with double qlearning","authors":"Hasselt","year":"2016","venue":"Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'16","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Hausknecht and Stone, 2016] Matthew Hausknecht and Pe- ter Stone. Deep reinforcement learning in parameterized action space. In Proceedings of the International Confer- ence on Learning Representations (ICLR), May 2016.","title":"Matthew Hausknecht and Peter Stone. Deep reinforcement learning in parameterized action space","authors":"Stone Hausknecht","year":"2016","venue":"Proceedings of the International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Hausknecht et al., 2016] Matthew Hausknecht, Prannoy Mupparaju, Sandeep Subramanian, Shivaram Kalyanakr- ishnan, and Peter Stone. Half field offense: An environ- ment for multiagent learning and ad hoc teamwork. In AAMAS Adaptive Learning Agents (ALA) Workshop, May 2016.","title":"Half field offense: An environment for multiagent learning and ad hoc teamwork","authors":"Matthew Hausknecht, Prannoy Mupparaju, Sandeep Subramanian, Shivaram Kalyanakrishnan, Peter Stone","year":"2016","venue":"AAMAS Adaptive Learning Agents (ALA) Workshop","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Heess et al., 2015] Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in NIPS 28. Curran Associates, Inc., 2015.","title":"Learning continuous control policies by stochastic value gradients","authors":"Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, Yuval Tassa","year":"2015","venue":"Advances in NIPS 28","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Kober et al., 2013] Jens Kober, J. Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11):1238- 1274, 2013.","title":"Reinforcement learning in robotics: A survey","authors":"Jens Kober, J Andrew Bagnell, Jan Peters","year":"2013","venue":"The International Journal of Robotics Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Konda and Tsitsiklis, 2000] Vijay R. Konda and John N. Tsitsiklis. Actor-critic algorithms. In S. A. Solla, T. K. Leen, and K. Müller, editors, Advances in Neural Informa- tion Processing Systems 12, pages 1008-1014. MIT Press, 2000.","title":"Actor-critic algorithms","authors":"Vijay R Konda, John N Tsitsiklis","year":"2000","venue":"Advances in Neural Information Processing Systems 12","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Lillicrap et al., 2016] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yu- val Tassa, David Silver, and Daan Wierstra. Continu- ous control with deep reinforcement learning. CoRR, abs/1509.02971, 2016.","title":"Continuous control with deep reinforcement learning","authors":"Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, Daan Wierstra","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Masson et al., 2016] Warwick Masson, Pravesh Ranchod, and George Konidaris. Reinforcement learning with pa- rameterized actions. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'16, pages 1934-1940. AAAI Press, 2016.","title":"Reinforcement learning with parameterized actions","authors":"Masson","year":"2016","venue":"Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'16","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. In NIPS Deep Learning Work- shop. 2013.","title":"Playing atari with deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller","year":"2013","venue":"NIPS Deep Learning Workshop","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asyn- chronous methods for deep reinforcement learning. In Proceedings of The 33rd International Conference on Ma- chine Learning, Proceedings of Machine Learning Re- search, New York, New York, USA, 2016. PMLR.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":"Proceedings of The 33rd International Conference on Machine Learning, Proceedings of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Con- ference on Machine Learning, Proceedings of Machine Learning Research, Lille, France, 2015. PMLR.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, Philipp Moritz","year":"2015","venue":"Proceedings of the 32nd International Conference on Machine Learning, Proceedings of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Sherstov and Stone, 2005] Alexander A. Sherstov and Pe- ter Stone. Function approximation via tile coding: Au- tomating parameter choice. In Jean-Daniel Zucker and Lorenza Saitta, editors, Abstraction, Reformulation and Approximation, pages 194-205, Berlin, Heidelberg, 2005. Springer Berlin Heidelberg.","title":"Function approximation via tile coding: Automating parameter choice","authors":"Alexander A Sherstov, Peter Stone","year":"2005","venue":"Abstraction, Reformulation and Approximation","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Silver et al., 2014] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In Proceed- ings of the 31st International Conference on International Conference on Machine Learning -Volume 32, ICML'14, pages I-387-I-395. JMLR.org, 2014.","title":"Deterministic policy gradient algorithms","authors":"David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, Martin Riedmiller","year":"2014","venue":"Proceedings of the 31st International Conference on International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Christopher J. Maddison, et al. Mastering the game of go with deep neu- ral networks and tree search. Nature, 529:484-503, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Christopher J Maddison","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Silver et al., 2017] David Silver, Julian Schrittwieser, Karen Simonyan, et al. Mastering the game of go without human knowledge. Nature, 550:354-, October 2017.","title":"Mastering the game of go without human knowledge","authors":"David Silver, Julian Schrittwieser, Karen Simonyan","year":"2017","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sutton et al., 2000] Richard S Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approx- imation. In S. A. Solla, T. K. Leen, and K. Müller, editors, Advances in Neural Information Processing Systems 12, pages 1057-1063. MIT Press, 2000.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"David A Richard S Sutton, Mcallester, P Satinder, Yishay Singh, Mansour","year":"2000","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Vinyals et al., 2017] Oriol Vinyals, Timo Ewalds, Sergey Bartunov, et al. Starcraft II: A new challenge for rein- forcement learning. CoRR, abs/1708.04782, 2017.","title":"Starcraft II: A new challenge for reinforcement learning","authors":"Oriol Vinyals, Timo Ewalds, Sergey Bartunov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2016] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Duel- ing network architectures for deep reinforcement learning. In Proceedings of The 33rd International Conference on Machine Learning, Proceedings of Machine Learning Re- search, New York, New York, USA, 2016. PMLR.","title":"Dueling network architectures for deep reinforcement learning","authors":"Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, Nando Freitas","year":"2016","venue":"Proceedings of The 33rd International Conference on Machine Learning, Proceedings of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Watkins and Dayan, 1992] Christopher J. C. H. Watkins and Peter Dayan. Q-learning. In Machine Learning, pages 279-292, 1992.","title":"Q-learning","authors":"J C H Christopher, Peter Watkins, Dayan","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wei et al., 2018] Ermo Wei, Drew Wicke, and Sean Luke. Hierarchical approaches for reinforcement learning in pa- rameterized action space. CoRR, abs/1810.09656, 2018.","title":"Hierarchical approaches for reinforcement learning in parameterized action space","authors":"Ermo Wei, Drew Wicke, Sean Luke","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xiong et al., 2018] Jiechao Xiong, Qing Wang, Zhuoran Yang, Peng Sun, Lei Han, Yang Zheng, Haobo Fu, Tong Zhang, Ji Liu, and Han Liu. Parametrized deep q- networks learning: Reinforcement learning with discrete- continuous hybrid action space. CoRR, abs/1810.06394, 2018.","title":"Parametrized deep qnetworks learning: Reinforcement learning with discretecontinuous hybrid action space","authors":"Jiechao Xiong, Qing Wang, Zhuoran Yang, Peng Sun, Lei Han, Yang Zheng, Haobo Fu, Tong Zhang, Ji Liu, Han Liu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}