{"type":"ScholarlyArticle","title":"Deep Multi-Agent Reinforcement Learning with Discrete-Continuous Hybrid Action Spaces","authors":[{"name":"Haotian Fu","slug":"haotian-fu","given":"Haotian","family":"Fu","affiliation":"College of Intelligence and Computing , Tianjin University"},{"name":"Hongyao Tang","slug":"hongyao-tang","given":"Hongyao","family":"Tang","affiliation":"College of Intelligence and Computing , Tianjin University"},{"name":"Jianye Hao","slug":"jianye-hao","given":"Jianye","family":"Hao","affiliation":"College of Intelligence and Computing , Tianjin University"},{"name":"Zihan Lei","slug":"zihan-lei","given":"Zihan","family":"Lei","affiliation":"Fuxi AI Lab in Netease"},{"name":"Yingfeng Chen","slug":"yingfeng-chen","given":"Yingfeng","family":"Chen","affiliation":"Fuxi AI Lab in Netease"},{"name":"Changjie Fan","slug":"changjie-fan","given":"Changjie","family":"Fan","affiliation":"Fuxi AI Lab in Netease"}],"abstract":"Deep Reinforcement Learning (DRL) has been applied to address a variety of cooperative multi-agent problems with either discrete action spaces or continuous action spaces. However, to the best of our knowledge, no previous work has ever succeeded in applying DRL to multi-agent problems with discrete-continuous hybrid (or parameterized) action spaces which is very common in practice. Our work fills this gap by proposing two novel algorithms: Deep Multi-Agent Parameterized Q-Networks (Deep MAPQN) and Deep Multi-Agent Hierarchical Hybrid Q-Networks (Deep MAHHQN). We follow the centralized training but decentralized execution paradigm: different levels of communication between different agents are used to facilitate the training process, while each agent executes its policy independently based on local observations during execution. Our empirical results on several challenging tasks (simulated RoboCup Soccer and game Ghost Story) show that both Deep MAPQN and Deep MAHHQN are effective and significantly outperform existing independent deep parameterized Q-learning method.","doi":"10.24963/ijcai.2019/323","url":"https://ijcai.org/proceedings/2019/323","pdf":"https://ijcai.org/proceedings/2019/0323.pdf","markdown":"https://ijcai.org/proceedings/2019/323.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/323","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":323,"pages":{"first":2329,"last":2335},"keywords":["Agent-based and Multi-agent Systems: Coordination and Cooperation","Agent-based and Multi-agent Systems: Multi-agent Learning","Machine Learning: Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Cao et al., 2013] Yongcan Cao, Wenwu Yu, Wei Ren, and Guanrong Chen. An overview of recent progress in the study of distributed multi-agent coordination. IEEE Trans- actions on Industrial Informatics, 9:427-438, 2013.","title":"An overview of recent progress in the study of distributed multi-agent coordination","authors":"Cao","year":"2013","venue":"IEEE Transactions on Industrial Informatics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Foerster et al., 2016] Jakob N. Foerster, Yannis M. Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learn- ing. In Proceedings of NeurIPS, pages 2137-2145, 2016.","title":"Learning to communicate with deep multi-agent reinforcement learning","authors":"Jakob N Foerster, M Yannis, Nando Assael, Shimon De Freitas, Whiteson","year":"2016","venue":"Proceedings of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Hausknecht and Stone, 2016] Matthew J. Hausknecht and Peter Stone. Deep reinforcement learning in parameter- ized action space. In Proceedings of ICLR, pages 861-868, 2016.","title":"Deep reinforcement learning in parameterized action space","authors":"Matthew J Hausknecht, Peter Stone","year":"2016","venue":"Proceedings of ICLR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Hausknecht, 2016] Matthew J. Hausknecht. Half field of- fense : An environment for multiagent learning and ad hoc teamwork. In Proceedings of ALA, pages 1391-1398, 2016.","title":"Half field offense : An environment for multiagent learning and ad hoc teamwork","authors":"Matthew J Hausknecht","year":"2016","venue":"Proceedings of ALA","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Kulkarni et al., 2016] Tejas D. Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrat- ing temporal abstraction and intrinsic motivation. In Proceedings of NeurIPS, pages 3675-3683, 2016.","title":"Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation","authors":"D Tejas, Karthik Kulkarni, Ardavan Narasimhan, Josh Saeedi, Tenenbaum","year":"2016","venue":"Proceedings of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Lillicrap et al., 2016] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yu- val Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. Proceedings of ICLR, pages 1052-1059, 2016.","title":"Continuous control with deep reinforcement learning","authors":"Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, Daan Wierstra","year":"2016","venue":"Proceedings of ICLR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Lowe et al., 2017] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environ- ments. In Proceedings of NeurIPS, pages 6382-6393, 2017.","title":"Multi-agent actor-critic for mixed cooperative-competitive environments","authors":"Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, Igor Mordatch","year":"2017","venue":"Proceedings of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Masson et al., 2016] Warwick Masson, Pravesh Ranchod, and George Konidaris. Reinforcement learning with pa- rameterized actions. In Proceedings of AAAI, pages 1934- 1940, 2016.","title":"Reinforcement learning with parameterized actions","authors":"Masson","year":"2016","venue":"Proceedings of AAAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013.","title":"Playing atari with deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin A Riedmiller","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fid- jeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Has- sabis. Human-level control through deep reinforcement learning. Nature, 518:529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin A Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, Demis Hassabis","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asyn- chronous methods for deep reinforcement learning. In Proceedings of ICML, pages 1928-1937, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":"Proceedings of ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Peng et al., 2017] Peng Peng, Quan Yuan, Ying Wen, Yaodong Yang, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent bidirectionally-coordinated nets for learning to play starcraft combat games. CoRR, abs/1703.10069, 2017.","title":"Multiagent bidirectionally-coordinated nets for learning to play starcraft combat games","authors":"Peng Peng, Quan Yuan, Ying Wen, Yaodong Yang, Zhenkun Tang, Haitao Long, Jun Wang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Rashid et al., 2018] Tabish Rashid, Mikayel Samvelyan, Christian Schröder de Witt, Gregory Farquhar, Jakob N. Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In Proceedings of ICML, pages 4292-4301, 2018.","title":"Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning","authors":"Tabish Rashid, Mikayel Samvelyan, Christian Schröder De Witt, Gregory Farquhar, Jakob N Foerster, Shimon Whiteson","year":"2018","venue":"Proceedings of ICML","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Silver et al., 2014] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin A. Riedmiller. Deterministic policy gradient algorithms. In Proceedings of ICML, pages 387-395, 2014.","title":"Deterministic policy gradient algorithms","authors":"David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, Martin A Riedmiller","year":"2014","venue":"Proceedings of ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sutton and Barto, 1988] Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. IEEE Transactions on Neural Networks, 16:285-286, 1988.","title":"Reinforcement learning: An introduction","authors":"Richard S Sutton, Andrew G Barto","year":"1988","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tang et al., 2018] Hongyao Tang, Jianye Hao, Tangjie Lv, Yingfeng Chen, Zongzhang Zhang, Hangtian Jia, Chunxu Ren, Yan Zheng, Changjie Fan, and Li Wang. Hierar- chical deep multiagent reinforcement learning. CoRR, abs/1809.09332, 2018.","title":"Hierarchical deep multiagent reinforcement learning","authors":"Hongyao Tang, Jianye Hao, Tangjie Lv, Yingfeng Chen, Zongzhang Zhang, Hangtian Jia, Chunxu Ren, Yan Zheng, Changjie Fan, Li Wang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2018] Qing Wang, Jiechao Xiong, Lei Han, Peng Sun, Han Liu, and Tong Zhang. Exponentially weighted imitation learning for batched historical data. In Proceedings of NeurIPS, pages 6291-6300, 2018.","title":"Exponentially weighted imitation learning for batched historical data","authors":"Qing Wang, Jiechao Xiong, Lei Han, Peng Sun, Han Liu, Tong Zhang","year":"2018","venue":"Proceedings of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wei et al., 2018a] Ermo Wei, Drew Wicke, David Freelan, and Sean Luke. Multiagent soft q-learning. In Proceedings of AAAI, 2018.","title":"Multiagent soft q-learning","authors":"Wei","year":"2018","venue":"Proceedings of AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wei et al., 2018b] Ermo Wei, Drew Wicke, and Sean Luke. Hierarchical approaches for reinforcement learning in pa- rameterized action space. In Proceedings of AAAI, 2018.","title":"Hierarchical approaches for reinforcement learning in parameterized action space","authors":"Wei","year":"2018","venue":"Proceedings of AAAI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Xiong et al., 2018] Jiechao Xiong, Qing Wang, Zhuoran Yang, Peter P Sun, Lei Han, Yang Zheng, Haobo Fu, Tong Zhang, Ji Liu, and Hao Liu. Parametrized deep q- networks learning: Reinforcement learning with discrete- continuous hybrid action space. CoRR, abs/1810.06394, 2018.","title":"Parametrized deep qnetworks learning: Reinforcement learning with discretecontinuous hybrid action space","authors":"Jiechao Xiong, Qing Wang, Zhuoran Yang, Lei Peter P Sun, Yang Han, Haobo Zheng, Tong Fu, Ji Zhang, Hao Liu, Liu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ye et al., 2015] Dayong Ye, Minjie Zhang, and Yun Yang. A multi-agent framework for packet routing in wireless sensor networks. Sensors, 15(5):10026-10047, 2015.","title":"A multi-agent framework for packet routing in wireless sensor networks","authors":"Dayong Ye, Minjie Zhang, Yun Yang","year":"2015","venue":"Sensors","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}