{"type":"ScholarlyArticle","title":"Learning Deep Decentralized Policy Network by Collective Rewards for Real-Time Combat Game","authors":[{"name":"Peixi Peng","slug":"peixi-peng","given":"Peixi","family":"Peng","affiliation":"Institute of Automation , Chinese Academy of Sciences"},{"name":"Junliang Xing","slug":"junliang-xing","given":"Junliang","family":"Xing","affiliation":"Institute of Automation , Chinese Academy of Sciences"},{"name":"Lili Cao","slug":"lili-cao","given":"Lili","family":"Cao","affiliation":"Institute of Automation , Chinese Academy of Sciences"},{"name":"Lisen Mu","slug":"lisen-mu","given":"Lisen","family":"Mu","affiliation":"Horizon Robotics"},{"name":"Chang Huang","slug":"chang-huang","given":"Chang","family":"Huang","affiliation":"Horizon Robotics"}],"abstract":"The task of real-time combat game is to coordinate multiple units to defeat their enemies controlled by the given opponent in a real-time combat scenario. It is difficult to design a high-level Artificial Intelligence (AI) program for such a task due to its extremely large state-action space and real-time requirements. This paper formulates this task as a collective decentralized partially observable Markov decision process, and designs a Deep Decentralized Policy Network (DDPN) to model the polices. To train DDPN effectively, a novel two-stage learning algorithm is proposed which combines imitation learning from opponent and reinforcement learning by no-regret dynamics.  Extensive experimental results on various combat scenarios indicate that proposed method can defeat different opponent models and significantly outperforms many state-of-the-art approaches.","doi":"10.24963/ijcai.2019/181","url":"https://ijcai.org/proceedings/2019/181","pdf":"https://ijcai.org/proceedings/2019/0181.pdf","markdown":"https://ijcai.org/proceedings/2019/181.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/181","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Heuristic Search and Game Playing","track":"Main track","number":181,"pages":{"first":1305,"last":1311},"keywords":["Heuristic Search and Game Playing: Game Playing","Heuristic Search and Game Playing: Game Playing and Machine Learning","Machine Learning Applications: Applications of Reinforcement Learning","Machine Learning Applications: Game Playing"],"videos":[],"references":[{"position":1,"raw":"[Cesa-Bianchi et al., 2007] Nicolo Cesa-Bianchi, Yishay Mansour, and Gilles Stoltz. Improved second-order bounds for prediction with expert advice. Machine Learning, 2007.","title":"Improved second-order bounds for prediction with expert advice","authors":"Cesa-Bianchi","year":"2007","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Churchill and Buro, 2013] David Churchill and Michael Buro. Portfolio greedy search and simulation for large- scale combat in starcraft. In IEEE CIG, 2013.","title":"Portfolio greedy search and simulation for largescale combat in starcraft","authors":"Churchill, Buro, David Churchill, Michael Buro","year":"2013","venue":"IEEE CIG","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Churchill et al., 2012] David Churchill, Abdallah Saffidine, and Michael Buro. Fast heuristic search for rts game combat scenarios. In AIIDE, 2012.","title":"Fast heuristic search for rts game combat scenarios","authors":"David Churchill, Abdallah Saffidine, Michael Buro","year":"2012","venue":"AIIDE","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Foerster et al., 2018] Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In AAAI, 2018.","title":"Counterfactual multi-agent policy gradients","authors":"Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, Shimon Whiteson","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hester et al., 2018] Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Horgan Dan, John Quan, Andrew Sendonaris, and Gabriel Dulacarnold. Deep q-learning from demonstrations. In AAAI, 2018.","title":"Deep q-learning from demonstrations","authors":"Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, John Horgan Dan, Andrew Quan, Gabriel Sendonaris, Dulacarnold","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ho and Ermon, 2016] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In NIPS, 2016.","title":"Generative adversarial imitation learning","authors":"Jonathan Ho, Stefano Ermon","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2018] Yue Hu, Juntao Li, Xi Li, Gang Pan, and Mingliang Xu. Knowledge-guided agent-tactic-aware learning for starcraft micromanagement. In IJCAI, 2018.","title":"Knowledge-guided agent-tactic-aware learning for starcraft micromanagement","authors":"Yue Hu, Juntao Li, Xi Li, Gang Pan, Mingliang Xu","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/204"},{"position":8,"raw":"[Kang et al., 2018] Bingyi Kang, Zequn Jie, and Jiashi Feng. Policy optimization with demonstrations. In ICML, 2018.","title":"Policy optimization with demonstrations","authors":"Kang","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kong et al., 2017] Xiangyu Kong, Bo Xin, Fangchen Liu, and Yizhou Wang. Revisiting the master-slave architecture in multi-agent deep reinforcement learning. arXiv, 2017.","title":"Revisiting the master-slave architecture in multi-agent deep reinforcement learning","authors":"Xiangyu Kong, Bo Xin, Fangchen Liu, Yizhou Wang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[LeCun et al., 2015] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 2015.","title":"Deep learning","authors":"Yann Lecun, Yoshua Bengio, Geoffrey Hinton","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lelis, 2017] Levi H. S. Lelis. Stratified strategy selection for unit control in real-time strategy games. In IJCAI, 2017.","title":"Stratified strategy selection for unit control in real-time strategy games","authors":"H S Levi, Lelis","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/522"},{"position":12,"raw":"[Li, 2018] Yuxi Li. Deep reinforcement learning. arXiv preprint arXiv:1810.06339, 2018.","title":"Deep reinforcement learning","authors":"Li ;, Yuxi Li","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lowe et al., 2017] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. In NIPS. 2017.","title":"Multi-agent actor-critic for mixed cooperative-competitive environments","authors":"Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Openai, Igor Abbeel, Mordatch","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Maas et al., 2013] Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlinearities improve neural network acoustic models. In ICML, 2013.","title":"Rectifier nonlinearities improve neural network acoustic models","authors":"Awni Y Andrew L Maas, Andrew Y Hannun, Ng","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv, 2013.","title":"Playing atari with deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Moraes and Lelis, 2018] Rubens O. Moraes and Levi H. S. Lelis. Asymmetric action abstractions for multi-unit control in adversarial real-time games. In AAAI, 2018.","title":"Asymmetric action abstractions for multi-unit control in adversarial real-time games","authors":"O Rubens, Moraes, H S Levi, Lelis","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nguyen et al., 2018] Duc Thien Nguyen, Akshat Kumar, and Hoong Chuin Lau. Credit assignment for collective multiagent rl with global rewards. In NIPS, 2018.","title":"Credit assignment for collective multiagent rl with global rewards","authors":"Akshat Duc Thien Nguyen, Hoong Chuin Kumar, Lau","year":"2018","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Oh et al., 2016] Junhyuk Oh, Valliappa Chockalingam, Satinder Singh, and Honglak Lee. Control of memory, active perception, and action in minecraft. arXiv, 2016.","title":"Junhyuk Oh, Valliappa Chockalingam, Satinder Singh, and Honglak Lee","authors":"Oh","year":"2016","venue":"Control of memory, active perception, and action in minecraft","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Oliehoek et al., 2008] Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate q- value functions for decentralized pomdps. Journal of Artificial Intelligence Research, 32:289-353, 2008.","title":"Optimal and approximate qvalue functions for decentralized pomdps","authors":"Oliehoek","year":"2008","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Peng et al., 2017] Peng Peng, Quan Yuan, Ying Wen, Yaodong Yang, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent bidirectionally-coordinated nets for learning to play starcraft combat games. arXiv, 2017.","title":"Multiagent bidirectionally-coordinated nets for learning to play starcraft combat games","authors":"Peng Peng, Quan Yuan, Ying Wen, Yaodong Yang, Zhenkun Tang, Haitao Long, Jun Wang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Pomerleau, 1991] D. A. Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 3(1):88-97, 1991.","title":"Efficient training of artificial neural networks for autonomous navigation","authors":"D A Pomerleau","year":"1991","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rashid et al., 2018] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning. In ICML, 2018.","title":"QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning","authors":"Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, Shimon Whiteson","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ross et al., 2011] Stephane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Aistats, 2011.","title":"A reduction of imitation learning and structured prediction to no-regret online learning","authors":"Stephane Ross, Geoffrey J Gordon, J Andrew Bagnell","year":"2011","venue":"Aistats","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Roughgarden, 2016] Tim Roughgarden. Twenty Lectures on Algorithmic Game Theory, pages 230-247. Cambridge University Press, 1st edition, 2016.","title":"Twenty Lectures on Algorithmic Game Theory","authors":"Roughgarden","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 2015.","title":"Imagenet large scale visual recognition challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein","year":"2015","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Schaal, 1997] Stefan Schaal. Learning from demonstration. In NIPS, 1997.","title":"Learning from demonstration","authors":"Stefan Schaal","year":"1997","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Silver et al., 2016] D Silver, A. Huang, C. J. Maddison, A Guez, L Sifre, den Driessche G Van, J Schrittwieser, I Antonoglou, V Panneershelvam, and M Lanctot. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484-489, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"A Silver, C J Huang, Maddison, L Guez, Sifre, G Den Driessche, J Van, I Schrittwieser, V Antonoglou, M Panneershelvam, Lanctot","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Song et al., 2018] Jiaming Song, Hongyu Ren, Dorsa Sadigh, and Stefano Ermon. Multi-agent generative adversarial imitation learning. In NIPS, 2018.","title":"Multi-agent generative adversarial imitation learning","authors":"Jiaming Song, Hongyu Ren, Dorsa Sadigh, Stefano Ermon","year":"2018","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Sukhbaatar and Fergus, 2016] Sainbayar Sukhbaatar and Rob Fergus. Learning multiagent communication with backpropagation. In NIPS, 2016.","title":"Sainbayar Sukhbaatar and Rob Fergus. Learning multiagent communication with backpropagation","authors":"Fergus Sukhbaatar","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sun et al., 2017] Wen Sun, Arun Venkatraman, Geoffrey J. Gordon, Byron Boots, and J. Andrew Bagnell. Deeply AggreVaTeD: Differentiable imitation learning for sequential prediction. In ICML, pages 3309-3318, 2017.","title":"Deeply AggreVaTeD: Differentiable imitation learning for sequential prediction","authors":"Wen Sun, Arun Venkatraman, Geoffrey J Gordon, Byron Boots, J Andrew Bagnell","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Tian et al., 2017] Yuandong Tian, Qucheng Gong, Wenling Shang, Yuxin Wu, and C. Lawrence Zitnick. Elf: An extensive, lightweight and flexible research platform for real-time strategy games. In NIPS, 2017.","title":"Elf: An extensive, lightweight and flexible research platform for real-time strategy games","authors":"Yuandong Tian, Qucheng Gong, Wenling Shang, Yuxin Wu, C Lawrence Zitnick","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Usunier et al., 2016] Nicolas Usunier, Gabriel Synnaeve, Zeming Lin, and Soumith Chintala. Episodic exploration for deep deterministic policies: An application to starcraft micromanagement tasks. arXiv, 2016.","title":"Episodic exploration for deep deterministic policies: An application to starcraft micromanagement tasks","authors":"Usunier","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Williams, 1992] Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. In Machine Learning, volume 8, pages 229-256. Springer, 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Ronald J Williams","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}