{"type":"ScholarlyArticle","title":"Hybrid Learning for Multi-agent Cooperation with Sub-optimal Demonstrations","authors":[{"name":"Peixi Peng","slug":"peixi-peng","given":"Peixi","family":"Peng","affiliation":"Institute of Automation , Chinese Academy of Sciences; Peking University"},{"name":"Junliang Xing","slug":"junliang-xing","given":"Junliang","family":"Xing","affiliation":"Institute of Automation , Chinese Academy of Sciences"},{"name":"Lili Cao","slug":"lili-cao","given":"Lili","family":"Cao","affiliation":"Institute of Automation , Chinese Academy of Sciences"}],"abstract":"This paper aims to learn multi-agent cooperation where each agent performs its actions in a decentralized way. In this case, it is very challenging to learn decentralized policies when the rewards are global and sparse. Recently, learning from demonstrations (LfD) provides a promising way to handle this challenge. However, in many practical tasks, the available demonstrations are often sub-optimal. To learn better policies from these sub-optimal demonstrations, this paper follows a centralized learning and decentralized execution framework and proposes a novel hybrid learning method based on multi-agent actor-critic. At first, the expert trajectory returns generated from demonstration actions are used to pre-train the centralized critic network. Then, multi-agent decisions are made by best response dynamics based on the critic and used to train the decentralized actor networks. Finally, the demonstrations are updated by the actor networks, and the critic and actor networks are learned jointly by running the above two steps alliteratively. We evaluate the proposed approach on a real-time strategy combat game. Experimental results show that the approach outperforms many competing demonstration-based methods.","doi":"10.24963/ijcai.2020/420","url":"https://ijcai.org/proceedings/2020/420","pdf":"https://ijcai.org/proceedings/2020/0420.pdf","markdown":"https://ijcai.org/proceedings/2020/420.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/420","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":420,"pages":{"first":3037,"last":3043},"keywords":["Agent-based and Multi-agent Systems: Multi-agent Learning","Machine Learning: Reinforcement Learning","Multidisciplinary Topics and Applications: Computer Games"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/26143/file_26143.m3u8","length_sec":299,"provider":"hls"}],"references":[{"position":1,"raw":"[Chen et al., 2017] Yu Fan Chen, Miao Liu, Michael Everett, and Jonathan P How. Decentralized non-communicating multiagent collision avoidance with deep reinforcement learning. In International Conference on Robotics and Au- tomation, pages 285-292, 2017.","title":"Decentralized non-communicating multiagent collision avoidance with deep reinforcement learning","authors":"Miao Yu Fan Chen, Michael Liu, Jonathan P Everett, How","year":"2017","venue":"International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Churchill and Buro, 2013] David Churchill and Michael Buro. Portfolio greedy search and simulation for large- scale combat in StarCraft. In IEEE's Conference on Com- putational Intelligence in Games, pages 1-8, 2013.","title":"Portfolio greedy search and simulation for largescale combat in StarCraft","authors":"Churchill, Buro, David Churchill, Michael Buro","year":"2013","venue":"IEEE's Conference on Computational Intelligence in Games","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Churchill et al., 2012] David Churchill, Abdallah Saffidine, and Michael Buro. Fast heuristic search for RTS game combat scenarios. In Artificial Intelligence and Interactive Digital Entertainment Conference, pages 112-117, 2012.","title":"Fast heuristic search for RTS game combat scenarios","authors":"David Churchill, Abdallah Saffidine, Michael Buro","year":"2012","venue":"Artificial Intelligence and Interactive Digital Entertainment Conference","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Foerster et al., 2018] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In AAAI Conference on Artificial Intelligence, pages 2974- 2982, 2018.","title":"Counterfactual multi-agent policy gradients","authors":"Foerster","year":"2018","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gupta et al., 2017] Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control us- ing deep reinforcement learning. In International Con- ference on Autonomous Agents and Multiagent Systems, pages 66-83, 2017.","title":"Cooperative multi-agent control us-ing deep reinforcement learning","authors":"Maxim Jayesh K Gupta, Mykel Egorov, Kochenderfer","year":"2017","venue":"International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hester et al., 2018] Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Horgan Dan, John Quan, Andrew Sendonaris, and Gabriel Du- lacarnold. Deep q-learning from demonstrations. In AAAI Conference on Artificial Intelligence, pages 3223-3230, 2018.","title":"Deep q-learning from demonstrations","authors":"Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, John Horgan Dan, Andrew Quan, Gabriel Sendonaris, Dulacarnold","year":"2018","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ho and Ermon, 2016] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pages 4565- 4573, 2016.","title":"Generative adversarial imitation learning","authors":"Jonathan Ho, Stefano Ermon","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu et al., 2018] Yue Hu, Juntao Li, Xi Li, Gang Pan, and Mingliang Xu. Knowledge-guided agent-tactic-aware learning for StarCraft micromanagement. In International Joint Conference on Artificial Intelligence, pages 1471- 1477, 2018.","title":"Knowledge-guided agent-tactic-aware learning for StarCraft micromanagement","authors":"Yue Hu, Juntao Li, Xi Li, Gang Pan, Mingliang Xu","year":"2018","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2018/204"},{"position":9,"raw":"[Lelis, 2017] Levi H. S. Lelis. Stratified strategy selection for unit control in real-time strategy games. In Interna- tional Joint Conference on Artificial Intelligence, pages 3735-3741, 2017.","title":"Stratified strategy selection for unit control in real-time strategy games","authors":"H S Levi, Lelis","year":"2017","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2017/522"},{"position":10,"raw":"[Li, 2018] Yuxi Li. Deep reinforcement learning. arXiv preprint arXiv:1810.06339, 2018.","title":"Deep reinforcement learning","authors":"Li ;, Yuxi Li","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lin et al., 2018] Xiaomin Lin, Peter A. Beling, and Randy Cogill. Multiagent inverse reinforcement learning for two- person zero-sum games. IEEE Transactions on Games, 10(1):56-68, 2018.","title":"Multiagent inverse reinforcement learning for twoperson zero-sum games","authors":"Xiaomin Lin, Peter A Beling, Randy Cogill","year":"2018","venue":"IEEE Transactions on Games","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lowe et al., 2017] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-competitive envi- ronments. In Advances in Neural Information Processing Systems, pages 6379-6390, 2017.","title":"Multiagent actor-critic for mixed cooperative-competitive environments","authors":"Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Openai, Igor Abbeel, Mordatch","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Moraes and Lelis, 2018] Rubens O. Moraes and Levi H. S. Lelis. Asymmetric action abstractions for multi-unit con- trol in adversarial real-time games. In AAAI Conference on Artificial Intelligence, pages 876-883, 2018.","title":"Asymmetric action abstractions for multi-unit control in adversarial real-time games","authors":"O Rubens, Moraes, H S Levi, Lelis","year":"2018","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ng and Russell, 2000] Andrew Y. Ng and Stuart J. Russell. Algorithms for inverse reinforcement learning. In Interna- tional Conference on Machine Learning, pages 663-670, 2000.","title":"Algorithms for inverse reinforcement learning","authors":"Andrew Y Ng, Stuart J Russell","year":"2000","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nguyen et al., 2018] Duc Thien Nguyen, Akshat Kumar, and Hoong Chuin Lau. Credit assignment for collective multiagent rl with global rewards. In Advances in Neural Information Processing Systems, pages 8102-8113, 2018.","title":"Credit assignment for collective multiagent rl with global rewards","authors":"Akshat Duc Thien Nguyen, Hoong Chuin Kumar, Lau","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Oliehoek and Amato, 2016] Frans Oliehoek and Christo- pher Amato. A concise introduction to decentralized POMDPs. Springer, 2016.","title":"Frans Oliehoek and Christopher Amato. A concise introduction to decentralized POMDPs","authors":"Amato Oliehoek","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Oliehoek et al., 2008] Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate q-value functions for decentralized pomdps. JAIR, 32(1):289-353, 2008.","title":"Optimal and approximate q-value functions for decentralized pomdps","authors":"Matthijs Frans A Oliehoek, Nikos Tj Spaan, Vlassis","year":"2008","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Peng et al., 2017] Peng Peng, Quan Yuan, Ying Wen, Yaodong Yang, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent bidirectionally-coordinated nets for learning to play StarCraft combat games. arXiv preprint arXiv:1703.10069, 2017.","title":"Multiagent bidirectionally-coordinated nets for learning to play StarCraft combat games","authors":"Peng Peng, Quan Yuan, Ying Wen, Yaodong Yang, Zhenkun Tang, Haitao Long, Jun Wang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pomerleau, 1991] Dean A. Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neu- ral Computation, 3(1):88-97, 1991.","title":"Efficient training of artificial neural networks for autonomous navigation","authors":"Dean A Pomerleau","year":"1991","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rashid et al., 2018] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pages 4292-4301, 2018.","title":"QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning","authors":"Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, Shimon Whiteson","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ross et al., 2011] Stephane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Inter- national Conference on Artificial Intelligence and Statis- tics, volume 15, pages 627-635, 2011.","title":"A reduction of imitation learning and structured prediction to no-regret online learning","authors":"Stephane Ross, Geoffrey J Gordon, J Andrew Bagnell","year":"2011","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Roughgarden, 2016] Tim Roughgarden. Twenty Lectures on Algorithmic Game Theory. Cambridge University Press, 2016.","title":"Twenty Lectures on Algorithmic Game Theory","authors":"Tim Roughgarden","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Schaal, 1996] Stefan Schaal. Learning from demonstration. In Advances in Neural Information Processing Systems, pages 1040-1046, 1996.","title":"Learning from demonstration","authors":"Stefan Schaal, Schaal","year":"1996","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J. Mad- dison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Ve- davyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mas- tering the game of Go with deep neural networks and tree search. Nature, 529(7587):484-489, 2016.","title":"Mastering the game of Go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, Demis Hassabis","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Song et al., 2018] Jiaming Song, Hongyu Ren, Dorsa Sadigh, and Stefano Ermon. Multi-agent generative ad- versarial imitation learning. In Advances in Neural Infor- mation Processing Systems, pages 7461-7472, 2018.","title":"Multi-agent generative adversarial imitation learning","authors":"Jiaming Song, Hongyu Ren, Dorsa Sadigh, Stefano Ermon","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sukhbaatar and Fergus, 2016] Sainbayar Sukhbaatar and Rob Fergus. Learning multiagent communication with backpropagation. In Advances in Neural Information Pro- cessing Systems, pages 2252-2260, 2016.","title":"Sainbayar Sukhbaatar and Rob Fergus. Learning multiagent communication with backpropagation","authors":"Fergus Sukhbaatar","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sun et al., 2017] Wen Sun, Arun Venkatraman, Geoffrey J. Gordon, Byron Boots, and J. Andrew Bagnell. Deeply aggrevated: Differentiable imitation learning for sequen- tial prediction. In International Conference on Machine Learning, pages 3309-3318, 2017.","title":"Deeply aggrevated: Differentiable imitation learning for sequential prediction","authors":"Wen Sun, Arun Venkatraman, Geoffrey J Gordon, Byron Boots, J Andrew Bagnell","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tumer and Agogino, 2007] Kagan Tumer and Adrian K. Agogino. Distributed agent-based air traffic flow manage- ment. In International Conference on Autonomous Agents and Multiagent Systems, pages 1-8, 2007.","title":"Distributed agent-based air traffic flow management","authors":"Kagan Tumer, Adrian K Agogino","year":"2007","venue":"International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Vecerik et al., 2017] Matej Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothorl, Thomas Lampe, and Martin Ried- miller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:1707.08817, 2017.","title":"Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards","authors":"Matej Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothorl, Thomas Lampe, Martin Riedmiller","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}