{"type":"ScholarlyArticle","title":"Knowledge-Guided Agent-Tactic-Aware Learning for StarCraft Micromanagement","authors":[{"name":"Yue Hu","slug":"yue-hu","given":"Yue","family":"Hu","affiliation":"College of Computer Science and Technology , Zhejiang University , Hangzhou , China"},{"name":"Juntao Li","slug":"juntao-li","given":"Juntao","family":"Li","affiliation":"College of Computer Science and Technology , Zhejiang University , Hangzhou , China"},{"name":"Xi Li","slug":"xi-li","given":"Xi","family":"Li","affiliation":"College of Computer Science and Technology , Zhejiang University , Hangzhou , China"},{"name":"Gang Pan","slug":"gang-pan","given":"Gang","family":"Pan","affiliation":"College of Computer Science and Technology , Zhejiang University , Hangzhou , China"},{"name":"Mingliang Xu","slug":"mingliang-xu","given":"Mingliang","family":"Xu","affiliation":"Zhengzhou University , Zhengzhou , China"}],"abstract":"As an important and challenging problem in artificial intelligence (AI) game playing, StarCraft micromanagement involves a dynamically adversarial game playing process with complex multi-agent control within a large action space. In this paper, we propose a novel knowledge-guided agent-tactic-aware learning scheme, that is, opponent-guided tactic learning (OGTL), to cope with this micromanagement problem. In principle, the proposed scheme takes a two-stage cascaded learning strategy which is capable of not only transferring the human tactic knowledge from the human-made opponent agents to our AI agents but also improving the adversarial ability. With the power of reinforcement learning, such a knowledge-guided agent-tactic-aware scheme has the ability to guide the AI agents to achieve high winning-rate performances while accelerating the policy exploration process in a tactic-interpretable fashion. Experimental results demonstrate the effectiveness of the proposed scheme against the state-of-the-art approaches in several benchmark combat scenarios.","doi":"10.24963/ijcai.2018/204","url":"https://ijcai.org/proceedings/2018/204","pdf":"https://ijcai.org/proceedings/2018/0204.pdf","markdown":"https://ijcai.org/proceedings/2018/204.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/204","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Heuristic Search and Game Playing","track":"Main track","number":204,"pages":{"first":1471,"last":1477},"keywords":["Heuristic Search and Game Playing: Game Playing and Machine Learning","Machine Learning Applications: Game Playing"],"videos":[],"references":[{"position":1,"raw":"[Foerster et al., 2016] Jakob Foerster, Yannis Assael, Nando de Freitas, and Shimon Whiteson. Learning to communi- cate with deep multi-agent reinforcement learning. In Ad- vances in Neural Information Processing Systems, pages 2137-2145, 2016.","title":"Learning to communicate with deep multi-agent reinforcement learning","authors":"Foerster","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Foerster et al., 2017a] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. arXiv preprint arXiv:1705.08926, 2017.","title":"Counterfactual multi-agent policy gradients","authors":"Foerster","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Foerster et al., 2017b] Jakob Foerster, Nantas Nardelli, Gregory Farquhar, Philip Torr, Pushmeet Kohli, Shimon Whiteson, et al. Stabilising experience replay for deep multi-agent reinforcement learning. International Confer- ence on Machine Learning, 2017.","title":"Stabilising experience replay for deep multi-agent reinforcement learning","authors":"Foerster","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Kong et al., 2017] Xiangyu Kong, Bo Xin, Fangchen Liu, and Yizhou Wang. Revisiting the master-slave architecture in multi-agent deep reinforcement learning. arXiv preprint arXiv:1712.07305, 2017.","title":"Revisiting the master-slave architecture in multi-agent deep reinforcement learning","authors":"Xiangyu Kong, Bo Xin, Fangchen Liu, Yizhou Wang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Levine et al., 2016] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-1373, 2016.","title":"End-to-end training of deep visuomotor policies","authors":"Sergey Levine, Chelsea Finn, Trevor Darrell, Pieter Abbeel","year":"2016","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Li et al., 2017] Xuijun Li, Yun-Nung Chen, Lihong Li, and Jianfeng Gao. End-to-end task-completion neural dialogue systems. arXiv preprint arXiv:1703.01008, 2017.","title":"End-to-end task-completion neural dialogue systems","authors":"Xuijun Li, Yun-Nung Chen, Lihong Li, Jianfeng Gao","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Li, 2017] Yuxi Li. Deep reinforcement learning: An overview. arXiv preprint arXiv:1701.07274, 2017.","title":"Deep reinforcement learning: An overview","authors":"Li ;, Yuxi Li","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lillicrap et al., 2015] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tas- sa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arX- iv:1509.02971, 2015.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lowe et al., 2017] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-competitive envi- ronments. In Advances in Neural Information Processing Systems, pages 6382-6393, 2017.","title":"Multiagent actor-critic for mixed cooperative-competitive environments","authors":"Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Openai, Igor Abbeel, Mordatch","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Mirowski et al., 2017] Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J Ballard, Andrea Ban- ino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, et al. Learning to navigate in complex en- vironments. International Conference on Learning Repre- sentations, 2017.","title":"Learning to navigate in complex environments","authors":"Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu","year":"2017","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Nisan et al., 2007] Noam Nisan, Tim Roughgarden, Eva Tardos, and Vijay V Vazirani. Algorithmic game theory, volume 1. Cambridge University Press Cambridge, 2007.","title":"Algorithmic game theory","authors":"Nisan","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ontanón et al., 2013] Santiago Ontanón, Gabriel Synnaeve, Alberto Uriarte, Florian Richoux, David Churchill, and Mike Preuss. A survey of real-time strategy game ai re- search and competition in starcraft. IEEE Transactions on Computational Intelligence and AI in games, 5(4):293- 311, 2013.","title":"A survey of real-time strategy game ai research and competition in starcraft","authors":"Ontanón","year":"2013","venue":"IEEE Transactions on Computational Intelligence and AI in games","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pasunuru and Bansal, 2017] Ramakanth Pasunuru and Mo- hit Bansal. Reinforced video captioning with entailment rewards. arXiv preprint arXiv:1708.02300, 2017.","title":"Ramakanth Pasunuru and Mohit Bansal. Reinforced video captioning with entailment rewards","authors":"Bansal Pasunuru","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Peng et al., 2017] Peng Peng, Quan Yuan, Ying Wen, Yaodong Yang, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent bidirectionally-coordinated nets for learning to play starcraft combat games. arXiv preprint arXiv:1703.10069, 2017.","title":"Multiagent bidirectionally-coordinated nets for learning to play starcraft combat games","authors":"Peng Peng, Quan Yuan, Ying Wen, Yaodong Yang, Zhenkun Tang, Haitao Long, Jun Wang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Schuster and Paliwal, 1997] Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11):2673-2681, 1997.","title":"Bidirectional recurrent neural networks","authors":"Mike Schuster, Kuldip K Paliwal","year":"1997","venue":"IEEE Transactions on Signal Processing","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shapley, 1953] Lloyd S Shapley. Stochastic games. Proceedings of the national academy of sciences, 39(10):1095-1100, 1953.","title":"Stochastic games","authors":"S Lloyd, Shapley","year":"1953","venue":"Proceedings of the national academy of sciences","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484-489, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Marc Driessche, Lanctot","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Silver et al., 2017] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354, 2017.","title":"Mastering the game of go without human knowledge","authors":"David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton","year":"2017","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sukhbaatar et al., 2016] Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backprop- agation. In Advances in Neural Information Processing Systems, pages 2244-2252, 2016.","title":"Learning multiagent communication with backpropagation","authors":"Sukhbaatar","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Synnaeve et al., 2016] Gabriel Synnaeve, Nantas Nardelli, Alex Auvolat, Soumith Chintala, Timothée Lacroix, Zem- ing Lin, Florian Richoux, and Nicolas Usunier. Torchcraft: a library for machine learning research on real-time strat- egy games. arXiv preprint arXiv:1611.00625, 2016.","title":"Torchcraft: a library for machine learning research on real-time strategy games","authors":"Nantas Gabriel Synnaeve, Alex Nardelli, Soumith Auvolat, Timothée Chintala, Zeming Lacroix, Florian Lin, Nicolas Richoux, Usunier","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Usunier et al., 2017] Nicolas Usunier, Gabriel Synnaeve, Zeming Lin, and Soumith Chintala. Episodic exploration for deep deterministic policies: An application to star- craft micromanagement tasks. International Conference on Learning Representations, 2017.","title":"Episodic exploration for deep deterministic policies: An application to starcraft micromanagement tasks","authors":"Usunier","year":"2017","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wender and Watson, 2012] Stefan Wender and Ian Watson. Applying reinforcement learning to small scale combat in the real-time strategy game starcraft: Broodwar. In Com- putational Intelligence and Games, 2012 IEEE Confer- ence on, pages 402-408. IEEE, 2012.","title":"Applying reinforcement learning to small scale combat in the real-time strategy game starcraft: Broodwar","authors":"Watson Wender, Stefan Wender, Ian Watson","year":"2012","venue":"Computational Intelligence and Games","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}