{"type":"ScholarlyArticle","title":"Explicitly Coordinated Policy Iteration","authors":[{"name":"Yujing Hu","slug":"yujing-hu","given":"Yujing","family":"Hu","affiliation":"Lab in Netease"},{"name":"Yingfeng Chen","slug":"yingfeng-chen","given":"Yingfeng","family":"Chen","affiliation":"Lab in Netease"},{"name":"Changjie Fan","slug":"changjie-fan","given":"Changjie","family":"Fan","affiliation":"Lab in Netease"},{"name":"Jianye Hao","slug":"jianye-hao","given":"Jianye","family":"Hao","affiliation":"Tianjin University"}],"abstract":"Coordination on an optimal policy between independent learners in fully cooperative stochastic games is difficult due to problems such as relative overgeneralization and miscoordination. Most state-of-the-art algorithms apply fusion heuristics on agents' optimistic and average rewards, by which coordination between agents can be achieved implicitly. However, such implicit coordination faces practical issues such as tedious parameter-tuning in real world applications. The lack of an explicit coordination mechanism may also lead to a low likelihood of coordination in problems with multiple optimal policies. Based on the necessary conditions of an optimal policy, we propose the explicitly coordinated policy iteration (EXCEL) algorithm which always forces agents to coordinate by comparing the agents' separated optimistic and average value functions. We also propose three solutions for deep reinforcement learning extensions of EXCEL. Extensive experiments in matrix games (from 2-agent 2-action games to 5-agent 20-action games) and stochastic games (from 2-agent games to 5-agent games) show that EXCEL has better performance than the state-of-the-art algorithms (such as faster convergence and better coordination).","doi":"10.24963/ijcai.2019/51","url":"https://ijcai.org/proceedings/2019/51","pdf":"https://ijcai.org/proceedings/2019/0051.pdf","markdown":"https://ijcai.org/proceedings/2019/51.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/51","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Agent-based and Multi-agent Systems","track":"Main track","number":51,"pages":{"first":357,"last":363},"keywords":["Machine Learning: Reinforcement Learning","Agent-based and Multi-agent Systems: Multi-agent Learning","Agent-based and Multi-agent Systems: Coordination and Cooperation","Agent-based and Multi-agent Systems: Cooperative Games"],"videos":[],"references":[{"position":1,"raw":"[Bowling and Veloso, 2002] Michael Bowling and Manuela Veloso. Multiagent learning using a variable learning rate. Artificial Intelligence, 136(2):215-250, 2002.","title":"Multiagent learning using a variable learning rate","authors":"Michael Bowling, Manuela Veloso","year":"2002","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bus ¸oniu et al., 2008] Lucian Bus ¸oniu, Robert Babuška, and Bart De Schutter. A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, 38(2):156-172, 2008.","title":"Lucian Bus ¸oniu, Robert Babuška, and Bart De Schutter. A comprehensive survey of multiagent reinforcement learning","authors":"Bus ¸oniu","year":"2008","venue":"IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Claus and Boutilier, 1998] Caroline Claus and Craig Boutilier. The dynamics of reinforcement learning in cooperative multiagent systems. In Proceedings of the National Conference on Artificial Intelligence, pages 746-752, 1998.","title":"The dynamics of reinforcement learning in cooperative multiagent systems","authors":"Claus, Boutilier, Caroline Claus, Craig Boutilier","year":"1998","venue":"Proceedings of the National Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Foerster et al., 2018] Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the 32nd AAAI Conference on Artificial In- telligence (AAAI'18), pages 2974-2982, 2018.","title":"Counterfactual multi-agent policy gradients","authors":"Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, Shimon Whiteson","year":"2018","venue":"Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI'18)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Greenwald and Hall, 2003] Amy Greenwald and Keith Hall. Correlated Q-learning. In Proceedings of the 20th In- ternational Conference on Machine Learning, pages 242- 249, 2003.","title":"Correlated Q-learning","authors":"Amy Greenwald, Keith Hall","year":"2003","venue":"Proceedings of the 20th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hu and Wellman, 2003] Junling Hu and Michael P. Well- man. Nash Q-learning for general-sum stochastic games. The Journal of Machine Learning Research, 4:1039-1069, 2003.","title":"Wellman. Nash Q-learning for general-sum stochastic games","authors":"Wellman Hu, Junling Hu, Michael P","year":"2003","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kapetanakis and Kudenko, 2002] Spiros Kapetanakis and Daniel Kudenko. Reinforcement learning of coordina- tion in cooperative multi-agent systems. In Proceedings of the 18th National Conference on Artificial Intelligence (AAAI'02), pages 326-331. York, 2002.","title":"Reinforcement learning of coordination in cooperative multi-agent systems","authors":"Kudenko Kapetanakis, Spiros Kapetanakis, Daniel Kudenko","year":"2002","venue":"Proceedings of the 18th National Conference on Artificial Intelligence (AAAI'02)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lauer and Riedmiller, 2000] Martin Lauer and Martin Ried- miller. An algorithm for distributed reinforcement learn- ing in cooperative multi-agent systems. In Proceedings of the 17th International Conference on Machine Learning, pages 535-542, 2000.","title":"An algorithm for distributed reinforcement learning in cooperative multi-agent systems","authors":"Riedmiller Lauer, Martin Lauer, Martin Riedmiller","year":"2000","venue":"Proceedings of the 17th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Littman, 1994] Michael L. Littman. Markov games as a framework for multi-agent reinforcement learning. In Pro- ceedings of the 11th International Conference on Machine Learning, pages 157-163, 1994.","title":"Markov games as a framework for multi-agent reinforcement learning","authors":"L Michael, Littman","year":"1994","venue":"Proceedings of the 11th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lowe et al., 2017] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-competitive envi- ronments. In Advances in Neural Information Processing Systems, pages 6379-6390, 2017.","title":"Multiagent actor-critic for mixed cooperative-competitive environments","authors":"Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Openai, Igor Abbeel, Mordatch","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Matignon et al., 2007] Laëtitia Matignon, Guillaume Lau- rent, and Nadine Le Fort-Piat. Hysteretic q-learning: an algorithm for decentralized reinforcement learning in co- operative multi-agent teams. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS'07), pages 64-69, 2007.","title":"Hysteretic q-learning: an algorithm for decentralized reinforcement learning in cooperative multi-agent teams","authors":"Laëtitia Matignon, Guillaume Laurent, Nadine Le Fort-Piat","year":"2007","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS'07)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Matignon et al., 2008] Laëtitia Matignon, Guillaume Lau- rent, and Nadine Le Fort-Piat. A study of FMQ heuristic in cooperative multi-agent games. In Proceedings of Multi- Agent Sequential Decision Making in Uncertain Multi- Agent Domains Workshop in the 7th International Confer- ence on Autonomous Agents and Multiagent Systems (AA- MAS'08), pages 77-91, 2008.","title":"A study of FMQ heuristic in cooperative multi-agent games","authors":"Laëtitia Matignon, Guillaume Laurent, Nadine Le Fort-Piat","year":"2008","venue":"Proceedings of Multi-Agent Sequential Decision Making in Uncertain Multi-Agent Domains Workshop in the 7th International Conference on Autonomous Agents and Multiagent Systems (AA-MAS'08)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Matignon et al., 2009] Laëtitia Matignon, Guillaume J Lau- rent, and Nadine Le Fort-Piat. Coordination of indepen- dent learners in cooperative markov games. Technical re- port, Institut FEMTO-ST, UniversitPé de Franche-Comté, 2009.","title":"Coordination of independent learners in cooperative markov games","authors":"Laëtitia Matignon, Guillaume J Laurent, Nadine Le Fort-Piat","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Matignon et al., 2012] Laetitia Matignon, Guillaume J Lau- rent, and Nadine Le Fort-Piat. Independent reinforcement learners in cooperative markov games: a survey regarding coordination problems. The Knowledge Engineering Re- view, 27(1):1-31, 2012.","title":"Independent reinforcement learners in cooperative markov games: a survey regarding coordination problems","authors":"Laetitia Matignon, Guillaume J Laurent, Nadine Le Fort-Piat","year":"2012","venue":"The Knowledge Engineering Review","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, et al. Human-Level Con- trol Through Deep Reinforcement Learning. Nature, 518(7540):529-533, 2015.","title":"Human-Level Control Through Deep Reinforcement Learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Omidshafiei et al., 2017] Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan P. How, and John Vian. Deep Decentralized Multi-task Multi-Agent Re- inforcement Learning Under Partial Observability. In Proceedings of the 34th International Conference on Machine Learning (ICML'17), pages 2681-2690, 2017.","title":"Deep Decentralized Multi-task Multi-Agent Reinforcement Learning Under Partial Observability","authors":"Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan P How, John Vian","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning (ICML'17)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Palmer et al., 2018] Gregory Palmer, Karl Tuyls, Daan Bloembergen, and Rahul Savani. Lenient multi-agent deep reinforcement learning. In Proceedings of the 17th In- ternational Conference on Autonomous Agents and Multi- Agent Systems (AAMAS'18), pages 443-451, 2018.","title":"Lenient multi-agent deep reinforcement learning","authors":"Gregory Palmer, Karl Tuyls, Daan Bloembergen, Rahul Savani","year":"2018","venue":"Proceedings of the 17th International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS'18)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Panait et al., 2006] Liviu Panait, Keith Sullivan, and Sean Luke. Lenient learners in cooperative multiagent sys- tems. In Proceedings of the 15th International Con- ference on Autonomous Agents and Multi-Agent Systems (AAMAS'06), pages 801-803, 2006.","title":"Lenient learners in cooperative multiagent systems","authors":"Liviu Panait, Keith Sullivan, Sean Luke","year":"2006","venue":"Proceedings of the 15th International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS'06)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shoham et al., 2003] Yoav Shoham, Rob Powers, and Trond Grenager. Multi-agent reinforcement learning: a critical survey. Unpublished survey. http://robotics. stan- ford. edu/shoham, 2003.","title":"Multi-agent reinforcement learning: a critical survey","authors":"Yoav Shoham, Rob Powers, Trond Grenager","year":null,"venue":null,"doi":null,"url":"http://robotics.stan-ford.edu/shoham,2003","internal_url":null},{"position":20,"raw":"[Tan, 1993] Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the 10th International Conference on Machine Learning, vol- ume 337, pages 330-337, 1993.","title":"Multi-agent reinforcement learning: Independent vs. cooperative agents","authors":"Ming Tan","year":"1993","venue":"Proceedings of the 10th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Watkins and Dayan, 1992] Christopher J.C.H. Watkins and Peter Dayan. Q-learning. Machine Learning, 8(3):279- 292, 1992.","title":"Q-learning","authors":"J C H Christopher, Peter Watkins, Dayan","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wei and Luke, 2016] Ermo Wei and Sean Luke. Le- nient learning in independent-learner stochastic coopera- tive games. The Journal of Machine Learning Research, 17(1):2914-2955, 2016.","title":"Lenient learning in independent-learner stochastic cooperative games","authors":"Ermo Wei, Sean Luke","year":"2016","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wiegand, 2003] R. Paul Wiegand. An Analysis of Coopera- tive Coevolutionary Algorithms. PhD thesis, George Ma- son University, 2003.","title":"An Analysis of Cooperative Coevolutionary Algorithms","authors":"R Paul Wiegand","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}