{"type":"ScholarlyArticle","title":"Value Function Transfer for Deep Multi-Agent Reinforcement Learning Based on N-Step Returns","authors":[{"name":"Yong Liu","slug":"yong-liu","given":"Yong","family":"Liu","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , China; Fuxi AI Lab in Netease"},{"name":"Yujing Hu","slug":"yujing-hu","given":"Yujing","family":"Hu","affiliation":"Fuxi AI Lab in Netease"},{"name":"Yang Gao","slug":"yang-gao","given":"Yang","family":"Gao","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , China"},{"name":"Yingfeng Chen","slug":"yingfeng-chen","given":"Yingfeng","family":"Chen","affiliation":"Fuxi AI Lab in Netease"},{"name":"Changjie Fan","slug":"changjie-fan","given":"Changjie","family":"Fan","affiliation":"Fuxi AI Lab in Netease"}],"abstract":"Many real-world problems, such as robot control and soccer game, are naturally modeled as sparse-interaction multi-agent systems. Reutilizing single-agent knowledge in multi-agent systems with sparse interactions can greatly accelerate the multi-agent learning process. Previous works rely on bisimulation metric to define Markov decision process (MDP) similarity for controlling knowledge transfer. However, bisimulation metric is costly to compute and is not suitable for high-dimensional state space problems. In this work, we propose more scalable transfer learning methods based on a novel MDP similarity concept. We start by defining the MDP similarity based on the N-step return (NSR) values of an MDP. Then, we propose two knowledge transfer methods based on deep neural networks called direct value function transfer and NSR-based value function transfer. We conduct experiments in image-based grid world, multi-agent particle environment (MPE) and Ms. Pac-Man game. The results indicate that the proposed methods can significantly accelerate multi-agent reinforcement learning and meanwhile get better asymptotic performance.","doi":"10.24963/ijcai.2019/65","url":"https://ijcai.org/proceedings/2019/65","pdf":"https://ijcai.org/proceedings/2019/0065.pdf","markdown":"https://ijcai.org/proceedings/2019/65.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/65","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Agent-based and Multi-agent Systems","track":"Main track","number":65,"pages":{"first":457,"last":463},"keywords":["Agent-based and Multi-agent Systems: Multi-agent Learning","Machine Learning: Reinforcement Learning","Machine Learning: Transfer, Adaptation, Multi-task Learning"],"videos":[],"references":[{"position":1,"raw":"[De Hauwere et al., 2010] Yann-Michaël De Hauwere, Peter Vrancx, and Ann Nowé. Learning multi-agent state space representations. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Sys- tems, pages 715-722, 2010.","title":"Learning multi-agent state space representations","authors":"Yann-Michaël De Hauwere, Peter Vrancx, Ann Nowé","year":"2010","venue":"Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ferns et al., 2004] Norm Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite Markov decision pro- cesses. In Proceedings of the 20th Conference in Uncer- tainty in Artificial Intelligence, pages 162-169, 2004.","title":"Metrics for finite Markov decision processes","authors":"Ferns","year":"2004","venue":"Proceedings of the 20th Conference in Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Foerster et al., 2018] Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.","title":"Counterfactual multi-agent policy gradients","authors":"Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, Shimon Whiteson","year":"2018","venue":"Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Guestrin et al., 2002a] Carlos Guestrin, Michail G. Lagoudakis, and Ronald Parr. Coordinated reinforce- ment learning. In Proceedings of the 9th International Conference on Machine Learning, pages 227-234, 2002.","title":"Coordinated reinforcement learning","authors":"Guestrin","year":"2002","venue":"Proceedings of the 9th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Guestrin et al., 2002b] Carlos Guestrin, Shobha Venkatara- man, and Daphne Koller. Context-specific multiagent co- ordination and planning with factored MDPs. In Proceed- ings of the 18th National Conference on Artificial Intelli- gence, pages 253-259, 2002.","title":"Context-specific multiagent coordination and planning with factored MDPs","authors":"Guestrin","year":"2002","venue":"Proceedings of the 18th National Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hauwere et al., 2011] Yann-Michaël De Hauwere, Peter Vrancx, and Ann Nowé. Solving sparse delayed coordi- nation problems in multi-agent reinforcement learning. In International Workshop on Adaptive and Learning Agents (ALA 2011), pages 114-133, 2011.","title":"Solving sparse delayed coordination problems in multi-agent reinforcement learning","authors":"Yann-Michaël De Hauwere, Peter Vrancx, Ann Nowé","year":"2011","venue":"International Workshop on Adaptive and Learning Agents (ALA 2011)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2015] Yujing Hu, Yang Gao, and Bo An. Learning in multi-agent systems with sparse interactions by knowl- edge transfer and game abstraction. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, pages 753-761, 2015.","title":"Learning in multi-agent systems with sparse interactions by knowledge transfer and game abstraction","authors":"Yujing Hu, Yang Gao, Bo An","year":"2015","venue":"Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kok and Vlassis, 2004] Jelle R. Kok and Nikos A. Vlas- sis. Sparse cooperative Q-learning. In Proceedings of the 21st International Conference on Machine Learning (ICML 2004), pages 61-68, 2004.","title":"Sparse cooperative Q-learning","authors":"R Jelle, Nikos A Kok, Vlassis","year":"2004","venue":"Proceedings of the 21st International Conference on Machine Learning (ICML 2004)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kok et al., 2005] Jelle R. Kok, Pieter Jan't Hoen, Bram Bakker, and Nikos A. Vlassis. Utile coordination: Learn- ing interdependencies among cooperative agents. In Pro- ceedings of the 2005 IEEE Symposium on Computational Intelligence and Games (CIG05), pages 29-36, 2005.","title":"Utile coordination: Learning interdependencies among cooperative agents","authors":"R Jelle, Pieter Kok, Bram Jan't Hoen, Nikos A Bakker, Vlassis","year":"2005","venue":"Proceedings of the 2005 IEEE Symposium on Computational Intelligence and Games (CIG05)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lowe et al., 2017] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environ- ments. Neural Information Processing Systems (NIPS), 2017.","title":"Multi-agent actor-critic for mixed cooperative-competitive environments","authors":"Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, Igor Mordatch","year":"2017","venue":"Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Melo and Veloso, 2009] Francisco S Melo and Manuela Veloso. Learning of coordination: Exploiting sparse in- teractions in multiagent systems. In Proceedings of the 8th International Conference on Autonomous Agents and Multiagent Systems, pages 773-780, 2009.","title":"Learning of coordination: Exploiting sparse interactions in multiagent systems","authors":"Veloso ; Melo, S Francisco, Manuela Melo, Veloso","year":"2009","venue":"Proceedings of the 8th International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Melo and Veloso, 2011] Francisco S. Melo and Manuela M. Veloso. Decentralized MDPs with sparse interactions. Ar- tifitial Intelligence, 175(11):1757-1789, 2011.","title":"Decentralized MDPs with sparse interactions","authors":"Melo, Francisco S Veloso, Manuela M Melo, Veloso","year":"2011","venue":"Artifitial Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asyn- chronous methods for deep reinforcement learning. In In- ternational conference on machine learning, pages 1928- 1937, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Peng et al., 2017] Peng Peng, Quan Yuan, Ying Wen, Yaodong Yang, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent bidirectionally-coordinated nets for learning to play starcraft combat games. arXiv preprint arXiv:1703.10069, 2017.","title":"Multiagent bidirectionally-coordinated nets for learning to play starcraft combat games","authors":"Peng Peng, Quan Yuan, Ying Wen, Yaodong Yang, Zhenkun Tang, Haitao Long, Jun Wang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rashid et al., 2018] Tabish Rashid, Mikayel Samvelyan, Christian Schröder de Witt, Gregory Farquhar, Jakob N. Foerster, and Shimon Whiteson. QMIX: monotonic value function factorisation for deep multi-agent rein- forcement learning. In Proceedings of the 35th Inter- national Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, pages 4292-4301, 2018.","title":"QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning","authors":"Tabish Rashid, Mikayel Samvelyan, Christian Schröder De Witt, Gregory Farquhar, Jakob N Foerster, Shimon Whiteson","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Song et al., 2016] Jinhua Song, Yang Gao, Hao Wang, and Bo An. Measuring the distance between finite Markov decision processes. In Proceedings of the 2016 Interna- tional Conference on Autonomous Agents and Multiagent Systems, pages 468-476, 2016.","title":"Measuring the distance between finite Markov decision processes","authors":"Jinhua Song, Yang Gao, Hao Wang, Bo An","year":"2016","venue":"Proceedings of the 2016 International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sukhbaatar et al., 2016] Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backprop- agation. In Advances in Neural Information Processing Systems, pages 2244-2252, 2016.","title":"Learning multiagent communication with backpropagation","authors":"Sukhbaatar","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sunehag et al., 2018] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning based on team reward. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pages 2085- 2087. International Foundation for Autonomous Agents and Multiagent Systems, 2018.","title":"Value-decomposition networks for cooperative multi-agent learning based on team reward","authors":"Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech, Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls","year":"2018","venue":"Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Watkins, 1989] C.J.C.H. Watkins. Learning from delayed rewards. PhD thesis, King's College, Cambridge, 1989.","title":"Learning from delayed rewards","authors":"C J C H Watkins","year":"1989","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yu et al., 2015] C Yu, M Zhang, F Ren, and G Tan. Mul- tiagent learning of coordination in loosely coupled mul- tiagent systems. IEEE Transactions on Cybernetics, 45(12):2853-2867, 2015.","title":"Multiagent learning of coordination in loosely coupled multiagent systems","authors":"M Yu, F Zhang, G Ren, Tan","year":"2015","venue":"IEEE Transactions on Cybernetics","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}