{"type":"ScholarlyArticle","title":"Goal Consistency: An Effective Multi-Agent Cooperative Method for Multistage Tasks","authors":[{"name":"Xinning Chen","slug":"xinning-chen","given":"Xinning","family":"Chen","affiliation":"College of Computer Science and Electronic Engineering , Hunan University , China"},{"name":"Xuan Liu","slug":"xuan-liu","given":"Xuan","family":"Liu","affiliation":"College of Computer Science and Electronic Engineering , Hunan University , China"},{"name":"Shigeng Zhang","slug":"shigeng-zhang","given":"Shigeng","family":"Zhang","affiliation":"School of Computer Science and Engineering , Central South University , China"},{"name":"Bo Ding","slug":"bo-ding","given":"Bo","family":"Ding","affiliation":"School of Computer Science , National University of Defense Technology , China"},{"name":"Kenli Li","slug":"kenli-li","given":"Kenli","family":"Li","affiliation":"College of Computer Science and Electronic Engineering , Hunan University , China"}],"abstract":"Although multistage tasks involving multiple sequential goals are common in real-world applications, they are not fully studied in multi-agent reinforcement learning (MARL). To accomplish a multi-stage task, agents have to achieve cooperation on different subtasks. Exploring the collaborative patterns of different subtasks and the sequence of completing the subtasks leads to an explosion in the search space, which poses great challenges to policy learning. Existing works designed for single-stage tasks where agents learn to cooperate only once usually suffer from low sample efficiency in multi-stage tasks as agents explore aimlessly. Inspired by human’s improving cooperation through goal consistency, we propose Multi-Agent Goal Consistency (MAGIC) framework to improve sample efficiency for learning in multi-stage tasks. MAGIC adopts a goal-oriented actor-critic model to learn both local and global views of goal cognition, which helps agents understand the task at the goal level so that they can conduct targeted exploration accordingly. Moreover, to improve exploration efficiency, MAGIC employs two-level goal consistency training to drive agents to formulate a consistent goal cognition. Experimental results show that MAGIC significantly improves sample efficiency and facilitates cooperation among agents compared with state-of-art MARL algorithms in several challenging multistage tasks.","doi":"10.24963/ijcai.2022/25","url":"https://ijcai.org/proceedings/2022/25","pdf":"https://ijcai.org/proceedings/2022/0025.pdf","markdown":"https://ijcai.org/proceedings/2022/25.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/25","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":25,"pages":{"first":172,"last":178},"keywords":["Agent-based and Multi-agent Systems: Coordination and Cooperation","Agent-based and Multi-agent Systems: Multi-agent Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986592","length_sec":90,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984735","length_sec":348,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Ding et al., 2020] Ziluo Ding, Tiejun Huang, and Zongqing Lu. Learning individually inferred communication for multi-agent cooperation. In Proceedings of Conference on Neural Information Processing Systems, 2020.","title":"Learning individually inferred communication for multi-agent cooperation","authors":"Ding","year":"2020","venue":"Proceedings of Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Gronauer and Diepold, 2021] S. Gronauer and K. Diepold. Multi-agent deep reinforcement learning: a survey. Artifi- cial Intelligence Review, pages 1-49, 2021.","title":"Multi-agent deep reinforcement learning: a survey","authors":"S Gronauer, K Diepold","year":"2021","venue":"Artificial Intelligence Review","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Hao et al., 2019] Xiaotian Hao, Weixun Wang, Jianye Hao, and Yaodong Yang. Independent generative adversarial self-imitation learning in cooperative multiagent systems. In Proceedings of the International Conference on Au- tonomous Agents and MultiAgent Systems, pages 1315- 1323, 2019.","title":"Independent generative adversarial self-imitation learning in cooperative multiagent systems","authors":"Xiaotian Hao, Weixun Wang, Jianye Hao, Yaodong Yang","year":"2019","venue":"Proceedings of the International Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Iqbal and Sha, 2019] Shariq Iqbal and Fei Sha. Actor- attention-critic for multi-agent reinforcement learning. In Proceedings of the 36th International Conference on Ma- chine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2961-2970, 2019.","title":"Actorattention-critic for multi-agent reinforcement learning","authors":"Shariq Iqbal, Fei Sha","year":"2019","venue":"Proceedings of the 36th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Jiang and Lu, 2018] Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent coop- eration. In Proceedings of Conference on Neural Informa- tion Processing Systems, pages 7265-7275, 2018.","title":"Learning attentional communication for multi-agent cooperation","authors":"Lu ; Jiang, Jiechuan Jiang, Zongqing Lu","year":"2018","venue":"Proceedings of Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Li et al., 2021a] Chenghao Li, Chengjie Wu, Tonghan Wang, Jun Yang, Qianchuan Zhao, and Chongjie Zhang. Celebrating diversity in shared multi-agent reinforcement learning. CoRR, abs/2106.02195, 2021.","title":"Celebrating diversity in shared multi-agent reinforcement learning","authors":"Li","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Li et al., 2021b] Wenhao Li, Xiangfeng Wang, Bo Jin, Junjie Sheng, Yun Hua, and Hongyuan Zha. Struc- tured diversification emergence via reinforced organiza- tion control and hierarchical consensus learning. CoRR, abs/2102.04775, 2021.","title":"Structured diversification emergence via reinforced organization control and hierarchical consensus learning","authors":"Li","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lillicrap et al., 2016] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yu- val Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016.","title":"Continuous control with deep reinforcement learning","authors":"Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, Daan Wierstra","year":"2016","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lin et al., 2021] Toru Lin, Minyoung Huh, Chris Stauffer, Ser-Nam Lim, and Phillip Isola. Learning to ground multi-agent communication with autoencoders. CoRR, abs/2110.15349, 2021.","title":"Learning to ground multi-agent communication with autoencoders","authors":"Toru Lin, Minyoung Huh, Chris Stauffer, Ser-Nam Lim, Phillip Isola","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liu et al., 2020] Yong Liu, Weixun Wang, Yujing Hu, Jianye Hao, Xingguo Chen, and Yang Gao. Multi-agent game abstraction via graph attention neural network. In Proceeding of Conference on Artificial Intelligence, pages 7211-7218, 2020.","title":"Multi-agent game abstraction via graph attention neural network","authors":"Yong Liu, Weixun Wang, Yujing Hu, Jianye Hao, Xingguo Chen, Yang Gao","year":"2020","venue":"Proceeding of Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2021] Iou-Jen Liu, Unnat Jain, Raymond A. Yeh, and Alexander G. Schwing. Cooperative exploration for multi-agent deep reinforcement learning. In Proceedings of the 38th International Conference on Machine Learn- ing, volume 139, pages 6826-6836, 2021.","title":"Cooperative exploration for multi-agent deep reinforcement learning","authors":"Iou-Jen Liu, Unnat Jain, Raymond A Yeh, Alexander G Schwing","year":"2021","venue":"Proceedings of the 38th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lowe et al., 2017] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environ- ments. In Proceeding of Neural Information Processing Systems, pages 6379-6390, 2017.","title":"Multi-agent actor-critic for mixed cooperative-competitive environments","authors":"Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, Igor Mordatch","year":"2017","venue":"Proceeding of Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mahajan et al., 2019] Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. MAVEN: multi-agent variational exploration. In Proceedings of Conference on Neural Information Processing Systems, pages 7611-7622, 2019.","title":"MAVEN: multi-agent variational exploration","authors":"Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, Shimon Whiteson","year":"2019","venue":"Proceedings of Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mao et al., 2019] Hangyu Mao, Zhengchao Zhang, Zhen Xiao, and Zhibo Gong. Modelling the dynamic joint pol- icy of teammates with attention multi-agent DDPG. In Proceedings of the 18th International Conference on Au- tonomous Agents and MultiAgent Systems, pages 1108- 1116, 2019.","title":"Modelling the dynamic joint policy of teammates with attention multi-agent DDPG","authors":"Mao","year":"2019","venue":"Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mao et al., 2020] Hangyu Mao, Wulong Liu, Jianye Hao, Jun Luo, Dong Li, Zhengchao Zhang, Jun Wang, and Zhen Xiao. Neighborhood cognition consistent multi-agent re- inforcement learning. In Proceeding of Conference on Ar- tificial Intelligence, pages 7219-7226. AAAI Press, 2020.","title":"Neighborhood cognition consistent multi-agent reinforcement learning","authors":"Mao","year":"2020","venue":"Proceeding of Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ryu et al., 2020] Heechang Ryu, Hayong Shin, and Jinkyoo Park. Multi-agent actor-critic with hierarchical graph at- tention network. In Proceeding of Conference on Artificial Intelligence, pages 7236-7243, 2020.","title":"Heechang Ryu, Hayong Shin, and Jinkyoo Park. Multi-agent actor-critic with hierarchical graph attention network","authors":"Ryu","year":"2020","venue":"Proceeding of Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sukhbaatar et al., 2016] Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. Learning multiagent com- munication with backpropagation. In Proceedings of Conference on Neural Information Processing Systems, pages 2244-2252, 2016.","title":"Learning multiagent communication with backpropagation","authors":"Sukhbaatar","year":"2016","venue":"Proceedings of Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Xu et al., 2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdi- nov, Richard S. Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual at- tention. In Proceedings of the 32nd International Confer- ence on Machine Learning, volume 37, pages 2048-2057, 2015.","title":"Show, attend and tell: Neural image caption generation with visual attention","authors":"Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C Courville, Ruslan Salakhutdinov, Richard S Zemel, Yoshua Bengio","year":"2015","venue":"Proceedings of the 32nd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Yang et al., 2018] Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi- agent reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning, pages 5567-5576, 2018.","title":"Mean field multiagent reinforcement learning","authors":"Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, Jun Wang","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yang et al., 2020] Jiachen Yang, Alireza Nakhaei, David Isele, Kikuo Fujimura, and Hongyuan Zha. CM3: coop- erative multi-goal multi-stage multi-agent reinforcement learning. In Proceeding of the 8th International Confer- ence on Learning Representations, 2020.","title":"CM3: cooperative multi-goal multi-stage multi-agent reinforcement learning","authors":"Yang","year":"2020","venue":"Proceeding of the 8th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}