{"type":"ScholarlyArticle","title":"STAR: Spatio-Temporal State Compression for Multi-Agent Tasks with Rich Observations","authors":[{"name":"Chao Li","slug":"chao-li","given":"Chao","family":"Li","affiliation":"State Key Laboratory for Novel Software Technology , Nanjing University"},{"name":"Yujing Hu","slug":"yujing-hu","given":"Yujing","family":"Hu","affiliation":"NetEase Fuxi AI Lab"},{"name":"Shangdong Yang","slug":"shangdong-yang","given":"Shangdong","family":"Yang","affiliation":"School of Computer Science , Nanjing University of Posts and Telecommunications; State Key Laboratory for Novel Software Technology , Nanjing University"},{"name":"Tangjie Lv","slug":"tangjie-lv","given":"Tangjie","family":"Lv","affiliation":"NetEase Fuxi AI Lab"},{"name":"Changjie Fan","slug":"changjie-fan","given":"Changjie","family":"Fan","affiliation":"NetEase Fuxi AI Lab"},{"name":"Wenbin Li","slug":"wenbin-li","given":"Wenbin","family":"Li","affiliation":"State Key Laboratory for Novel Software Technology , Nanjing University"},{"name":"Chongjie Zhang","slug":"chongjie-zhang","given":"Chongjie","family":"Zhang","affiliation":"Department of Computer Science & Engineering , Washington University in St. Louis"},{"name":"Yang Gao","slug":"yang-gao","given":"Yang","family":"Gao","affiliation":"State Key Laboratory for Novel Software Technology , Nanjing University"}],"abstract":"This paper focuses on the problem of learning compressed state representations for multi-agent tasks. Under the assumption of rich observation, we pinpoint that the state representations should be compressed both spatially and temporally to enable efficient prioritization of task-relevant features, while existing works typically fail. To overcome this limitation, we propose a novel method named Spatio-Temporal stAte compRession (STAR) that explicitly defines both spatial and temporal compression operations on the learned state representations to encode per-agent task-relevant features. Specifically, we first formalize this problem by introducing Task Informed Partially Observable Stochastic Game (TI-POSG). Then, we identify the spatial representation compression in it as encoding the latent states from the joint observations of all agents, and achieve this by learning representations that approximate the latent states based on the information theoretical principle. After that, we further extract the task-relevant features of each agent from these representations by aligning them based on their reward similarities, which is regarded as the temporal representation compression. Structurally, we implement these two compression by learning a set of agent-specific decoding functions and incorporate them into a critic shared by agents for scalable learning. We evaluate our method by developing decentralized policies on 12 maps of the StarCraft Multi-Agent Challenge benchmark, and the superior performance demonstrates its effectiveness.","doi":"10.24963/ijcai.2024/14","url":"https://ijcai.org/proceedings/2024/14","pdf":"https://ijcai.org/proceedings/2024/0014.pdf","markdown":"https://ijcai.org/proceedings/2024/14.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/14","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":14,"pages":{"first":120,"last":128},"keywords":["Machine Learning: ML: Reinforcement learning","Agent-based and Multi-agent Systems: MAS: Multi-agent learning"],"videos":[],"references":[{"position":1,"raw":"[Cao et al., 2012] Yongcan Cao, Wenwu Yu, Wei Ren, and Guanrong Chen. An overview of recent progress in the study of distributed multi-agent coordination. IEEE Trans- actions on Industrial informatics, 9(1):427-438, 2012.","title":"An overview of recent progress in the study of distributed multi-agent coordination","authors":"Cao","year":"2012","venue":"IEEE Transactions on Industrial informatics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Christianos et al., 2020] Filippos Christianos, Lukas Schäfer, and Stefano Albrecht. Shared experience actor- critic for multi-agent reinforcement learning. Advances in neural information processing systems, 33:10707-10717, 2020.","title":"Shared experience actorcritic for multi-agent reinforcement learning","authors":"Filippos Christianos, Lukas Schäfer, Stefano Albrecht","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[de Witt et al., 2020] Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.09533, 2020.","title":"Christian Schroeder de Witt","authors":"De Witt","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Du et al., 2019] Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik, and John Lang- ford. Provably efficient rl with rich observations via latent state decoding. In International Conference on Machine Learning, pages 1665-1674. PMLR, 2019.","title":"Provably efficient rl with rich observations via latent state decoding","authors":"Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik, John Langford","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ferns et al., 2004] Norm Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite markov decision pro- cesses. In UAI, volume 4, pages 162-169, 2004.","title":"Metrics for finite markov decision processes","authors":"Ferns","year":"2004","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Foerster et al., 2018] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelli- gence, volume 32, 2018.","title":"Counterfactual multi-agent policy gradients","authors":"Foerster","year":"2018","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fu et al., 2021] Xiang Fu, Ge Yang, Pulkit Agrawal, and Tommi Jaakkola. Learning task informed abstractions. In International Conference on Machine Learning, pages 3480-3491. PMLR, 2021.","title":"Learning task informed abstractions","authors":"Xiang Fu, Ge Yang, Pulkit Agrawal, Tommi Jaakkola","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gelada et al., 2019] Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Marc G Bellemare. Deep- mdp: Learning continuous latent space models for repre- sentation learning. In International Conference on Ma- chine Learning, pages 2170-2179. PMLR, 2019.","title":"Deepmdp: Learning continuous latent space models for representation learning","authors":"Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, Marc G Bellemare","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang et al., 2020] Yixin Huang, Shufan Wu, Zhongcheng Mu, Xiangyu Long, Sunhao Chu, and Guohong Zhao. A multi-agent reinforcement learning method for swarm robots in space collaborative exploration. In 2020 6th international conference on control, automation and robotics (ICCAR), pages 139-144. IEEE, 2020.","title":"A multi-agent reinforcement learning method for swarm robots in space collaborative exploration","authors":"Yixin Huang, Shufan Wu, Zhongcheng Mu, Xiangyu Long, Sunhao Chu, Guohong Zhao","year":"2020","venue":"2020 6th international conference on control, automation and robotics (ICCAR)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Iqbal and Sha, 2019] Shariq Iqbal and Fei Sha. Actor- attention-critic for multi-agent reinforcement learning. In International conference on machine learning, pages 2961-2970. PMLR, 2019.","title":"Actorattention-critic for multi-agent reinforcement learning","authors":"Sha ; Iqbal, Shariq Iqbal, Fei Sha","year":"2019","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lowe et al., 2017] Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-competitive en- vironments. Advances in Neural Information Processing Systems, 30, 2017.","title":"Multiagent actor-critic for mixed cooperative-competitive environments","authors":"Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, Pieter Openai, Igor Abbeel, Mordatch","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Osband et al., 2019] Ian Osband, Benjamin Van Roy, Daniel J Russo, and Zheng Wen. Deep exploration via randomized value functions. Journal of Machine Learn- ing Research, 20:1-62, 2019.","title":"Deep exploration via randomized value functions","authors":"Ian Osband, Benjamin Van Roy, Daniel J Russo, Zheng Wen","year":"2019","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Peng et al., 2021] Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Fac- mac: Factored multi-agent centralised policy gradients. Advances in Neural Information Processing Systems, 34:12208-12221, 2021.","title":"Facmac: Factored multi-agent centralised policy gradients","authors":"Bei Peng, Tabish Rashid, Christian Schroeder De Witt, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, Shimon Whiteson","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Rashid et al., 2020] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Fo- erster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. The Journal of Machine Learning Research, 21(1):7234- 7284, 2020.","title":"Monotonic value function factorisation for deep multi-agent reinforcement learning","authors":"Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De, Gregory Witt, Jakob Farquhar, Shimon Foerster, Whiteson","year":"2020","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Samvelyan et al., 2019] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. In International Con- ference on Autonomous Agents and MultiAgent Systems, pages 2186-2188, 2019.","title":"The starcraft multi-agent challenge","authors":"Tabish Mikayel Samvelyan, Christian Rashid, Gregory Schroeder De Witt, Nantas Farquhar, Tim Gj Nardelli, Chia-Man Rudner, Hung, Jakob Philip Hs Torr, Shimon Foerster, Whiteson","year":"2019","venue":"International Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Seo et al., 2021] Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State entropy maximization with random encoders for efficient explo- ration. In International Conference on Machine Learning, pages 9443-9454. PMLR, 2021.","title":"State entropy maximization with random encoders for efficient exploration","authors":"Seo","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sunehag et al., 2017] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.05296, 2017.","title":"Value-decomposition networks for cooperative multi-agent learning","authors":"Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech, Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tan, 1993] Ming Tan. Multi-agent reinforcement learn- ing: independent versus cooperative agents. In Interna- tional Conference on International Conference on Ma- chine Learning, pages 330-337, 1993.","title":"Multi-agent reinforcement learning: independent versus cooperative agents","authors":"Ming Tan","year":"1993","venue":"International Conference on International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tao et al., 2020] Ruo Yu Tao, Vincent Franc ¸ois-Lavet, and Joelle Pineau. Novelty search in representational space for sample efficient exploration. Advances in Neural Informa- tion Processing Systems, 33:8114-8126, 2020.","title":"Novelty search in representational space for sample efficient exploration","authors":"Ruo Yu Tao, Vincent Franc ¸ois-Lavet, Joelle Pineau","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tishby et al., 2000] Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.","title":"The information bottleneck method","authors":"Naftali Tishby, Fernando C Pereira, William Bialek","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2020a] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning. arXiv preprint arXiv:2008.01062, 2020.","title":"Qplex: Duplex dueling multi-agent q-learning","authors":"Wang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2020b] Xiaoqiang Wang, Liangjun Ke, Zhimin Qiao, and Xinghua Chai. Large-scale traffic signal control using a novel multiagent reinforcement learning. IEEE transactions on cybernetics, 51(1):174-187, 2020.","title":"Large-scale traffic signal control using a novel multiagent reinforcement learning","authors":"Wang","year":"2020","venue":"IEEE transactions on cybernetics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2020c] Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang. Dop: Off-policy multi-agent decomposed policy gradients. In International conference on learning representations, 2020.","title":"Dop: Off-policy multi-agent decomposed policy gradients","authors":"Wang","year":"2020","venue":"International conference on learning representations","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yu et al., 2022] Chao Yu, Akash Velu, Eugene Vinitsky, Ji- axuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Sys- tems, 35:24611-24624, 2022.","title":"The surprising effectiveness of ppo in cooperative multi-agent games","authors":"Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, Yi Wu","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang and Lesser, 2011] Chongjie Zhang and Victor Lesser. Coordinated multi-agent reinforcement learning in networked distributed pomdps. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 25, pages 764-770, 2011.","title":"Chongjie Zhang and Victor Lesser. Coordinated multi-agent reinforcement learning in networked distributed pomdps","authors":"Lesser Zhang","year":"2011","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2020] Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learn- ing invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006.10742, 2020.","title":"Learning invariant representations for reinforcement learning without reconstruction","authors":"Amy Zhang, Rowan Mcallister, Roberto Calandra, Yarin Gal, Sergey Levine","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}