{"type":"ScholarlyArticle","title":"GPLight: Grouped Multi-agent Reinforcement Learning for Large-scale Traffic Signal Control","authors":[{"name":"Yilin Liu","slug":"yilin-liu","given":"Yilin","family":"Liu","affiliation":"State Key Laboratory of Networking and Switching Technology , Beijing University of Posts and Telecommunications , Beijing 100876 , China; State Key Laboratory of Integrated Services Networks , Xidian University , Xi'an 710126 , China"},{"name":"Guiyang Luo","slug":"guiyang-luo","given":"Guiyang","family":"Luo","affiliation":"State Key Laboratory of Networking and Switching Technology , Beijing University of Posts and Telecommunications , Beijing 100876 , China"},{"name":"Quan Yuan","slug":"quan-yuan","given":"Quan","family":"Yuan","affiliation":"State Key Laboratory of Networking and Switching Technology , Beijing University of Posts and Telecommunications , Beijing 100876 , China; State Key Laboratory of Integrated Services Networks , Xidian University , Xi'an 710126 , China"},{"name":"Jinglin Li","slug":"jinglin-li","given":"Jinglin","family":"Li","affiliation":"State Key Laboratory of Networking and Switching Technology , Beijing University of Posts and Telecommunications , Beijing 100876 , China"},{"name":"Lei Jin","slug":"lei-jin","given":"Lei","family":"Jin","affiliation":"School of Electronic Engineering , Beijing University of Posts and Telecommunications , Beijing 100876 , China"},{"name":"Bo Chen","slug":"bo-chen","given":"Bo","family":"Chen","affiliation":"State Key Laboratory of Networking and Switching Technology , Beijing University of Posts and Telecommunications , Beijing 100876 , China"},{"name":"Rui Pan","slug":"rui-pan","given":"Rui","family":"Pan","affiliation":"State Key Laboratory of Networking and Switching Technology , Beijing University of Posts and Telecommunications , Beijing 100876 , China"}],"abstract":"The use of multi-agent reinforcement learning (MARL) methods in coordinating traffic lights (CTL) has become increasingly popular, treating each intersection as an agent. However, existing MARL approaches either treat each agent absolutely homogeneous, i.e., same network and parameter for each agent, or treat each agent completely heterogeneous, i.e., different networks and parameters for each agent. This creates a difficult balance between accuracy and complexity, especially in large-scale CTL. To address this challenge, we propose a grouped MARL method named GPLight. We first mine the similarity between agent environment considering both real-time traffic flow and static fine-grained road topology. Then we propose two loss functions to maintain a learnable and dynamic clustering, one that uses mutual information estimation for better stability, and the other that maximizes separability between groups. Finally, GPLight enforces the agents in a group to share the same network and parameters. This approach reduces complexity by promoting cooperation within the same group of agents while reflecting differences between groups to ensure accuracy. To verify the effectiveness of our method, we conduct experiments on both synthetic and real-world datasets, with up to 1,089 intersections. Compared with state-of-the-art methods, experiment results demonstrate the superiority of our proposed method, especially in large-scale CTL.","doi":"10.24963/ijcai.2023/23","url":"https://ijcai.org/proceedings/2023/23","pdf":"https://ijcai.org/proceedings/2023/0023.pdf","markdown":"https://ijcai.org/proceedings/2023/23.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/23","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":23,"pages":{"first":199,"last":207},"keywords":["Machine Learning: ML: Deep reinforcement learning","Agent-based and Multi-agent Systems: MAS: Applications"],"videos":[],"references":[{"position":1,"raw":"[Alemi et al., 2016] Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410, 2016.","title":"Deep variational information bottleneck","authors":"Ian Alexander A Alemi, Joshua V Fischer, Kevin Dillon, Murphy","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2020] Chacha Chen, Hua Wei, Nan Xu, Guanjie Zheng, Ming Yang, Yuanhao Xiong, Kai Xu, and Zhenhui Li. Toward a thousand lights: Decentralized deep rein- forcement learning for large-scale traffic signal control. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 34, pages 3414-3421, 2020.","title":"Toward a thousand lights: Decentralized deep reinforcement learning for large-scale traffic signal control","authors":"Chacha Chen, Hua Wei, Nan Xu, Guanjie Zheng, Ming Yang, Yuanhao Xiong, Kai Xu, Zhenhui Li","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.","title":"Learning phrase representations using rnn encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Jamil et al., 2020] Abu Rafe Md Jamil, Kishan Kumar Gan- guly, and Naushin Nower. Adaptive traffic signal control system using composite reward architecture based deep re- inforcement learning. IET Intelligent Transport Systems, 14(14):2030-2041, 2020.","title":"Adaptive traffic signal control system using composite reward architecture based deep reinforcement learning","authors":"Jamil","year":"2020","venue":"IET Intelligent Transport Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Jiang et al., 2021] Qize Jiang, Jingze Li, Weiwei Sun SUN, and Baihua Zheng. Dynamic lane traffic signal control with group attention and multi-timescale reinforcement learning. IJCAI, 2021.","title":"Dynamic lane traffic signal control with group attention and multi-timescale reinforcement learning","authors":"Jiang","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/501"},{"position":6,"raw":"[Kipf and Welling, 2016] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.","title":"Semi-supervised classification with graph convolutional networks","authors":"N Thomas, Max Kipf, Welling","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Koonce and Rodegerdts, 2008] Peter Koonce and Lee Rodegerdts. Traffic signal timing manual. Technical report, United States. Federal Highway Administration, 2008.","title":"Traffic signal timing manual","authors":"Rodegerdts Koonce, Peter Koonce, Lee Rodegerdts","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lhaksmana et al., 2018] Kemas M Lhaksmana, Yohei Mu- rakami, and Toru Ishida. Role-based modeling for design- ing agent behavior in self-organizing multi-agent systems. International Journal of Software Engineering and Knowl- edge Engineering, 28(01):79-96, 2018.","title":"Role-based modeling for designing agent behavior in self-organizing multi-agent systems","authors":"Yohei Kemas M Lhaksmana, Toru Murakami, Ishida","year":"2018","venue":"International Journal of Software Engineering and Knowledge Engineering","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Liang et al., 2022] Enming Liang, Zicheng Su, Chilin Fang, and Renxin Zhong. Oam: An option-action reinforcement learning framework for universal multi-intersection con- trol. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 4550-4558, 2022.","title":"Oam: An option-action reinforcement learning framework for universal multi-intersection control","authors":"Liang","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lopez et al., 2018] Pablo Alvarez Lopez, Michael Behrisch, Laura Bieker-Walz, Jakob Erdmann, Yun- Pang Flötteröd, Robert Hilbrich, Leonhard Lücken, Johannes Rummel, Peter Wagner, and Evamarie Wießner. Microscopic traffic simulation using sumo. In 2018 21st international conference on intelligent transportation systems (ITSC), pages 2575-2582. IEEE, 2018.","title":"Microscopic traffic simulation using sumo","authors":"Lopez","year":"2018","venue":"2018 21st international conference on intelligent transportation systems (ITSC)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Luo et al., 2020] Guiyang Luo, Hui Zhang, Haibo He, Jinglin Li, and Fei-Yue Wang. Multiagent adversarial col- laborative learning via mean-field theory. IEEE Transac- tions on Cybernetics, 51(10):4994-5007, 2020.","title":"Multiagent adversarial collaborative learning via mean-field theory","authors":"Guiyang Luo, Hui Zhang, Haibo He, Jinglin Li, Fei-Yue Wang","year":"2020","venue":"IEEE Transactions on Cybernetics","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Luo et al., 2022a] Guiyang Luo, Hui Zhang, Xiao Wang, Quan Yuan, Jinglin Li, and Fei-Yue Wang. Acp based large-scale coordinated route planning: From perspective of cyber-physical-social systems. In 2022 IEEE 25th Inter- national Conference on Intelligent Transportation Systems (ITSC), pages 1842-1849. IEEE, 2022.","title":"Acp based large-scale coordinated route planning: From perspective of cyber-physical-social systems","authors":"Luo","year":"2022","venue":"IEEE 25th International Conference on Intelligent Transportation Systems (ITSC)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Luo et al., 2022b] Guiyang Luo, Hui Zhang, Quan Yuan, Jinglin Li, and Fei-Yue Wang. Estnet: embedded spatial- temporal network for modeling traffic flow dynamics. IEEE transactions on intelligent transportation systems, 23(10):19201-19212, 2022.","title":"Estnet: embedded spatialtemporal network for modeling traffic flow dynamics","authors":"Luo","year":"2022","venue":"IEEE transactions on intelligent transportation systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Luo et al., 2023] Guiyang Luo, Hui Zhang, Quan Yuan, Jinglin Li, Wendong Wang, and Fei-Yue Wang. One size fits all: A unified traffic predictor for capturing the es- sential spatial-temporal dependency. IEEE Transactions on Neural Networks and Learning Systems, pages 1-15, 2023.","title":"One size fits all: A unified traffic predictor for capturing the essential spatial-temporal dependency","authors":"Guiyang Luo, Hui Zhang, Quan Yuan, Jinglin Li, Wendong Wang, Fei-Yue Wang","year":"2023","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Oliehoek and Amato, 2016] Frans A Oliehoek and Christo- pher Amato. A concise introduction to decentralized POMDPs. Springer, 2016.","title":"A concise introduction to decentralized POMDPs","authors":"A Frans, Christopher Oliehoek, Amato","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rashid et al., 2018] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pages 4295-4304. PMLR, 2018.","title":"Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning","authors":"Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, Shimon Whiteson","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Smith, 1937] Adam Smith. The wealth of nations [1776], volume 11937. na, 1937.","title":"The wealth of nations","authors":"Adam Smith","year":"1776","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Su et al., 2022] Haoran Su, Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Emvlight: A de- centralized reinforcement learning framework for efficient passage of emergency vehicles. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 4593-4601, 2022.","title":"Emvlight: A decentralized reinforcement learning framework for efficient passage of emergency vehicles","authors":"Haoran Su, Yaofeng Desmond Zhong, Biswadip Dey, Amit Chakraborty","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Varaiya, 2013] Pravin Varaiya. The max-pressure controller for arbitrary networks of signalized intersections. In Ad- vances in dynamic network modeling in complex trans- portation systems, pages 27-66. Springer, 2013.","title":"The max-pressure controller for arbitrary networks of signalized intersections","authors":"Pravin Varaiya","year":"2013","venue":"Advances in dynamic network modeling in complex transportation systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Velickovic et al., 2017] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. stat, 1050:20, 2017.","title":"Graph attention networks","authors":"Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Bengio","year":"2017","venue":"stat","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wainwright and Jordan, 2008] Martin J Wainwright and Michael Irwin Jordan. Graphical models, exponential fam- ilies, and variational inference. Now Publishers Inc, 2008.","title":"Graphical models, exponential families, and variational inference","authors":"J Martin, Michael Wainwright, Jordan Irwin","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wei et al., 2018] Hua Wei, Guanjie Zheng, Huaxiu Yao, and Zhenhui Li. Intellilight: A reinforcement learning ap- proach for intelligent traffic light control. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2496-2505, 2018.","title":"Intellilight: A reinforcement learning approach for intelligent traffic light control","authors":"Hua Wei, Guanjie Zheng, Huaxiu Yao, Zhenhui Li","year":"2018","venue":"Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wei et al., 2019a] Hua Wei, Chacha Chen, Guanjie Zheng, Kan Wu, Vikash Gayah, Kai Xu, and Zhenhui Li. Presslight: Learning max pressure control to coordinate traffic signals in arterial network. In Proceedings of the 25th ACM SIGKDD International Conference on Knowl- edge Discovery & Data Mining, pages 1290-1298, 2019.","title":"Presslight: Learning max pressure control to coordinate traffic signals in arterial network","authors":"Wei","year":"2019","venue":"Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wei et al., 2019b] Hua Wei, Nan Xu, Huichu Zhang, Guan- jie Zheng, Xinshi Zang, Chacha Chen, Weinan Zhang, Yanmin Zhu, Kai Xu, and Zhenhui Li. Colight: Learn- ing network-level cooperation for traffic signal control. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 1913- 1922, 2019.","title":"Colight: Learning network-level cooperation for traffic signal control","authors":"Wei","year":"2019","venue":"Proceedings of the 28th ACM International Conference on Information and Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wei et al., 2019c] Hua Wei, Guanjie Zheng, Vikash Gayah, and Zhenhui Li. A survey on traffic signal control meth- ods. arXiv preprint arXiv:1904.08117, 2019.","title":"A survey on traffic signal control methods","authors":"Wei","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Ye et al., 2021] Yutong Ye, Wupan Zhao, Tongquan Wei, Shiyan Hu, and Mingsong Chen. Fedlight: Federated rein- forcement learning for autonomous multi-intersection traf- fic signal control. In 2021 58th ACM/IEEE Design Au- tomation Conference (DAC), pages 847-852. IEEE, 2021.","title":"Fedlight: Federated reinforcement learning for autonomous multi-intersection traffic signal control","authors":"Yutong Ye, Wupan Zhao, Tongquan Wei, Shiyan Hu, Mingsong Chen","year":"2021","venue":"58th ACM/IEEE Design Automation Conference (DAC)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yu et al., 2021] Zhengxu Yu, Shuxian Liang, Long Wei, Zhongming Jin, Jianqiang Huang, Deng Cai, Xiaofei He, and Xian-Sheng Hua. Macar: Urban traffic light control via active multi-agent communication and action rectifi- cation. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artifi- cial Intelligence, pages 2491-2497, 2021.","title":"Macar: Urban traffic light control via active multi-agent communication and action rectification","authors":"Yu","year":"2021","venue":"Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2020/345"},{"position":28,"raw":"[Zhang et al., 2019] Huichu Zhang, Siyuan Feng, Chang Liu, Yaoyao Ding, Yichen Zhu, Zihan Zhou, Weinan Zhang, Yong Yu, Haiming Jin, and Zhenhui Li. Cityflow: A multi-agent reinforcement learning environment for large scale city traffic scenario. In The world wide web conference, pages 3620-3624, 2019.","title":"Cityflow: A multi-agent reinforcement learning environment for large scale city traffic scenario","authors":"Zhang","year":"2019","venue":"The world wide web conference","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2020] Huichu Zhang, Chang Liu, Weinan Zhang, Guanjie Zheng, and Yong Yu. Generalight: Im- proving environment generalization of traffic signal con- trol via meta reinforcement learning. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 1783-1792, 2020.","title":"Generalight: Improving environment generalization of traffic signal control via meta reinforcement learning","authors":"Huichu Zhang, Chang Liu, Weinan Zhang, Guanjie Zheng, Yong Yu","year":"2020","venue":"Proceedings of the 29th ACM International Conference on Information & Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zheng et al., 2019] Guanjie Zheng, Yuanhao Xiong, Xinshi Zang, Jie Feng, Hua Wei, Huichu Zhang, Yong Li, Kai Xu, and Zhenhui Li. Learning phase competition for traffic sig- nal control. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 1963-1972, 2019.","title":"Learning phase competition for traffic signal control","authors":"Zheng","year":"2019","venue":"Proceedings of the 28th ACM International Conference on Information and Knowledge Management","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}