{"type":"ScholarlyArticle","title":"Multi-Agent Reinforcement Learning for Traffic Signal Control through Universal Communication Method","authors":[{"name":"Qize Jiang","slug":"qize-jiang","given":"Qize","family":"Jiang","affiliation":"School of Computer Science , Fudan University; Shanghai Key Laboratory of Data Science , Fudan University; Shanghai Institute of Intelligent Electronics & Systems"},{"name":"Minhao Qin","slug":"minhao-qin","given":"Minhao","family":"Qin","affiliation":"School of Computer Science , Fudan University; Shanghai Key Laboratory of Data Science , Fudan University; Shanghai Institute of Intelligent Electronics & Systems"},{"name":"Shengmin Shi","slug":"shengmin-shi","given":"Shengmin","family":"Shi","affiliation":"School of Computer Science , Fudan University; Shanghai Key Laboratory of Data Science , Fudan University; Shanghai Institute of Intelligent Electronics & Systems"},{"name":"Weiwei Sun","slug":"weiwei-sun","given":"Weiwei","family":"Sun","affiliation":"School of Computer Science , Fudan University; Shanghai Key Laboratory of Data Science , Fudan University; Shanghai Institute of Intelligent Electronics & Systems"},{"name":"Baihua Zheng","slug":"baihua-zheng","given":"Baihua","family":"Zheng","affiliation":"School of Computing and Information Systems , Singapore Management University"}],"abstract":"How to coordinate the communication among intersections effectively in real complex traffic scenarios with multi-intersection is challenging.  Existing approaches only enable the communication in a heuristic manner without considering the content/importance of information to be shared. In this paper, we propose a universal communication form UniComm between intersections. UniComm embeds massive observations collected at one agent into crucial predictions of their impact on its neighbors, which improves the communication efficiency and is universal across existing methods. We also propose a concise network UniLight to make full use of communications enabled by UniComm.  Experimental results on real datasets demonstrate that UniComm universally improves the performance of existing state-of-the-art methods, and UniLight significantly outperforms existing methods on a wide range of traffic situations. Source codes are available at https://github.com/zyr17/UniLight.","doi":"10.24963/ijcai.2022/535","url":"https://ijcai.org/proceedings/2022/535","pdf":"https://ijcai.org/proceedings/2022/0535.pdf","markdown":"https://ijcai.org/proceedings/2022/535.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/535","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Multidisciplinary Topics and Applications","track":"Main Track","number":535,"pages":{"first":3854,"last":3860},"keywords":["Multidisciplinary Topics and Applications: Transportation"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986662","length_sec":80,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984805","length_sec":327,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Arel et al., 2010] Itamar Arel, Cong Liu, Tom Urbanik, and Airton G Kohls. Reinforcement learning-based multi- agent system for network traffic signal control. IET In- telligent Transport Systems, 4(2):128-135, 2010.","title":"Reinforcement learning-based multiagent system for network traffic signal control","authors":"Itamar Arel, Cong Liu, Tom Urbanik, Airton G Kohls","year":"2010","venue":"IET Intelligent Transport Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ault et al., 2020] James Ault, Josiah P. Hanna, and Guni Sharon. Learning an interpretable traffic signal control policy. In AAMAS 2020, pages 88-96, 2020.","title":"Learning an interpretable traffic signal control policy","authors":"James Ault, Josiah P Hanna, Guni Sharon","year":"2020","venue":"AAMAS 2020","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bishop, 2006] Christopher M Bishop. Pattern recognition and machine learning. springer, 2006.","title":"Pattern recognition and machine learning","authors":"M Christopher, Bishop","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2020] Chacha Chen, Hua Wei, Nan Xu, Guanjie Zheng, Ming Yang, Yuanhao Xiong, Kai Xu, and Zhenhui Li. Toward a thousand lights: Decentralized deep rein- forcement learning for large-scale traffic signal control. In AAAI 2020, pages 3414-3421, 2020.","title":"Toward a thousand lights: Decentralized deep reinforcement learning for large-scale traffic signal control","authors":"Chacha Chen, Hua Wei, Nan Xu, Guanjie Zheng, Ming Yang, Yuanhao Xiong, Kai Xu, Zhenhui Li","year":"2020","venue":"AAAI 2020","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cools et al., 2013] Seung-Bae Cools, Carlos Gershenson, and Bart D'Hooghe. Self-Organizing Traffic Lights: A Re- alistic Simulation, pages 45-55. Springer London, Lon- don, 2013.","title":"Self-Organizing Traffic Lights: A Realistic Simulation","authors":"Seung-Bae Cools, Carlos Gershenson, Bart D' Hooghe","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[El-Tantawy et al., 2013] Samah El-Tantawy, Baher Abdul- hai, and Hossam Abdelgawad. Multiagent reinforcement learning for integrated network of adaptive traffic signal controllers (marlin-atsc): methodology and large-scale ap- plication on downtown toronto. IEEE Transactions on In- telligent Transportation Systems, 14(3):1140-1150, 2013.","title":"Multiagent reinforcement learning for integrated network of adaptive traffic signal controllers (marlin-atsc): methodology and large-scale application on downtown toronto","authors":"Samah El-Tantawy, Baher Abdulhai, Hossam Abdelgawad","year":"2013","venue":"IEEE Transactions on Intelligent Transportation Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In ICML 2018, pages 1856-1865, 2018.","title":"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor","authors":"Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine","year":"2018","venue":"ICML 2018","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jiang et al., 2021] Chun-Yao Jiang, Xiao-Min Hu, and Wei- Neng Chen. An urban traffic signal control system based on traffic flow prediction. In ICACI 2021, pages 259-265. IEEE, 2021.","title":"An urban traffic signal control system based on traffic flow prediction","authors":"Chun-Yao Jiang, Xiao-Min Hu, Wei-Neng Chen","year":"2021","venue":"ICACI 2021","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jiang et al., 2022] Qize Jiang, Minhao Qin, Shengmin Shi, Weiwei Sun, and Baihua Zheng. Multi-agent rein- forcement learning for traffic signal control through uni- versal communication method (long version). CoRR, abs/2204.12190, 2022.","title":"Multi-agent reinforcement learning for traffic signal control through universal communication method","authors":"Jiang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Little et al., 1981] John DC Little, Mark D Kelson, and Nathan H Gartner. Maxband: A versatile program for set- ting signals on arteries and triangular networks. Trans- portation Research Record Journal, 1981.","title":"Maxband: A versatile program for setting signals on arteries and triangular networks","authors":"John Dc Little, Nathan H Mark D Kelson, Gartner","year":"1981","venue":"Transportation Research Record Journal","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asyn- chronous methods for deep reinforcement learning. In ICML 2016, pages 1928-1937, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":"ICML 2016","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Nishi et al., 2018] Tomoki Nishi, Keisuke Otaki, Keiichiro Hayakawa, and Takayoshi Yoshimura. Traffic signal con- trol based on reinforcement learning with graph convolu- tional neural nets. In ITSC 2018, pages 877-883, 2018.","title":"Traffic signal control based on reinforcement learning with graph convolutional neural nets","authors":"Nishi","year":"2018","venue":"ITSC 2018","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Nowé et al., 2012] Ann Nowé, Peter Vrancx, and Yann- Michaël De Hauwere. Game theory and multi-agent re- inforcement learning. Springer, 2012.","title":"Game theory and multi-agent reinforcement learning","authors":"Ann Nowé, Peter Vrancx, Yann-Michaël De Hauwere","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Oliehoek et al., 2016] Frans A Oliehoek, Christopher Am- ato, et al. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016.","title":"A concise introduction to decentralized POMDPs","authors":"Oliehoek","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Oroojlooy et al., 2020] Afshin Oroojlooy, MohammadReza Nazari, Davood Hajinezhad, and Jorge Silva. Attendlight: Universal attention-based reinforcement learning model for traffic signal control. In NeurIPS 2020, 2020.","title":"Attendlight: Universal attention-based reinforcement learning model for traffic signal control","authors":"Afshin Oroojlooy, Mohammadreza Nazari, Davood Hajinezhad, Jorge Silva","year":"2020","venue":"NeurIPS 2020","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[van Hasselt et al., 2016] Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with dou- ble q-learning. In Dale Schuurmans and Michael P. Well- man, editors, AAAI 2016, pages 2094-2100, 2016.","title":"Deep reinforcement learning with double q-learning","authors":"Van Hasselt","year":"2016","venue":"AAAI 2016","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Varaiya, 2013] Pravin Varaiya. Max pressure control of a network of signalized intersections. Transportation Re- search Part C: Emerging Technologies, 36:177-195, 2013.","title":"Max pressure control of a network of signalized intersections","authors":"Pravin Varaiya","year":"2013","venue":"Transportation Research Part C: Emerging Technologies","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS 2017, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS 2017","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wei et al., 2019a] Hua Wei, Chacha Chen, Guanjie Zheng, Kan Wu, Vikash Gayah, Kai Xu, and Zhenhui Li. Presslight: Learning max pressure control to coordinate traffic signals in arterial network. In SIGKDD 2019, pages 1290-1298, 2019.","title":"Presslight: Learning max pressure control to coordinate traffic signals in arterial network","authors":"Wei","year":"2019","venue":"SIGKDD 2019","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wei et al., 2019b] Hua Wei, Nan Xu, Huichu Zhang, Guan- jie Zheng, Xinshi Zang, Chacha Chen, Weinan Zhang, Yanmin Zhu, Kai Xu, and Zhenhui Li. Colight: Learn- ing network-level cooperation for traffic signal control. In CIKM 2019, pages 1913-1922, 2019.","title":"Colight: Learning network-level cooperation for traffic signal control","authors":"Wei","year":"2019","venue":"CIKM 2019","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yu et al., 2020] Zhengxu Yu, Shuxian Liang, Long Wei, Zhongming Jin, Jianqiang Huang, Deng Cai, Xiaofei He, and Xian-Sheng Hua. Macar: Urban traffic light control via active multi-agent communication and action rectifica- tion. In IJCAI 2020, pages 2491-2497, 7 2020.","title":"Macar: Urban traffic light control via active multi-agent communication and action rectification","authors":"Yu","year":"2020","venue":"IJCAI 2020","doi":null,"url":null,"internal_url":"/proceedings/2020/345"},{"position":23,"raw":"[Zang et al., 2020] Xinshi Zang, Huaxiu Yao, Guanjie Zheng, Nan Xu, Kai Xu, and Zhenhui Li. Metalight: Value-based meta-reinforcement learning for traffic signal control. In AAAI 2020, pages 1153-1160, 2020.","title":"Metalight: Value-based meta-reinforcement learning for traffic signal control","authors":"Xinshi Zang, Huaxiu Yao, Guanjie Zheng, Nan Xu, Kai Xu, Zhenhui Li","year":"2020","venue":"AAAI 2020","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2019] Huichu Zhang, Siyuan Feng, Chang Liu, Yaoyao Ding, Yichen Zhu, Zihan Zhou, Weinan Zhang, Yong Yu, Haiming Jin, and Zhenhui Li. Cityflow: A multi-agent reinforcement learning environment for large scale city traffic scenario. In WWW 2019, pages 3620-3624, 2019.","title":"Cityflow: A multi-agent reinforcement learning environment for large scale city traffic scenario","authors":"Zhang","year":"2019","venue":"WWW 2019","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zheng et al., 2019a] Guanjie Zheng, Yuanhao Xiong, Xin- shi Zang, Jie Feng, Hua Wei, Huichu Zhang, Yong Li, Kai Xu, and Zhenhui Li. Learning phase competition for traffic signal control. In CIKM 2019, pages 1963-1972, 2019.","title":"Learning phase competition for traffic signal control","authors":"Zheng","year":"2019","venue":"CIKM 2019","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zheng et al., 2019b] Guanjie Zheng, Xinshi Zang, Nan Xu, Hua Wei, Zhengyao Yu, Vikash V. Gayah, Kai Xu, and Zhenhui Li. Diagnosing reinforcement learning for traffic signal control. CoRR, abs/1905.04716, 2019.","title":"Diagnosing reinforcement learning for traffic signal control","authors":"Zheng","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}