{"type":"ScholarlyArticle","title":"MaCAR: Urban Traffic Light Control via Active Multi-agent Communication and Action Rectification","authors":[{"name":"Zhengxu Yu","slug":"zhengxu-yu","given":"Zhengxu","family":"Yu","affiliation":"State Key Lab of CAD&CG , Zhejiang University , Hangzhou , China"},{"name":"Shuxian Liang","slug":"shuxian-liang","given":"Shuxian","family":"Liang","affiliation":"Computer Science Department , Zhejiang University , Hangzhou , China; DAMO Academy , Alibaba Group , Hangzhou , China"},{"name":"Long Wei","slug":"long-wei","given":"Long","family":"Wei","affiliation":"State Key Lab of CAD&CG , Zhejiang University , Hangzhou , China"},{"name":"Zhongming Jin","slug":"zhongming-jin","given":"Zhongming","family":"Jin","affiliation":"DAMO Academy , Alibaba Group , Hangzhou , China"},{"name":"Jianqiang Huang","slug":"jianqiang-huang","given":"Jianqiang","family":"Huang","affiliation":"DAMO Academy , Alibaba Group , Hangzhou , China"},{"name":"Deng Cai","slug":"deng-cai","given":"Deng","family":"Cai","affiliation":"State Key Lab of CAD&CG , Zhejiang University , Hangzhou , China"},{"name":"Xiaofei He","slug":"xiaofei-he","given":"Xiaofei","family":"He","affiliation":"State Key Lab of CAD&CG , Zhejiang University , Hangzhou , China; Fabu Inc. , Hangzhou , China"},{"name":"Xian-Sheng Hua","slug":"xian-sheng-hua","given":"Xian-Sheng","family":"Hua","affiliation":"DAMO Academy , Alibaba Group , Hangzhou , China"}],"abstract":"Urban traffic light control is an important and challenging real-world problem. By regarding intersections as agents, most of the Reinforcement Learning (RL) based methods generate actions of agents independently. They can cause action conflict and result in overflow or road resource waste in adjacent intersections. Recently, some collaborative methods have alleviated the above problems by extending the observable surroundings of agents, which can be considered as inactive cross-agent communication methods. However, when agents act synchronously in these works, the perceived action value is biased and the information exchanged is insufficient. In this work, we propose a novel Multi-agent Communication and Action Rectification (MaCAR) framework. It enables active communication between agents by considering the impact of synchronous actions of agents. MaCAR consists of two parts: (1) an active Communication Agent Network (CAN) involving a Message Propagation Graph Neural Network (MPGNN); (2) a Traffic Forecasting Network (TFN) which learns to predict the traffic after agents' synchronous actions and the corresponding action values. By using predicted information, we mitigate the action value bias during training to help rectify agents' future actions. In experiments, we show that our proposal can outperforms state-of-the-art methods on both synthetic and real-world datasets.","doi":"10.24963/ijcai.2020/345","url":"https://ijcai.org/proceedings/2020/345","pdf":"https://ijcai.org/proceedings/2020/0345.pdf","markdown":"https://ijcai.org/proceedings/2020/345.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/345","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":345,"pages":{"first":2491,"last":2497},"keywords":["Agent-based and Multi-agent Systems: Coordination and Cooperation","Machine Learning: Deep Reinforcement Learning","Machine Learning Applications: Applications of Reinforcement Learning","Multidisciplinary Topics and Applications: Transportation"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24374/file_24374.m3u8","length_sec":293,"provider":"hls"}],"references":[{"position":1,"raw":"[Arel et al., 2010] Itamar Arel, Cong Liu, Tom Urbanik, and Airton G Kohls. Reinforcement learning-based multi- agent system for network traffic signal control. IET In- telligent Transport Systems, 4(2):128-135, 2010.","title":"Reinforcement learning-based multiagent system for network traffic signal control","authors":"Itamar Arel, Cong Liu, Tom Urbanik, Airton G Kohls","year":"2010","venue":"IET Intelligent Transport Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Aslani et al., 2017] Mohammad Aslani, Mohammad Saadi Mesgari, and Marco Wiering. Adaptive traffic signal con- trol with actor-critic methods in a real-world traffic net- work with different traffic disruption events. Transporta- tion Research Part C: Emerging Technologies, 85:732- 752, 2017.","title":"Adaptive traffic signal control with actor-critic methods in a real-world traffic network with different traffic disruption events","authors":"Mohammad Aslani, Mohammad Saadi Mesgari, Marco Wiering","year":"2017","venue":"Transportation Research Part C: Emerging Technologies","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Battaglia et al., 2018] Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinícius Flores Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, C ¸aglar Gülc ¸ehre, H. Francis Song, Andrew J. Ballard, Justin Gilmer, George E. Dahl, Ashish Vaswani, Kelsey R. Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matthew Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. CoRR, abs/1806.01261, 2018.","title":"Relational inductive biases, deep learning, and graph networks","authors":"Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinícius Flores Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, C ¸aglar Gülc ¸ehre, H Francis Song, Andrew J Ballard, Justin Gilmer, George E Dahl, Ashish Vaswani, Kelsey R Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matthew Botvinick, Oriol Vinyals, Yujia Li, Razvan Pascanu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Casas, 2017] Noe Casas. Deep deterministic policy gra- dient for urban traffic light control. arXiv preprint arXiv:1703.09035, 2017.","title":"Deep deterministic policy gradient for urban traffic light control","authors":"Noe Casas","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2020] Chacha Chen, Hua Wei, Nan Xu, Guanjie Zheng, Ming Yang, Yuanhao Xiong, Kai Xu, and Zhen- hui. Toward a thousand lights: Decentralized deep rein- forcement learning for large-scale traffic signal control. In AAAI 2020, 2020.","title":"Toward a thousand lights: Decentralized deep reinforcement learning for large-scale traffic signal control","authors":"Chacha Chen, Hua Wei, Nan Xu, Guanjie Zheng, Ming Yang, Yuanhao Xiong, Kai Xu, Zhenhui","year":"2020","venue":"AAAI 2020","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chin et al., 2012] Yit Kwong Chin, Wei Yeang Kow, Wei Leong Khong, Min Keng Tan, and Kenneth Tze Kin Teo. Q-learning traffic signal optimization within multiple intersections traffic network. In 2012 Sixth UKSim/AMSS European Symposium on Computer Modeling and Simula- tion, pages 343-348. IEEE, 2012.","title":"Q-learning traffic signal optimization within multiple intersections traffic network","authors":"Yit Kwong, Chin, Wei Yeang Kow, Wei Leong Khong, Min Keng Tan, Kenneth Tze, Kin Teo","year":"2012","venue":"Sixth UKSim/AMSS European Symposium on Computer Modeling and Simulation","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chu et al., 2019] Tianshu Chu, Jie Wang, Lara Codecà, and Zhaojian Li. Multi-agent deep reinforcement learning for large-scale traffic signal control. IEEE Transactions on Intelligent Transportation Systems, 2019. Cools et al., 2013]","title":"Multi-agent deep reinforcement learning for large-scale traffic signal control","authors":"Tianshu Chu, Jie Wang, Lara Codecà, Zhaojian Li","year":"2013","venue":"IEEE Transactions on Intelligent Transportation Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"Seung-Bae Cools, Carlos Gershenson, and Bart D'Hooghe. Self-Organizing Traffic Lights: A Re- alistic Simulation, pages 45-55. Springer London, Lon- don, 2013.","title":"Self-Organizing Traffic Lights: A Realistic Simulation","authors":"Seung-Bae Cools, Carlos Gershenson, Bart D' Hooghe","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lee et al., 2019] Donghwan Lee, Niao He, Parameswaran Kamalaruban, and Volkan Cevher. Optimization for re- inforcement learning: From single agent to cooperative agents. arXiv preprint arXiv:1912.00498, 2019.","title":"Optimization for reinforcement learning: From single agent to cooperative agents","authors":"Lee","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2018] Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural net- work: Data-driven traffic forecasting. In International Conference on Learning Representations (ICLR '18), 2018.","title":"Diffusion convolutional recurrent neural network: Data-driven traffic forecasting","authors":"Yaguang Li, Rose Yu, Cyrus Shahabi, Yan Liu","year":"2018","venue":"International Conference on Learning Representations (ICLR '18)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Nishi et al., 2018] Tomoki Nishi, Keisuke Otaki, Keiichiro Hayakawa, and Takayoshi Yoshimura. Traffic signal con- trol based on reinforcement learning with graph convolu- tional neural nets. In 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pages 877- 883. IEEE, 2018.","title":"Traffic signal con-trol based on reinforcement learning with graph convolutional neural nets","authors":"Nishi","year":"2018","venue":"21st International Conference on Intelligent Transportation Systems (ITSC)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Van der Pol and Oliehoek, 2016] Elise Van der Pol and Frans A Oliehoek. Coordinated deep reinforcement learn- ers for traffic light control. Proceedings of Learning, Infer- ence and Control of Multi-Agent Systems (at NIPS 2016), 2016.","title":"Coordinated deep reinforcement learners for traffic light control","authors":"Pol Van Der, Elise Oliehoek, Frans A Van Der Pol, Oliehoek","year":"2016","venue":"Proceedings of Learning, Inference and Control of Multi-Agent Systems (at NIPS 2016)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Wei et al., 2018] Hua Wei, Guanjie Zheng, Huaxiu Yao, and Zhenhui Li. Intellilight: A reinforcement learning ap- proach for intelligent traffic light control. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2496-2505. ACM, 2018.","title":"Intellilight: A reinforcement learning approach for intelligent traffic light control","authors":"Hua Wei, Guanjie Zheng, Huaxiu Yao, Zhenhui Li","year":"2018","venue":"Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Wei et al., 2019a] Hua Wei, Chacha Chen, Guanjie Zheng, Kan Wu, Vikash Gayah, Kai Xu, and Zhenhui Li. Presslight: Learning max pressure control to coordinate traffic signals in arterial network. In Proceedings of the 25th ACM SIGKDD International Conference on Knowl- edge Discovery & Data Mining, KDD '19, pages 1290- 1298, 2019.","title":"Presslight: Learning max pressure control to coordinate traffic signals in arterial network","authors":"Wei","year":"2019","venue":"Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '19","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Wei et al., 2019b] Hua Wei, Nan Xu, Huichu Zhang, Guan- jie Zheng, Xinshi Zang, Chacha Chen, Weinan Zhang, Yamin Zhu, Kai Xu, and Zhenhui Li. Colight: Learning network-level cooperation for traffic signal control. In Pro- ceedings of the 28th ACM International Conference on In- formation and Knowledge Management, CIKM '19, 2019.","title":"Colight: Learning network-level cooperation for traffic signal control","authors":"Wei","year":"2019","venue":"Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM '19","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Wei et al., 2019c] L. Wei, Z. Yu, Z. Jin, L. Xie, J. Huang, D. Cai, X. He, and X. Hua. Dual graph for traffic forecast- ing. IEEE Access, pages 1-1, 2019.","title":"Dual graph for traffic forecasting","authors":"Wei","year":"2019","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Xu et al., 2019] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural net- works? In International Conference on Learning Rep- resentations, 2019.","title":"How powerful are graph neural networks","authors":"Xu","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Yan et al., 2018] Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton-based action recognition. In AAAI, 2018.","title":"Spatial temporal graph convolutional networks for skeleton-based action recognition","authors":"Sijie Yan, Yuanjun Xiong, Dahua Lin","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Yu et al., 2019] Bing Yu, Haoteng Yin, and Zhanxing Zhu. St-unet: A spatio-temporal u-network for graph-structured time series modeling. CoRR, abs/1903.05631, 2019.","title":"St-unet: A spatio-temporal u-network for graph-structured time series modeling","authors":"Yu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zheng et al., 2019] Guanjie Zheng, Xinshi Zang, Nan Xu, Hua Wei, Zhengyao Yu, Vikash V. Gayah, Kai Xu, and Zhenhui Li. Diagnosing reinforcement learning for traffic signal control. CoRR, abs/1905.04716, 2019.","title":"Diagnosing reinforcement learning for traffic signal control","authors":"Zheng","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhou et al., 2018] Jie Zhou, Ganqu Cui, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Graph neu- ral networks: A review of methods and applications. arXiv preprint arXiv:1812.08434, 2018.","title":"Graph neural networks: A review of methods and applications","authors":"Jie Zhou, Ganqu Cui, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Maosong Sun","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}