{"type":"ScholarlyArticle","title":"Reinforcement Learning Approaches for Traffic Signal Control under Missing Data","authors":[{"name":"Hao Mei","slug":"hao-mei","given":"Hao","family":"Mei","affiliation":"New Jersey Institute of Technology"},{"name":"Junxian Li","slug":"junxian-li","given":"Junxian","family":"Li","affiliation":"Xi'an Jiaotong University"},{"name":"Bin Shi","slug":"bin-shi","given":"Bin","family":"Shi","affiliation":"Xi'an Jiaotong University"},{"name":"Hua Wei","slug":"hua-wei","given":"Hua","family":"Wei","affiliation":"New Jersey Institute of Technology; Arizona State University"}],"abstract":"The emergence of reinforcement learning (RL) methods in traffic signal control (TSC) tasks has achieved promising results. Most RL approaches require the observation of the environment for the agent to decide which action is optimal for a long-term reward. However, in real-world urban scenarios, missing observation of traffic states may frequently occur due to the lack of sensors, which makes existing RL methods inapplicable on road networks with missing observation. In this work, we aim to control the traffic signals in a real-world setting, where some of the intersections in the road network are not installed with sensors and thus with no direct observations around them. To the best of our knowledge, we are the first to use RL methods to tackle the TSC problem in this real-world setting. Specifically, we propose two solutions: 1) imputes the traffic states to enable adaptive control. 2) imputes both states and rewards to enable adaptive control and the training of RL agents. Through extensive experiments on both synthetic and real-world road network traffic, we reveal that our method outperforms conventional approaches and performs consistently with different missing rates. We also investigate how missing data influences the performance of our model.","doi":"10.24963/ijcai.2023/251","url":"https://ijcai.org/proceedings/2023/251","pdf":"https://ijcai.org/proceedings/2023/0251.pdf","markdown":"https://ijcai.org/proceedings/2023/251.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/251","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Data Mining","track":"Main Track","number":251,"pages":{"first":2261,"last":2269},"keywords":["Agent-based and Multi-agent Systems: MAS: Multi-agent learning","Machine Learning: ML: Reinforcement learning","Data Mining: DM: Mining spatial and/or temporal data"],"videos":[],"references":[{"position":1,"raw":"[Aboudolas et al., 2009] K. Aboudolas, M. Papageorgiou, and E. Kosmatopoulos. Store-and-forward based meth- ods for the signal control problem in large-scale congested urban road networks. Transportation Research Part C: Emerging Technologies, 17(2):163-174, 2009. Selected papers from the Sixth Triennial Symposium on Trans- portation Analysis (TRISTAN VI).","title":"Store-and-forward based methods for the signal control problem in large-scale congested urban road networks","authors":"K Aboudolas, M Papageorgiou, E Kosmatopoulos","year":"2009","venue":"Sixth Triennial Symposium on Transportation Analysis (TRISTAN VI)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arel et al., 2010] Itamar Arel, Cong Liu, Tom Urbanik, and Airton G Kohls. Reinforcement learning-based multi- agent system for network traffic signal control. IET In- telligent Transport Systems, 4(2):128-135, 2010.","title":"Reinforcement learning-based multiagent system for network traffic signal control","authors":"Itamar Arel, Cong Liu, Tom Urbanik, Airton G Kohls","year":"2010","venue":"IET Intelligent Transport Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen and Jiang, 2019] Jinglin Chen and Nan Jiang. Information-theoretic considerations in batch reinforce- ment learning. In International Conference on Machine Learning, pages 1042-1051. PMLR, 2019.","title":"Information-theoretic considerations in batch reinforcement learning","authors":"Jinglin Chen, Nan Jiang","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2019] Xinyu Chen, Zhaocheng He, and Lijun Sun. A bayesian tensor decomposition approach for spa- tiotemporal traffic data imputation. Transportation Re- search Part C: Emerging Technologies, 98:73-84, 2019.","title":"A bayesian tensor decomposition approach for spatiotemporal traffic data imputation","authors":"Xinyu Chen, Zhaocheng He, Lijun Sun","year":"2019","venue":"Transportation Research Part C: Emerging Technologies","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2020] Chacha Chen, Hua Wei, Nan Xu, Guanjie Zheng, Ming Yang, Yuanhao Xiong, Kai Xu, and Zhenhui Li. Toward a thousand lights: Decentralized deep rein- forcement learning for large-scale traffic signal control. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 34, pages 3414-3421, 2020.","title":"Toward a thousand lights: Decentralized deep reinforcement learning for large-scale traffic signal control","authors":"Chacha Chen, Hua Wei, Nan Xu, Guanjie Zheng, Ming Yang, Yuanhao Xiong, Kai Xu, Zhenhui Li","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cui et al., 2020] Zhiyong Cui, Ruimin Ke, Ziyuan Pu, and Yinhai Wang. Stacked bidirectional and unidirectional lstm recurrent neural network for forecasting network- wide traffic state with missing values. Transportation Re- search Part C: Emerging Technologies, 118:102674, 2020.","title":"Stacked bidirectional and unidirectional lstm recurrent neural network for forecasting networkwide traffic state with missing values","authors":"Zhiyong Cui, Ruimin Ke, Ziyuan Pu, Yinhai Wang","year":"2020","venue":"Transportation Research Part C: Emerging Technologies","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Duan et al., 2016] Yanjie Duan, Yisheng Lv, Yu-Liang Liu, and Fei-Yue Wang. An efficient realization of deep learn- ing for traffic data imputation. Transportation Research Part C: Emerging Technologies, 72:168-181, 2016.","title":"An efficient realization of deep learning for traffic data imputation","authors":"Yanjie Duan, Yisheng Lv, Yu-Liang Liu, Fei-Yue Wang","year":"2016","venue":"Transportation Research Part C: Emerging Technologies","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[El-Tantawy et al., 2013] Samah El-Tantawy, Baher Abdul- hai, and Hossam Abdelgawad. Multiagent reinforcement learning for integrated network of adaptive traffic signal controllers (marlin-atsc): methodology and large-scale ap- plication on downtown toronto. IEEE Transactions on In- telligent Transportation Systems, 14(3):1140-1150, 2013.","title":"Multiagent reinforcement learning for integrated network of adaptive traffic signal controllers (marlin-atsc): methodology and large-scale application on downtown toronto","authors":"Samah El-Tantawy, Baher Abdulhai, Hossam Abdelgawad","year":"2013","venue":"IEEE Transactions on Intelligent Transportation Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gan et al., 2015] Min Gan, C. L. Philip Chen, Han-Xiong Li, and Long Chen. Gradient radial basis function based varying-coefficient autoregressive model for nonlinear and nonstationary time series. IEEE Signal Processing Letters, 22(7):809-812, 2015.","title":"Gradient radial basis function based varying-coefficient autoregressive model for nonlinear and nonstationary time series","authors":"Gan","year":"2015","venue":"IEEE Signal Processing Letters","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2021] Xingshuai Huang, Di Wu, Michael Jenkin, and Benoit Boulet. Modellight: Model-based meta-reinforcement learning for traffic signal control. arXiv preprint arXiv:2111.08067, 2021.","title":"Modellight: Model-based meta-reinforcement learning for traffic signal control","authors":"Xingshuai Huang, Di Wu, Michael Jenkin, Benoit Boulet","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[HUNT, 1981] PB HUNT. Scoot-a traffic responsive method of coordinating signals. TRRL Laboratory Report, 1014, 1981.","title":"Scoot-a traffic responsive method of coordinating signals","authors":"Hunt ; Pb Hunt","year":"1981","venue":"TRRL Laboratory Report","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Koonce and Rodegerdts, 2008] Peter Koonce and Lee Rodegerdts. Traffic signal timing manual. Technical report, United States. Federal Highway Administration, 2008.","title":"Traffic signal timing manual","authors":"Rodegerdts Koonce, Peter Koonce, Lee Rodegerdts","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2020] Xiaoshuang Li, Zhongzheng Guo, Xingyuan Dai, Yilun Lin, Junchen Jin, Fenghua Zhu, and Fei-Yue Wang. Deep imitation learning for traffic signal control and operations based on graph convolutional neural networks. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pages 1-6. IEEE, 2020.","title":"Deep imitation learning for traffic signal control and operations based on graph convolutional neural networks","authors":"Li","year":"2020","venue":"IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Luo et al., 2022] Fan-Ming Luo, Tian Xu, Hang Lai, Xiong-Hui Chen, Weinan Zhang, and Yang Yu. A survey on model-based reinforcement learning. arXiv preprint arXiv:2206.09328, 2022.","title":"A survey on model-based reinforcement learning","authors":"Fan-Ming Luo, Tian Xu, Hang Lai, Xiong-Hui Chen, Weinan Zhang, Yang Yu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lv et al., 2015] Yisheng Lv, Yanjie Duan, Wenwen Kang, Zhengxi Li, and Fei-Yue Wang. Traffic flow prediction with big data: A deep learning approach. IEEE Trans- actions on Intelligent Transportation Systems, 16(2):865- 873, 2015.","title":"Traffic flow prediction with big data: A deep learning approach","authors":"Yisheng Lv, Yanjie Duan, Wenwen Kang, Zhengxi Li, Fei-Yue Wang","year":"2015","venue":"IEEE Transactions on Intelligent Transportation Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mai et al., 2019] Tien Mai, Quoc Phong Nguyen, Kian Hsiang Low, and Patrick Jaillet. Inverse rein- forcement learning with missing data. arXiv preprint arXiv:1911.06930, 2019.","title":"Inverse reinforcement learning with missing data","authors":"Tien Mai, Quoc Phong Nguyen, Kian Hsiang Low, Patrick Jaillet","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Oroojlooy et al., 2020] Afshin Oroojlooy, Mohammadreza Nazari, Davood Hajinezhad, and Jorge Silva. Attendlight: Universal attention-based reinforcement learning model for traffic signal control. Advances in Neural Information Processing Systems, 33:4079-4090, 2020.","title":"Attendlight: Universal attention-based reinforcement learning model for traffic signal control","authors":"Afshin Oroojlooy, Mohammadreza Nazari, Davood Hajinezhad, Jorge Silva","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sims and Dobinson, 1980] Arthur G Sims and Kenneth W Dobinson. The sydney coordinated adaptive traffic (scat) system philosophy and benefits. IEEE Transactions on ve- hicular technology, 29(2):130-137, 1980.","title":"The sydney coordinated adaptive traffic (scat) system philosophy and benefits","authors":"G Arthur, Kenneth W Sims, Dobinson","year":"1980","venue":"IEEE Transactions on vehicular technology","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sutton, 1991] Richard S Sutton. Dyna, an integrated archi- tecture for learning, planning, and reacting. ACM Sigart Bulletin, 2(4):160-163, 1991.","title":"Dyna, an integrated architecture for learning, planning, and reacting","authors":"Richard S Sutton","year":"1991","venue":"ACM Sigart Bulletin","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tirone, 2022] Onathan Tirone. Congestion pricing, the route more cities are taking. Bloomberg, 2022.","title":"Congestion pricing, the route more cities are taking","authors":"Onathan Tirone","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Varaiya, 2013] Pravin Varaiya. The max-pressure controller for arbitrary networks of signalized intersections. In Ad- vances in dynamic network modeling in complex trans- portation systems, pages 27-66. Springer, 2013.","title":"The max-pressure controller for arbitrary networks of signalized intersections","authors":"Pravin Varaiya","year":"2013","venue":"Advances in dynamic network modeling in complex transportation systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2022] Peixiao Wang, Tong Zhang, Yueming Zheng, and Tao Hu. A multi-view bidirectional spatiotem- poral graph network for urban traffic flow imputation. In- ternational Journal of Geographical Information Science, 36(6):1231-1257, 2022.","title":"A multi-view bidirectional spatiotemporal graph network for urban traffic flow imputation","authors":"Peixiao Wang, Tong Zhang, Yueming Zheng, Tao Hu","year":"2022","venue":"International Journal of Geographical Information Science","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wei et al., 2018] Hua Wei, Guanjie Zheng, Huaxiu Yao, and Zhenhui Li. Intellilight: A reinforcement learning ap- proach for intelligent traffic light control. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2496-2505, 2018.","title":"Intellilight: A reinforcement learning approach for intelligent traffic light control","authors":"Hua Wei, Guanjie Zheng, Huaxiu Yao, Zhenhui Li","year":"2018","venue":"Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wei et al., 2019a] Hua Wei, Chacha Chen, Guanjie Zheng, Kan Wu, Vikash Gayah, Kai Xu, and Zhenhui Li. Presslight: Learning max pressure control to coordinate traffic signals in arterial network. In Proceedings of the 25th ACM SIGKDD International Conference on Knowl- edge Discovery & Data Mining, KDD '19, pages 1290- 1298, 2019.","title":"Presslight: Learning max pressure control to coordinate traffic signals in arterial network","authors":"Wei","year":"2019","venue":"Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '19","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wei et al., 2019b] Hua Wei, Nan Xu, Huichu Zhang, Guan- jie Zheng, Xinshi Zang, Chacha Chen, Weinan Zhang, Yanmin Zhu, Kai Xu, and Zhenhui Li. Colight: Learn- ing network-level cooperation for traffic signal control. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 1913- 1922, 2019.","title":"Colight: Learning network-level cooperation for traffic signal control","authors":"Wei","year":"2019","venue":"Proceedings of the 28th ACM International Conference on Information and Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wu et al., 2021] Libing Wu, Min Wang, Dan Wu, and Jia Wu. Dynstgat: Dynamic spatial-temporal graph atten- tion network for traffic signal control. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 2150-2159, 2021.","title":"Dynstgat: Dynamic spatial-temporal graph attention network for traffic signal control","authors":"Wu","year":"2021","venue":"Proceedings of the 30th ACM International Conference on Information & Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yao et al., 2018] Huaxiu Yao, Fei Wu, Jintao Ke, Xianfeng Tang, Yitian Jia, Siyu Lu, Pinghua Gong, Jieping Ye, and Zhenhui Li. Deep multi-view spatial-temporal network for taxi demand prediction. Proceedings of the AAAI Confer- ence on Artificial Intelligence, 32(1), Apr. 2018.","title":"Deep multi-view spatial-temporal network for taxi demand prediction","authors":"Huaxiu Yao, Fei Wu, Jintao Ke, Xianfeng Tang, Yitian Jia, Siyu Lu, Pinghua Gong, Jieping Ye, Zhenhui Li","year":"2018","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2019] Huichu Zhang, Siyuan Feng, Chang Liu, Yaoyao Ding, Yichen Zhu, Zihan Zhou, Weinan Zhang, Yong Yu, Haiming Jin, and Zhenhui Li. Cityflow: A multi-agent reinforcement learning environment for large scale city traffic scenario. In The world wide web conference, pages 3620-3624, 2019.","title":"Cityflow: A multi-agent reinforcement learning environment for large scale city traffic scenario","authors":"Zhang","year":"2019","venue":"The world wide web conference","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2021] Weibin Zhang, Pulin Zhang, Yinghao Yu, Xiying Li, Salvatore Antonio Biancardo, and Junyi Zhang. Missing data repairs for traffic flow with self- attention generative adversarial imputation net. IEEE Transactions on Intelligent Transportation Systems, 2021.","title":"Missing data repairs for traffic flow with selfattention generative adversarial imputation net","authors":"Weibin Zhang, Pulin Zhang, Yinghao Yu, Xiying Li, Salvatore Antonio Biancardo, Junyi Zhang","year":"2021","venue":"IEEE Transactions on Intelligent Transportation Systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhao et al., 2020] Yangyang Zhao, Zhenyu Wang, Kai Yin, Rui Zhang, Zhenhua Huang, and Pei Wang. Dynamic reward-based dueling deep dyna-q: Robust policy learning in noisy environments. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 34, pages 9676- 9684, 2020.","title":"Dynamic reward-based dueling deep dyna-q: Robust policy learning in noisy environments","authors":"Yangyang Zhao, Zhenyu Wang, Kai Yin, Rui Zhang, Zhenhua Huang, Pei Wang","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhong et al., 2004] Ming Zhong, Pawan Lingras, and Satish Sharma. Estimation of missing traffic counts using factor, genetic, neural, and regression techniques. Transportation Research Part C: Emerging Technologies, 12(2):139-166, 2004.","title":"Estimation of missing traffic counts using factor, genetic, neural, and regression techniques","authors":"Ming Zhong, Pawan Lingras, Satish Sharma","year":"2004","venue":"Transportation Research Part C: Emerging Technologies","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}