{"type":"ScholarlyArticle","title":"Dynamic Electronic Toll Collection via Multi-Agent Deep Reinforcement Learning with Edge-Based Graph Convolutional Networks","authors":[{"name":"Wei Qiu","slug":"wei-qiu","given":"Wei","family":"Qiu","affiliation":"School of Computer Science and Engineering , Nanyang Technological University , Singapore"},{"name":"Haipeng Chen","slug":"haipeng-chen","given":"Haipeng","family":"Chen","affiliation":"Department of Computer Science , Dartmouth College , USA"},{"name":"Bo An","slug":"bo-an","given":"Bo","family":"An","affiliation":"School of Computer Science and Engineering , Nanyang Technological University , Singapore"}],"abstract":"Over the past decades, Electronic Toll Collection (ETC) systems have been proved the capability of alleviating traffic congestion in urban areas. Dynamic Electronic Toll Collection (DETC) was recently proposed to further improve the efficiency of ETC, where tolls are dynamically set based on traffic dynamics. However, computing the optimal DETC scheme is computationally difficult and existing approaches are limited to small scale or partial road networks, which significantly restricts the adoption of DETC. To this end, we propose a novel multi-agent reinforcement learning (RL) approach for DETC. We make several key contributions: i) an enhancement over the state-of-the-art RL-based method with a deep neural network  representation of the policy and value functions and a temporal difference learning framework to accelerate the update of target values, ii) a novel edge-based graph convolutional neural network (eGCN) to extract the spatio-temporal correlations of the road network state features, iii) a novel cooperative multi-agent reinforcement learning (MARL) which divides the whole road network into partitions according to their geographic and economic characteristics and trains a tolling agent for each partition. Experimental results show that our approach can scale up to realistic-sized problems with robust performance and significantly outperform the state-of-the-art method.","doi":"10.24963/ijcai.2019/635","url":"https://ijcai.org/proceedings/2019/635","pdf":"https://ijcai.org/proceedings/2019/0635.pdf","markdown":"https://ijcai.org/proceedings/2019/635.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/635","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning Applications","track":"Main track","number":635,"pages":{"first":4568,"last":4574},"keywords":["Machine Learning Applications: Applications of Reinforcement Learning","Multidisciplinary Topics and Applications: Transportation"],"videos":[],"references":[{"position":1,"raw":"[BPR, 1964] BPR. Traffic assignment manual. US Depart- ment of Commerce, 1964.","title":"BPR. Traffic assignment manual","authors":"Bpr","year":"1964","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2018] Haipeng Chen, Bo An, Guni Sharon, Josiah P Hanna, Peter Stone, Chunyan Miao, and Yeng Chai Soh. Dyetc: Dynamic electronic toll collection for traffic congestion alleviation. AAAI, pages 757-765, 2018.","title":"Dyetc: Dynamic electronic toll collection for traffic congestion alleviation","authors":"Haipeng Chen, Bo An, Guni Sharon, Josiah P Hanna, Peter Stone, Chunyan Miao, Yeng Chai Soh","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Defferrard et al., 2016] Michaël Defferrard, Xavier Bres- son, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering, 2016.","title":"Convolutional neural networks on graphs with fast localized spectral filtering","authors":"Michaël Defferrard, Xavier Bresson, Pierre Vandergheynst","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Foerster et al., 2017] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. arXiv preprint arXiv:1705.08926, 2017.","title":"Counterfactual multi-agent policy gradients","authors":"Foerster","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gupta et al., 2017] Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control us- ing deep reinforcement learning. In AAMAS, pages 66-83, 2017.","title":"Cooperative multi-agent control using deep reinforcement learning","authors":"Maxim Jayesh K Gupta, Mykel Egorov, Kochenderfer","year":"2017","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Henaff et al., 2015] Mikael Henaff, Joan Bruna, and Yann LeCun. Deep convolutional networks on graph-structured data. arXiv preprint arXiv:1506.05163, 2015.","title":"Deep convolutional networks on graph-structured data","authors":"Mikael Henaff, Joan Bruna, Yann Lecun","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Huang and Li, 2007] Hai-Jun Huang and Zhi-Chun Li. A multiclass, multicriteria logit-based traffic equilibrium as- signment model under atis. European Journal of Opera- tional Research, 176(3):1464-1477, 2007.","title":"A multiclass, multicriteria logit-based traffic equilibrium assignment model under atis","authors":"Li ; Hai-Jun Huang, Zhi-Chun Huang, Li","year":"2007","venue":"European Journal of Operational Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Joksimovic et al., 2005] Dusica Joksimovic, Michiel CJ Bliemer, Piet HL Bovy, and Zofia Verwater-Lukszo. Dy- namic road pricing for optimizing network performance with heterogeneous users. In Networking, Sensing and Control, 2005. IEEE, pages 407-412, 2005.","title":"Dynamic road pricing for optimizing network performance with heterogeneous users","authors":"Dusica Joksimovic, C J Michiel, Piet Hl Bliemer, Zofia Bovy, Verwater-Lukszo","year":"2005","venue":"Networking, Sensing and Control","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kipf and Welling, 2016] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.","title":"Semi-supervised classification with graph convolutional networks","authors":"Thomas N Kipf, Max Welling","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Konda and Borkar, 1999] Vijaymohan R Konda and Vivek S Borkar. Actor-critic-type learning algorithms for markov decision processes. SIAM Journal on Control and Optimization, 38(1):94-123, 1999.","title":"Actor-critic-type learning algorithms for markov decision processes","authors":"Borkar ; Konda, R Vijaymohan, Konda, S Vivek, Borkar","year":"1999","venue":"SIAM Journal on Control and Optimization","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Konda and Tsitsiklis, 2000] Vijay R Konda and John N Tsitsiklis. Actor-critic algorithms. In NIPS, pages 1008- 1014, 2000.","title":"Actor-critic algorithms","authors":"R Vijay, John N Konda, Tsitsiklis","year":"2000","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lillicrap et al., 2015] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous con- trol with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lo and Szeto, 2002] Hong K Lo and WY Szeto. A method- ology for sustainable traveler information services. Trans- portation Research Part B: Methodological, 36(2):113- 130, 2002.","title":"A methodology for sustainable traveler information services","authors":"Szeto ; ] Lo, K Hong, W Y Lo, Szeto","year":"2002","venue":"Transportation Research Part B: Methodological","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lowe et al., 2017] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-competitive envi- ronments. In NIPS, pages 6379-6390, 2017.","title":"Multiagent actor-critic for mixed cooperative-competitive environments","authors":"Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Openai, Igor Abbeel, Mordatch","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[LTA, 2017] LTA. Annual motor vehicle population by ve- hicle quota categories. https://data.gov.sg/dataset/annual- motor-vehicle-population-by-vehicle-quota-category, 2017. Accessed: 2016-04-12.","title":"LTA. Annual motor vehicle population by vehicle quota categories","authors":null,"year":"2016","venue":null,"doi":null,"url":"https://data.gov.sg/dataset/annual-motor-vehicle-population-by-vehicle-quota-category","internal_url":null},{"position":16,"raw":"[Lu et al., 2008] Chung-Cheng Lu, Hani S Mahmassani, and Xuesong Zhou. A bi-criterion dynamic user equilibrium traffic assignment model and solution algorithm for evalu- ating dynamic road pricing strategies. TR Part C: Emerg- ing Technologies, 16(4):371-389, 2008.","title":"A bi-criterion dynamic user equilibrium traffic assignment model and solution algorithm for evaluating dynamic road pricing strategies","authors":"Chung-Cheng Lu, Hani S Mahmassani, Xuesong Zhou","year":"2008","venue":"TR Part C: Emerging Technologies","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mirzaei et al., 2018] Hamid Mirzaei, Guni Sharon, Stephen Boyles, Tony Givargis, and Peter Stone. Link-based pa- rameterized micro-tolling scheme for optimal traffic man- agement. In AAMAS, pages 2013-2015, 2018.","title":"Link-based parameterized micro-tolling scheme for optimal traffic management","authors":"Hamid Mirzaei, Guni Sharon, Stephen Boyles, Tony Givargis, Peter Stone","year":"2018","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Niepert et al., 2016] Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. Learning convolutional neural networks for graphs. In ICML, pages 2014-2023, 2016.","title":"Learning convolutional neural networks for graphs","authors":"Mathias Niepert, Mohamed Ahmed, Konstantin Kutzkov","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rummery and Niranjan, 1994] Gavin A Rummery and Ma- hesan Niranjan. On-line Q-learning using Connectionist Systems, volume 37. University of Cambridge, 1994.","title":"On-line Q-learning using Connectionist Systems","authors":"A Gavin, Mahesan Rummery, Niranjan","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In ICML, pages 1889-1897, 2015.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, Philipp Moritz","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sharon et al., 2017] Guni Sharon, Josiah P Hanna, Tarun Rambha, Michael W Levin, Michael Albert, Stephen D Boyles, and Peter Stone. Real-time adaptive tolling scheme for optimized social welfare in traffic networks. In AAMAS, pages 828-836, 2017.","title":"Real-time adaptive tolling scheme for optimized social welfare in traffic networks","authors":"Guni Sharon, Josiah P Hanna, Tarun Rambha, Michael Michael W Levin, Stephen D Albert, Peter Boyles, Stone","year":"2017","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Marc Driessche, Lanctot","year":"2016","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sukhbaatar et al., 2016] Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backprop- agation. In NIPS, pages 2244-2252, 2016.","title":"Learning multiagent communication with backpropagation","authors":"Sukhbaatar","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Watkins and Dayan, 1992] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine Learning, 8(3-4):279- 292, 1992.","title":"Q-learning","authors":"Jch Christopher, Peter Watkins, Dayan","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Williams, 1992] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine Learning, 8(3-4):229-256, 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Williams Ronald","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2013] Kuilin Zhang, Hani S Mahmassani, and Chung-Cheng Lu. Dynamic pricing, heterogeneous users and perception error: Probit-based bi-criterion dynamic stochastic user equilibrium assignment. Transportation Research Part C: Emerging Technologies, 27:189-204, 2013.","title":"Dynamic pricing, heterogeneous users and perception error: Probit-based bi-criterion dynamic stochastic user equilibrium assignment","authors":"Zhang","year":"2013","venue":"Transportation Research Part C: Emerging Technologies","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}