{"type":"ScholarlyArticle","title":"Reinforcement Learning from Optimization Proxy for Ride-Hailing Vehicle Relocation (Extended Abstract)","authors":[{"name":"Enpeng Yuan","slug":"enpeng-yuan","given":"Enpeng","family":"Yuan","affiliation":"Georgia Institute of Technology; Artificial Intel- ligence Research [Yuan et"},{"name":"Wenbo Chen","slug":"wenbo-chen","given":"Wenbo","family":"Chen","affiliation":"Georgia Institute of Technology"},{"name":"Pascal Van Hentenryck","slug":"pascal-van-hentenryck","given":"Pascal","family":"Van Hentenryck","affiliation":"Georgia Institute of Technology"}],"abstract":"Idle vehicle relocation is crucial for addressing demand-supply imbalance that frequently arises in the ride-hailing system. Current mainstream methodologies - optimization and reinforcement learning - suffer from obvious computational drawbacks. Optimization models need to be solved in real-time and often trade off model fidelity (hence quality of solutions) for computational efficiency. Reinforcement learning is expensive to train and often struggles to achieve coordination among a large fleet. This paper designs a hybrid approach that leverages the strengths of the two while overcoming their drawbacks. Specifically, it trains an optimization proxy, i.e., a machine-learning model that approximates an optimization model, and refines the proxy with reinforcement learning. This Reinforcement Learning from Optimization Proxy (RLOP) approach is efficient to train and deploy, and achieves better results than RL or optimization alone. Numerical experiments on the New York City dataset show that the RLOP approach reduces both the relocation costs and computation time significantly compared to the optimization model, while pure reinforcement learning fails to converge due to computational complexity.","doi":"10.24963/ijcai.2023/796","url":"https://ijcai.org/proceedings/2023/796","pdf":"https://ijcai.org/proceedings/2023/0796.pdf","markdown":"https://ijcai.org/proceedings/2023/796.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/796","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Journal Track","track":"Journal Track","number":796,"pages":{"first":6990,"last":6994},"keywords":["Planning and Scheduling: PS: Real-time planning","Machine Learning: ML: Reinforcement learning","Machine Learning: ML: Applications"],"videos":[],"references":[{"position":1,"raw":"[Guériau and Dusparic, 2018] Maxime Guériau and Ivana Dusparic. Samod: Shared autonomous mobility-on- demand using decentralized reinforcement learning. In 2018 21st International Conference on Intelligent Trans- portation Systems (ITSC), pages 1558-1563, 2018.","title":"Samod: Shared autonomous mobility-ondemand using decentralized reinforcement learning","authors":"Dusparic Guériau, Maxime Guériau, Ivana Dusparic","year":"2018","venue":"21st International Conference on Intelligent Transportation Systems (ITSC)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Holler et al., 2019] John Holler, Risto Vuorio, Zhi- wei (Tony) Qin, Xiaocheng Tang, Yan Jiao, Tiancheng Jin, Satinder Singh, Chenxi Wang, and Jieping Ye. Deep reinforcement learning for multi-driver vehicle dispatch- ing and repositioning problem. In Jianyong Wang, Kyuseok Shim, and Xindong Wu, editors, 2019 IEEE International Conference on Data Mining, ICDM 2019, Beijing, China, November 8-11, 2019, pages 1090-1095. IEEE, 2019.","title":"Deep reinforcement learning for multi-driver vehicle dispatching and repositioning problem","authors":"John Holler, Risto Vuorio, ( Zhiwei, Xiaocheng Tony) Qin, Yan Tang, Tiancheng Jiao, Satinder Jin, Chenxi Singh, Jieping Wang, Ye","year":"2019","venue":"IEEE International Conference on Data Mining, ICDM 2019","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Iglesias et al., 2017] Ramón Iglesias, Federico Rossi, Kevin Wang, David Hallac, Jure Leskovec, and Marco Pavone. Data-driven model predictive control of autonomous mobility-on-demand systems. CoRR, abs/1709.07032, 2017.","title":"Data-driven model predictive control of autonomous mobility-on-demand systems","authors":"Ramón Iglesias, Federico Rossi, Kevin Wang, David Hallac, Jure Leskovec, Marco Pavone","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Jiao et al., 2021] Yan Jiao, Xiaocheng Tang, Zhiwei Qin, Shuaiji Li, Fangfang Zhang, Hongtu Zhu, and Jie ping Ye. Real-world ride-hailing vehicle repositioning using deep reinforcement learning. ArXiv, abs/2103.04555, 2021.","title":"Real-world ride-hailing vehicle repositioning using deep reinforcement learning","authors":"Yan Jiao, Xiaocheng Tang, Zhiwei Qin, Shuaiji Li, Fangfang Zhang, Hongtu Zhu, Jie Ping, Ye","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Lin et al., 2018] Kaixiang Lin, Renyu Zhao, Zhe Xu, and Jiayu Zhou. Efficient large-scale fleet management via multi-agent deep reinforcement learning. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '18, page 1774-1783, New York, NY, USA, 2018. Association for Computing Machinery.","title":"Efficient large-scale fleet management via multi-agent deep reinforcement learning","authors":"Lin","year":"2018","venue":"Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '18","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Mao et al., 2020] Chao Mao, Yulin Liu, and Zuo-Jun (Max) Shen. Dispatch of autonomous vehicles for taxi services: A deep reinforcement learning approach. Transportation Research Part C: Emerging Technologies, 115:102626, 2020.","title":"Dispatch of autonomous vehicles for taxi services: A deep reinforcement learning approach","authors":"Chao Mao, Yulin Liu, Zuo-Jun ( Max, ) Shen","year":"2020","venue":"Transportation Research Part C: Emerging Technologies","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Miao et al., 2015] Fei Miao, Shan Lin, Sirajum Munir, John Stankovic, Hua Huang, Desheng Zhang, Tian He, and George Pappas. Taxi dispatch with real-time sensing data in metropolitan areas -a receding horizon control ap- proach. IEEE Transactions on Automation Science and Engineering, 13, 04 2015.","title":"Taxi dispatch with real-time sensing data in metropolitan areas -a receding horizon control approach","authors":"Fei Miao, Shan Lin, Sirajum Munir, John Stankovic, Hua Huang, Desheng Zhang, Tian He, George Pappas","year":null,"venue":"IEEE Transactions on Automation Science and Engineering","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Miao et al., 2017] Fei Miao, Shuo Han, Abdeltawab M. Hendawi, Mohamed E Khalefa, John A. Stankovic, and George J. Pappas. Data-driven distributionally robust vehicle balancing using dynamic region partitions. In 2017 ACM/IEEE 8th International Conference on Cyber- Physical Systems (ICCPS), 2017.","title":"Data-driven distributionally robust vehicle balancing using dynamic region partitions","authors":"Fei Miao, Shuo Han, Abdeltawab M Hendawi, Mohamed E Khalefa, John A Stankovic, George J Pappas","year":"2017","venue":"ACM/IEEE 8th International Conference on Cyber-Physical Systems (ICCPS)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[NYC, 2019] NYC. NYC Taxi & Limousine Commis- sion -trip record data. https://www.nyc.gov/site/tlc/about/ tlc-trip-record-data.page, 2019. Accessed: 2020-10-01.","title":"NYC. NYC Taxi & Limousine Commission -trip record data","authors":"Nyc","year":"2019","venue":null,"doi":null,"url":"https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page","internal_url":null},{"position":10,"raw":"[Oda and Joe-Wong, 2018] Takuma Oda and Carlee Joe- Wong. Movi: A model-free approach to dynamic fleet management. In IEEE INFOCOM 2018 -IEEE Confer- ence on Computer Communications, pages 2708-2716, 2018.","title":"Movi: A model-free approach to dynamic fleet management","authors":"Oda, Joe-Wong ; ] Takuma, Oda, Carlee Joe-Wong","year":"2018","venue":"IEEE INFOCOM 2018 -IEEE Conference on Computer Communications","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ramírez et al., 2021] Jorge Ramírez, Wen Yu, and Adolfo Perrusquía. Model-free reinforcement learning from ex- pert demonstrations: a survey. Artificial Intelligence Re- view, 1(1), 2021.","title":"Model-free reinforcement learning from expert demonstrations: a survey","authors":"Jorge Ramírez, Wen Yu, Adolfo Perrusquía","year":"2021","venue":"Artificial Intelligence Review","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Riley et al., 2020] Connor Riley, Pascal Van Hentenryck, and Enpeng Yuan. Real-time dispatching of large-scale ride-sharing systems: Integrating optimization, machine learning, and model predictive control. In Proceedings of the Twenty-Ninth International Joint Conference on Ar- tificial Intelligence, IJCAI-20, pages 4417-4423. Interna- tional Joint Conferences on Artificial Intelligence Organi- zation, 7 2020.","title":"Real-time dispatching of large-scale ride-sharing systems: Integrating optimization, machine learning, and model predictive control","authors":"Connor Riley, Pascal Van Hentenryck, Enpeng Yuan","year":"2020","venue":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20","doi":null,"url":null,"internal_url":"/proceedings/2020/609"},{"position":13,"raw":"[Sutton and Barto, 2018] Richard Sutton and Andrew Barto. Reinforcement Learning: An Introduction. The MIT Press., Cambridge, Massachusetts., 2018.","title":"Reinforcement Learning: An Introduction","authors":"Barto Sutton, Richard Sutton, Andrew Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Tsao et al., 2018] Matthew Tsao, Ramon Iglesias, and Marco Pavone. Stochastic model predictive control for au- tonomous mobility on demand. In 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pages 3941-3948, 2018.","title":"Stochastic model predictive control for autonomous mobility on demand","authors":"Matthew Tsao, Ramon Iglesias, Marco Pavone","year":"2018","venue":"21st International Conference on Intelligent Transportation Systems (ITSC)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Verma et al., 2017] Tanvi Verma, Pradeep Varakantham, Sarit Kraus, and Hoong Chuin Lau. Augmenting decisions of taxi drivers through reinforcement learning for improv- ing revenues. In ICAPS, 2017.","title":"Augmenting decisions of taxi drivers through reinforcement learning for improving revenues","authors":"Tanvi Verma, Pradeep Varakantham, Sarit Kraus, Hoong Chuin Lau","year":"2017","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Wen et al., 2017] Jian Wen, Jinhua Zhao, and Patrick Jail- let. Rebalancing shared mobility-on-demand systems: A reinforcement learning approach. In 2017 IEEE 20th Inter- national Conference on Intelligent Transportation Systems (ITSC), pages 220-225, 2017.","title":"Rebalancing shared mobility-on-demand systems: A reinforcement learning approach","authors":"Jian Wen, Jinhua Zhao, Patrick Jaillet","year":"2017","venue":"IEEE 20th International Conference on Intelligent Transportation Systems (ITSC)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Yuan et al., 2022] Enpeng Yuan, Wenbo Chen, and Pas- cal Van Hentenryck. Reinforcement learning from opti- mization proxy for ride-hailing vehicle relocation. J. Artif. Intell. Res., 75:985-1002, 2022.","title":"Reinforcement learning from optimization proxy for ride-hailing vehicle relocation","authors":"Yuan","year":"2022","venue":"J. Artif. Intell. Res","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}