{"type":"ScholarlyArticle","title":"Learning to Send Reinforcements: Coordinating Multi-Agent Dynamic Police Patrol Dispatching and Rescheduling via Reinforcement Learning","authors":[{"name":"Waldy Joe","slug":"waldy-joe","given":"Waldy","family":"Joe","affiliation":"School of Computing and Information Systems , Singapore Management University"},{"name":"Hoong Chuin Lau","slug":"hoong-chuin-lau","given":"Hoong Chuin","family":"Lau","affiliation":"School of Computing and Information Systems , Singapore Management University"}],"abstract":"We address the problem of coordinating multiple agents in a dynamic police patrol scheduling via a Reinforcement Learning (RL) approach. Our approach utilizes Multi-Agent Value Function Approximation (MAVFA) with a rescheduling heuristic to learn dispatching and rescheduling policies jointly. Often, police operations are divided into multiple sectors for more effective and efficient operations. In a dynamic setting, incidents occur throughout the day across different sectors, disrupting initially-planned patrol schedules. To maximize policing effectiveness, police agents from different sectors cooperate by sending reinforcements to support one another in their incident response and even routine patrol. This poses an interesting research challenge on how to make such complex decision of dispatching and rescheduling involving multiple agents in a coordinated fashion within an operationally reasonable time. Unlike existing Multi-Agent RL (MARL) approaches which solve similar problems by either decomposing the problem or action into multiple components, our approach learns the dispatching and rescheduling policies jointly without any decomposition step. In addition, instead of directly searching over the joint action space, we incorporate an iterative best response procedure as a decentralized optimization heuristic and an explicit coordination mechanism for a scalable and coordinated decision-making. We evaluate our approach against the commonly adopted two-stage approach and conduct a series of ablation studies to ascertain the effectiveness of our proposed learning and coordination mechanisms.","doi":"10.24963/ijcai.2023/18","url":"https://ijcai.org/proceedings/2023/18","pdf":"https://ijcai.org/proceedings/2023/0018.pdf","markdown":"https://ijcai.org/proceedings/2023/18.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/18","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":18,"pages":{"first":153,"last":161},"keywords":["Planning and Scheduling: PS: Applications","Agent-based and Multi-agent Systems: MAS: Multi-agent learning","Planning and Scheduling: PS: Learning in planning and scheduling"],"videos":[],"references":[{"position":1,"raw":"[Amador et al., 2014] Sofia Amador, Steven Okamoto, and Roie Zivan. Dynamic multi-agent task allocation with spa- tial and temporal constraints. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 28, 2014.","title":"Dynamic multi-agent task allocation with spatial and temporal constraints","authors":"Amador","year":"2014","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2019] Yujie Chen, Yu Qian, Yichen Yao, Zili Wu, Rongqi Li, Yinzhi Zhou, Haoyuan Hu, and Yinghui Xu. Can sophisticated dispatching strategy acquired by re- inforcement learning? In Proceedings of the 18th Interna- tional Conference on Autonomous Agents and MultiAgent Systems, pages 1395-1403, 2019.","title":"Can sophisticated dispatching strategy acquired by reinforcement learning?","authors":"Yujie Chen, Yu Qian, Yichen Yao, Zili Wu, Rongqi Li, Yinzhi Zhou, Haoyuan Hu, Yinghui Xu","year":"2019","venue":"Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2021] Jiayu Chen, Abhishek K Umrawal, Tian Lan, and Vaneet Aggarwal. Deepfreight: A model-free deep-reinforcement-learning-based algorithm for multi- transfer freight delivery. In Proceedings of the Interna- tional Conference on Automated Planning and Scheduling, volume 31, pages 510-518, 2021.","title":"Deepfreight: A model-free deep-reinforcement-learning-based algorithm for multitransfer freight delivery","authors":"Jiayu Chen, K Abhishek, Tian Umrawal, Vaneet Lan, Aggarwal","year":"2021","venue":"Proceedings of the Interna-tional Conference on Automated Planning and Scheduling","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2022] Xinwei Chen, Marlin W Ulmer, and Bar- rett W Thomas. Deep q-learning for same-day delivery with vehicles and drones. European Journal of Opera- tional Research, 298(3):939-952, 2022.","title":"Deep q-learning for same-day delivery with vehicles and drones","authors":"Xinwei Chen, Marlin W Ulmer, Barrett W Thomas","year":"2022","venue":"European Journal of Operational Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Das et al., 2019] Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communi- cation. In International Conference on Machine Learning, pages 1538-1546. PMLR, 2019.","title":"Tarmac: Targeted multi-agent communication","authors":"Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, Joelle Pineau","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dewinter et al., 2020] Maite Dewinter, Christophe Vande- viver, Tom Vander Beken, and Frank Witlox. Analysing the police patrol routing problem: A review. ISPRS Inter- national Journal of Geo-Information, 9(3):157, 2020.","title":"Analysing the police patrol routing problem: A review","authors":"Maite Dewinter, Christophe Vandeviver, Tom Vander Beken, Frank Witlox","year":"2020","venue":"ISPRS International Journal of Geo-Information","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Doyle et al., 2016] Maria Doyle, Louise Frogner, Henrik Andershed, and Anna-Karin Andershed. Feelings of safety in the presence of the police, security guards, and police volunteers. European journal on criminal policy and re- search, 22:19-40, 2016.","title":"Feelings of safety in the presence of the police, security guards, and police volunteers","authors":"Maria Doyle, Louise Frogner, Henrik Andershed, Anna-Karin Andershed","year":"2016","venue":"European journal on criminal policy and research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Foerster et al., 2018] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelli- gence, volume 32, 2018.","title":"Counterfactual multi-agent policy gradients","authors":"Foerster","year":"2018","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jiang and Lu, 2018] Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent coop- eration. Advances in neural information processing sys- tems, 31, 2018.","title":"Learning attentional communication for multi-agent cooperation","authors":"Lu ; Jiang, Jiechuan Jiang, Zongqing Lu","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jiang et al., 2020] Jiechuan Jiang, Chen Dun, Tiejun Huang, and Zongqing Lu. Graph convolutional reinforce- ment learning. In International Conference on Learning Representations, 2020.","title":"Graph convolutional reinforcement learning","authors":"Jiechuan Jiang, Chen Dun, Tiejun Huang, Zongqing Lu","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Joe and Lau, 2020] Waldy Joe and Hoong Chuin Lau. Deep reinforcement learning approach to solve dynamic vehicle routing problem with stochastic customers. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 30, pages 394-402, 2020.","title":"Deep reinforcement learning approach to solve dynamic vehicle routing problem with stochastic customers","authors":"Joe, Lau ; ] Waldy Joe, Hoong Chuin Lau","year":"2020","venue":"Proceedings of the International Conference on Automated Planning and Scheduling","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Joe and Lau, 2023] Waldy Joe and Hoong Chuin Lau. Coor- dinating multi-party vehicle routing with location conges- tion via iterative best response. In Multi-Agent Systems: 18th European Conference, EUMAS 2021. Special issue in SN Computer Science, 4:157. Springer Nature, 2023.","title":"Coordinating multi-party vehicle routing with location congestion via iterative best response","authors":"Joe, Lau ; ] Waldy Joe, Hoong Chuin Lau","year":"2021","venue":"Multi-Agent Systems: 18th European Conference","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Joe et al., 2022] Waldy Joe, Hoong Chuin Lau, and Jonathan Pan. Reinforcement learning approach to solve dynamic bi-objective police patrol dispatching and rescheduling problem. In Proceedings of the International Conference on Automated Planning and Scheduling, vol- ume 32, pages 453-461, 2022.","title":"Reinforcement learning approach to solve dynamic bi-objective police patrol dispatching and rescheduling problem","authors":"Waldy Joe, Hoong, Chuin Lau, Jonathan Pan","year":"2022","venue":"Proceedings of the International Conference on Automated Planning and Scheduling","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lambert Iii et al., 2005] Theodore J Lambert Iii, Marina A Epelman, and Robert L Smith. A fictitious play ap- proach to large-scale optimization. Operations Research, 53(3):477-489, 2005.","title":"A fictitious play approach to large-scale optimization","authors":"Lambert Theodore, Marina A Iii, Robert L Epelman, Smith","year":"2005","venue":"Operations Research","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Los et al., 2020] Johan Los, Frederik Schulte, Margaretha Gansterer, Richard F Hartl, Matthijs TJ Spaan, and Rudy R Negenborn. Decentralized combinatorial auctions for dy- namic and large-scale collaborative vehicle routing. In In- ternational Conference on Computational Logistics, pages 215-230. Springer, 2020.","title":"Decentralized combinatorial auctions for dynamic and large-scale collaborative vehicle routing","authors":"Johan Los, Frederik Schulte, Margaretha Gansterer, Richard F Hartl, Matthijs Tj Spaan, Rudy R Negenborn","year":"2020","venue":"International Conference on Computational Logistics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lowe et al., 2017] Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-competitive envi- ronments. Advances in neural information processing sys- tems, 30, 2017.","title":"Multiagent actor-critic for mixed cooperative-competitive environments","authors":"Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, Pieter Openai, Igor Abbeel, Mordatch","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ma et al., 2021] Yi Ma, Xiaotian Hao, Jianye Hao, Jiawen Lu, Xing Liu, Tong Xialiang, Mingxuan Yuan, Zhigang Li, Jie Tang, and Zhaopeng Meng. A hierarchical reinforce- ment learning based optimization framework for large- scale dynamic pickup and delivery problems. Advances in Neural Information Processing Systems, 34:23609-23620, 2021.","title":"A hierarchical reinforcement learning based optimization framework for largescale dynamic pickup and delivery problems","authors":"Yi Ma, Xiaotian Hao, Jianye Hao, Jiawen Lu, Xing Liu, Tong Xialiang, Mingxuan Yuan, Zhigang Li, Jie Tang, Zhaopeng Meng","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Monderer and Shapley, 1996] Dov Monderer and Lloyd S Shapley. Potential games. Games and economic behav- ior, 14(1):124-143, 1996.","title":"Dov Monderer and Lloyd S Shapley","authors":"Shapley Monderer","year":"1996","venue":"Games and economic behavior","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mukhopadhyay et al., 2016] Ayan Mukhopadhyay, Chao Zhang, Yevgeniy Vorobeychik, Milind Tambe, Kenneth Pence, and Paul Speer. Optimal allocation of police pa- trol resources using a continuous-time crime model. In International conference on decision and game theory for security, pages 139-158. Springer, 2016.","title":"Optimal allocation of police patrol resources using a continuous-time crime model","authors":"Ayan Mukhopadhyay, Chao Zhang, Yevgeniy Vorobeychik, Milind Tambe, Kenneth Pence, Paul Speer","year":"2016","venue":"International conference on decision and game theory for security","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nelke et al., 2020] Sofia Amador Nelke, Steven Okamoto, and Roie Zivan. Market clearing-based dynamic multi- agent task allocation. ACM Transactions on Intelligent Systems and Technology (TIST), 11(1):1-25, 2020.","title":"Market clearing-based dynamic multiagent task allocation","authors":"Nelke","year":"2020","venue":"ACM Transactions on Intelligent Systems and Technology (TIST)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pettet et al., 2022] Geoffrey Pettet, Ayan Mukhopadhyay, Mykel J Kochenderfer, and Abhishek Dubey. Hierarchi- cal planning for dynamic resource allocation in smart and connected communities. ACM Transactions on Cyber- Physical Systems, 6(4):1-26, 2022.","title":"Hierarchical planning for dynamic resource allocation in smart and connected communities","authors":"Geoffrey Pettet, Ayan Mukhopadhyay, J Mykel, Abhishek Kochenderfer, Dubey","year":"2022","venue":"ACM Transactions on Cyber-Physical Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Powell, 2019] Warren B Powell. A unified framework for stochastic optimization. European Journal of Operational Research, 275(3):795-821, 2019.","title":"A unified framework for stochastic optimization","authors":"B Warren, Powell","year":"2019","venue":"European Journal of Operational Research","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rashid et al., 2018] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pages 4295-4304. PMLR, 2018.","title":"Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning","authors":"Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, Shimon Whiteson","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Richardson et al., 2019] Rashia Richardson, Jason Schultz, and Kate Crawford. Dirty data, bad predictions: How civil rights violations impact police data, predictive polic- ing systems, and justice. New York University Law Review Online, 94(192), 2019.","title":"Dirty data, bad predictions: How civil rights violations impact police data, predictive policing systems, and justice","authors":"Richardson","year":"2019","venue":"New York University Law Review Online","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ritzinger et al., 2016] Ulrike Ritzinger, Jakob Puchinger, and Richard F Hartl. A survey on dynamic and stochastic vehicle routing problems. International Journal of Pro- duction Research, 54(1):215-231, 2016.","title":"A survey on dynamic and stochastic vehicle routing problems","authors":"Ulrike Ritzinger, Jakob Puchinger, Richard F Hartl","year":"2016","venue":"International Journal of Production Research","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Ruan et al., 2022] Jingqing Ruan, Linghui Meng, Xuantang Xiong, Dengpeng Xing, and Bo Xu. Learning multi- agent action coordination via electing first-move agent. In Proceedings of the International Conference on Auto- mated Planning and Scheduling, volume 32, pages 624- 628, 2022.","title":"Learning multiagent action coordination via electing first-move agent","authors":"Ruan","year":"2022","venue":"Proceedings of the International Conference on Automated Planning and Scheduling","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Rumi et al., 2020] Shakila Khan Rumi, Wei Shao, and Flora D Salim. Realtime predictive patrolling and routing with mobility and emergency calls data. In Proceedings of the international AAAI conference on web and social media, volume 14, pages 964-968, 2020.","title":"Realtime predictive patrolling and routing with mobility and emergency calls data","authors":"Wei Shakila Khan Rumi, Flora D Shao, Salim","year":"2020","venue":"Proceedings of the international AAAI conference on web and social media","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Santana et al., 2004] Hugo Santana, Geber Ramalho, Vin- cent Corruble, and Bohdana Ratitch. Multi-agent pa- trolling with reinforcement learning. In Autonomous Agents and Multiagent Systems, International Joint Con- ference on, volume 4, pages 1122-1129. IEEE Computer Society, 2004.","title":"Hugo Santana, Geber Ramalho, Vincent Corruble, and Bohdana Ratitch. Multi-agent patrolling with reinforcement learning","authors":"Santana","year":"2004","venue":"Autonomous Agents and Multiagent Systems, International Joint Conference on","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Son et al., 2019] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learn- ing to factorize with transformation for cooperative multi- agent reinforcement learning. In International Conference on Machine Learning, pages 5887-5896. PMLR, 2019.","title":"Qtran: Learning to factorize with transformation for cooperative multiagent reinforcement learning","authors":"Son","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Sunehag et al., 2018] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning based on team reward. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pages 2085- 2087, 2018.","title":"Value-decomposition networks for cooperative multi-agent learning based on team reward","authors":"Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech, Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls","year":"2018","venue":"Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Tkach and Amador, 2021] Itshak Tkach and Sofia Amador. Towards addressing dynamic multi-agent task allocation in law enforcement. Autonomous Agents and Multi-Agent Systems, 35(1):1-18, 2021.","title":"Itshak Tkach and Sofia Amador. Towards addressing dynamic multi-agent task allocation in law enforcement","authors":"Amador Tkach","year":"2021","venue":"Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Ulmer et al., 2020] Marlin W Ulmer, Justin C Goodson, Dirk C Mattfeld, and Barrett W Thomas. On modeling stochastic dynamic vehicle routing problems. EURO Jour- nal on Transportation and Logistics, 9(2):100008, 2020.","title":"On modeling stochastic dynamic vehicle routing problems","authors":"Marlin W Ulmer, Justin C Goodson, Dirk C Mattfeld, Barrett W Thomas","year":"2020","venue":"EURO Journal on Transportation and Logistics","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wang and Kopfer, 2013] Xin Wang and Herbert Kopfer. Dynamic collaborative transportation planning: A rolling horizon planning approach. In International Conference on Computational Logistics, pages 128-142. Springer, 2013.","title":"Dynamic collaborative transportation planning: A rolling horizon planning approach","authors":"Xin Wang, Herbert Kopfer","year":"2013","venue":"International Conference on Computational Logistics","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Wang et al., 2019] Wanyuan Wang, Zichen Dong, Bo An, and Yichuan Jiang. Toward efficient city-scale patrol plan- ning using decomposition and grafting. IEEE Transactions on Intelligent Transportation Systems, 2019.","title":"Toward efficient city-scale patrol planning using decomposition and grafting","authors":"Wang","year":"2019","venue":"IEEE Transactions on Intelligent Transportation Systems","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Wang et al., 2022] Wanyuan Wang, Hansi Tao, and Yichuan Jiang. Efficient online city-scale patrolling by exploiting offline model-based coordination policy. IEEE Transac- tions on Intelligent Transportation Systems, 23(8):13805- 13818, 2022.","title":"Efficient online city-scale patrolling by exploiting offline model-based coordination policy","authors":"Wang","year":"2022","venue":"IEEE Transactions on Intelligent Transportation Systems","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhang and Dietterich, 2000] Wei Zhang and Thomas G Di- etterich. Solving combinatorial optimization tasks by re- inforcement learning: A general methodology applied to resource-constrained scheduling. Journal of Artificial In- telligence Reseach, 1:1-38, 2000.","title":"Solving combinatorial optimization tasks by reinforcement learning: A general methodology applied to resource-constrained scheduling","authors":"Wei Zhang, Thomas G Dietterich","year":"2000","venue":"Journal of Artificial Intelligence Reseach","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}