{"type":"ScholarlyArticle","title":"Semi-Markov Reinforcement Learning for Stochastic Resource Collection","authors":[{"name":"Sebastian Schmoll","slug":"sebastian-schmoll","given":"Sebastian","family":"Schmoll","affiliation":"LMU Munich"},{"name":"Matthias Schubert","slug":"matthias-schubert","given":"Matthias","family":"Schubert","affiliation":"LMU Munich"}],"abstract":"We show that the task of collecting stochastic, spatially distributed resources (Stochastic Resource Collection, SRC) may be considered as a Semi-Markov-Decision-Process. Our Deep-Q-Network (DQN) based approach uses a novel scalable and transferable artificial neural network architecture. The concrete use-case of the SRC is an officer (single agent) trying to maximize the amount of fined parking violations in his area. We evaluate our approach on a environment based on the real-world parking data of the city of Melbourne. In small, hence simple, settings with short distances between resources and few simultaneous violations, our approach is comparable to previous work. When the size of the network grows (and hence the amount of resources) our solution significantly outperforms preceding methods. Moreover, applying a trained agent to a non-overlapping new area outperforms existing approaches.","doi":"10.24963/ijcai.2020/463","url":"https://ijcai.org/proceedings/2020/463","pdf":"https://ijcai.org/proceedings/2020/0463.pdf","markdown":"https://ijcai.org/proceedings/2020/463.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/463","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning Applications","track":"Main track","number":463,"pages":{"first":3349,"last":3355},"keywords":["Multidisciplinary Topics and Applications: Transportation","Machine Learning: Deep Reinforcement Learning","Machine Learning Applications: Applications of Reinforcement Learning","Planning and Scheduling: Markov Decisions Processes"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25156/file_25156.m3u8","length_sec":288,"provider":"hls"}],"references":[{"position":1,"raw":"[Alabbasi et al., 2019] Abubakr Alabbasi, Arnob Ghosh, and Vaneet Aggarwal. Deeppool: Distributed model-free algorithm for ride-sharing using deep reinforcement learn- ing. arXiv preprint arXiv:1903.03882, 2019.","title":"Deeppool: Distributed model-free algorithm for ride-sharing using deep reinforcement learning","authors":"Abubakr Alabbasi, Arnob Ghosh, Vaneet Aggarwal","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alshamsi et al., 2009] Aamena Alshamsi, Sherief Abdal- lah, and Iyad Rahwan. Multiagent self-organization for a taxi dispatch system. In 8th AAMAS, Hungary, 2009.","title":"Aamena Alshamsi, Sherief Abdallah, and Iyad Rahwan. Multiagent self-organization for a taxi dispatch system","authors":"Alshamsi","year":"2009","venue":"8th AAMAS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Baird, 1995] Leemon Baird. Residual algorithms: Rein- forcement learning with function approximation. In Ma- chine Learning Proceedings 1995. Elsevier, 1995.","title":"Residual algorithms: Reinforcement learning with function approximation","authors":"Leemon Baird","year":"1995","venue":"Machine Learning Proceedings","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Barto and Mahadevan, 2003] Andrew G Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforce- ment learning. Discrete event dynamic systems, 2003.","title":"Recent advances in hierarchical reinforcement learning","authors":"G Andrew, Sridhar Barto, Mahadevan","year":"2003","venue":"Discrete event dynamic systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bellman et al., 1957] R. Bellman, R.E. Bellman, and Rand Corporation. Dynamic Programming. Rand Corporation research study. Princeton University Press, 1957.","title":"Dynamic Programming","authors":"R Bellman, R E Bellman","year":"1957","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bradtke and Duff, 1995] Steven J Bradtke and Michael O Duff. Reinforcement learning methods for continuous- time markov decision problems. In NIPS, 1995.","title":"Reinforcement learning methods for continuoustime markov decision problems","authors":"J Steven, Michael O Bradtke, Duff","year":"1995","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Crites and Barto, 1998] Robert H Crites and Andrew G Barto. Elevator group control using multiple reinforce- ment learning agents. Machine learning, 1998.","title":"Elevator group control using multiple reinforcement learning agents","authors":"H Robert, Andrew G Crites, Barto","year":"1998","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gendreau et al., 1996] Michel Gendreau, Gilbert Laporte, and René Séguin. Stochastic vehicle routing. EJOR, 1996.","title":"Stochastic vehicle routing","authors":"Michel Gendreau, Gilbert Laporte, René Séguin","year":"1996","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Godfrey and Powell, 2002] Gregory A Godfrey and War- ren B Powell. An adaptive dynamic programming al- gorithm for dynamic fleet management, i: Single period travel times. Transportation Science, 36(1):21-39, 2002.","title":"An adaptive dynamic programming algorithm for dynamic fleet management, i: Single period travel times","authors":"A Gregory, Warren B Godfrey, Powell","year":"2002","venue":"Transportation Science","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Khalil et al., 2017] Elias Khalil, Hanjun Dai, Yuyu Zhang, Bistra Dilkina, and Le Song. Learning combinatorial opti- mization algorithms over graphs. In NIPS, 2017.","title":"Learning combinatorial optimization algorithms over graphs","authors":"Elias Khalil, Hanjun Dai, Yuyu Zhang, Bistra Dilkina, Le Song","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kim et al., 2019] Joon-Seok Kim, Dieter Pfoser, and An- dreas Züfle. Distance-aware competitive spatiotemporal searching using spatiotemporal resource matrix factoriza- tion. In 27th ACM SIGSPATIAL, pages 624-627, 2019.","title":"Distance-aware competitive spatiotemporal searching using spatiotemporal resource matrix factorization","authors":"Joon-Seok Kim, Dieter Pfoser, Andreas Züfle","year":"2019","venue":"27th ACM SIGSPATIAL","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kool et al., 2018] Wouter Kool, Herke van Hoof, and Max Welling. Attention, learn to solve routing problems! arXiv preprint arXiv:1803.08475, 2018.","title":"Attention, learn to solve routing problems!","authors":"Wouter Kool, Herke Van Hoof, Max Welling","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2019] Minne Li, Zhiwei Qin, Yan Jiao, Yaodong Yang, Jun Wang, Chenxi Wang, Guobin Wu, and Jieping Ye. Efficient ridesharing order dispatching with mean field multi-agent reinforcement learning. In WWW, 2019.","title":"Efficient ridesharing order dispatching with mean field multi-agent reinforcement learning","authors":"Minne Li, Zhiwei Qin, Yan Jiao, Yaodong Yang, Jun Wang, Chenxi Wang, Guobin Wu, Jieping Ye","year":"2019","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lillicrap et al., 2015] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv:1509.02971, 2015.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lin et al., 2018] Kaixiang Lin, Renyu Zhao, Zhe Xu, and Jiayu Zhou. Efficient large-scale fleet management via multi-agent deep reinforcement learning. In 24th ACM SIGKDD, pages 1774-1783. ACM, 2018.","title":"Efficient large-scale fleet management via multi-agent deep reinforcement learning","authors":"Lin","year":"2018","venue":"24th ACM SIGKDD","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Little et al., 2007] Iain Little, Sylvie Thiebaux, et al. Prob- abilistic planning vs. replanning. In ICAPS Workshop on IPC: Past, Present and Future, 2007.","title":"Probabilistic planning vs. replanning","authors":"Iain Little, Sylvie Thiebaux","year":"2007","venue":"ICAPS Workshop on IPC: Past, Present and Future","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. preprint arXiv:1312.5602, 2013.","title":"Playing atari with deep reinforcement learning","authors":"Mnih","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asyn- chronous methods for deep reinforcement learning. In In- ternational conference on machine learning, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schaul et al., 2015] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience re- play. arXiv preprint arXiv:1511.05952, 2015.","title":"Prioritized experience replay","authors":"Tom Schaul, John Quan, Ioannis Antonoglou, David Silver","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Schmoll and Schubert, 2018] Sebastian Schmoll and Matthias Schubert. Dynamic resource routing using real-time dynamic programming. In 27th IJCAI 2018, Stockholm, Sweden, pages 4822-4828, 2018.","title":"Dynamic resource routing using real-time dynamic programming","authors":"Sebastian Schmoll, Matthias Schubert","year":"2018","venue":"27th IJCAI 2018","doi":null,"url":null,"internal_url":"/proceedings/2018/670"},{"position":21,"raw":"[Schmoll et al., 2019] Sebastian Schmoll, Sabrina Friedl, and Matthias Schubert. Scaling the dynamic resource rout- ing problem. In 16th SSTD, Vienna, Austria, 2019.","title":"Scaling the dynamic resource routing problem","authors":"Sebastian Schmoll, Sabrina Friedl, Matthias Schubert","year":"2019","venue":"16th SSTD","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Shao et al., 2017] Wei Shao, Flora D Salim, Tao Gu, Ngoc- Thanh Dinh, and Jeffrey Chan. Travelling officer problem: Managing car parking violations efficiently using sensor data. IEEE Internet of Things Journal, 2017.","title":"Travelling officer problem: Managing car parking violations efficiently using sensor data","authors":"Wei Shao, Flora D Salim, Tao Gu, Ngoc-Thanh Dinh, Jeffrey Chan","year":"2017","venue":"IEEE Internet of Things Journal","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shao et al., 2019] Wei Shao, Flora D. Salim, Jeffrey Chan, Sean Morrison, and Fabio Zambetta. Approximating opti- misation solutions for travelling officer problem with cus- tomised deep learning network. CoRR, 2019.","title":"Approximating optimisation solutions for travelling officer problem with customised deep learning network","authors":"Wei Shao, Flora D Salim, Jeffrey Chan, Sean Morrison, Fabio Zambetta","year":"2019","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Solomon and Desrosiers, 1988] Marius M Solomon and Jacques Desrosiers. Survey paper-time window con- strained routing and scheduling problems. Transportation science, 22(1):1-13, 1988.","title":"Survey paper-time window constrained routing and scheduling problems","authors":"M Marius, Jacques Solomon, Desrosiers","year":"1988","venue":"Transportation science","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tang et al., 2019] Xiaocheng Tang, Zhiwei (Tony) Qin, Fan Zhang, Zhaodong Wang, Zhe Xu, Yintai Ma, Hongtu Zhu, and Jieping Ye. A deep value-network based approach for multi-driver order dispatching. In 25th ACM SIGKDD, KDD 2019, Anchorage, USA, pages 1780-1790, 2019.","title":"A deep value-network based approach for multi-driver order dispatching","authors":"Tang","year":"2019","venue":"25th ACM SIGKDD, KDD 2019","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Van Hasselt et al., 2016] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with dou- ble q-learning. In 13th AAAI, 2016.","title":"Deep reinforcement learning with double q-learning","authors":"Van Hasselt","year":"2016","venue":"13th AAAI","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Vinyals et al., 2015] Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In Advances in Neural Information Processing Systems, pages 2692-2700, 2015.","title":"Pointer networks","authors":"Oriol Vinyals, Meire Fortunato, Navdeep Jaitly","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Watkins, 1989] Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. 1989.","title":"Learning from delayed rewards","authors":"Christopher John, Cornish Hellaby Watkins","year":"1989","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Xu et al., 2018] Zhe Xu, Zhixin Li, Qingwen Guan, Ding- shui Zhang, Qiang Li, Junxiao Nan, Chunyang Liu, Wei Bian, and Jieping Ye. Large-scale order dispatch in on- demand ride-hailing platforms: A learning and planning approach. In 24th ACM SIGKDD, pages 905-913, 2018.","title":"Large-scale order dispatch in ondemand ride-hailing platforms: A learning and planning approach","authors":"Zhe Xu, Zhixin Li, Qingwen Guan, Dingshui Zhang, Qiang Li, Junxiao Nan, Chunyang Liu, Wei Bian, Jieping Ye","year":"2018","venue":"24th ACM SIGKDD","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}