{"type":"ScholarlyArticle","title":"Dynamic Resource Routing using Real-Time Dynamic Programming","authors":[{"name":"Sebastian Schmoll","slug":"sebastian-schmoll","given":"Sebastian","family":"Schmoll","affiliation":"Ludwig-Maximilians-Universität , Munich , Germany"},{"name":"Matthias Schubert","slug":"matthias-schubert","given":"Matthias","family":"Schubert","affiliation":"Ludwig-Maximilians-Universität , Munich , Germany"}],"abstract":"Acquiring available resources in stochastic environments becomes more and more important to future mobility. For instance, cities like Melbourne, Canberra and San Francisco install sensors that detect in real-time whether a parking spot (resource) is available or not. In such environments, the current state of the resources may be fully observable, although the future development is stochastic. In order to reduce the traffic, such cities want to fully exploit parking spots, such that the amount of searching cars is minimized. Thus, we formulate a problem setting where the expected seek time for each driver is minimized. This problem can be modeled by a Markov Decision Process (MDP) and solved using standard algorithms. In this paper, we focus on the setting, where pre-computation is not possible and search policies have to be computed on the fly. Our approach is based on state-of-the-art Real-Time Dynamic Programming (RTDP) approaches. However, standard RTDP approaches do not perform well on this specific problem setting as shown in our experiments. We introduce adapted bounds and approximations that exploit the specific nature of the problem in order to improve the performance significantly.","doi":"10.24963/ijcai.2018/670","url":"https://ijcai.org/proceedings/2018/670","pdf":"https://ijcai.org/proceedings/2018/0670.pdf","markdown":"https://ijcai.org/proceedings/2018/670.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/670","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Planning and Scheduling","track":"Main track","number":670,"pages":{"first":4822,"last":4828},"keywords":["Planning and Scheduling: Markov Decisions Processes","Planning and Scheduling: Planning under Uncertainty","Planning and Scheduling: Real-time Planning","Planning and Scheduling: Applications of Planning"],"videos":[],"references":[{"position":1,"raw":"[Barto et al., 1995] Andrew G Barto, Steven J Bradtke, and Satinder P Singh. Learning to act using real-time dynamic programming. Artificial Intelligence, 72(1):81-138, 1995.","title":"Learning to act using real-time dynamic programming","authors":"Steven J Andrew G Barto, Bradtke, P Satinder, Singh","year":"1995","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bellman, 1957] Richard Bellman. A markovian decision process. Journal of Mathematics and Mechanics, pages 679-684, 1957.","title":"A markovian decision process","authors":"Richard Bellman","year":"1957","venue":"Journal of Mathematics and Mechanics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bertsekas and Tsitsiklis, 1991] Dimitri P Bertsekas and John N Tsitsiklis. An analysis of stochastic shortest path problems. Mathematics of Operations Research, 16(3):580-595, 1991.","title":"An analysis of stochastic shortest path problems","authors":"P Dimitri, John N Bertsekas, Tsitsiklis","year":"1991","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bonet and Geffner, 2003a] Blai Bonet and Hector Geffner. Faster heuristic search algorithms for planning with un- certainty and full feedback. In IJCAI, pages 1233-1238, 2003.","title":"Faster heuristic search algorithms for planning with uncertainty and full feedback","authors":"Geffner Bonet, Blai Bonet, Hector Geffner","year":"2003","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2003/175"},{"position":5,"raw":"[Bonet and Geffner, 2003b] Blai Bonet and Hector Geffner. Labeled rtdp: Improving the convergence of real-time dy- namic programming. In ICAPS, volume 3, pages 12-21, 2003.","title":"Labeled rtdp: Improving the convergence of real-time dynamic programming","authors":"Geffner Bonet, Blai Bonet, Hector Geffner","year":"2003","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Guo and Wolfson, 2016] Qing Guo and Ouri Wolfson. Probabilistic spatio-temporal resource search. GeoInfor- matica, pages 1-29, 2016.","title":"Probabilistic spatio-temporal resource search","authors":"Wolfson Guo, Qing Guo, Ouri Wolfson","year":"2016","venue":"GeoInformatica","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hansen and Zilberstein, 2001] Eric A Hansen and Shlomo Zilberstein. Lao*: A heuristic search algorithm that finds solutions with loops. Artificial Intelligence, 129(1-2):35- 62, 2001.","title":"Lao*: A heuristic search algorithm that finds solutions with loops","authors":"Zilberstein Hansen, Eric A Hansen, Shlomo Zilberstein","year":"2001","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Howard, 1960] Ronald A Howard. Dynamic programming and markov processes.. 1960.","title":"Dynamic programming and markov processes","authors":"A Ronald, Howard","year":"1960","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jossé et al., 2015] G. Jossé, K. A. Schmid, and M. Schubert. Probabilistic resource route queries with reappearance. In #EDBT15#, 2015.","title":"Probabilistic resource route queries with reappearance","authors":"G Jossé, K A Schmid, M Schubert","year":"2015","venue":"#EDBT15#","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[McMahan et al., 2005] H Brendan McMahan, Maxim Likhachev, and Geoffrey J Gordon. Bounded real-time dynamic programming: Rtdp with monotone upper bounds and performance guarantees. In Proceedings of the 22nd international conference on Machine learning, pages 569-576. ACM, 2005.","title":"Bounded real-time dynamic programming: Rtdp with monotone upper bounds and performance guarantees","authors":"Maxim H Brendan Mcmahan, Geoffrey J Likhachev, Gordon","year":"2005","venue":"Proceedings of the 22nd international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Puterman, 1994] Martin L Puterman. Markov decision pro- cesses: Discrete stochastic dynamic programming. 1994.","title":"Markov decision processes: Discrete stochastic dynamic programming","authors":"Martin L Puterman","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Sanner et al., 2009] Scott Sanner, Robby Goetschalckx, Kurt Driessens, Guy Shani, et al. Bayesian real-time dy- namic programming. In IJCAI, pages 1784-1789. Cite- seer, 2009.","title":"Bayesian real-time dynamic programming","authors":"Scott Sanner, Robby Goetschalckx, Kurt Driessens, Guy Shani","year":"2009","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2009/297"},{"position":13,"raw":"[Schmoll and Schubert, 2018] Sebastian Schmoll and Matthias Schubert. Dynamic resource routing using real-time information. In Proceedings of the 21th Inter- national Conference on Extending Database Technology, EDBT 2018, Vienna, Austria, March 26-29, 2018., pages 501-504, 2018.","title":"Dynamic resource routing using real-time information","authors":"Sebastian Schmoll, Matthias Schubert","year":"2018","venue":"Proceedings of the 21th International Conference on Extending Database Technology, EDBT 2018","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Smith and Simmons, 2006] Trey Smith and Reid Simmons. Focused real-time dynamic programming for mdps: Squeezing more out of a heuristic. In AAAI, pages 1227- 1232, 2006.","title":"Focused real-time dynamic programming for mdps: Squeezing more out of a heuristic","authors":"Simmons Smith, Trey Smith, Reid Simmons","year":"2006","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}