{"type":"ScholarlyArticle","title":"Goal-HSVI: Heuristic Search Value Iteration for Goal POMDPs","authors":[{"name":"Karel Horák","slug":"karel-horak","given":"Karel","family":"Horák","affiliation":"Department of Computer Science , FEE , Czech Technical University in Prague"},{"name":"Branislav Bošanský","slug":"branislav-bosansky","given":"Branislav","family":"Bošanský","affiliation":"Department of Computer Science , FEE , Czech Technical University in Prague"},{"name":"Krishnendu Chatterjee","slug":"krishnendu-chatterjee","given":"Krishnendu","family":"Chatterjee","affiliation":"IST Austria , Klosterneuburg , Austria"}],"abstract":"Partially observable Markov decision processes (POMDPs) are the standard models for planning under uncertainty with both finite and infinite horizon. Besides the well-known discounted-sum objective, indefinite-horizon objective (aka Goal-POMDPs) is another classical objective for POMDPs. In this case, given a set of target states and a positive cost for each transition, the optimization objective is to minimize the expected total cost until a target state is reached.\r\nIn the literature, RTDP-Bel or heuristic search value iteration (HSVI) have been used for solving Goal-POMDPs. Neither of these algorithms has theoretical convergence guarantees, and HSVI may even fail to terminate its trials. We give the following contributions: (1) We discuss the challenges introduced in Goal-POMDPs and illustrate how they prevent the original HSVI from converging. (2) We present a novel algorithm inspired by HSVI, termed Goal-HSVI, and show that our algorithm has convergence guarantees. (3) We show that Goal-HSVI outperforms RTDP-Bel on a set of well-known examples.","doi":"10.24963/ijcai.2018/662","url":"https://ijcai.org/proceedings/2018/662","pdf":"https://ijcai.org/proceedings/2018/0662.pdf","markdown":"https://ijcai.org/proceedings/2018/662.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/662","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Planning and Scheduling","track":"Main track","number":662,"pages":{"first":4764,"last":4770},"keywords":["Planning and Scheduling: Planning under Uncertainty","Planning and Scheduling: Theoretical Foundations of Planning","Planning and Scheduling: POMDPs"],"videos":[],"references":[{"position":1,"raw":"[Barto et al., 1995] A. G. Barto, S. J. Bradtke, and S. P. Singh. Learning to act using real-time dynamic program- ming. Artificial intelligence, 72(1-2):81-138, 1995.","title":"Learning to act using real-time dynamic programming","authors":"A G Barto, S J Bradtke, S P Singh","year":"1995","venue":"Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bertsekas and Tsitsiklis, 1996] D. P. Bertsekas and J. N. Tsitsiklis. Neuro-dynamic programming. Athena Scien- tific, 1996.","title":"Neuro-dynamic programming","authors":"D P Bertsekas, J N Tsitsiklis","year":"1996","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bertsekas, 2005] D. P. Bertsekas. Dynamic programming and optimal control, 3rd Edition. Athena Scientific, 2005.","title":"Dynamic programming and optimal control, 3rd Edition","authors":"D P Bertsekas","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bonet and Geffner, 2009] B. Bonet and H. Geffner. Solv- ing POMDPs: RTDP-Bel vs. point-based algorithms. In IJCAI, pages 1641-1646, 2009.","title":"Solving POMDPs: RTDP-Bel vs. point-based algorithms","authors":"B Bonet, H Geffner","year":"2009","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2009/274"},{"position":5,"raw":"[Bonet, 1998] B. Bonet. Solving large POMDPs using real time dynamic programming. In In Proc. AAAI Fall Symp. on POMDPs. Citeseer, 1998.","title":"Solving large POMDPs using real time dynamic programming","authors":"B Bonet","year":"1998","venue":"Proc. AAAI Fall Symp. on POMDPs. Citeseer","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Černý et al., 2011] P. Černý, K. Chatterjee, T. A. Henzinger, A. Radhakrishna, and R. Singh. Quantitative synthesis for concurrent programs. In Proc. of CAV, LNCS 6806, pages 243-259. Springer, 2011.","title":"Quantitative synthesis for concurrent programs","authors":"P Černý, K Chatterjee, T A Henzinger, A Radhakrishna, R Singh","year":"2011","venue":"Proc. of CAV, LNCS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chatterjee et al., 2016] K. Chatterjee, M. Chmelík, R. Gupta, and A. Kanodia. Optimal cost almost-sure reachability in POMDPs. Artificial Intelligence, 234:26- 48, 2016.","title":"Optimal cost almost-sure reachability in POMDPs","authors":"K Chatterjee, M Chmelík, R Gupta, A Kanodia","year":"2016","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Culik and Kari, 1997] K. Culik and J. Kari. Digital images and formal languages. Handbook of formal languages, pages 599-616, 1997.","title":"Digital images and formal languages. Handbook of formal languages","authors":"K Culik, J Kari","year":"1997","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Durbin et al., 1998] R. Durbin, S. Eddy, A. Krogh, and G. Mitchison. Biological sequence analysis: probabilis- tic models of proteins and nucleic acids. Cambridge Univ. Press, 1998.","title":"Biological sequence analysis: probabilistic models of proteins and nucleic acids","authors":"R Durbin, S Eddy, A Krogh, G Mitchison","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Filar and Vrieze, 1997] J. Filar and K. Vrieze. Competitive Markov Decision Processes. Springer-Verlag, 1997.","title":"Competitive Markov Decision Processes","authors":"Filar, ] J Vrieze, K Filar, Vrieze","year":"1997","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hauskrecht, 2000] M. Hauskrecht. Value-function approx- imations for partially observable Markov decision pro- cesses. Journal of Artificial Intelligence Research, 13:33- 94, 2000.","title":"Value-function approximations for partially observable Markov decision processes","authors":"M Hauskrecht","year":"2000","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Howard, 1960] R. A. Howard. Dynamic Programming and Markov Processes. MIT Press, 1960.","title":"Dynamic Programming and Markov Processes","authors":"R A Howard","year":"1960","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kaelbling et al., 1996] L. P. Kaelbling, M. L. Littman, and A. W. Moore. Reinforcement learning: A survey. JAIR, 4:237-285, 1996.","title":"Reinforcement learning: A survey","authors":"L P Kaelbling, M L Littman, A W Moore","year":"1996","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kolobov et al., 2011] A. Kolobov, Mausam, D.S. Weld, and H. Geffner. Heuristic search for generalized stochastic shortest path MDPs. In ICAPS, 2011.","title":"Heuristic search for generalized stochastic shortest path MDPs","authors":"A Kolobov, D S Mausam, H Weld, Geffner","year":"2011","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kress-Gazit et al., 2009] H. Kress-Gazit, G. E. Fainekos, and G. J. Pappas. Temporal-logic-based reactive mission and motion planning. IEEE Transactions on Robotics, 25(6):1370-1381, 2009.","title":"Temporal-logic-based reactive mission and motion planning","authors":"H Kress-Gazit, G E Fainekos, G J Pappas","year":"2009","venue":"IEEE Transactions on Robotics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kurniawati et al., 2008] H. Kurniawati, D. Hsu, and W.S. Lee. SARSOP: Efficient point-based POMDP planning by approximating optimally reachable belief spaces. In Robotics: Science and Systems, pages 65-72, 2008.","title":"SARSOP: Efficient point-based POMDP planning by approximating optimally reachable belief spaces","authors":"H Kurniawati, D Hsu, W S Lee","year":"2008","venue":"Robotics: Science and Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Littman et al., 1995] M. L. Littman, A. R. Cassandra, and L. P Kaelbling. Learning policies for partially observable environments: Scaling up. In ICML, pages 362-370, 1995.","title":"Learning policies for partially observable environments: Scaling up","authors":"M L Littman, A R Cassandra, L P Kaelbling","year":"1995","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Littman, 1996] M. L. Littman. Algorithms for Sequential Decision Making. PhD thesis, Brown University, 1996.","title":"Algorithms for Sequential Decision Making","authors":"M L Littman","year":"1996","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mohri, 1997] M. Mohri. Finite-state transducers in lan- guage and speech processing. Comp. Linguistics, 23(2):269-311, 1997.","title":"Finite-state transducers in language and speech processing","authors":"M Mohri","year":"1997","venue":"Comp. Linguistics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Papadimitriou and Tsitsiklis, 1987] C. H. Papadimitriou and J. N. Tsitsiklis. The complexity of Markov deci- sion processes. Mathematics of Operations Research, 12:441-450, 1987.","title":"The complexity of Markov decision processes","authors":"C H Papadimitriou, J N Tsitsiklis","year":"1987","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Patek, 2001] S. D. Patek. On partially observed stochastic shortest path problems. In Proceedings of the 40th IEEE Conference on Decision and Control, volume 5, pages 5050-5055. IEEE, 2001.","title":"On partially observed stochastic shortest path problems","authors":"S D Patek","year":"2001","venue":"Proceedings of the 40th IEEE Conference on Decision and Control","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Puterman, 1994] M. L. Puterman. Markov Decision Pro- cesses. John Wiley and Sons, 1994.","title":"Markov Decision Processes","authors":"M L Puterman","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Smith and Simmons, 2004] T. Smith and R. Simmons. Heuristic search value iteration for POMDPs. In UAI, pages 520-527. AUAI Press, 2004.","title":"Heuristic search value iteration for POMDPs","authors":"T Smith, R Simmons","year":"2004","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Smith and Simmons, 2005] T. Smith and R. Simmons. Point-based POMDP algorithms: improved analysis and implementation. In UAI, pages 542-549. AUAI Press, 2005.","title":"Point-based POMDP algorithms: improved analysis and implementation","authors":"T Smith, R Simmons","year":"2005","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Smith, 2007] T. Smith. Probabilistic planning for robotic exploration. Carnegie Mellon University, 2007.","title":"Probabilistic planning for robotic exploration","authors":"T Smith","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Warnquist et al., 2013] H. Warnquist, J. Kvarnström, and P. Doherty. Exploiting Fully Observable and Deterministic Structures in Goal POMDPs. In ICAPS, pages 242-250, 2013.","title":"Exploiting Fully Observable and Deterministic Structures in Goal POMDPs","authors":"H Warnquist, J Kvarnström, P Doherty","year":"2013","venue":"ICAPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}