{"type":"ScholarlyArticle","title":"Approximability of Constant-horizon Constrained POMDP","authors":[{"name":"Majid Khonji","slug":"majid-khonji","given":"Majid","family":"Khonji","affiliation":"EECS Department , KU Center for Autonomous Robotic Systems , Khalifa University , Abu Dhabi , UAE; Computer Science and Artificial Intelligence Laboratory , Massachusetts Institute of Technology , Cambridge , MA , USA"},{"name":"Ashkan Jasour","slug":"ashkan-jasour","given":"Ashkan","family":"Jasour","affiliation":"Computer Science and Artificial Intelligence Laboratory , Massachusetts Institute of Technology , Cambridge , MA , USA"},{"name":"Brian Williams","slug":"brian-williams","given":"Brian","family":"Williams","affiliation":"Computer Science and Artificial Intelligence Laboratory , Massachusetts Institute of Technology , Cambridge , MA , USA"}],"abstract":"Partially Observable Markov Decision Process (POMDP) is a fundamental framework for planning and decision making under uncertainty. POMDP is known to be intractable to solve or even approximate when the planning horizon is long (i.e., within a polynomial number of time steps). Constrained POMDP (C-POMDP) allows constraints to be specified on some aspects of the policy in addition to the objective function. When the constraints involve bounding the probability of failure, the problem is called Chance-Constrained POMDP (CC-POMDP). Our first contribution is a reduction from CC-POMDP to C-POMDP and a novel Integer Linear Programming (ILP) formulation. Thus, any algorithm for the later problem can be utilized to solve any instance of the former. Second, we show that unlike POMDP, when the length of the planning horizon is constant, (C)C-POMDP is NP-Hard. Third, we present the first Fully Polynomial Time Approximation Scheme (FPTAS) that computes (near) optimal deterministic policies for constant-horizon (C)C-POMDP in polynomial time.","doi":"10.24963/ijcai.2019/775","url":"https://ijcai.org/proceedings/2019/775","pdf":"https://ijcai.org/proceedings/2019/0775.pdf","markdown":"https://ijcai.org/proceedings/2019/775.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/775","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Planning and Scheduling","track":"Main track","number":775,"pages":{"first":5583,"last":5590},"keywords":["Uncertainty in AI: Markov Decision Processes","Planning and Scheduling: Planning under Uncertainty","Planning and Scheduling: POMDPs"],"videos":[],"references":[{"position":1,"raw":"[Altman, 1999] Eitan Altman. Constrained Markov decision processes, volume 7. CRC Press, 1999.","title":"Constrained Markov decision processes","authors":"Eitan Altman","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Aras et al., 2007] Raghav Aras, Alain Dutech, Franc ¸ois Charpillet, et al. Mixed integer linear programming for exact finite-horizon planning in decentralized pomdps. In ICAPS, pages 18-25, 2007.","title":"Mixed integer linear programming for exact finite-horizon planning in decentralized pomdps","authors":"Raghav Aras, Alain Dutech, Franc ¸ois Charpillet","year":"2007","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dolgov and Durfee, 2005] Dmitri Dolgov and Edmund Durfee. Stationary deterministic policies for constrained mdps with multiple rewards, costs, and discount factors. In International Joint Conference on Artificial Intelligence, volume 19, page 1326, 2005.","title":"Stationary deterministic policies for constrained mdps with multiple rewards, costs, and discount factors","authors":"Dmitri Dolgov, Edmund Durfee","year":"2005","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2005/236"},{"position":4,"raw":"[Feinberg and Shwartz, 1996] Eugene A Feinberg and Adam Shwartz. Constrained discounted dynamic program- ming. Mathematics of Operations Research, 21(4):922- 945, 1996.","title":"Constrained discounted dynamic programming","authors":"Eugene A Feinberg, Adam Shwartz","year":"1996","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Howard, 1960] Ronald A Howard. Dynamic programming and markov processes. 1960.","title":"Dynamic programming and markov processes","authors":"A Ronald, Howard","year":"1960","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Isom et al., 2008] Joshua D Isom, Sean P Meyn, and Richard D Braatz. Piecewise linear dynamic programming for constrained pomdps. In AAAI, volume 1, pages 291- 296, 2008.","title":"Piecewise linear dynamic programming for constrained pomdps","authors":"Joshua D Isom, Sean P Meyn, Richard D Braatz","year":"2008","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Johnson and Niemi, 1983] David S Johnson and KA Niemi. On knapsacks, partitions, and a new dynamic program- ming technique for trees. Mathematics of Operations Re- search, 8(1):1-14, 1983.","title":"On knapsacks, partitions, and a new dynamic programming technique for trees","authors":"S David, K A Johnson, Niemi","year":"1983","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kaelbling et al., 1996] Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. Reinforcement learn- ing: A survey. Journal of artificial intelligence research, 4:237-285, 1996.","title":"Reinforcement learning: A survey","authors":"Leslie Pack, Kaelbling Michael L Littman, Andrew W Moore","year":"1996","venue":"Journal of artificial intelligence research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kaelbling et al., 1998] Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intel- ligence, 101(1-2):99-134, 1998.","title":"Planning and acting in partially observable stochastic domains","authors":"Leslie Pack, Kaelbling Michael L Littman, Anthony R Cassandra","year":"1998","venue":"Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kellerer et al., 2004] Hans Kellerer, Ulrich Pferschy, and David Pisinger. Knapsack Problems. Springer, 2004.","title":"Knapsack Problems","authors":"Kellerer","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kim et al., 2011] Dongho Kim, Jaesong Lee, Kee-Eung Kim, and Pascal Poupart. Point-based value iteration for constrained pomdps. In Twenty-Second International Joint Conference on Artificial Intelligence, 2011.","title":"Point-based value iteration for constrained pomdps","authors":"Dongho Kim, Jaesong Lee, Kee-Eung Kim, Pascal Poupart","year":"2011","venue":"Twenty-Second International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2011/329"},{"position":12,"raw":"[Kress-Gazit et al., 2009] Hadas Kress-Gazit, Georgios E Fainekos, and George J Pappas. Temporal-logic-based re- active mission and motion planning. IEEE transactions on robotics, 25(6):1370-1381, 2009.","title":"Temporal-logic-based reactive mission and motion planning","authors":"Kress-Gazit","year":"2009","venue":"IEEE transactions on robotics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lusena et al., 2001] Christopher Lusena, Judy Goldsmith, and Martin Mundhenk. Nonapproximability results for partially observable markov decision processes. Journal of artificial intelligence research, 14:83-103, 2001.","title":"Nonapproximability results for partially observable markov decision processes","authors":"Christopher Lusena, Judy Goldsmith, Martin Mundhenk","year":"2001","venue":"Journal of artificial intelligence research","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ono et al., 2012] Masahiro Ono, Yoshiaki Kuwata, and J Balaram. Joint chance-constrained dynamic program- ming. In 2012 IEEE 51st IEEE Conference on Decision and Control (CDC), pages 1915-1922. IEEE, 2012.","title":"Joint chance-constrained dynamic programming","authors":"Masahiro Ono, Yoshiaki Kuwata, Balaram","year":"2012","venue":"IEEE 51st IEEE Conference on Decision and Control (CDC)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Papadimitriou and Tsitsiklis, 1987] Christos H Papadim- itriou and John N Tsitsiklis. The complexity of markov decision processes. Mathematics of operations research, 12(3):441-450, 1987.","title":"The complexity of markov decision processes","authors":"Christos H Papadimitriou, John N Tsitsiklis","year":"1987","venue":"Mathematics of operations research","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Poupart et al., 2015] Pascal Poupart, Aarti Malhotra, Pei Pei, Kee-Eung Kim, Bongseok Goh, and Michael Bowl- ing. Approximate linear programming for constrained par- tially observable markov decision processes. In Twenty- Ninth AAAI Conference on Artificial Intelligence, 2015.","title":"Approximate linear programming for constrained partially observable markov decision processes","authors":"Pascal Poupart, Aarti Malhotra, Pei Pei, Kee-Eung Kim, Bongseok Goh, Michael Bowling","year":"2015","venue":"Twenty-Ninth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Puterman, 2014] Martin L Puterman. Markov decision pro- cesses: discrete stochastic dynamic programming. John Wiley & Sons, 2014.","title":"Markov decision processes: discrete stochastic dynamic programming","authors":"Martin L Puterman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Russell and Norvig, 2016] Stuart J Russell and Peter Norvig. Artificial intelligence: a modern approach. Malaysia; Pearson Education Limited,, 2016.","title":"Artificial intelligence: a modern approach","authors":"J Stuart, Peter Russell, Norvig","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Santana and Williams, 2015] Pedro Henrique Santana and Brian C Williams. Dynamic execution of temporal plans with sensing actions and bounded risk. In Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.","title":"Dynamic execution of temporal plans with sensing actions and bounded risk","authors":"Pedro, Henrique Santana, Brian C Williams","year":"2015","venue":"Twenty-Fourth International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2015/644"},{"position":20,"raw":"[Santana et al., 2016] Pedro Santana, Sylvie Thiébaux, and Brian Williams. Rao*: an algorithm for chance con- strained pomdps. In Proc. AAAI Conference on Artificial Intelligence, 2016.","title":"Rao*: an algorithm for chance constrained pomdps","authors":"Pedro Santana, Sylvie Thiébaux, Brian Williams","year":"2016","venue":"Proc. AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sondik, 1971] Edward J Sondik. The optimal control of par- tially observable markov decision processes. PhD the sis, Stanford University, 1971.","title":"The optimal control of partially observable markov decision processes","authors":"J Edward, Sondik","year":"1971","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Timmons and Williams, 2015] Eric Timmons and Brian C Williams. Enumerating preferred solutions to conditional simple temporal networks quickly using bounding con- flicts. In Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.","title":"Enumerating preferred solutions to conditional simple temporal networks quickly using bounding conflicts","authors":"Eric Timmons, Brian C Williams","year":"2015","venue":"Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Trevizan et al., 2016] Felipe Trevizan, Sylvie Thiébaux, Pe- dro Santana, and Brian Williams. Heuristic search in dual space for constrained stochastic shortest path problems. In Twenty-Sixth International Conference on Automated Planning and Scheduling, 2016.","title":"Heuristic search in dual space for constrained stochastic shortest path problems","authors":"Felipe Trevizan, Sylvie Thiébaux, Pedro Santana, Brian Williams","year":"2016","venue":"Twenty-Sixth International Conference on Automated Planning and Scheduling","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Undurti and How, 2010] Aditya Undurti and Jonathan P How. An online algorithm for constrained pomdps. In 2010 IEEE International Conference on Robotics and Au- tomation, pages 3966-3973. IEEE, 2010.","title":"An online algorithm for constrained pomdps","authors":"Aditya Undurti, Jonathan P How","year":"2010","venue":"2010 IEEE International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Vazirani, 2013] Vijay V Vazirani. Approximation algo- rithms. Springer Science & Business Media, 2013.","title":"Approximation algorithms","authors":"V Vijay, Vazirani","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}