{"type":"ScholarlyArticle","title":"Expectation Optimization with Probabilistic Guarantees in POMDPs with Discounted-Sum Objectives","authors":[{"name":"Krishnendu Chatterjee","slug":"krishnendu-chatterjee","given":"Krishnendu","family":"Chatterjee","affiliation":"Institute of Science and Technology Austria , Klosterneuburg , Austria"},{"name":"Adrián Elgyütt","slug":"adrian-elgyutt","given":"Adrián","family":"Elgyütt","affiliation":"Institute of Science and Technology Austria , Klosterneuburg , Austria"},{"name":"Petr Novotný","slug":"petr-novotny","given":"Petr","family":"Novotný","affiliation":"Institute of Science and Technology Austria , Klosterneuburg , Austria"},{"name":"Owen Rouillé","slug":"owen-rouille","given":"Owen","family":"Rouillé","affiliation":"École Normale Supérieure de Rennes , Rennes , France"}],"abstract":"Partially-observable Markov decision processes (POMDPs) with discounted-sum payoff are a standard framework to model a wide range of problems related to decision making under uncertainty. Traditionally, the goal has been to obtain policies that optimize the expectation of the discounted-sum payoff. A key drawback of the expectation measure is that even low probability events with extreme payoff can significantly affect the expectation, and thus the obtained policies are not necessarily risk averse. An alternate approach is to optimize the probability that the payoff is above a certain threshold, which allows to obtain risk-averse policies, but ignore optimization of the expectation. We consider the expectation optimization with probabilistic guarantee (EOPG) problem where the goal is to optimize the expectation ensuring that the payoff is above a given threshold with at least a specified probability. We present several results on the EOPG problem, including the first algorithm to solve it.","doi":"10.24963/ijcai.2018/652","url":"https://ijcai.org/proceedings/2018/652","pdf":"https://ijcai.org/proceedings/2018/0652.pdf","markdown":"https://ijcai.org/proceedings/2018/652.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/652","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Planning and Scheduling","track":"Main track","number":652,"pages":{"first":4692,"last":4699},"keywords":["Agent-based and Multi-agent Systems: Formal Verification, Validation and Synthesis","Uncertainty in AI: Markov Decision Processes","Planning and Scheduling: Planning with Incomplete information","Planning and Scheduling: POMDPs","Planning and Scheduling: Planning under Uncertainty","Planning and Scheduling: Theoretical Foundations of Planning"],"videos":[],"references":[{"position":1,"raw":"[AI-Toolbox, 2017] AI-Toolbox. AI-Toolbox [Computer Software]. https://github.com/Svalorzen/AI-Toolbox, 2017.","title":"AI-Toolbox. AI-Toolbox","authors":null,"year":"2017","venue":null,"doi":null,"url":"https://github.com/Svalorzen/AI-Toolbox,2017","internal_url":null},{"position":2,"raw":"[Altman, 1999] Eitan Altman. Constrained Markov decision processes, volume 7. CRC Press, 1999.","title":"Constrained Markov decision processes","authors":"Eitan Altman","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bruyère et al., 2014] Véronique Bruyère, Emmanuel Filiot, Mickael Randour, and Jean-Franc ¸ois Raskin. Meet your expectations with guarantees: Beyond worst-case synthesis in quantitative games. In STACS, pages 199-213. Schloss Dagstuhl -Leibniz-Zentrum fuer Informatik, 2014.","title":"Meet your expectations with guarantees: Beyond worst-case synthesis in quantitative games","authors":"Véronique Bruyère, Emmanuel Filiot, Mickael Randour, Jean-Franc ¸ois Raskin","year":"2014","venue":"STACS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cassandra, 1998] A.R. Cassandra. Exact and approximate algorithms for partially observable Markov decision pro- cesses. PhD thesis, 1998.","title":"Exact and approximate algorithms for partially observable Markov decision processes","authors":"A R Cassandra","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chatterjee et al., 2015a] K. Chatterjee, M. Chmelik, R. Gupta, and A. Kanodia. Optimal cost almost-sure reachability in POMDPs. In AAAI. AAAI Press, 2015.","title":"Optimal cost almost-sure reachability in POMDPs","authors":"Chatterjee","year":"2015","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chatterjee et al., 2015b] Krishnendu Chatterjee, Zuzana Komárková, and Jan Kretínský. Unifying two views on multiple mean-payoff objectives in Markov decision pro- cesses. In LICS, pages 244-256. IEEE Computer Society, 2015.","title":"Unifying two views on multiple mean-payoff objectives in Markov decision processes","authors":"Chatterjee","year":"2015","venue":"LICS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chatterjee et al., 2017] Krishnendu Chatterjee, Petr Novotný, Guillermo A. Pérez, Jean-Franc ¸ois Raskin, and Dorde Zikelic. Optimizing expectation with guarantees in POMDPs. In AAAI, pages 3725-3732. AAAI Press, 2017.","title":"Optimizing expectation with guarantees in POMDPs","authors":"Krishnendu Chatterjee, Petr Novotný, Guillermo A Pérez, Jean-Franc ¸ois Raskin, Dorde Zikelic","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chatterjee et al., 2018] K. Chatterjee, A. Elgyütt, P. Novotný, and O. Rouillé. Expectation optimiza- tion with probabilistic guarantees in POMDPs with discounted-sum objectives. CoRR, abs/1804.10601, 2018.","title":"Expectation optimization with probabilistic guarantees in POMDPs with discounted-sum objectives","authors":"K Chatterjee, A Elgyütt, P Novotný, O Rouillé","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Defourny et al., 2008] Boris Defourny, Damien Ernst, and Louis Wehenkel. Risk-aware decision making and dynamic programming. Technical report, 2008.","title":"Risk-aware decision making and dynamic programming","authors":"Boris Defourny, Damien Ernst, Louis Wehenkel","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Feinberg and Shwartz, 1995] Eugene A Feinberg and Adam Shwartz. Constrained markov decision models with weighted discounted rewards. Mathematics of Operations Research, 20(2):302-320, 1995.","title":"Constrained markov decision models with weighted discounted rewards","authors":"Eugene A Feinberg, Adam Shwartz","year":"1995","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hou et al., 2016] Ping Hou, William Yeoh, and Pradeep Varakantham. Solving risk-sensitive POMDPs with and without cost observations. In AAAI, pages 3138-3144. AAAI Press, 2016.","title":"Solving risk-sensitive POMDPs with and without cost observations","authors":"Hou","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Howard, 1960] H. Howard. Dynamic Programming and Markov Processes. MIT Press, 1960.","title":"Dynamic Programming and Markov Processes","authors":"H Howard","year":"1960","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kaelbling et al., 1996] L. P. Kaelbling, M. L. Littman, and A. W. Moore. Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4:237-285, 1996.","title":"Reinforcement learning: A survey","authors":"L P Kaelbling, M L Littman, A W Moore","year":"1996","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kaelbling et al., 1998] L. P. Kaelbling, M. L. Littman, and A. R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1):99-134, 1998.","title":"Planning and acting in partially observable stochastic domains","authors":"L P Kaelbling, M L Littman, A R Cassandra","year":"1998","venue":"Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kim et al., 2011] Dongho Kim, Jaesong Lee, Kee-Eung Kim, and Pascal Poupart. Point-based value iteration for constrained POMDPs. In IJCAI, pages 1968-1974, 2011.","title":"Point-based value iteration for constrained POMDPs","authors":"Dongho Kim, Jaesong Lee, Kee-Eung Kim, Pascal Poupart","year":"2011","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2011/329"},{"position":16,"raw":"[Kocsis and Szepesvári, 2006] Levente Kocsis and Csaba Szepesvári. Bandit Based Monte-Carlo Planning. In ECML, volume 4212 of LNCS, pages 282-293. Springer, 2006.","title":"Bandit Based Monte-Carlo Planning","authors":"Szepesvári Kocsis, Levente Kocsis, Csaba Szepesvári","year":"2006","venue":"ECML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kress-Gazit et al., 2009] H. Kress-Gazit, G. E. Fainekos, and G. J. Pappas. Temporal-logic-based reactive mission and motion planning. IEEE Transactions on Robotics, 25(6):1370-1381, 2009.","title":"Temporal-logic-based reactive mission and motion planning","authors":"H Kress-Gazit, G E Fainekos, G J Pappas","year":"2009","venue":"IEEE Transactions on Robotics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kurniawati et al., 2008] H. Kurniawati, D. Hsu, and W.S. Lee. SARSOP: Efficient point-based POMDP planning by approximating optimally reachable belief spaces. In Robotics: Science and Systems, pages 65-72, 2008.","title":"SARSOP: Efficient point-based POMDP planning by approximating optimally reachable belief spaces","authors":"H Kurniawati, D Hsu, W S Lee","year":"2008","venue":"Robotics: Science and Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Littman, 1996] M. L. Littman. Algorithms for Sequential Decision Making. PhD thesis, Brown University, 1996.","title":"Algorithms for Sequential Decision Making","authors":"M L Littman","year":"1996","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Papadimitriou and Tsitsiklis, 1987] C. H. Papadimitriou and J. N. Tsitsiklis. The complexity of Markov decision pro- cesses. Math. Oper. Res., 12:441-450, 1987.","title":"The complexity of Markov decision processes","authors":"C H Papadimitriou, J N Tsitsiklis","year":"1987","venue":"Math. Oper. Res","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Poupart et al., 2015] Pascal Poupart, Aarti Malhotra, Pei Pei, Kee-Eung Kim, Bongseok Goh, and Michael Bowling. Ap- proximate linear programming for constrained partially observable Markov decision processes. In AAAI, pages 3342-3348. AAAI Press, 2015.","title":"Approximate linear programming for constrained partially observable Markov decision processes","authors":"Pascal Poupart, Aarti Malhotra, Pei Pei, Kee-Eung Kim, Bongseok Goh, Michael Bowling","year":"2015","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Puterman, 2005] M L Puterman. Markov Decision Pro- cesses. Wiley-Interscience, 2005.","title":"Markov Decision Processes","authors":"M L Puterman","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Randour et al., 2015] Mickael Randour, Jean-Franc ¸ois Raskin, and Ocan Sankur. Variations on the stochastic shortest path problem. In VMCAI, volume 8931 of LNCS, pages 1-18. Springer, 2015.","title":"Variations on the stochastic shortest path problem","authors":"Mickael Randour, Jean-Franc ¸ois Raskin, Ocan Sankur","year":"2015","venue":"VMCAI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ross et al., 2008] Stéphane Ross, Joelle Pineau, Sébastien Paquet, and Brahim Chaib-draa. Online planning algo- rithms for POMDPs. J. Artif. Intell. Res. (JAIR), 32:663- 704, 2008.","title":"Online planning algorithms for POMDPs","authors":"Stéphane Ross, Joelle Pineau, Sébastien Paquet, Brahim Chaib-Draa","year":"2008","venue":"J. Artif. Intell. Res. (JAIR)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Russell and Norvig, 2010] Stuart J. Russell and Peter Norvig. Artificial Intelligence -A Modern Approach (3. internat. ed.). Pearson Education, 2010.","title":"Artificial Intelligence -A Modern Approach","authors":"Stuart J Russell, Peter Norvig","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Santana et al., 2016] Pedro Santana, Sylvie Thiébaux, and Brian C. Williams. RAO*: An algorithm for chance- constrained POMDP's. In AAAI, pages 3308-3314. AAAI Press, 2016.","title":"An algorithm for chanceconstrained POMDP's","authors":"Pedro Santana, Sylvie Thiébaux, Brian C Williams, Rao","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Silver and Veness, 2010] David Silver and Joel Veness. Monte-Carlo planning in large POMDPs. In NIPS 23, pages 2164-2172.","title":"Monte-Carlo planning in large POMDPs","authors":"Veness Silver, David Silver, Joel Veness","year":"2010","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"Curran Associates, Inc., 2010.","title":null,"authors":"Curran Associates, Inc","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Smith and Simmons, 2004] T. Smith and R. Simmons. Heuristic search value iteration for POMDPs. In UAI, pages 520-527. AUAI Press, 2004.","title":"Heuristic search value iteration for POMDPs","authors":"T Smith, R Simmons","year":"2004","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Sprauel et al., 2014] Jonathan Sprauel, Andrey Kolobov, and Florent Teichteil-Königsbuch. Saturated path-constrained mdp: Planning under uncertainty and deterministic model- checking constraints. In AAAI, pages 2367-2373, 2014.","title":"Saturated path-constrained mdp: Planning under uncertainty and deterministic modelchecking constraints","authors":"Jonathan Sprauel, Andrey Kolobov, Florent Teichteil-Königsbuch","year":"2014","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Undurti and How, 2010] Aditya Undurti and Jonathan P How. An online algorithm for constrained POMDPs. In ICRA, pages 3966-3973. IEEE, 2010.","title":"An online algorithm for constrained POMDPs","authors":"Aditya Undurti, Jonathan P How","year":"2010","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Ye et al., 2017] Nan Ye, Adhiraj Somani, David Hsu, and Wee Sun Lee. DESPOT: Online POMDP planning with regularization. J. Artif. Intell. Res., 58:231-266, 2017.","title":"DESPOT: Online POMDP planning with regularization","authors":"Nan Ye, Adhiraj Somani, David Hsu, Wee Sun, Lee","year":"2017","venue":"J. Artif. Intell. Res","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}