{"type":"ScholarlyArticle","title":"Simplified Risk-aware Decision Making with Belief-dependent Rewards in Partially Observable Domains (Extended Abstract)","authors":[{"name":"Andrey Zhitnikov","slug":"andrey-zhitnikov","given":"Andrey","family":"Zhitnikov","affiliation":"Technion Autonomous Systems Program (TASP) , Technion -Israel Institute of Technology , Haifa 32000 , Israel"},{"name":"Vadim Indelman","slug":"vadim-indelman","given":"Vadim","family":"Indelman","affiliation":"Department of Aerospace Engineering , Technion -Israel Institute of Technology , Haifa 32000 , Israel"}],"abstract":"It is a long-standing objective to ease the computation burden incurred by the decision-making problem under partial observability. Identifying the sensitivity to simplification of various components of the original problem has tremendous ramifications. Yet, algorithms for decision-making under uncertainty usually lean on approximations or heuristics without quantifying their effect. Therefore, challenging scenarios could severely impair the performance of such methods. In this paper, we extend the decision-making mechanism to the whole by removing standard approximations and considering all previously suppressed stochastic sources of variability. On top of this extension,  we scrutinize the distribution of the return. We begin from a return given a single candidate policy and continue to the pair of returns given a corresponding pair of candidate policies. Furthermore, we present novel stochastic bounds on the return and novel tools, Probabilistic Loss (PLoss) and its online accessible counterpart (PbLoss), to characterize the effect of a simplification.","doi":"10.24963/ijcai.2023/798","url":"https://ijcai.org/proceedings/2023/798","pdf":"https://ijcai.org/proceedings/2023/0798.pdf","markdown":"https://ijcai.org/proceedings/2023/798.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/798","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Journal Track","track":"Journal Track","number":798,"pages":{"first":7001,"last":7005},"keywords":["Planning and Scheduling: PS: POMDPs","Uncertainty in AI: UAI: Sequential decision making","Robotics: ROB: Motion and path planning","Planning and Scheduling: PS: Planning under uncertainty"],"videos":[],"references":[{"position":1,"raw":"[Araya et al., 2010] Mauricio Araya, Olivier Buffet, Vincent Thomas, and Franc ¸cois Charpillet. A pomdp extension with belief-dependent rewards. In Advances in Neural In- formation Processing Systems (NIPS), pages 64-72, 2010.","title":"A pomdp extension with belief-dependent rewards","authors":"Mauricio Araya, Olivier Buffet, Vincent Thomas, Franc ¸cois, Charpillet","year":"2010","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bertsekas, 1995] Dimitri Bertsekas. Dynamic programming and optimal control, volume 1. Athena scientific Belmont, MA, 1995.","title":"Dynamic programming and optimal control","authors":"Dimitri Bertsekas","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Defourny et al., 2008] Boris Defourny, Damien Ernst, and Louis Wehenkel. Risk-aware decision making and dy- namic programming. In NIPS Workshop on Model Un- certainty and Risk in RL, 2008.","title":"Risk-aware decision making and dynamic programming","authors":"Boris Defourny, Damien Ernst, Louis Wehenkel","year":"2008","venue":"NIPS Workshop on Model Uncertainty and Risk in RL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dressel and Kochenderfer, 2017] Louis Dressel and Mykel J. Kochenderfer. Efficient decision-theoretic target localization. In Laura Barbulescu, Jeremy Frank, Mausam, and Stephen F. Smith, editors, Proceedings of the Twenty-Seventh International Conference on Auto- mated Planning and Scheduling, ICAPS 2017, Pittsburgh, Pennsylvania, USA, June 18-23, 2017, pages 70-78. AAAI Press, 2017.","title":"Efficient decision-theoretic target localization","authors":"Kochenderfer Dressel, Louis Dressel, Mykel J Kochenderfer","year":"2017","venue":"Proceedings of the Twenty-Seventh International Conference on Automated Planning and Scheduling, ICAPS 2017","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Elimelech and Indelman, 2022] Khen Elimelech and Vadim Indelman. Simplified decision making in the belief space using belief sparsification. The International Journal of Robotics Research, 41(5):470-496, 2022.","title":"Khen Elimelech and Vadim Indelman. Simplified decision making in the belief space using belief sparsification","authors":"Indelman Elimelech","year":"2022","venue":"The International Journal of Robotics Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fehr et al., 2018] Mathieu Fehr, Olivier Buffet, Vincent Thomas, and Jilles Dibangoye. rho-pomdps have lipschitz- continuous epsilon-optimal value functions. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 6933-6943.","title":"rho-pomdps have lipschitzcontinuous epsilon-optimal value functions","authors":"Mathieu Fehr, Olivier Buffet, Vincent Thomas, Jilles Dibangoye","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Indelman et al., 2015] V. Indelman, L. Carlone, and F. Del- laert. Planning in the continuous domain: a general- ized belief space approach for autonomous navigation in unknown environments. Intl. J. of Robotics Research, 34(7):849-882, 2015.","title":"Planning in the continuous domain: a generalized belief space approach for autonomous navigation in unknown environments","authors":"V Indelman, L Carlone, F Dellaert","year":"2015","venue":"Intl. J. of Robotics Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Indelman, 2016] V. Indelman. No correlations involved: Decision making under uncertainty in a conservative sparse information space. IEEE Robotics and Automation Letters (RA-L), 1(1):407-414, 2016.","title":"No correlations involved: Decision making under uncertainty in a conservative sparse information space","authors":"V Indelman","year":"2016","venue":"IEEE Robotics and Automation Letters (RA-L)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kearns et al., 2002] Michael Kearns, Yishay Mansour, and Andrew Y Ng. A sparse sampling algorithm for near- optimal planning in large markov decision processes. Ma- chine learning, 49(2):193-208, 2002.","title":"A sparse sampling algorithm for nearoptimal planning in large markov decision processes","authors":"Michael Kearns, Yishay Mansour, Andrew Y Ng","year":"2002","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kitanov and Indelman, 2019] A. Kitanov and V. Indel- man. Topological information-theoretic belief space planning with optimality guarantees. arXiv preprint arXiv:1903.00927, 3 2019.","title":"Topological information-theoretic belief space planning with optimality guarantees","authors":"A Kitanov, V Indelman","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Papadimitriou and Tsitsiklis, 1987] C. Papadimitriou and J. Tsitsiklis. The complexity of Markov decision pro- cesses. Mathematics of operations research, 12(3):441- 450, 1987.","title":"The complexity of Markov decision processes","authors":"C Papadimitriou, J Tsitsiklis","year":"1987","venue":"Mathematics of operations research","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Shienman and Indelman, 2022a] M. Shienman and V. In- delman. D2a-bsp: Distilled data association belief space planning with performance guarantees under budget con- straints. In IEEE Intl. Conf. on Robotics and Automation (ICRA), 2022.","title":"Indelman. D2a-bsp: Distilled data association belief space planning with performance guarantees under budget constraints","authors":"Indelman Shienman, M Shienman, V","year":"2022","venue":"IEEE Intl. Conf. on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Shienman and Indelman, 2022b] M. Shienman and V. In- delman. Nonmyopic distilled data association belief space planning under budget constraints. In Proc. of the Intl. Symp. of Robotics Research (ISRR), 2022.","title":"Indelman. Nonmyopic distilled data association belief space planning under budget constraints","authors":"Indelman Shienman, M Shienman, V","year":"2022","venue":"Proc. of the Intl. Symp. of Robotics Research (ISRR)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Sunberg and Kochenderfer, 2018] Zachary Sunberg and Mykel Kochenderfer. Online algorithms for pomdps with continuous state, action, and observation spaces. In Pro- ceedings of the International Conference on Automated Planning and Scheduling, volume 28, 2018.","title":"Online algorithms for pomdps with continuous state, action, and observation spaces","authors":"Kochenderfer Sunberg, Zachary Sunberg, Mykel Kochenderfer","year":"2018","venue":"Proceedings of the International Conference on Automated Planning and Scheduling","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sztyglic and Indelman, 2021] Ori Sztyglic and Vadim In- delman. Online pomdp planning via simplification. arXiv preprint arXiv:2105.05296, 2021.","title":"Ori Sztyglic and Vadim Indelman","authors":"Indelman Sztyglic","year":"2021","venue":"Online pomdp planning via simplification","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sztyglic and Indelman, 2022] Ori Sztyglic and Vadim In- delman. Speeding up online pomdp planning via simplifi- cation. In IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), 2022.","title":"Ori Sztyglic and Vadim Indelman. Speeding up online pomdp planning via simplification","authors":"Indelman Sztyglic","year":"2022","venue":"IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Thrun et al., 2005] S. Thrun, W. Burgard, and D. Fox. Prob- abilistic Robotics. The MIT press, Cambridge, MA, 2005.","title":"Probabilistic Robotics","authors":"S Thrun, W Burgard, D Fox","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Zhitnikov and Indelman, 2022] A. Zhitnikov and V. Indel- man. Simplified risk aware decision making with be- lief dependent rewards in partially observable domains. Artificial Intelligence, Special Issue on \"Risk-Aware Au- tonomous Systems: Theory and Practice\", 2022.","title":"Simplified risk aware decision making with belief dependent rewards in partially observable domains","authors":"A Zhitnikov, V Indelman","year":"2022","venue":"Artificial Intelligence, Special Issue on \"Risk-Aware Autonomous Systems: Theory and Practice","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}