{"type":"ScholarlyArticle","title":"A Rigorous Risk-aware Linear Approach to Extended Markov Ratio Decision Processes with Embedded Learning","authors":[{"name":"Alexander Zadorojniy","slug":"alexander-zadorojniy","given":"Alexander","family":"Zadorojniy","affiliation":"IBM Research - Israel"},{"name":"Takayuki Osogami","slug":"takayuki-osogami","given":"Takayuki","family":"Osogami","affiliation":"IBM Research -Tokyo"},{"name":"Orit Davidovich","slug":"orit-davidovich","given":"Orit","family":"Davidovich","affiliation":"IBM Research - Israel"}],"abstract":"We consider the problem of risk-aware Markov Decision Processes (MDPs) for Safe AI. We introduce a theoretical framework, Extended Markov Ratio Decision Processes (EMRDP), that incorporates risk into MDPs and embeds environment learning into this framework. We propose an algorithm to find the optimal policy for EMRDP with theoretical guarantees. Under a certain monotonicity assumption, this algorithm runs in strongly-polynomial time both in the discounted and expected average reward models. We validate our algorithm empirically on a Grid World benchmark, evaluating its solution quality, required number of steps, and numerical stability. We find its solution quality to be stable under data noising, while its required number of steps grows with added noise. We observe its numerical stability compared to global methods.","doi":"10.24963/ijcai.2023/608","url":"https://ijcai.org/proceedings/2023/608","pdf":"https://ijcai.org/proceedings/2023/0608.pdf","markdown":"https://ijcai.org/proceedings/2023/608.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/608","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Planning and Scheduling","track":"Main Track","number":608,"pages":{"first":5475,"last":5483},"keywords":["Planning and Scheduling: PS: Markov decisions processes","Uncertainty in AI: UAI: Sequential decision making","Machine Learning: ML: Reinforcement learning"],"videos":[],"references":[{"position":1,"raw":"[Aggarwal et al., 1977] Vijay Aggarwal, Ramaswamy Chan- drasekaran, and Kunhiraman P.K. Nair. Markov ratio de- cision processes. Journal of Optimization Theory and Ap- plications, 21(1):27-37, 1977.","title":"Markov ratio decision processes","authors":"Vijay Aggarwal, Ramaswamy Chandrasekaran, P K Kunhiraman, Nair","year":"1977","venue":"Journal of Optimization Theory and Applications","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Altman, 1999] Eitan Altman. Constrained Markov Deci- sion Processes, volume 7. CRC Press, 1999.","title":"Constrained Markov Decision Processes","authors":"Eitan Altman","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Borgwardt, 2020] Steffen Borgwardt. An LP-based, strongly-polynomial 2-approximation algorithm for sparse wasserstein barycenters. Operational Research, pages 1- 41, 2020.","title":"An LP-based, strongly-polynomial 2-approximation algorithm for sparse wasserstein barycenters","authors":"Steffen Borgwardt","year":"2020","venue":"Operational Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Borkar and Jain, 2014] Vivek Borkar and Rahul Jain. Risk- constrained Markov decision processes. IEEE Transac- tions on Automatic Control, 59(9):2574-2579, 2014.","title":"Riskconstrained Markov decision processes","authors":"Vivek Borkar, Rahul Jain","year":"2014","venue":"IEEE Transactions on Automatic Control","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chow et al., 2015] Yinlam Chow, Aviv Tamar, Shie Man- nor, and Marco Pavone. Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Gar- nett, editors, Advances in Neural Information Processing Systems, volume 28.","title":"Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach","authors":"Chow","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"Curran Associates, Inc., 2015.","title":null,"authors":"Curran Associates, Inc","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[de Ghellinck, 1960] Guy de Ghellinck. Les problémes de décisions séquentielles. Cahiers Centre Etudes Rech. Op- erationnelle, 2:161-179, 1960.","title":"Guy de Ghellinck. Les problémes de décisions séquentielles","authors":"Ghellinck De","year":"1960","venue":"Cahiers Centre Etudes Rech. Operationnelle","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[d'Epenoux, 1963] Franc ¸ois d'Epenoux. A probabilistic pro- duction and inventory problem. Management Science, pages 98-108, 1963.","title":"Franc ¸ois d'Epenoux. A probabilistic production and inventory problem","authors":"Epenoux","year":"1963","venue":"Management Science","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Derman, 1962] Cyrus Derman. On sequential decisions and Markov chains. Management Science, pages 16-24, 1962.","title":"Cyrus Derman. On sequential decisions and Markov chains","authors":"Derman","year":"1962","venue":"Management Science","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Even and Zadorojniy, 2012] Guy Even and Alexander Zadorojniy. Strong polynomiality of the Gass-Saaty shadow-vertex pivoting rule for controlled random walks. Annals of Operations Research, 201(1):159-167, 2012.","title":"Guy Even and Alexander Zadorojniy. Strong polynomiality of the Gass-Saaty shadow-vertex pivoting rule for controlled random walks","authors":"Zadorojniy Even","year":"2012","venue":"Annals of Operations Research","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Feinberg and Huang, 2014] Eugene A. Feinberg and Jeffer- son Huang. The value iteration algorithm is not strongly polynomial for discounted dynamic programming. Oper- ations Research Letters, 42(2):130-131, 2014.","title":"The value iteration algorithm is not strongly polynomial for discounted dynamic programming","authors":"Eugene A Feinberg, Jefferson Huang","year":"2014","venue":"Operations Research Letters","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Feinberg and Huang, 2018] Eugene A. Feinberg and Jeffer- son Huang. Reduction of total-cost and average-cost MDPs with weakly continuous transition probabilities to discounted MDPs. Operations research letters, 46(2):179- 184, 2018.","title":"Reduction of total-cost and average-cost MDPs with weakly continuous transition probabilities to discounted MDPs","authors":"Eugene A Feinberg, Jefferson Huang","year":"2018","venue":"Operations research letters","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Friedmann, 2009] Oliver Friedmann. An exponential lower bound for the parity game strategy improvement algorithm as we know it. In 2009 24th Annual IEEE Symposium on Logic In Computer Science, pages 145-156. IEEE, 2009.","title":"An exponential lower bound for the parity game strategy improvement algorithm as we know it","authors":"Oliver Friedmann","year":"2009","venue":"24th Annual IEEE Symposium on Logic In Computer Science","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Gill and Murray, 1973] Philip E. Gill and Walter Murray. A numerically stable form of the simplex algorithm. Linear algebra and its applications, 7(2):99-138, 1973.","title":"A numerically stable form of the simplex algorithm","authors":"Philip E Gill, Walter Murray","year":"1973","venue":"Linear algebra and its applications","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hansen et al., 2013] Thomas D. Hansen, Peter B. Miltersen, and Uri Zwick. Strategy iteration is strongly polynomial for 2-player turn-based stochastic games with a constant discount factor. Journal of the ACM (JACM), 60(1):1-16, 2013.","title":"Strategy iteration is strongly polynomial for 2-player turn-based stochastic games with a constant discount factor","authors":"D Thomas, Peter B Hansen, Uri Miltersen, Zwick","year":"2013","venue":"Journal of the ACM (JACM)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hollanders et al., 2012] Romain Hollanders, Jean-Charles Delvenne, and Raphaël M Jungers. The complexity of pol- icy iteration is exponential for discounted markov decision processes. In 2012 IEEE 51st IEEE Conference on Deci- sion and Control (CDC), pages 5997-6002. IEEE, 2012.","title":"The complexity of policy iteration is exponential for discounted markov decision processes","authors":"Hollanders","year":"2012","venue":"IEEE 51st IEEE Conference on Decision and Control (CDC)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Khachiyan, 1979] Leonid G. Khachiyan. A polynomial al- gorithms in linear programming. In Doklady Akademii Nauk, volume 244, pages 1093-1096. Russian Academy of Sciences, 1979.","title":"A polynomial algorithms in linear programming","authors":"Leonid G Khachiyan","year":"1979","venue":"Doklady Akademii Nauk","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kostrikov and Nachum, 2020] Ilya Kostrikov and Ofir Nachum. Statistical Bootstrapping for Uncertainty Esti- mation in Off-Policy Evaluation. CoRR, abs/2007.13609, 2020.","title":"Ilya Kostrikov and Ofir Nachum. Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluation","authors":"Nachum Kostrikov","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Meggendorfer, 2022] Tobias Meggendorfer. Risk-Aware Stochastic Shortest Path. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 36, pages 9858- 9867, 2022.","title":"Tobias Meggendorfer. Risk-Aware Stochastic Shortest Path","authors":"Meggendorfer","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Megiddo and Chandrasekaran, 1989] Nimrod Megiddo and Ramaswamy Chandrasekaran. On the ε-perturbation method for avoiding degeneracy. Operations Research Letters, 8(6):305-308, 1989.","title":"Nimrod Megiddo and Ramaswamy Chandrasekaran. On the ε-perturbation method for avoiding degeneracy","authors":"Chandrasekaran Megiddo","year":"1989","venue":"Operations Research Letters","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Meyer, 2000] Carl D. Meyer. Matrix analysis and applied linear algebra, volume 71. Siam, 2000.","title":"Matrix analysis and applied linear algebra","authors":"Carl D Meyer","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Moritz et al., 2018] Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica. Ray: A Distributed Framework for Emerging AI Applications. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pages 561-577, Carlsbad, CA, 2018. USENIX Association.","title":"Ray: A Distributed Framework for Emerging AI Applications","authors":"Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I Jordan, Ion Stoica","year":"2018","venue":"13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Olver and Végh, 2020] Neil Olver and LászlóA Végh. A simpler and faster strongly polynomial algorithm for gen- eralized flow maximization. Journal of the ACM (JACM), 67(2):1-26, 2020.","title":"A simpler and faster strongly polynomial algorithm for generalized flow maximization","authors":"Neil Olver, Lászlóa Végh","year":"2020","venue":"Journal of the ACM (JACM)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Puterman, 1994] Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming.","title":"Markov Decision Processes: Discrete Stochastic Dynamic Programming","authors":"Martin L Puterman","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Scherrer, 2016] Bruno Scherrer. Improved and generalized upper bounds on the complexity of policy iteration. Math- ematics of Operations Research, 41(3):758-774, 2016.","title":"Improved and generalized upper bounds on the complexity of policy iteration","authors":"Bruno Scherrer","year":"2016","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Schrijver, 1998] Alexander Schrijver. Theory of Linear and Integer Programming. John Wiley & Sons, 1998.","title":"Theory of Linear and Integer Programming","authors":"Alexander Schrijver","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Voloshin et al., 2021] Cameron Voloshin, Hoang Minh Le, Nan Jiang, and Yisong Yue. Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning. In Thirty- fifth Conference on Neural Information Processing Sys- tems Datasets and Benchmarks Track (Round 1), 2021.","title":"Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning","authors":"Cameron Voloshin, Minh Hoang, Nan Le, Yisong Jiang, Yue","year":"2021","venue":"Thirtyfifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Walkup and Wets, 1969] David W. Walkup and Roger J.-B. Wets. Lifting projections of convex polyhedra. Pacific Journal of Mathematics, 28(2):465-475, 1969.","title":"Lifting projections of convex polyhedra","authors":"David W Walkup, Roger J.-B Wets","year":"1969","venue":"Pacific Journal of Mathematics","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Ye, 2005] Yinyu Ye. A New Complexity Result on Solving the Markov Decision Problem. Mathematics of Operations Research, 30(3):733-749, 2005.","title":"Yinyu Ye. A New Complexity Result on Solving the Markov Decision Problem","authors":"Ye","year":"2005","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Ye, 2011] Yinyu Ye. The Simplex and Policy-Iteration Methods are Strongly Polynomial for the Markov Deci- sion Problem with a Fixed Discount Rate. Mathematics of Operations Research, 36(4):593-603, 2011.","title":"Yinyu Ye. The Simplex and Policy-Iteration Methods are Strongly Polynomial for the Markov Decision Problem with a Fixed Discount Rate","authors":"Ye","year":"2011","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zadorojniy et al., 2009] Alexander Zadorojniy, Guy Even, and Adam Shwartz. A Strongly Polynomial Algorithm for Controlled Queues. Mathematics of Operations Research, 34(4):992-1007, 2009.","title":"A Strongly Polynomial Algorithm for Controlled Queues","authors":"Alexander Zadorojniy, Guy Even, Adam Shwartz","year":"2009","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}