{"type":"ScholarlyArticle","title":"Minimax-Regret Querying on Side Effects for Safe Optimality in Factored Markov Decision Processes","authors":[{"name":"Shun Zhang","slug":"shun-zhang","given":"Shun","family":"Zhang","affiliation":"Computer Science and Engineering , University of Michigan"},{"name":"Edmund H. Durfee","slug":"edmund-h-durfee","given":"Edmund H.","family":"Durfee","affiliation":"Computer Science and Engineering , University of Michigan"},{"name":"Satinder Singh","slug":"satinder-singh","given":"Satinder","family":"Singh","affiliation":"Computer Science and Engineering , University of Michigan"}],"abstract":"As it achieves a goal on behalf of its human user, an autonomous agent's actions may have side effects that change features of its environment in ways that negatively surprise its user. An agent that can be trusted to operate safely should thus only change features the user has explicitly permitted.  We formalize this problem, and develop a planning algorithm that avoids potentially negative side effects given what the agent knows about (un)changeable features. Further, we formulate a provably minimax-regret querying strategy for the agent to selectively ask the user about features that it hasn't explicitly been told about. We empirically show how much faster it is than a more exhaustive approach and how much better its queries are than those found by the best known heuristic.","doi":"10.24963/ijcai.2018/676","url":"https://ijcai.org/proceedings/2018/676","pdf":"https://ijcai.org/proceedings/2018/0676.pdf","markdown":"https://ijcai.org/proceedings/2018/676.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/676","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Planning and Scheduling","track":"Main track","number":676,"pages":{"first":4867,"last":4873},"keywords":["Planning and Scheduling: Markov Decisions Processes","Planning and Scheduling: Planning with Incomplete information","Humans and AI: Human-AI Collaboration"],"videos":[],"references":[{"position":1,"raw":"[Achiam et al., 2017] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimiza- tion. In Proc. Int. Conf. on Machine Learning (ICML), pages 22-31, 2017.","title":"Constrained policy optimization","authors":"Achiam","year":"2017","venue":"Proc. Int. Conf. on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Amin et al., 2017] Kareem Amin, Nan Jiang, and Satinder Singh. Repeated inverse reinforcement learning. In Adv. in Neural Info. Proc. Sys. (NIPS), pages 1813-1822, 2017.","title":"Repeated inverse reinforcement learning","authors":"Kareem Amin, Nan Jiang, Satinder Singh","year":"2017","venue":"Adv. in Neural Info. Proc. Sys. (NIPS)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Amodei et al., 2016] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565, 2016.","title":"Concrete problems in AI safety","authors":"Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Boutilier et al., 1999] Craig Boutilier, Thomas Dean, and Steve Hanks. Decision-theoretic planning: Structural as- sumptions and computational leverage. J. of Artificial In- telligence Research (JAIR), 11(1):94, 1999.","title":"Decision-theoretic planning: Structural assumptions and computational leverage","authors":"Craig Boutilier, Thomas Dean, Steve Hanks","year":"1999","venue":"J. of Artificial Intelligence Research (JAIR)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cohn et al., 2011] Robert Cohn, Edmund Durfee, and Satinder Singh. Comparing action-query strategies in semi-autonomous agents. In Int. Conf. on Autonomous Agents and Multiagent Systems, pages 1287-1288, 2011.","title":"Comparing action-query strategies in semi-autonomous agents","authors":"Robert Cohn, Edmund Durfee, Satinder Singh","year":"2011","venue":"Int. Conf. on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dolgov and Durfee, 2006] Dmitri A. Dolgov and Ed- mund H. Durfee. Symmetric approximate linear program- ming for factored MDPs with application to constrained problems. Annals of Mathematics and Artificial Intelli- gence, 47(3):273-293, Aug 2006.","title":"Symmetric approximate linear programming for factored MDPs with application to constrained problems","authors":"Dmitri A Dolgov, Edmund H Durfee","year":"2006","venue":"Annals of Mathematics and Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Garcıa and Fernández, 2015] Javier Garcıa and Fernando Fernández. A comprehensive survey on safe reinforce- ment learning. J. of Machine Learning Research (JMLR), 16(1):1437-1480, 2015.","title":"Javier Garcıa and Fernando Fernández. A comprehensive survey on safe reinforcement learning","authors":"Fernández Garcıa","year":"2015","venue":"J. of Machine Learning Research (JMLR)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hadfield-Menell et al., 2017] Dylan Hadfield-Menell, Smitha Milli, Stuart J Russell, Pieter Abbeel, and Anca Dragan. Inverse reward design. In Adv. in Neural Info. Processing Systems (NIPS), pages 6749-6758, 2017.","title":"Inverse reward design","authors":"Hadfield-Menell","year":"2017","venue":"Adv. in Neural Info. Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hans et al., 2008] Alexander Hans, Daniel Schneegaß, An- ton Maximilian Schäfer, and Steffen Udluft. Safe explo- ration for reinforcement learning. In Euro. Symp. on Arti- ficial Neural Networks (ESANN), pages 143-148, 2008.","title":"Safe exploration for reinforcement learning","authors":"Alexander Hans, Daniel Schneegaß, Anton, Maximilian Schäfer, Steffen Udluft","year":"2008","venue":"Euro. Symp. on Artificial Neural Networks (ESANN)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kolobov et al., 2012] Andrey Kolobov, Mausam, and Daniel S. Weld. A theory of goal-oriented MDPs with dead ends. In Proc. Conf. on Uncertainty in Artificial Intelligence (UAI), pages 438-447, 2012.","title":"A theory of goal-oriented MDPs with dead ends","authors":"Andrey Kolobov, Daniel S Mausam, Weld","year":"2012","venue":"Proc. Conf. on Uncertainty in Artificial Intelligence (UAI)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Laskey et al., 2016] Michael Laskey, Sam Staszak, Wesley Yu-Shu Hsieh, Jeffrey Mahler, Florian T Pokorny, Anca D Dragan, and Ken Goldberg. SHIV: Reducing supervisor burden in DAgger using support vectors for efficient learn- ing from demonstrations in high dimensional state spaces. In IEEE Int. Conf. on Robotics and Automation (ICRA), pages 462-469, 2016.","title":"SHIV: Reducing supervisor burden in DAgger using support vectors for efficient learning from demonstrations in high dimensional state spaces","authors":"Michael Laskey, Sam Staszak, Wesley, Yu-Shu Hsieh, Jeffrey Mahler, Florian T Pokorny, Anca D Dragan, Ken Goldberg","year":"2016","venue":"IEEE Int. Conf. on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Leike et al., 2017] Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A Ortega, Tom Everitt, Andrew Lefrancq, Laurent Orseau, and Shane Legg. AI safety grid- worlds. arXiv preprint arXiv:1711.09883, 2017.","title":"AI safety gridworlds","authors":"Leike","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Milli et al., 2017] Smitha Milli, Dylan Hadfield-Menell, Anca D. Dragan, and Stuart J. Russell. Should robots be obedient? In Proc. Int. Joint Conf. on Artificial Intelli- gence (IJCAI), pages 4754-4760, 2017.","title":"Should robots be obedient?","authors":"Smitha Milli, Dylan Hadfield-Menell, Anca D Dragan, Stuart J Russell","year":"2017","venue":"Proc. Int. Joint Conf. on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2017/662"},{"position":14,"raw":"[Moldovan and Abbeel, 2012] Teodor M. Moldovan and Pieter Abbeel. Safe exploration in Markov decision pro- cesses. In Proc. Int. Conf. on Machine Learning (ICML), pages 1711-1718, 2012.","title":"Safe exploration in Markov decision processes","authors":"Teodor M Moldovan, Pieter Abbeel","year":"2012","venue":"Proc. Int. Conf. on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nilim and El Ghaoui, 2005] Arnab Nilim and Laurent El Ghaoui. Robust control of Markov decision processes with uncertain transition matrices. Operations Research, 53(5):780-798, 2005.","title":"Arnab Nilim and Laurent El Ghaoui. Robust control of Markov decision processes with uncertain transition matrices","authors":"El Nilim, Ghaoui","year":"2005","venue":"Operations Research","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Regan and Boutilier, 2009] Kevin Regan and Craig Boutilier. Regret-based reward elicitation for Markov decision processes. In Proc. Conf. on Uncertainty in Artificial Intelligence (UAI), pages 444-451, 2009.","title":"Regret-based reward elicitation for Markov decision processes","authors":"Kevin Regan, Craig Boutilier","year":"2009","venue":"Proc. Conf. on Uncertainty in Artificial Intelligence (UAI)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Regan and Boutilier, 2010] Kevin Regan and Craig Boutilier. Robust policy computation in reward-uncertain MDPs using nondominated policies. In Assoc. for Adv. of Artificial Intelligence (AAAI), pages 1127-1133, 2010.","title":"Robust policy computation in reward-uncertain MDPs using nondominated policies","authors":"Kevin Regan, Craig Boutilier","year":"2010","venue":"Assoc. for Adv. of Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Teichteil-Königsbuch, 2012] Florent Teichteil-Königsbuch. Stochastic safest and shortest path problems. In Proc. As- soc. for Adv. of Artificial Intelligence (AAAI), pages 1825- 1831, 2012.","title":"Stochastic safest and shortest path problems","authors":"Florent Teichteil-Königsbuch","year":"2012","venue":"Proc. Assoc. for Adv. of Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Viappiani and Boutilier, 2009] Paolo Viappiani and Craig Boutilier. Regret-based optimal recommendation sets in conversational recommender systems. In Proc. ACM Conf. on Recommender Systems, pages 101-108, 2009.","title":"Paolo Viappiani and Craig Boutilier. Regret-based optimal recommendation sets in conversational recommender systems","authors":"Boutilier Viappiani","year":"2009","venue":"Proc. ACM Conf. on Recommender Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Weng and Zanuttini, 2013] Paul Weng and Bruno Zanuttini. Interactive value iteration for Markov decision processes with unknown rewards. In Proc. Int. Joint Conf. on Artifi- cial Intelligence (IJCAI), pages 2415-2421, 2013.","title":"Interactive value iteration for Markov decision processes with unknown rewards","authors":"Paul Weng, Bruno Zanuttini","year":"2013","venue":"Proc. Int. Joint Conf. on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2013/355"},{"position":21,"raw":"[Witwicki and Durfee, 2010] Stefan J Witwicki and Ed- mund H Durfee. Influence-based policy abstraction for weakly-coupled Dec-POMDPs. In Proc. Int. Conf. Auto. Planning and Scheduling (ICAPS), pages 185-192, 2010.","title":"Influence-based policy abstraction for weakly-coupled Dec-POMDPs","authors":"J Stefan, Edmund H Witwicki, Durfee","year":"2010","venue":"Proc. Int. Conf. Auto. Planning and Scheduling (ICAPS)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang et al., 2017] Shun Zhang, Edmund Durfee, and Satinder Singh. Approximately-optimal queries for plan- ning in reward-uncertain Markov decision processes. In Proc. Int. Conf. on Automated Planning and Scheduling (ICAPS), pages 339-347, 2017.","title":"Approximately-optimal queries for planning in reward-uncertain Markov decision processes","authors":"Zhang","year":"2017","venue":"Proc. Int. Conf. on Automated Planning and Scheduling (ICAPS)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}