{"type":"ScholarlyArticle","title":"Should Robots be Obedient?","authors":[{"name":"Smitha Milli","slug":"smitha-milli","given":"Smitha","family":"Milli","affiliation":"University of California , Berkeley"},{"name":"Dylan Hadfield-Menell","slug":"dylan-hadfield-menell","given":"Dylan","family":"Hadfield-Menell","affiliation":"University of California , Berkeley"},{"name":"Anca Dragan","slug":"anca-dragan","given":"Anca","family":"Dragan","affiliation":"University of California , Berkeley"},{"name":"Stuart Russell","slug":"stuart-russell","given":"Stuart","family":"Russell","affiliation":"University of California , Berkeley"}],"abstract":"Intuitively, obedience -- following the order that a human gives -- seems like a good property for a robot to have. But, we humans are not perfect and we may give orders that are not best aligned to our preferences. We show that when a human is not perfectly rational then a robot that tries to infer and act according to the human's underlying preferences can always perform better than a robot that simply follows the human's literal order. Thus, there is a tradeoff between the obedience of a robot and the value it can attain for its owner. We investigate how this tradeoff is impacted by the way the robot infers the human's preferences, showing that some methods err more on the side of obedience than others. We then analyze how performance degrades when the robot has a misspecified model of the features that the human cares about or the level of rationality of the human. Finally, we study how robots can start detecting such model misspecification. Overall, our work suggests that there might be a middle ground in which robots intelligently decide when to obey human orders, but err on the side of obedience.","doi":"10.24963/ijcai.2017/662","url":"https://ijcai.org/proceedings/2017/662","pdf":"https://ijcai.org/proceedings/2017/0662.pdf","markdown":"https://ijcai.org/proceedings/2017/662.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/662","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"AI and autonomy track","track":"AI and autonomy track","number":662,"pages":{"first":4754,"last":4760},"keywords":["Technical: Techniques","Social: Human-machine Interaction","Technical: Models"],"videos":[],"references":[{"position":1,"raw":"[Abbeel and Ng, 2004] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learn- ing. In International Conference on Machine learning, page 1. ACM, 2004.","title":"Apprenticeship learning via inverse reinforcement learning","authors":"Abbeel, ; Ng, Pieter Abbeel, Andrew Y Ng","year":"2004","venue":"International Conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Asaro, 2006] Peter M Asaro. What Should We Want From a Robot Ethic. International Review of Information Ethics, 6(12):9-16, 2006.","title":"What Should We Want From a Robot Ethic","authors":"M Peter, Asaro","year":"2006","venue":"International Review of Information Ethics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bostrom, 2014] Nick Bostrom. Superintelligence: Paths, Dangers, Strategies. OUP Oxford, 2014.","title":"Superintelligence: Paths, Dangers, Strategies","authors":"Nick Bostrom","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Choi and Kim, 2011] Jaedeug Choi and Kee-Eung Kim. MAP Inference for Bayesian Inverse Reinforcement Learning. In Advances in Neural Information Processing Systems, pages 1989-1997, 2011.","title":"MAP Inference for Bayesian Inverse Reinforcement Learning","authors":"Kim ; Choi, Jaedeug Choi, Kee-Eung Kim","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Diaconis and Freedman, 1986] Persi Diaconis and David Freedman. On the consistency of Bayes estimates. The Annals of Statistics, pages 1-26, 1986.","title":"On the consistency of Bayes estimates","authors":"Persi Diaconis, David Freedman","year":"1986","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dragan and Srinivasa, 2012] Anca D Dragan and Sid- dhartha S Srinivasa. Formalizing assistive teleoperation. MIT Press, July, 2012.","title":"Formalizing assistive teleoperation","authors":"D Anca, Siddhartha S Dragan, Srinivasa","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Evans et al., 2016] Owain Evans, Andreas Stuhlmüller, and Noah D Goodman. Learning the preferences of igno- rant, inconsistent agents. In Proceedings of the Thirti- eth AAAI Conference on Artificial Intelligence, pages 323- 329. AAAI Press, 2016.","title":"Learning the preferences of ignorant, inconsistent agents","authors":"Owain Evans, Andreas Stuhlmüller, Noah D Goodman","year":"2016","venue":"Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hadfield-Menell et al., 2016] Dylan Hadfield-Menell, Stu- art J Russell, Pieter Abbeel, and Anca Dragan. Coopera- tive Inverse Reinforcement Learning. In Advances in Neu- ral Information Processing Systems, pages 3909-3917, 2016.","title":"Cooperative Inverse Reinforcement Learning","authors":"Hadfield-Menell","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hadfield-Menell et al., 2017] Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell. The Off-Switch Game. In IJCAI, 2017.","title":"The Off-Switch Game","authors":"Hadfield-Menell","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Javdani et al., 2015] Shervin Javdani, Siddhartha S Srini- vasa, and J Andrew Bagnell. Shared autonomy via hind- sight optimization. In Robotics: Science and Systems, 2015.","title":"Shared autonomy via hindsight optimization","authors":"Shervin Javdani, Siddhartha S Srinivasa, Andrew Bagnell","year":"2015","venue":"Robotics: Science and Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kaelbling et al., 1996] Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. Reinforcement Learn- ing: A Survey. Journal of Artificial Intelligence Research, 4:237-285, 1996.","title":"Reinforcement Learning: A Survey","authors":"Leslie Pack, Kaelbling Michael L Littman, Andrew W Moore","year":"1996","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lewis, 2014] Johin Lewis. The Case for Regulating Fully Autonomous Weapons. Yale Law Journal, 124:1309, 2014.","title":"Johin Lewis. The Case for Regulating Fully Autonomous Weapons","authors":"Lewis","year":"2014","venue":"Yale Law Journal","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Michini and How, 2012] Bernard Michini and Jonathan P How. Improving the Efficiency of Bayesian Inverse Re- inforcement Learning. In IEEE International Conference on Robotics and Automation (ICRA), pages 3651-3656. IEEE, 2012.","title":"Improving the Efficiency of Bayesian Inverse Reinforcement Learning","authors":"How Michini, Bernard Michini, Jonathan P How","year":"2012","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ng et al., 2000] Andrew Y Ng, Stuart J Russell, et al. Algo- rithms for inverse reinforcement learning. In International Conference on Machine Learning, pages 663-670, 2000.","title":"Algorithms for inverse reinforcement learning","authors":"Y Andrew, Stuart J Ng, Russell","year":"2000","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Orseau and Armstrong, 2016] Laurent Orseau and Stuart Armstrong. Safely Interruptible Agents. In UAI, 2016.","title":"Safely Interruptible Agents","authors":"Laurent Orseau, Stuart Armstrong","year":"2016","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ramachandran and Amir, 2007] Deepak Ramachandran and Eyal Amir. Bayesian Inverse Reinforcement Learning. In IJCAI, 2007.","title":"Bayesian Inverse Reinforcement Learning","authors":"Deepak Ramachandran, Eyal Amir","year":"2007","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2007/416"},{"position":17,"raw":"[Russell et al., 2015] Stuart Russell, Daniel Dewey, and Max Tegmark. Research Priorities for Robust and Beneficial Artificial Intelligence. AI Magazine, 36(4):105-114, 2015.","title":"Research Priorities for Robust and Beneficial Artificial Intelligence","authors":"Stuart Russell, Daniel Dewey, Max Tegmark","year":"2015","venue":"AI Magazine","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Scheutz and Crowell, 2007] Matthias Scheutz and Charles Crowell. The Burden of Embodied Autonomy: Some Reflections on the Social and Ethical Implications of Au- tonomous Robots. In Workshop on Roboethics at the Inter- national Conference on Robotics and Automation, Rome, 2007.","title":"The Burden of Embodied Autonomy: Some Reflections on the Social and Ethical Implications of Autonomous Robots","authors":"Crowell Scheutz, Matthias Scheutz, Charles Crowell","year":"2007","venue":"Workshop on Roboethics at the International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Soares et al., 2015] Nate Soares, Benja Fallenstein, Stuart Armstrong, and Eliezer Yudkowsky. Corrigibility. In Workshops at the Twenty-Ninth AAAI Conference on Ar- tificial Intelligence, 2015.","title":"Corrigibility","authors":"Nate Soares, Benja Fallenstein, Stuart Armstrong, Eliezer Yudkowsky","year":"2015","venue":"Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Teitelman, 1970] Warren Teitelman. Toward a Program- ming Laboratory. Software Engineering Techniques, page 108, 1970.","title":"Toward a Programming Laboratory","authors":"Warren Teitelman","year":"1970","venue":"Software Engineering Techniques","doi":null,"url":null,"internal_url":"/proceedings/1969/1"},{"position":21,"raw":"[Weld and Etzioni, 1994] Daniel Weld and Oren Etzioni. The First Law of Robotics (a call to arms). In AAAI, vol- ume 94, pages 1042-1047, 1994.","title":"The First Law of Robotics (a call to arms)","authors":"Etzioni Weld, Daniel Weld, Oren Etzioni","year":"1994","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ziebart et al., 2008] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. Maximum En- tropy Inverse Reinforcement Learning. In AAAI, volume 8, pages 1433-1438. Chicago, IL, USA, 2008.","title":"Maximum Entropy Inverse Reinforcement Learning","authors":"Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey","year":"2008","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}