{"type":"ScholarlyArticle","title":"The Off-Switch Game","authors":[{"name":"Dylan Hadfield-Menell","slug":"dylan-hadfield-menell","given":"Dylan","family":"Hadfield-Menell","affiliation":"University of California , Berkeley ,"},{"name":"Anca Dragan","slug":"anca-dragan","given":"Anca","family":"Dragan","affiliation":"University of California , Berkeley ,"},{"name":"Pieter Abbeel","slug":"pieter-abbeel","given":"Pieter","family":"Abbeel","affiliation":"University of California , Berkeley ,; OpenAI ,; International Computer Science Institute (ICSI)"},{"name":"Stuart Russell","slug":"stuart-russell","given":"Stuart","family":"Russell","affiliation":"University of California , Berkeley ,"}],"abstract":"It is clear that one of the primary tools we can use to mitigate the potential risk from a misbehaving AI system is the ability to turn the system off. As the capabilities of AI systems improve, it is important to ensure that such systems do not adopt subgoals that prevent a human from switching the system off. This is a challenge because many formulations of rational agents create strong incentives for self-preservation. This is not caused by a built-in instinct, but because a rational agent will maximize expected utility and cannot achieve whatever objective it has been given if it is dead. Our goal is to study the incentives an agent has to allow itself to be switched off. We analyze a simple game between a human H and a robot R, where H can press R’s off switch but R can disable the off switch. A traditional agent takes its reward function for granted: we show that such agents have an incentive to disable the off switch, except in the special case where H is perfectly rational. Our key insight is that for R to want to preserve its off switch, it needs to be uncertain about the utility associated with the outcome, and to treat H’s actions as important observations about that utility. (R also has no incentive to switch itself off in this setting.) We conclude that giving machines an appropriate level of uncertainty about their objectives leads to safer designs, and we argue that this setting is a useful generalization of the classical AI paradigm of rational agents.","doi":"10.24963/ijcai.2017/32","url":"https://ijcai.org/proceedings/2017/32","pdf":"https://ijcai.org/proceedings/2017/0032.pdf","markdown":"https://ijcai.org/proceedings/2017/32.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/32","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Agent-based and Multi-agent Systems","track":"Main track","number":32,"pages":{"first":220,"last":227},"keywords":["Agent-based and Multi-agent Systems: Agent Theories and Models","Agent-based and Multi-agent Systems: Coordination and cooperation","Agent-based and Multi-agent Systems: Cooperative Games"],"videos":[],"references":[{"position":1,"raw":"[Del Prado, 2015] Guia Marie Del Prado. Here's what Face- book's artificial intelligence expert thinks about the future. Tech Insider 9/23/15, 2015.","title":"Guia Marie Del Prado. Here's what Facebook's artificial intelligence expert thinks about the future","authors":"Del Prado","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Fern et al., 2014] Alan Fern, Sriraam Natarajan, Kshitij Ju- dah, and Prasad Tadepalli. A decision-theoretic model of assistance. Journal of Artificial Intelligence Research, 50(1):71-104, 2014.","title":"A decision-theoretic model of assistance","authors":"Alan Fern, Sriraam Natarajan, Kshitij Judah, Prasad Tadepalli","year":"2014","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Gibbons, 1998] Robert Gibbons. Incentives in organiza- tions. 1998.","title":"Incentives in organizations","authors":"Robert Gibbons","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Hadfield-Menell et al., 2016] Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell. Cooperative in- verse reinforcement learning. In Neural Information Pro- cessing Systems, 2016.","title":"Cooperative inverse reinforcement learning","authors":"Hadfield-Menell","year":"2016","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"ITIF. Are super intelligent computers really a threat to humanity? Debate at the Information Technology Innovation Foundation, 6/30/15, 2015.","title":"Are super intelligent computers really a threat to humanity? Debate at the Information Technology Innovation Foundation","authors":"Itif","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kerr, 1975] Steven Kerr. On the folly of rewarding a, while hoping for b. Academy of Management Journal, 18(4):769-783, 1975.","title":"On the folly of rewarding a, while hoping for b","authors":"Steven Kerr","year":"1975","venue":"Academy of Management Journal","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Omohundro, 2008] Stephen M. Omohundro. The basic AI drives. In Proceedings of the First Conference on Artificial General Intelligence, 2008.","title":"The basic AI drives","authors":"M Stephen, Omohundro","year":"2008","venue":"Proceedings of the First Conference on Artificial General Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Orseau and Armstrong, 2016] Laurent Orseau and Stuart Armstrong. Safely interruptible agents. In Uncertainty in Artificial Intelligence, 2016.","title":"Laurent Orseau and Stuart Armstrong. Safely interruptible agents","authors":"Armstrong Orseau","year":"2016","venue":"Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Russell and Norvig, 2010] Stuart Russell and Peter Norvig. Artificial Intelligence: A Modern Approach. Pearson, 2010.","title":"Artificial Intelligence: A Modern Approach","authors":"Stuart Russell, Peter Norvig","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Russell, 2016] Stuart Russell. Should we fear supersmart robots? Scientific American, 314(June):58-59, 2016.","title":"Should we fear supersmart robots?","authors":"Stuart Russell","year":"2016","venue":"Scientific American","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Soares et al., 2015] Nate Soares, Benja Fallenstein, Stuart Armstrong, and Eliezer Yudkowsky. Corrigibility. In Workshops at the Twenty-Ninth AAAI Conference on Ar- tificial Intelligence, 2015.","title":"Corrigibility","authors":"Nate Soares, Benja Fallenstein, Stuart Armstrong, Eliezer Yudkowsky","year":"2015","venue":"Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Tirole, 2009] Jean Tirole. Cognition and incomplete con- tracts. The American Economic Review, 99(1):265-294, 2009.","title":"Cognition and incomplete contracts","authors":"Tirole","year":"2009","venue":"The American Economic Review","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Turing, 1951] Alan M. Turing. Can digital machines think? Lecture broadcast on BBC Third Programme; typescript at turingarchive.org., 1951.","title":"Can digital machines think? Lecture broadcast on BBC Third Programme","authors":"Alan M Turing","year":"1951","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}