{"type":"ScholarlyArticle","title":"AGI Safety Literature Review","authors":[{"name":"Tom Everitt","slug":"tom-everitt","given":"Tom","family":"Everitt","affiliation":"Australian National University"},{"name":"Gary Lea","slug":"gary-lea","given":"Gary","family":"Lea","affiliation":"Australian National University"},{"name":"Marcus Hutter","slug":"marcus-hutter","given":"Marcus","family":"Hutter","affiliation":"Australian National University"}],"abstract":"The development of Artificial General Intelligence (AGI) promises to be a major event. Along with its many potential benefits, it also raises serious safety concerns. The intention of this paper is to provide an easily accessible and up-to-date collection of references for the emerging field of AGI safety. A significant number of safety problems for AGI have been identified. We list these, and survey recent research on solving them. We also cover works on how best to think of AGI from the limited knowledge we have today, predictions for when AGI will first be created, and what will happen after its creation. Finally, we review the current public policy on AGI.","doi":"10.24963/ijcai.2018/768","url":"https://ijcai.org/proceedings/2018/768","pdf":"https://ijcai.org/proceedings/2018/0768.pdf","markdown":"https://ijcai.org/proceedings/2018/768.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/768","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Survey track","track":"Survey track","number":768,"pages":{"first":5441,"last":5449},"keywords":["Humans and AI: Ethical Issues in AI","Machine Learning: Reinforcement Learning","Multidisciplinary Topics and Applications: Philosophical and Ethical Issues"],"videos":[],"references":[{"position":1,"raw":"[Amodei et al., 2016] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete Problems in AI Safety, 2016.","title":"Concrete Problems in AI Safety","authors":"Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Armstrong et al., 2016] Stuart Armstrong, Nick Bostrom, and Carl Shulman. Racing to the Precipice: A Model of Artificial Intelligence Development. AI and Society, 31(2):201-206, 2016.","title":"Racing to the Precipice: A Model of Artificial Intelligence Development","authors":"Stuart Armstrong, Nick Bostrom, Carl Shulman","year":"2016","venue":"AI and Society","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Baum, 2017] Seth D. Baum. A Survey of Artificial General Intelligence Projects for Ethics, Risk, and Policy. Techni- cal Report November, Global Catastrophic Risk Institute, 2017.","title":"A Survey of Artificial General Intelligence Projects for Ethics, Risk, and Policy","authors":"Seth D Baum","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bostrom, 2012] Nick Bostrom. The superintelligent will: Motivation and instrumental rationality in advanced artifi- cial agents. Minds and Machines, pages 1-16, 2012.","title":"The superintelligent will: Motivation and instrumental rationality in advanced artificial agents","authors":"Nick Bostrom","year":"2012","venue":"Minds and Machines","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bostrom, 2014] Nick Bostrom. Superintelligence: Paths, Dangers, Strategies. Oxford University Press, 2014.","title":"Superintelligence: Paths, Dangers, Strategies","authors":"Nick Bostrom","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Brundage et al., 2018] Miles Brundage, Shahar Avin, et al. The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation, 2018.","title":"The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation","authors":"Miles Brundage, Shahar Avin","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chalmers, 2010] David J Chalmers. The singularity: A philosophical analysis. Journal of Consciousness Studies, 17(9-1):7-65, 2010.","title":"The singularity: A philosophical analysis","authors":"Chalmers David","year":"2010","venue":"Journal of Consciousness Studies","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Christiano et al., 2017] Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In NIPS., pages 4302-4310, 2017.","title":"Deep reinforcement learning from human preferences","authors":"Paul Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, Dario Amodei","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Davis, 2015] Ernest Davis. Ethical guidelines for a superin- telligence. Artificial Intelligence, 220:121-124, 2015.","title":"Ernest Davis. Ethical guidelines for a superintelligence","authors":"Davis","year":"2015","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Everitt et al., 2016] Tom Everitt, Daniel Filan, Mayank Daswani, and Marcus Hutter. Self-modification of policy and utility function in rational agents. In AGI, pages 1-11, 2016.","title":"Self-modification of policy and utility function in rational agents","authors":"Everitt","year":"2016","venue":"AGI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Everitt et al., 2017] Tom Everitt, Victoria Krakovna, Lau- rent Orseau, Marcus Hutter, and Shane Legg. Reinforce- ment Learning with Corrupted Reward Signal. In IJCAI, pages 4705-4713, 2017.","title":"Reinforcement Learning with Corrupted Reward Signal","authors":"Everitt","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Garrabrant et al., 2017] Scott Garrabrant, Tsvi Benson- Tilsen, Andrew Critch, Nate Soares, and Jessica Tay- lor. A Formal Approach to the Problem of Logical Non- Omniscience. Electronic Proceedings in Theoretical Com- puter Science, 251:221-235, 2017.","title":"A Formal Approach to the Problem of Logical Non-Omniscience","authors":"Scott Garrabrant, Tsvi Benson-Tilsen, Andrew Critch, Nate Soares, Jessica Taylor","year":"2017","venue":"Electronic Proceedings in Theoretical Computer Science","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hadfield-Menell et al., 2016] Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart J Russell. Cooperative Inverse Reinforcement Learning. In NIPS, pages 3909- 3917, 2016.","title":"Cooperative Inverse Reinforcement Learning","authors":"Hadfield-Menell","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hadfield-Menell et al., 2017a] Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart J Russell. The Off-Switch Game. In IJCAI, pages 220-227, 2017.","title":"The Off-Switch Game","authors":"Hadfield-Menell","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hadfield-Menell et al., 2017b] Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan. Inverse Reward Design. In NIPS, pages 6768-6777, 2017.","title":"Inverse Reward Design","authors":"Hadfield-Menell","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hutter, 2005] Marcus Hutter. Universal Artificial Intelli- gence. Springer-Verlag, Berlin, 2005.","title":"Universal Artificial Intelligence","authors":"Marcus Hutter","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hutter, 2012] Marcus Hutter. Can Intelligence Explode? Journal of Consciousness Studies, 19(1-2):143-146, 2012.","title":"Can Intelligence Explode?","authors":"Marcus Hutter, Hutter","year":"2012","venue":"Journal of Consciousness Studies","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Katz et al., 2017] Guy Katz, Clark Barrett, David L. Dill, Kyle Julian, and Mykel J. Kochenderfer. Reluplex: An efficient SMT solver for verifying deep neural networks. In (CAV), pages 97-117, 2017.","title":"Reluplex: An efficient SMT solver for verifying deep neural networks","authors":"Katz","year":"2017","venue":"CAV)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kurzweil, 2005] Ray Kurzweil. The Singularity Is Near. Viking, 2005.","title":"The Singularity Is Near","authors":"Ray Kurzweil","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Legg and Hutter, 2007] Shane Legg and Marcus Hutter. Universal Intelligence: A definition of machine intelli- gence. Minds & Machines, 17(4):391-444, 2007.","title":"Universal Intelligence: A definition of machine intelligence","authors":"Hutter Legg, Shane Legg, Marcus Hutter","year":"2007","venue":"Minds & Machines","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Leike et al., 2017] Jan Leike, Miljan Martic, et al. AI Safety Gridworlds, 2017.","title":"AI Safety Gridworlds","authors":"Jan Leike, Miljan Martic","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Milli et al., 2017] Smitha Milli, Dylan Hadfield-Menell, Anca Dragan, and Stuart J Russell. Should robots be obe- dient? In IJCAI, pages 4754-4760, 2017.","title":"Should robots be obedient?","authors":"Smitha Milli, Dylan Hadfield-Menell, Anca Dragan, Stuart J Russell","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/662"},{"position":23,"raw":"[Omohundro, 2008] Stephen M Omohundro. The Basic AI Drives. In P. Wang, B. Goertzel, and S. Franklin, editors, Artificial General Intelligence, volume 171, pages 483- 493. IOS Press, 2008.","title":"The Basic AI Drives","authors":"M Stephen, Omohundro","year":"2008","venue":"Artificial General Intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Orseau and Armstrong, 2016] Laurent Orseau and Stuart Armstrong. Safely interruptible agents. In 32nd Confer- ence on Uncertainty in Artificial Intelligence., 2016.","title":"Laurent Orseau and Stuart Armstrong. Safely interruptible agents","authors":"Armstrong Orseau","year":"2016","venue":"Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Orseau, 2014] Laurent Orseau. Universal Knowledge- seeking Agents. Theoretical Computer Science, 519:127- 139, 2014.","title":"Laurent Orseau. Universal Knowledgeseeking Agents","authors":"Orseau","year":"2014","venue":"Theoretical Computer Science","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Ring and Orseau, 2011] Mark Ring and Laurent Orseau. Delusion, Survival, and Intelligent Agents. In AGI, pages 1-11. Springer, 2011.","title":"Delusion, Survival, and Intelligent Agents","authors":"Mark Ring, Laurent Orseau","year":"2011","venue":"AGI","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Russell et al., 2016] Stuart J Russell, Daniel Dewey, and Max Tegmark. Research Priorities for Robust and Ben- eficial Artificial Intelligence. AI Magazine, 36(4), 2016.","title":"Research Priorities for Robust and Beneficial Artificial Intelligence","authors":"J Stuart, Daniel Russell, Max Dewey, Tegmark","year":"2016","venue":"AI Magazine","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Soares and Fallenstein, 2017] Nate Soares and Benya Fal- lenstein. Agent Foundations for Aligning Machine In- telligence with Human Interests: A Technical Research Agenda. In V Callaghan et al. editors, The Technologi- cal Singularity: Managing the Journey, chapter 5, pages 103-125. Springer, 2017.","title":"Agent Foundations for Aligning Machine Intelligence with Human Interests: A Technical Research Agenda","authors":"Fallenstein ; Benya Soares, Fallenstein","year":"2017","venue":"The Technological Singularity: Managing the Journey","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Soares et al., 2015] Nate Soares, Benya Fallenstein, Eliezer S Yudkowsky, and Stuart Armstrong. Corrigibil- ity. In AAAI Workshop on AI and Ethics, pages 74-82, 2015.","title":"Corrigibility","authors":"Nate Soares, Benya Fallenstein, S Eliezer, Stuart Yudkowsky, Armstrong","year":"2015","venue":"AAAI Workshop on AI and Ethics","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Sotala and Yampolskiy, 2014] Kaj Sotala and Roman V Yampolskiy. Responses to catastrophic AGI risk: a sur- vey. Physica Scripta, 90(1), 2014.","title":"Kaj Sotala and Roman V Yampolskiy. Responses to catastrophic AGI risk: a survey","authors":"Yampolskiy Sotala","year":"2014","venue":"Physica Scripta","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sotala, 2017] Kaj Sotala. How feasible is the rapid devel- opment of artificial superintelligence? Physica Scripta, 92(11):1-29, 2017.","title":"How feasible is the rapid development of artificial superintelligence?","authors":"Kaj Sotala, Sotala","year":"2017","venue":"Physica Scripta","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Stoica et al., 2017] Ion Stoica, Dawn Song, et al. A Berke- ley View of Systems Challenges for AI. Technical re- port, EECS Department, University of California, Berke- ley, 2017.","title":"A Berkeley View of Systems Challenges for AI","authors":"Ion Stoica, Dawn Song","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Taylor et al., 2016] Jessica Taylor, Eliezer S Yudkowsky, Patrick LaVictoire, and Andrew Critch. Alignment for Advanced Machine Learning Systems. Technical report, MIRI, 2016.","title":"Alignment for Advanced Machine Learning Systems","authors":"Jessica Taylor, S Eliezer, Patrick Yudkowsky, Andrew Lavictoire, Critch","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}