{"type":"ScholarlyArticle","title":"Pitfalls of Learning a Reward Function Online","authors":[{"name":"Stuart Armstrong","slug":"stuart-armstrong","given":"Stuart","family":"Armstrong","affiliation":"Future of Humanity Institute , Oxford University , UK; Machine Intelligence Research Institute , Berkeley , USA"},{"name":"Jan Leike","slug":"jan-leike","given":"Jan","family":"Leike","affiliation":"DeepMind , London , UK"},{"name":"Laurent Orseau","slug":"laurent-orseau","given":"Laurent","family":"Orseau","affiliation":"DeepMind , London , UK"},{"name":"Shane Legg","slug":"shane-legg","given":"Shane","family":"Legg","affiliation":"DeepMind , London , UK"}],"abstract":"In some agent designs like inverse reinforcement learning an agent needs to learn its own reward function. Learning the reward function and optimising for it are typically two different processes, usually performed at different stages. We consider a continual (``one life'') learning approach where the agent both learns the reward function and optimises for it at the same time. We show that this comes with a number of pitfalls, such as deliberately manipulating the learning process in one direction, refusing to learn, ``learning'' facts already known to the agent, and making decisions that are strictly dominated (for all relevant reward functions). We formally introduce two desirable properties: the first is `unriggability', which prevents the agent from steering the learning process in the direction of a reward function that is easier to optimise. The second is `uninfluenceability', whereby the reward-function learning process operates by learning facts about the environment. We show that an uninfluenceable process is automatically unriggable, and if the set of possible environments is sufficiently large, the converse is true too.","doi":"10.24963/ijcai.2020/221","url":"https://ijcai.org/proceedings/2020/221","pdf":"https://ijcai.org/proceedings/2020/0221.pdf","markdown":"https://ijcai.org/proceedings/2020/221.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/221","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Knowledge Representation and Reasoning","track":"Main track","number":221,"pages":{"first":1592,"last":1600},"keywords":["Knowledge Representation and Reasoning: Belief Change, Belief Merging","Machine Learning: Reinforcement Learning","Agent-based and Multi-agent Systems: Human-Agent Interaction"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25169/file_25169.m3u8","length_sec":284,"provider":"hls"}],"references":[{"position":1,"raw":"[Abbeel and Ng, 2004] Pieter Abbeel and Andrew Ng. Ap- prenticeship Learning Via Inverse Reinforcement Learn- ing. In International Conference on Machine Learning, pages 1-8, 2004.","title":"Apprenticeship Learning Via Inverse Reinforcement Learning","authors":"Ng Abbeel, Pieter Abbeel, Andrew Ng","year":"2004","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Abel et al., 2017] David Abel, John Salvatier, Andreas Stuhlmüller, and Owain Evans. Agent-Agnostic Human- in-the-Loop Reinforcement Learning. arXiv preprint arXiv:1701.04079, 2017.","title":"Agent-Agnostic Humanin-the-Loop Reinforcement Learning","authors":"David Abel, John Salvatier, Andreas Stuhlmüller, Owain Evans","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Akrour et al., 2012] Riad Akrour, Marc Schoenauer, and Michèle Sebag. April: Active Preference Learning-based Reinforcement Learning. In Joint European Confer- ence on Machine Learning and Knowledge Discovery in Databases, pages 116-131. Springer, 2012.","title":"April: Active Preference Learning-based Reinforcement Learning","authors":"Akrour","year":"2012","venue":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Amin and Singh, 2016] Kareem Amin and Satinder Singh. Towards Resolving Unidentifiability in Inverse Reinforce- ment Learning. arXiv preprint, arXiv:1601.06569, 2016.","title":"Towards Resolving Unidentifiability in Inverse Reinforcement Learning","authors":"Singh Amin, Kareem Amin, Satinder Singh","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Armstrong et al., 2020] Stuart Armstrong, Jan Leike, Lau- rent Orseau, and Shane Legg. Pitfalls of Learning a Re- ward Function Online. arXiv e-prints, April 2020.","title":"Pitfalls of Learning a Reward Function Online","authors":"Stuart Armstrong, Jan Leike, Laurent Orseau, Shane Legg","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Armstrong, 2017] Stuart Armstrong. Good and Safe Uses of AI Oracles. arXiv preprint arXiv:1711.05541, 2017.","title":"Good and Safe Uses of AI Oracles","authors":"Stuart Armstrong","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bostrom, 2014] Nick Bostrom. Superintelligence: Paths, dangers, strategies. Oxford University Press, 2014.","title":"Superintelligence: Paths, dangers, strategies","authors":"Nick Bostrom","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Buntine, 1994] Wray L Buntine. Operations for Learning with Graphical Models. Journal of Artificial Intelligence Research, 2:159-225, 1994.","title":"Operations for Learning with Graphical Models","authors":"Wray L Buntine","year":"1994","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Choi and Kim, 2011] Jaedeug Choi and Kee-Eung Kim. In- verse Reinforcement Learning in Partially Observable En- vironments. Journal of Machine Learning Research, 12:691-730, 2011.","title":"Inverse Reinforcement Learning in Partially Observable Environments","authors":"Kim ; Choi, Jaedeug Choi, Kee-Eung Kim","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Christiano et al., 2017] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep Reinforcement Learning from Human Preferences. In Advances in Neural Information Processing Systems, pages 4302-4310, 2017.","title":"Deep Reinforcement Learning from Human Preferences","authors":"Jan Paul F Christiano, Tom Leike, Miljan Brown, Shane Martic, Dario Legg, Amodei","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Everitt and Hutter, 2016] Tom Everitt and Marcus Hutter. Avoiding Wireheading with Value Reinforcement Learn- ing. In International Conference on Artificial General In- telligence, pages 12-22. Springer, 2016.","title":"Avoiding Wireheading with Value Reinforcement Learning","authors":"Hutter Everitt, Tom Everitt, Marcus Hutter","year":"2016","venue":"International Conference on Artificial General Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Everitt and Hutter, 2019] Tom Everitt and Marcus Hutter. Reward Tampering Problems and Solutions in Reinforce- ment Learning: A Causal Influence Diagram Perspective. arXiv preprint arXiv:1908.04734, 2019.","title":"Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective","authors":"Hutter Everitt, Tom Everitt, Marcus Hutter","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Everitt, 2018] Tom Everitt. Towards Safe Artificial General Intelligence. PhD thesis, The Australian National Univer- sity, 2018.","title":"Towards Safe Artificial General Intelligence","authors":"Tom Everitt","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hadfield-Menell et al., 2016] Dylan Hadfield-Menell, Anca Draga, Pieter Abbeel, and Stuart Russell. Cooperative In- verse Reinforcement Learning. In Advanced in Neural In- formation Processing Systems, pages 3909-3917, 2016.","title":"Cooperative Inverse Reinforcement Learning","authors":"Hadfield-Menell","year":"2016","venue":"Advanced in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hadfield-Menell et al., 2017] Dylan Hadfield-Menell, Smitha Milli, Stuart J Russell, Pieter Abbeel, and Anca Dragan. Inverse Reward Design. In Advances in Neural Information Processing Systems, pages 6749-6758, 2017.","title":"Inverse Reward Design","authors":"Hadfield-Menell","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hutter, 2004] Marcus Hutter. Universal Artificial Intelli- gence: Sequential Decisions Based on Algorithmic Prob- ability. Springer, 2004.","title":"Universal Artificial Intelligence: Sequential Decisions Based on Algorithmic Probability","authors":"Marcus Hutter, Hutter","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ibarz et al., 2018] Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward Learning from Human Preferences and Demonstrations in Atari. In Advances in Neural Information Processing Sys- tems, pages 8011-8023, 2018.","title":"Reward Learning from Human Preferences and Demonstrations in Atari","authors":"Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, Dario Amodei","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kahneman, 2011] Daniel Kahneman. Thinking, Fast and Slow. Farra, Straus and Giroux, 2011.","title":"Thinking, Fast and Slow","authors":"Daniel Kahneman","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lee et al., 2017] Chun-I Lee, I-Ping Chen, Chi-Min Hsieh, and Chia-Ning Liao. Design Aspects of Scoring Systems in Game. Art and Design Review, 05(01):26-43, 2017.","title":"Design Aspects of Scoring Systems in Game","authors":"Chun-I Lee, I-Ping Chen, Chi-Min Hsieh, Chia-Ning Liao","year":"2017","venue":"Art and Design Review","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Leike et al., 2017] Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A Ortega, Tom Everitt, Andrew Lefrancq, Laurent Orseau, and Shane Legg. AI Safety Gridworlds. arXiv preprint arXiv:1711.09883, 2017.","title":"AI Safety Gridworlds","authors":"Leike","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[MacGlashan et al., 2017] James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng, David Roberts, Matthew E Taylor, and Michael L Littman. Interactive Learning from Policy-dependent Human Feedback. arXiv preprint arXiv:1701.06049, 2017.","title":"Interactive Learning from Policy-dependent Human Feedback","authors":"James Macglashan, K Mark, Robert Ho, Bei Loftin, David Peng, Matthew E Roberts, Taylor, Michael L Littman","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ng and Russell, 2000] Andrew Y. Ng and Stuart J. Russell. Algorithms for Inverse Reinforcement Learning. In In- ternational Conference on Machine Learning, pages 663- 670, 2000.","title":"Algorithms for Inverse Reinforcement Learning","authors":"Andrew Y Ng, Stuart J Russell","year":"2000","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pearl, 2009] Judea Pearl. Causality. Cambridge University Press, 2009.","title":"Judea Pearl. Causality","authors":"Pearl","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Pilarski et al., 2011] Patrick M Pilarski, Michael R Daw- son, Thomas Degris, Farbod Fahimi, Jason P Carey, and Richard S Sutton. Online Human Training of a Myoelec- tric Prosthesis Controller Via Actor-Critic Reinforcement Learning. In Rehabilitation Robotics, pages 1-7, 2011.","title":"Online Human Training of a Myoelectric Prosthesis Controller Via Actor-Critic Reinforcement Learning","authors":"Patrick M Pilarski, Thomas Michael R Dawson, Farbod Degris, Jason P Fahimi, Richard S Carey, Sutton","year":"2011","venue":"Rehabilitation Robotics","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sutton and Barto, 1998] Richard Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.","title":"Reinforcement Learning: An Introduction","authors":"Richard Sutton, Andrew G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Vogt, 2007] Paul Vogt. Language Evolution and Robotics: Issues on Symbol Grounding and Language Acquisition. In Artificial cognition systems, pages 176-209. IGI Global, 2007.","title":"Language Evolution and Robotics: Issues on Symbol Grounding and Language Acquisition","authors":"Vogt","year":"2007","venue":"Artificial cognition systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yudkowsky, 2008] Eliezer Yudkowsky. Artificial intelli- gence as a positive and negative factor in global risk. In Nick Bostrom and Milan M. Ćirković, editors, Global catastrophic risks, pages 308-345, New York, 2008. Ox- ford University Press.","title":"Artificial intelligence as a positive and negative factor in global risk","authors":"Eliezer Yudkowsky, Yudkowsky","year":"2008","venue":"Global catastrophic risks","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}