{"type":"ScholarlyArticle","title":"Reinforcement Learning with a Corrupted Reward Channel","authors":[{"name":"Tom Everitt","slug":"tom-everitt","given":"Tom","family":"Everitt","affiliation":"Australian National University"},{"name":"Victoria Krakovna","slug":"victoria-krakovna","given":"Victoria","family":"Krakovna","affiliation":"Australian National University"},{"name":"Laurent Orseau","slug":"laurent-orseau","given":"Laurent","family":"Orseau","affiliation":"Australian National University"},{"name":"Shane Legg","slug":"shane-legg","given":"Shane","family":"Legg","affiliation":"Australian National University"}],"abstract":"No real-world reward function is perfect. Sensory errors and software bugs may result in agents getting higher (or lower) rewards than they should. For example, a reinforcement learning agent may prefer states where a sensory error gives it the maximum reward, but where the true reward is actually small. We formalise this problem as a generalised Markov Decision Problem called Corrupt Reward MDP. Traditional RL methods fare poorly in CRMDPs, even under strong simplifying assumptions and when trying to compensate for the possibly corrupt rewards. Two ways around the problem are investigated. First, by giving the agent richer data, such as in inverse reinforcement learning and semi-supervised reinforcement learning, reward corruption stemming from systematic sensory errors may sometimes be completely managed. Second, by using randomisation to blunt the agent's optimisation, reward corruption can be partially managed under some assumptions.","doi":"10.24963/ijcai.2017/656","url":"https://ijcai.org/proceedings/2017/656","pdf":"https://ijcai.org/proceedings/2017/0656.pdf","markdown":"https://ijcai.org/proceedings/2017/656.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/656","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra, IIIA-CSIC","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"AI and autonomy track","track":"AI and autonomy track","number":656,"pages":{"first":4705,"last":4713},"keywords":["Technical: Techniques","Technical: Models","Social: Security & Privacy","Philosophical: Other"],"videos":[],"references":[{"position":1,"raw":"Dario Amodei and Jack Clark. Faulty Reward Func- tions in the Wild. https://openai.com/blog/ faulty-reward-functions/, 2016. Accessed: 2017-02-18.","title":"Faulty Reward Functions in the Wild","authors":"Dario Amodei, Jack Clark","year":"2017","venue":null,"doi":null,"url":"https://openai.com/blog/faulty-reward-functions/,2016","internal_url":null},{"position":2,"raw":"Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete Problems in AI Safety. CoRR, 1606.06565, 2016.","title":"Concrete Problems in AI Safety","authors":"Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané","year":"2016","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"John Aslanides, Jan Leike, and Marcus Hutter. Universal reinforcement learning algorithms: Survey and experiments. In IJCAI-17. AAAI Press, 2017.","title":"Universal reinforcement learning algorithms: Survey and experiments","authors":"John Aslanides, Jan Leike, Marcus Hutter","year":"2017","venue":"IJCAI-17","doi":null,"url":null,"internal_url":"/proceedings/2017/194"},{"position":4,"raw":"Donald A Berry and Bert Fristedt. Bandit Problems: Sequen- tial Allocation of Experiments. Springer, 1985.","title":"Bandit Problems: Sequential Allocation of Experiments","authors":"A Donald, Bert Berry, Fristedt","year":"1985","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"Nick Bostrom. Superintelligence: Paths, Dangers, Strategies. Oxford University Press, 2014.","title":"Superintelligence: Paths, Dangers, Strategies","authors":"Nick Bostrom","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"Owain Evans, Andreas Stuhlmuller, and Noah D Goodman. Learning the Preferences of Ignorant, Inconsistent Agents. In AAAI-16, 2016.","title":"Learning the Preferences of Ignorant, Inconsistent Agents","authors":"Owain Evans, Andreas Stuhlmuller, Noah D Goodman","year":"2016","venue":"AAAI-16","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"Tom Everitt, Victoria Krakovna, Laurent Orseau, Marcus Hut- ter, and Shane Legg. Reinforcement Learning with Cor- rupted Reward Channel. Corr, 1705.08417, 2017.","title":"Reinforcement Learning with Corrupted Reward Channel","authors":"Tom Everitt, Victoria Krakovna, Laurent Orseau, Marcus Hutter, Shane Legg","year":"2017","venue":"Corr","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stu- art Russell. Cooperative Inverse Reinforcement Learning. Advances in Neural Information Processing Systems (NIPS), 2016.","title":"Cooperative Inverse Reinforcement Learning","authors":"Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, Stuart Russell","year":"2016","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell. The Off-Switch Game. In AAAI Workshop on AI, Ethics and Society, 2017.","title":"The Off-Switch Game","authors":"Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, Stuart Russell","year":"2017","venue":"AAAI Workshop on AI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"Marcus Hutter. Universal Artificial Intelligence: Sequential Decisions based on Algorithmic Probability. Lecture Notes in Artificial Intelligence (LNAI 2167). Springer, 2005.","title":"Universal Artificial Intelligence: Sequential Decisions based on Algorithmic Probability","authors":"Marcus Hutter","year":"2005","venue":"LNAI 2167","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"Tommi Jaakkola, Michael I Jordan, and Satinder P Singh. On the Convergence of Stochastic Iterative Dynamic Program- ming Algorithms. Neural Computation, 6(6):1185-1201, 1994.","title":"On the Convergence of Stochastic Iterative Dynamic Programming Algorithms","authors":"Tommi Jaakkola, Michael I Jordan, P Satinder, Singh","year":"1994","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and Acting in Partially Observable Stochastic Domains. Artificial Intelligence, 101(1-2):99- 134, 1998.","title":"Planning and Acting in Partially Observable Stochastic Domains","authors":"Leslie Pack Kaelbling, Michael L Littman, Anthony R Cassandra","year":"1998","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"Andrew Ng and Stuart Russell. Algorithms for Inverse Re- inforcement Learning. In Proceedings of the Seventeenth International Conference on Machine Learning, pages 663- 670, 2000.","title":"Algorithms for Inverse Reinforcement Learning","authors":"Andrew Ng, Stuart Russell","year":"2000","venue":"Proceedings of the Seventeenth International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"Mark O Riedl and Brent Harrison. Using Stories to Teach Human Values to Artificial Agents. In AAAI Workshop on AI, Ethics, and Society, 2016.","title":"Using Stories to Teach Human Values to Artificial Agents","authors":"O Mark, Brent Riedl, Harrison","year":"2016","venue":"AAAI Workshop on AI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"Mark Ring and Laurent Orseau. Delusion, Survival, and In- telligent Agents. In Artificial General Intelligence, pages 11-20. Springer Berlin Heidelberg, 2011.","title":"Delusion, Survival, and Intelligent Agents","authors":"Mark Ring, Laurent Orseau","year":"2011","venue":"Artificial General Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"Richard S Sutton and Andrew G Barto. Reinforcement Learn- ing: An Introduction. MIT Press, 1998.","title":"Reinforcement Learning: An Introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"Jessica Taylor. Quantilizers: A Safer Alternative to Maximiz- ers for Limited Optimization. In AAAI Workshop on AI, Ethics and Society, 2016.","title":"Quantilizers: A Safer Alternative to Maximizers for Limited Optimization","authors":"Jessica Taylor","year":"2016","venue":"AAAI Workshop on AI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"David H Wolpert and William G Macready. No Free Lunch Theorems for Optimization. IEEE Transactions on Evolu- tionary Computation, 1(1):270-283, 1997.","title":"No Free Lunch Theorems for Optimization","authors":"H David, William G Wolpert, Macready","year":"1997","venue":"IEEE Transactions on Evolutionary Computation","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"Roman V. Yampolskiy. Utility Function Security in Artificially Intelligent Agents. Journal of Experimental & Theoretical Artificial Intelligence, pages 373-389, 2014.","title":"Utility Function Security in Artificially Intelligent Agents","authors":"Roman V Yampolskiy","year":"2014","venue":"Journal of Experimental & Theoretical Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}