{"type":"ScholarlyArticle","title":"On the Expressivity of Markov Reward (Extended Abstract)","authors":[{"name":"David Abel","slug":"david-abel","given":"David","family":"Abel","affiliation":""},{"name":"Will Dabney","slug":"will-dabney","given":"Will","family":"Dabney","affiliation":""},{"name":"Anna Harutyunyan","slug":"anna-harutyunyan","given":"Anna","family":"Harutyunyan","affiliation":""},{"name":"Mark K. Ho","slug":"mark-k-ho","given":"Mark K.","family":"Ho","affiliation":"Princeton University"},{"name":"Michael L. Littman","slug":"michael-l-littman","given":"Michael L.","family":"Littman","affiliation":"Brown University"},{"name":"Doina Precup","slug":"doina-precup","given":"Doina","family":"Precup","affiliation":""},{"name":"Satinder Singh","slug":"satinder-singh","given":"Satinder","family":"Singh","affiliation":""}],"abstract":"Reward  is  the  driving  force  for  reinforcement-learning agents.  We here set out to understand the expressivity of Markov reward as a way to capture tasks that we would want an agent to perform.  We frame this study around three new abstract notions of \"task\":  (1) a set of acceptable behaviors, (2) a partial ordering over behaviors, or (3) a partial ordering over trajectories. Our main results prove that while reward can express many of these tasks, there exist instances  of  each  task  type  that  no  Markov reward  function  can  capture.   We  then  provide  a set of polynomial-time algorithms that construct a Markov reward function that allows an agent to perform each task type, and correctly determine when no such reward function exists.","doi":"10.24963/ijcai.2022/730","url":"https://ijcai.org/proceedings/2022/730","pdf":"https://ijcai.org/proceedings/2022/0730.pdf","markdown":"https://ijcai.org/proceedings/2022/730.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/730","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Sister Conferences Best Papers","track":"Sister Conferences Best Papers","number":730,"pages":{"first":5254,"last":5258},"keywords":["Artificial Intelligence: General"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987250","length_sec":81,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985393","length_sec":706,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Abel et al., 2021] David Abel, Will Dabney, Anna Haru- tyunyan, Mark K. Ho, Michael L. Littman, Doina Precup, and Satinder Singh. On the expressivity of Markov re- ward. In Advances in Neural Information Processing Sys- tems, 2021.","title":"On the expressivity of Markov reward","authors":"David Abel, Will Dabney, Anna Harutyunyan, Mark K Ho, Michael L Littman, Doina Precup, Satinder Singh","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ackley and Littman, 1992] David Ackley and Michael L. Littman. Interactions between learning and evolution. Ar- tificial Life II, 1992.","title":"Interactions between learning and evolution","authors":"David Ackley, Michael L Littman","year":"1992","venue":"Artificial Life","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Christian, 2021] Brian Christian. The Alignment Problem: Machine Learning and Human Values, pages 130-131. At- lantic Books, 2021.","title":"The Alignment Problem: Machine Learning and Human Values","authors":"Brian Christian","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dewey, 2014] Daniel Dewey. Reinforcement learning and the reward engineering principle. In Proceedings of the AAAI Spring Symposium Series, 2014.","title":"Reinforcement learning and the reward engineering principle","authors":"Daniel Dewey, Dewey","year":"2014","venue":"Proceedings of the AAAI Spring Symposium Series","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Littman, 2017] Michael L. Littman. The re- ward hypothesis. https://www.coursera.org/ lecture/fundamentals-of-reinforcement-learning/ michael-littman-the-reward-hypothesis-q6x0e, 2017.","title":"The reward hypothesis","authors":"L Michael, Littman","year":"2017","venue":null,"doi":null,"url":"https://www.coursera.org/lecture/fundamentals-of-reinforcement-learning/michael-littman-the-reward-hypothesis-q6x0e","internal_url":null},{"position":6,"raw":"[Mataric, 1994] Maja J. Mataric. Reward functions for accel- erated learning. In Proceedings of the International Con- ference on Machine Learning, 1994.","title":"Reward functions for accelerated learning","authors":"J Maja, Mataric","year":"1994","venue":"Proceedings of the International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ng et al., 1999] Andrew Y. Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceed- ings of the International Conference on Machine Learn- ing, 1999.","title":"Policy invariance under reward transformations: Theory and application to reward shaping","authors":"Andrew Y Ng, Daishi Harada, Stuart Russell","year":"1999","venue":"Proceedings of the International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Puterman, 2014] Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, 2014.","title":"Markov Decision Processes: Discrete Stochastic Dynamic Programming","authors":"Martin L Puterman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Singh et al., 2009] Satinder Singh, Richard L Lewis, and Andrew G Barto. Where do rewards come from? In Pro- ceedings of the Annual Conference of the Cognitive Sci- ence Society, 2009.","title":"Where do rewards come from?","authors":"Satinder Singh, Richard L Lewis, Andrew G Barto","year":"2009","venue":"Proceedings of the Annual Conference of the Cognitive Science Society","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Sorg et al., 2010] Jonathan Sorg, Richard L. Lewis, and Satinder Singh. Reward design via online gradient as- cent. Advances in Neural Information Processing Systems, 2010.","title":"Reward design via online gradient ascent","authors":"Jonathan Sorg, Richard L Lewis, Satinder Singh","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Sorg, 2011] Jonathan Sorg. The Optimal Reward Problem: Designing Effective Reward for Bounded Agents. PhD the- sis, University of Michigan, 2011.","title":"The Optimal Reward Problem: Designing Effective Reward for Bounded Agents","authors":"Jonathan Sorg","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Sutton and Barto, 2018] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2018.","title":"Reinforcement Learning: An Introduction","authors":"Richard S Sutton, Andrew G Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Sutton, 2004] Richard S. Sutton. The reward hypothe- sis. http://incompleteideas.net/rlai.cs.ualberta.ca/RLAI/ rewardhypothesis.html, 2004.","title":"The reward hypothesis","authors":"Richard S Sutton","year":"2004","venue":null,"doi":null,"url":"http://incompleteideas.net/rlai.cs.ualberta.ca/RLAI/rewardhypothesis.html","internal_url":null},{"position":14,"raw":"[Wilson et al., 2012] Aaron Wilson, Alan Fern, and Prasad Tadepalli. A Bayesian approach for policy learning from trajectory preference queries. In Advances in Neural In- formation Processing Systems, 2012.","title":"A Bayesian approach for policy learning from trajectory preference queries","authors":"Aaron Wilson, Alan Fern, Prasad Tadepalli","year":"2012","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Wirth et al., 2017] Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A survey of preference-based reinforcement learning methods. The Journal of Machine Learning Research, 18(1):4945-4990, 2017.","title":"A survey of preference-based reinforcement learning methods","authors":"Wirth","year":"2017","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}