{"type":"ScholarlyArticle","title":"Leveraging Human Knowledge in Tabular Reinforcement Learning: A Study of Human Subjects","authors":[{"name":"Ariel Rosenfeld","slug":"ariel-rosenfeld","given":"Ariel","family":"Rosenfeld","affiliation":"Department of Computer Science , Bar-Ilan University , Ramat-Gan , Israel."},{"name":"Matthew E. Taylor","slug":"matthew-e-taylor","given":"Matthew E.","family":"Taylor","affiliation":"Department of Computer Science , Washington State University , Pullman , Washington , USA."},{"name":"Sarit Kraus","slug":"sarit-kraus","given":"Sarit","family":"Kraus","affiliation":"Department of Computer Science , Bar-Ilan University , Ramat-Gan , Israel."}],"abstract":"Reinforcement Learning  (RL) can be extremely effective in  solving complex, real-world problems. However, injecting human knowledge into an RL agent may require extensive effort on the human designer's part. To date, human factors are generally not considered in the development and evaluation of possible approaches. In this paper, we propose and evaluate a novel method, based on human psychology literature, which we show to be both effective and efficient, for both expert and non-expert designers, in injecting human knowledge for speeding up tabular RL.","doi":"10.24963/ijcai.2017/534","url":"https://ijcai.org/proceedings/2017/534","pdf":"https://ijcai.org/proceedings/2017/0534.pdf","markdown":"https://ijcai.org/proceedings/2017/534.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/534","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Multidisciplinary Topics and Applications","track":"Main track","number":534,"pages":{"first":3823,"last":3830},"keywords":["Machine Learning: Reinforcement Learning","Multidisciplinary Topics and Applications: Human-Computer Interaction","Multidisciplinary Topics and Applications: Knowledge-based Software Engineering"],"videos":[],"references":[{"position":1,"raw":"[Albus, 1981] James Sacra Albus. Brains, Behavior and Robotics. McGraw-Hill, Inc., New York, NY, USA, 1981.","title":"Brains, Behavior and Robotics","authors":"James Sacra, Albus","year":"1981","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Azaria et al., 2015] Amos Azaria, Ariel Rosenfeld, Sarit Kraus, Claudia V Goldman, and Omer Tsimhoni. Advice provision for energy saving in automobile climate-control system. AI Magazine, 36(3):61-72, 2015.","title":"Advice provision for energy saving in automobile climate-control system","authors":"Amos Azaria, Ariel Rosenfeld, Sarit Kraus, Claudia V Goldman, Omer Tsimhoni","year":"2015","venue":"AI Magazine","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Benda, 1985] Miroslav Benda. On optimal cooperation of knowledge sources. Tech.l Report BCS-G2010-28, 1985.","title":"On optimal cooperation of knowledge sources","authors":"Miroslav Benda, Benda","year":"1985","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bruner, 1957] Jerome S Bruner. Going beyond the infor- mation given. Contemporary approaches to cognition, 1(1):119-160, 1957.","title":"Going beyond the information given","authors":"Jerome S Bruner","year":"1957","venue":"Contemporary approaches to cognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brys et al., 2014] Tim Brys, Ann Nowé, Daniel Kudenko, and Matthew E Taylor. Combining multiple correlated reward and shaping signals by measuring confidence. In AAAI, pages 1687-1693, 2014.","title":"Combining multiple correlated reward and shaping signals by measuring confidence","authors":"Tim Brys, Ann Nowé, Daniel Kudenko, Matthew E Taylor","year":"2014","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Brys et al., 2015] Tim Brys, Anna Harutyunyan, Halit Bener Suay, Sonia Chernova, Matthew E Taylor, and Ann Nowé. Reinforcement learning from demonstration through shaping. In IJCAI, pages 3352-3358, 2015.","title":"Reinforcement learning from demonstration through shaping","authors":"Tim Brys, Anna Harutyunyan, Halit Bener Suay, Sonia Chernova, Matthew E Taylor, Ann Nowé","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/472"},{"position":7,"raw":"[Busoniu et al., 2010] Lucian Busoniu, Robert Babuska, Bart De Schutter, and Damien Ernst. Reinforcement learn- ing and dynamic programming using function approxima- tors, volume 39. CRC press, 2010.","title":"Reinforcement learning and dynamic programming using function approximators","authors":"Lucian Busoniu, Robert Babuska, Bart De Schutter, Damien Ernst","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Girgin et al., 2007] Sertan Girgin, Faruk Polat, and Reda Alhajj. Positive impact of state similarity on reinforcement learning performance. IEEE Transactions on Cybernetics, 37(5):1256-1270, 2007.","title":"Positive impact of state similarity on reinforcement learning performance","authors":"Girgin","year":"2007","venue":"IEEE Transactions on Cybernetics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hart and Staveland, 1988] Sandra G Hart and Lowell E Staveland. Development of nasa-tlx (task load index): Re- sults of empirical and theoretical research. Advances in psychology, 52:139-183, 1988.","title":"Development of nasa-tlx (task load index): Results of empirical and theoretical research","authors":"G Sandra, Lowell E Hart, Staveland","year":"1988","venue":"Advances in psychology","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hester and Stone, 2013] Todd Hester and Peter Stone. Tex- plore: real-time sample-efficient reinforcement learning for robots. Machine learning, 90(3):385-429, 2013.","title":"Texplore: real-time sample-efficient reinforcement learning for robots","authors":"Stone Hester, Todd Hester, Peter Stone","year":"2013","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jong and Stone, 2007] Nicholas K Jong and Peter Stone. Model-based function approximation in reinforcement learning. In AAMAS, page 95. ACM, 2007.","title":"Model-based function approximation in reinforcement learning","authors":"K Nicholas, Peter Jong, Stone","year":"2007","venue":"AAMAS, page 95","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Karakovskiy and Togelius, 2012] Sergey Karakovskiy and Julian Togelius. The mario AI benchmark and competi- tions. IEEE Transactions on Computational Intelligence and AI in Games, 4(1):55-67, 2012.","title":"The mario AI benchmark and competitions","authors":"Sergey Karakovskiy, Julian Togelius","year":"2012","venue":"IEEE Transactions on Computational Intelligence and AI in Games","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kelly, 1955] George Kelly. Personal construct psychology. New York: Norton, 1955.","title":"Personal construct psychology","authors":"George Kelly","year":"1955","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Knox and Stone, 2010] W. Bradley Knox and Peter Stone. Combining manual feedback with subsequent MDP re- ward signals for reinforcement learning. In Proc. of AA- MAS, May 2010.","title":"Combining manual feedback with subsequent MDP reward signals for reinforcement learning","authors":"W, Bradley Knox, Peter Stone","year":"2010","venue":"Proc. of AA-MAS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Leffler et al., 2007] Bethany R Leffler, Michael L Littman, and Timothy Edmunds. Efficient reinforcement learning with relocatable action models. In AAAI, 2007.","title":"Efficient reinforcement learning with relocatable action models","authors":"Bethany R Leffler, Timothy Michael L Littman, Edmunds","year":"2007","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Levy and Sarne, 2016] Priel Levy and David Sarne. Intelli- gent advice provisioning for repeated interaction. In Thir- tieth AAAI Conference on Artificial Intelligence, 2016.","title":"Intelligent advice provisioning for repeated interaction","authors":"Sarne ; Levy, Priel Levy, David Sarne","year":"2016","venue":"Thirtieth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Littman, 1994] Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In ICML, volume 157, pages 157-163, 1994.","title":"Markov games as a framework for multi-agent reinforcement learning","authors":"Michael L Littman","year":"1994","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Martins and Bianchi, 2013] Murilo Fernandes Martins and Reinaldo AC Bianchi. Heuristically-accelerated reinforce- ment learning: A comparative analysis of performance. In Conference Towards Autonomous Robotic Systems, 2013.","title":"Heuristically-accelerated reinforcement learning: A comparative analysis of performance","authors":"Murilo Fernandes, Martins, Reinaldo Ac Bianchi","year":"2013","venue":"Conference Towards Autonomous Robotic Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Narayanamurthy and Ravindran, 2008] Shravan Matthur Narayanamurthy and Balaraman Ravindran. On the hard- ness of finding symmetries in markov decision processes. In ICML, pages 688-695, 2008.","title":"Shravan Matthur Narayanamurthy and Balaraman Ravindran. On the hardness of finding symmetries in markov decision processes","authors":"Ravindran Narayanamurthy","year":"2008","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Peng et al., 2016] Bei Peng, James MacGlashan, Robert Loftin, Michael L Littman, David L Roberts, and Matthew E Taylor. A need for speed: Adapting agent ac- tion speed to improve task learning from non-expert hu- mans. In AAMAS, pages 957-965, 2016.","title":"A need for speed: Adapting agent action speed to improve task learning from non-expert humans","authors":"Bei Peng, James Macglashan, Robert Loftin, David L Michael L Littman, Matthew E Roberts, Taylor","year":"2016","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ribeiro and Szepesvári, 1996] Carlos Ribeiro and Csaba Szepesvári. Q-learning combined with spreading: Con- vergence and results. In Proc. of Intelligent and Cognitive Systems, 1996.","title":"Q-learning combined with spreading: Convergence and results","authors":"Carlos Ribeiro, Csaba Szepesvári","year":"1996","venue":"Proc. of Intelligent and Cognitive Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ribeiro, 1995] Carlos HC Ribeiro. Attentional mechanisms as a strategy for generalisation in the Q-learning algorithm. In Proceedings of ICANN, 1995.","title":"Attentional mechanisms as a strategy for generalisation in the Q-learning algorithm","authors":"Carlos Hc, Ribeiro","year":"1995","venue":"Proceedings of ICANN","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Rosenfeld and Kraus, 2016] Ariel Rosenfeld and Sarit Kraus. Providing arguments in discussions on the basis of the prediction of human argumentative behavior. TiiS, 6(4):30:1-30:33, 2016.","title":"Providing arguments in discussions on the basis of the prediction of human argumentative behavior","authors":"Kraus Rosenfeld, Ariel Rosenfeld, Sarit Kraus","year":"2016","venue":"TiiS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Rosenfeld et al., 2015a] Ariel Rosenfeld, Noa Agmon, Oleg Maksimov, Amos Azaria, and Sarit Kraus. Intelli- gent agent supporting human-multi-robot team collabora- tion. In Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI, Buenos Aires, Argentina, July 25-31, 2015, pages 1902-1908. AAAI Press, 2015.","title":"Intelligent agent supporting human-multi-robot team collaboration","authors":"Rosenfeld","year":"2015","venue":"Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/270"},{"position":26,"raw":"[Rosenfeld et al., 2015b] Ariel Rosenfeld, Amos Azaria, Sarit Kraus, Claudia V. Goldman, and Omer Tsimhoni. Adaptive advice in automobile climate control systems. In AAMAS, pages 543-551, 2015.","title":"Adaptive advice in automobile climate control systems","authors":"Rosenfeld","year":"2015","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Rosenfeld et al., 2016] Ariel Rosenfeld, Joseph Keshet, Claudia V. Goldman, and Sarit Kraus. Online prediction of exponential decay time series with human-agent appli- cation. In ECAI, pages 595-603, 2016.","title":"Online prediction of exponential decay time series with human-agent application","authors":"Ariel Rosenfeld, Joseph Keshet, Claudia V Goldman, Sarit Kraus","year":"2016","venue":"ECAI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Rosenfeld et al., 2017] Ariel Rosenfeld, Matthew E Taylor, and Sarit Kraus. Speeding up tabular reinforcement learn- ing using state-action similarities. In AAMAS, pages 1722- 1724, 2017.","title":"Speeding up tabular reinforcement learning using state-action similarities","authors":"Rosenfeld","year":"2017","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sequeira et al., 2013] Pedro Sequeira, Francisco S Melo, and Ana Paiva. An associative state-space metric for learn- ing in factored mdps. In Portuguese Conference on Artifi- cial Intelligence, pages 163-174. Springer, 2013.","title":"An associative state-space metric for learning in factored mdps","authors":"Pedro Sequeira, Francisco S Melo, Ana Paiva","year":"2013","venue":"Portuguese Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Stone et al., 2006] Peter Stone, Gregory Kuhlmann, Matthew E. Taylor, and Yaxin Liu. Keepaway soc- cer: From machine learning testbed to benchmark. In RoboCup-2005: Robot Soccer World Cup IX. 2006.","title":"Keepaway soccer: From machine learning testbed to benchmark","authors":"Peter Stone, Gregory Kuhlmann, Matthew E Taylor, Yaxin Liu","year":"2006","venue":"RoboCup-2005: Robot Soccer World Cup IX","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Suay et al., 2016] Halit Bener Suay, Tim Brys, Matthew E Taylor, and Sonia Chernova. Learning from demonstra- tion for shaping through inverse reinforcement learning. In AAMAS, pages 429-437, 2016.","title":"Learning from demonstration for shaping through inverse reinforcement learning","authors":"Halit Bener Suay, Tim Brys, Matthew E Taylor, Sonia Chernova","year":"2016","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Sutton and Barto, 1998] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 1998.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Sutton, 1991] Richard S Sutton. Dyna, an integrated archi- tecture for learning, planning, and reacting. ACM SIGART Bulletin, 2(4):160-163, 1991.","title":"Dyna, an integrated architecture for learning, planning, and reacting","authors":"Richard S Sutton","year":"1991","venue":"ACM SIGART Bulletin","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Szepesvári and Littman, 1999] Csaba Szepesvári and Michael L Littman. A unified analysis of value- function-based reinforcement-learning algorithms. Neural computation, 11(8):2017-2060, 1999.","title":"A unified analysis of valuefunction-based reinforcement-learning algorithms","authors":"Csaba Szepesvári, Michael L Littman","year":"1999","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Tamassia et al., 2016] Marco Tamassia, Fabio Zambetta, William Raffe, Florian Mueller, and Xiaodong Li. Dy- namic choice of state abstraction in q-learning. In ECAI), 2016.","title":"Dynamic choice of state abstraction in q-learning","authors":"Marco Tamassia, Fabio Zambetta, William Raffe, Florian Mueller, Xiaodong Li","year":"2016","venue":"ECAI)","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Watkins, 1989] Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. PhD thesis, University of Cambridge England, 1989.","title":"Learning from delayed rewards","authors":"Christopher John, Cornish Hellaby Watkins","year":"1989","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Witten et al., 2016] Ian H Witten, Eibe Frank, Mark A Hall, and Christopher J Pal. Data Mining: Practical machine learning tools and techniques. Morgan Kaufmann, 2016.","title":"Data Mining: Practical machine learning tools and techniques","authors":"Eibe Ian H Witten, Mark A Frank, Christopher J Hall, Pal","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zinkevich and Balch, 2001] Martin Zinkevich and Tucker Balch. Symmetry in markov decision processes and its implications for single agent and multi agent learning. In ICML, 2001.","title":"Symmetry in markov decision processes and its implications for single agent and multi agent learning","authors":"Balch Zinkevich, Martin Zinkevich, Tucker Balch","year":"2001","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}