{"type":"ScholarlyArticle","title":"Interactive Teaching Algorithms for Inverse Reinforcement Learning","authors":[{"name":"Parameswaran Kamalaruban","slug":"parameswaran-kamalaruban","given":"Parameswaran","family":"Kamalaruban","affiliation":"Max Planck Institute for Software Systems (MPI-SWS)"},{"name":"Rati Devidze","slug":"rati-devidze","given":"Rati","family":"Devidze","affiliation":"Max Planck Institute for Software Systems (MPI-SWS)"},{"name":"Volkan Cevher","slug":"volkan-cevher","given":"Volkan","family":"Cevher","affiliation":"Max Planck Institute for Software Systems (MPI-SWS)"},{"name":"Adish Singla","slug":"adish-singla","given":"Adish","family":"Singla","affiliation":"Max Planck Institute for Software Systems (MPI-SWS)"}],"abstract":"We study the problem of inverse reinforcement learning (IRL) with the added twist that the learner is assisted by a helpful teacher. More formally, we tackle the following algorithmic question: How could a teacher provide an informative sequence of demonstrations to an IRL learner to speed up the learning process? We present an interactive teaching framework where a teacher adaptively chooses the next demonstration based on learner's current policy. In particular, we design teaching algorithms for two concrete settings: an omniscient setting where a teacher has full knowledge about the learner's dynamics and a blackbox setting where the teacher has minimal knowledge. Then, we study a sequential variant of the popular MCE-IRL learner and prove convergence guarantees of our teaching algorithm in the omniscient setting. Extensive experiments with a car driving simulator environment show that the learning progress can be speeded up drastically as compared to an uninformative teacher.","doi":"10.24963/ijcai.2019/374","url":"https://ijcai.org/proceedings/2019/374","pdf":"https://ijcai.org/proceedings/2019/0374.pdf","markdown":"https://ijcai.org/proceedings/2019/374.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/374","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":374,"pages":{"first":2692,"last":2700},"keywords":["Humans and AI: Computer-Aided Education","Machine Learning: Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Abbeel and Ng, 2004] Pieter Abbeel and Andrew Y Ng. Ap- prenticeship learning via inverse reinforcement learning. In ICML, 2004.","title":"Apprenticeship learning via inverse reinforcement learning","authors":"Ng Abbeel, Pieter Abbeel, Andrew Y Ng","year":"2004","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Amin et al., 2017] Kareem Amin, Nan Jiang, and Satinder P. Singh. Repeated inverse reinforcement learning. In NIPS, pages 1813-1822, 2017.","title":"Repeated inverse reinforcement learning","authors":"Kareem Amin, Nan Jiang, Satinder P Singh","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Amir et al., 2016] Ofra Amir, Ece Kamar, Andrey Kolobov, and Barbara J. Grosz. Interactive teaching strategies for agent training. In IJCAI, pages 804-811, 2016.","title":"Interactive teaching strategies for agent training","authors":"Ofra Amir, Ece Kamar, Andrey Kolobov, Barbara J Grosz","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Argall et al., 2009] Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 2009.","title":"A survey of robot learning from demonstration","authors":"Sonia Brenna D Argall, Manuela Chernova, Brett Veloso, Browning","year":"2009","venue":"Robotics and autonomous systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bain and Sommut, 1999] Michael Bain and Claude Sommut. A framework for hehavioural claning. Machine Intelligence 15, 15:103, 1999.","title":"A framework for hehavioural claning","authors":"Michael Bain, Claude Sommut","year":"1999","venue":"Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Billard et al., 2008] Aude Billard, Sylvain Calinon, Ruedi- ger Dillmann, and Stefan Schaal. Robot programming by demonstration. In Springer handbook of robotics. 2008.","title":"Robot programming by demonstration","authors":"Aude Billard, Sylvain Calinon, Ruediger Dillmann, Stefan Schaal","year":"2008","venue":"Springer handbook of robotics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Boularias et al., 2011] Abdeslam Boularias, Jens Kober, and Jan Peters. Relative entropy inverse reinforcement learning. In AISTATS, pages 182-189, 2011.","title":"Relative entropy inverse reinforcement learning","authors":"Abdeslam Boularias, Jens Kober, Jan Peters","year":"2011","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Brown and Niekum, 2019] Daniel S. Brown and Scott Niekum. Machine teaching for inverse reinforcement learn- ing: Algorithms and applications. In AAAI, 2019.","title":"Machine teaching for inverse reinforcement learning: Algorithms and applications","authors":"Daniel S Brown, Scott Niekum","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Buchsbaum et al., 2011] Daphna Buchsbaum, Alison Gop- nik, Thomas L Griffiths, and Patrick Shafto. Children's imitation of causal action sequences is influenced by statis- tical and pedagogical evidence. Cognition, 2011.","title":"Children's imitation of causal action sequences is influenced by statistical and pedagogical evidence","authors":"Daphna Buchsbaum, Alison Gopnik, Thomas L Griffiths, Patrick Shafto","year":"2011","venue":"Cognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Cakmak and Lopes, 2012] Maya Cakmak and Manuel Lopes. Algorithmic and human teaching of sequential decision tasks. In AAAI, 2012.","title":"Algorithmic and human teaching of sequential decision tasks","authors":"Lopes Cakmak, Maya Cakmak, Manuel Lopes","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Cakmak and Thomaz, 2014] Maya Cakmak and Andrea L Thomaz. Eliciting good teaching from humans for machine learners. Artificial Intelligence, 217:198-215, 2014.","title":"Eliciting good teaching from humans for machine learners","authors":"Thomaz Cakmak, Maya Cakmak, Andrea L Thomaz","year":"2014","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Chen et al., 2018] Yuxin Chen, Adish Singla, Oisin Mac Aodha, Pietro Perona, and Yisong Yue. Understand- ing the role of adaptivity in machine teaching: The case of version space learners. In NeurIPS, 2018.","title":"Understanding the role of adaptivity in machine teaching: The case of version space learners","authors":"Yuxin Chen, Adish Singla, Oisin Mac Aodha, Pietro Perona, Yisong Yue","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Chernova and Thomaz, 2014] Sonia Chernova and Andrea L Thomaz. Robot learning from human teachers. Synthesis Lectures on Artificial Intelligence and Machine Learning, 2014.","title":"Robot learning from human teachers","authors":"Sonia Chernova, Andrea L Thomaz","year":"2014","venue":"Synthesis Lectures on Artificial Intelligence and Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Dorsa Sadigh et al., 2017] Anca D Dragan Dorsa Sadigh, Shankar Sastry, and Sanjit A Seshia. Active preference- based learning of reward functions. In RSS, 2017.","title":"Active preferencebased learning of reward functions","authors":"Anca D Dragan Dorsa, Shankar Sadigh, Sanjit A Sastry, Seshia","year":"2017","venue":"RSS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Finn et al., 2016] Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In ICML, pages 49-58, 2016.","title":"Guided cost learning: Deep inverse optimal control via policy optimization","authors":"Finn","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Goldman and Kearns, 1995] Sally A Goldman and Michael J Kearns. On the complexity of teaching. Journal of Computer and System Sciences, 50(1):20-31, 1995.","title":"On the complexity of teaching","authors":"A Sally, Michael J Goldman, Kearns","year":"1995","venue":"Journal of Computer and System Sciences","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hadfield-Menell et al., 2016] Dylan Hadfield-Menell, Stu- art J Russell, Pieter Abbeel, and Anca Dragan. Cooperative inverse reinforcement learning. In NIPS, 2016.","title":"Cooperative inverse reinforcement learning","authors":"Hadfield-Menell","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Haug et al., 2018] Luis Haug, Sebastian Tschiatschek, and Adish Singla. Teaching Inverse Reinforcement Learners via Features and Demonstrations. In NeurIPS, 2018.","title":"Teaching Inverse Reinforcement Learners via Features and Demonstrations","authors":"Luis Haug, Sebastian Tschiatschek, Adish Singla","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ho and Ermon, 2016] Jonathan Ho and Stefano Ermon. Gen- erative adversarial imitation learning. In NIPS, 2016.","title":"Generative adversarial imitation learning","authors":"Jonathan Ho, Stefano Ermon","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ho et al., 2016] Mark K Ho, Michael Littman, James Mac- Glashan, Fiery Cushman, and Joseph L Austerweil. Show- ing versus doing: Teaching by demonstration. In NIPS, 2016.","title":"Showing versus doing: Teaching by demonstration","authors":"K Mark, Michael Ho, James Littman, Fiery Mac-Glashan, Joseph L Cushman, Austerweil","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Hunziker et al., 2018] Anette Hunziker, Yuxin Chen, Oisin Mac Aodha, Manuel Gomez-Rodriguez, Andreas Krause, Pietro Perona, Yisong Yue, and Adish Singla. Teaching mul- tiple concepts to a forgetful learner. CoRR, abs/1805.08322, 2018.","title":"Teaching multiple concepts to a forgetful learner","authors":"Anette Hunziker, Yuxin Chen, Oisin Mac Aodha, Manuel Gomez-Rodriguez, Andreas Krause, Pietro Perona, Yisong Yue, Adish Singla","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Kamalaruban et al., 2019] Parameswaran Kamalaruban, Rati Devidze, Volkan Cevher, and Adish Singla. Interactive teaching algorithms for inverse reinforcement learning. CoRR, abs/1905.11867, 2019.","title":"Interactive teaching algorithms for inverse reinforcement learning","authors":"Parameswaran Kamalaruban, Rati Devidze, Volkan Cevher, Adish Singla","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Levine et al., 2010] Sergey Levine, Zoran Popovic, and Vladlen Koltun. Feature construction for inverse reinforce- ment learning. In NIPS, pages 1342-1350, 2010.","title":"Feature construction for inverse reinforcement learning","authors":"Sergey Levine, Zoran Popovic, Vladlen Koltun","year":"2010","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Liu et al., 2017] Weiyang Liu, Bo Dai, Ahmad Humayun, Charlene Tay, Chen Yu, Linda B. Smith, James M. Rehg, and Le Song. Iterative machine teaching. In ICML, 2017.","title":"Iterative machine teaching","authors":"Weiyang Liu, Bo Dai, Ahmad Humayun, Charlene Tay, Chen Yu, Linda B Smith, James M Rehg, Le Song","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Liu et al., 2018] Weiyang Liu, Bo Dai, Xingguo Li, Zhen Liu, James Rehg, and Le Song. Towards black-box iterative machine teaching. In ICML, 2018.","title":"Towards black-box iterative machine teaching","authors":"Weiyang Liu, Bo Dai, Xingguo Li, Zhen Liu, James Rehg, Le Song","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Melo et al., 2018] Francisco S Melo, Carla Guerra, and Manuel Lopes. Interactive optimal teaching with unknown learners. In IJCAI, pages 2567-2573, 2018.","title":"Interactive optimal teaching with unknown learners","authors":"Melo","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/356"},{"position":27,"raw":"[Ng and Russell, 2000] Andrew Y Ng and Stuart J Russell. Algorithms for inverse reinforcement learning. In ICML, 2000.","title":"Algorithms for inverse reinforcement learning","authors":"Y Andrew, Stuart J Ng, Russell","year":"2000","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Ng et al., 1999] Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, 1999.","title":"Policy invariance under reward transformations: Theory and application to reward shaping","authors":"Daishi Andrew Y Ng, Stuart Harada, Russell","year":"1999","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Osa et al., 2018] Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, Jan Peters, et al. An algorithmic perspective on imitation learning. Foundations and Trends® in Robotics, 7(1-2):1-179, 2018.","title":"An algorithmic perspective on imitation learning","authors":"Takayuki Osa, Joni Pajarinen, Gerhard Neumann, Andrew Bagnell, Pieter Abbeel, Jan Peters","year":"2018","venue":"Foundations and Trends® in Robotics","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Patil et al., 2014] Kaustubh R Patil, Xiaojin Zhu, Łukasz Kopeć, and Bradley C Love. Optimal teaching for limited- capacity human learners. In NIPS, pages 2465-2473, 2014.","title":"Optimal teaching for limitedcapacity human learners","authors":"Xiaojin Kaustubh R Patil, Łukasz Zhu, Bradley C Kopeć, Love","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Rafferty et al., 2016] Anna N Rafferty, Emma Brunskill, Thomas L Griffiths, and Patrick Shafto. Faster teaching via pomdp planning. Cognitive science, 2016.","title":"Faster teaching via pomdp planning","authors":"Anna N Rafferty, Emma Brunskill, Thomas L Griffiths, Patrick Shafto","year":"2016","venue":"Cognitive science","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Ratliff et al., 2006] Nathan D Ratliff, J Andrew Bagnell, and Martin A Zinkevich. Maximum margin planning. In ICML, pages 729-736, 2006.","title":"Maximum margin planning","authors":"J Nathan D Ratliff, Martin A Andrew Bagnell, Zinkevich","year":"2006","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Rhinehart and Kitani, 2017] Nicholas Rhinehart and Kris M Kitani. First-person activity forecasting with online inverse reinforcement learning. In ICCV, pages 3696-3705, 2017.","title":"First-person activity forecasting with online inverse reinforcement learning","authors":"Kitani Rhinehart, Nicholas Rhinehart, Kris M Kitani","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Russell, 1998] Stuart Russell. Learning agents for uncertain environments. In COLT, pages 101-103, 1998.","title":"Learning agents for uncertain environments","authors":"Stuart Russell","year":"1998","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Schaal, 1997] Stefan Schaal. Learning from demonstration. In NIPS, pages 1040-1046, 1997.","title":"Learning from demonstration","authors":"Stefan Schaal","year":"1997","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Shafto et al., 2014] Patrick Shafto, Noah D Goodman, and Thomas L Griffiths. A rational account of pedagogical reasoning: Teaching by, and learning from, examples. Cog- nitive psychology, 71:55-89, 2014.","title":"A rational account of pedagogical reasoning: Teaching by, and learning from, examples","authors":"Patrick Shafto, Thomas L Noah D Goodman, Griffiths","year":"2014","venue":"Cognitive psychology","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Singla et al., 2013] Adish Singla, Ilija Bogunovic, G Bartók, A Karbasi, and A Krause. On actively teaching the crowd to classify. In NIPS Workshop on Data Driven Education, 2013.","title":"On actively teaching the crowd to classify","authors":"Adish Singla, Ilija Bogunovic, G Bartók, Karbasi, Krause","year":"2013","venue":"NIPS Workshop on Data Driven Education","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Singla et al., 2014] Adish Singla, Ilija Bogunovic, Gábor Bartók, Amin Karbasi, and Andreas Krause. Near- optimally teaching the crowd to classify. In ICML, 2014.","title":"Nearoptimally teaching the crowd to classify","authors":"Adish Singla, Ilija Bogunovic, Gábor Bartók, Amin Karbasi, Andreas Krause","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Sorg et al., 2010] Jonathan Sorg, Satinder P. Singh, and Richard L. Lewis. Reward design via online gradient ascent. In NIPS, pages 2190-2198, 2010.","title":"Reward design via online gradient ascent","authors":"Jonathan Sorg, P Satinder, Richard L Singh, Lewis","year":"2010","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Tadepalli, 2008] Prasad Tadepalli. Learning to solve prob- lems from exercises. Computational Intelligence, 2008.","title":"Learning to solve problems from exercises","authors":"Prasad Tadepalli","year":"2008","venue":"Computational Intelligence","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Torrey and Taylor, 2013] Lisa Torrey and Matthew Taylor. Teaching on a budget: Agents advising agents in reinforce- ment learning. In AAMAS, pages 1053-1060, 2013.","title":"Lisa Torrey and Matthew Taylor. Teaching on a budget: Agents advising agents in reinforcement learning","authors":"Torrey, Taylor","year":"2013","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Walsh and Goschin, 2012] Thomas J. Walsh and Sergiu Goschin. Dynamic teaching in sequential decision making environments. In UAI, 2012.","title":"Dynamic teaching in sequential decision making environments","authors":"Thomas J Walsh, Sergiu Goschin","year":"2012","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Wulfmeier et al., 2015] Markus Wulfmeier, Peter Ondruska, and Ingmar Posner. Maximum entropy deep inverse rein- forcement learning. CoRR, abs/1507.04888, 2015.","title":"Maximum entropy deep inverse reinforcement learning","authors":"Markus Wulfmeier, Peter Ondruska, Ingmar Posner","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Yeo et al., 2019] Teresa Yeo, Parameswaran Kamalaruban, Adish Singla, Arpit Merchant, Thibault Asselborn, Louis Faucon, Pierre Dillenbourg, and Volkan Cevher. Iterative classroom teaching. In AAAI, 2019.","title":"Iterative classroom teaching","authors":"Teresa Yeo, Parameswaran Kamalaruban, Adish Singla, Arpit Merchant, Louis Thibault Asselborn, Pierre Faucon, Volkan Dillenbourg, Cevher","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Zhang et al., 2009] Haoqi Zhang, David C Parkes, and Yil- ing Chen. Policy teaching through reward function learning. In EC, pages 295-304, 2009.","title":"Policy teaching through reward function learning","authors":"Zhang","year":"2009","venue":"EC","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Zhou et al., 2018] Zhengyuan Zhou, Michael Bloem, and Nicholas Bambos. Infinite time horizon maximum causal entropy inverse reinforcement learning. IEEE Trans. Au- tomat. Contr., 63(9):2787-2802, 2018.","title":"Infinite time horizon maximum causal entropy inverse reinforcement learning","authors":"Zhengyuan Zhou, Michael Bloem, Nicholas Bambos","year":"2018","venue":"IEEE Trans. Automat. Contr","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Zhu et al., 2018] Xiaojin Zhu, Adish Singla, Sandra Zilles, and Anna N. Rafferty. An overview of machine teaching. CoRR, abs/1801.05927, 2018.","title":"An overview of machine teaching","authors":"Xiaojin Zhu, Adish Singla, Sandra Zilles, Anna N Rafferty","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Ziebart et al., 2008] Brian D Ziebart, Andrew L Maas, J An- drew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In AAAI, 2008.","title":"Maximum entropy inverse reinforcement learning","authors":"Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey","year":"2008","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Ziebart, 2010] Brian D Ziebart. Modeling purposeful adap- tive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010.","title":"Modeling purposeful adaptive behavior with the principle of maximum causal entropy","authors":"Brian D Ziebart","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}