{"type":"ScholarlyArticle","title":"Active Learning within Constrained Environments through Imitation of an Expert Questioner","authors":[{"name":"Kalesha Bullard","slug":"kalesha-bullard","given":"Kalesha","family":"Bullard","affiliation":"Georgia Institute of Technology"},{"name":"Yannick Schroecker","slug":"yannick-schroecker","given":"Yannick","family":"Schroecker","affiliation":"Georgia Institute of Technology"},{"name":"Sonia Chernova","slug":"sonia-chernova","given":"Sonia","family":"Chernova","affiliation":"Georgia Institute of Technology"}],"abstract":"Active learning agents typically employ a query selection algorithm which solely considers the agent's learning objectives.  However, this may be insufficient in more realistic human domains.  This work uses imitation learning to enable an agent in a constrained environment to concurrently reason about both its internal learning goals and environmental constraints externally imposed, all within its objective function.  Experiments are conducted on a concept learning task to test generalization of the proposed algorithm to different environmental conditions and analyze how time and resource constraints impact efficacy of solving the learning problem.  Our findings show the environmentally-aware learning agent is able to statistically outperform all other active learners explored under most of the constrained conditions.  A key implication is adaptation for active learning agents to more realistic human environments, where constraints are often externally imposed on the learner.","doi":"10.24963/ijcai.2019/283","url":"https://ijcai.org/proceedings/2019/283","pdf":"https://ijcai.org/proceedings/2019/0283.pdf","markdown":"https://ijcai.org/proceedings/2019/283.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/283","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":283,"pages":{"first":2045,"last":2052},"keywords":["Robotics: Human Robot Interaction","Machine Learning: Active Learning"],"videos":[],"references":[{"position":1,"raw":"[Abbeel and Ng, 2004] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learn- ing. In Proceedings of the twenty-first international con- ference on Machine learning, page 1. ACM, 2004.","title":"Apprenticeship learning via inverse reinforcement learning","authors":"Abbeel, ; Ng, Pieter Abbeel, Andrew Y Ng","year":"2004","venue":"Proceedings of the twenty-first international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Basu et al., 2018] Chandrayee Basu, Mukesh Singhal, and Anca D Dragan. Learning from richer human guid- ance: Augmenting comparison-based learning with fea- ture queries. In Proceedings of the 2018 ACM/IEEE Inter- national Conference on Human-Robot Interaction, pages 132-140. ACM, 2018.","title":"Learning from richer human guidance: Augmenting comparison-based learning with feature queries","authors":"Basu","year":"2018","venue":"Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bullard et al., 2018a] Kalesha Bullard, Sonia Chernova, and Andrea L Thomaz. Human-driven feature selec- tion for a robotic agent learning classification tasks from demonstration. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 6923-6930. IEEE, 2018.","title":"Human-driven feature selection for a robotic agent learning classification tasks from demonstration","authors":"Bullard","year":"2018","venue":"IEEE International Conference on Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bullard et al., 2018b] Kalesha Bullard, Andrea L Thomaz, and Sonia Chernova. Towards intelligent arbitration of diverse active learning queries. In 2018 IEEE/RSJ In- ternational Conference on Intelligent Robots and Systems (IROS), pages 6049-6056. IEEE, 2018.","title":"Towards intelligent arbitration of diverse active learning queries","authors":"Bullard","year":"2018","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cakmak and Thomaz, 2012] Maya Cakmak and Andrea L Thomaz. Designing robot learners that ask good ques- tions. In ACM/IEEE Int Conf on Human-Robot Interac- tion, pages 17-24, 2012.","title":"Designing robot learners that ask good questions","authors":"Thomaz Cakmak, Maya Cakmak, Andrea L Thomaz","year":"2012","venue":"ACM/IEEE Int Conf on Human-Robot Interaction","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cakmak et al., 2010] Maya Cakmak, Crystal Chao, and An- drea L Thomaz. Designing interactions for robot active learners. Autonomous Mental Development, IEEE Trans- actions on, 2(2):108-118, 2010.","title":"Maya Cakmak, Crystal Chao, and Andrea L Thomaz. Designing interactions for robot active learners","authors":"Cakmak","year":"2010","venue":"Autonomous Mental Development","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chao et al., 2010] Crystal Chao, Maya Cakmak, and An- drea Lockerd Thomaz. Transparent active learning for robots. In ACM/IEEE Int. Conf. on Human-Robot Inter- action, pages 317-324, 2010.","title":"Crystal Chao, Maya Cakmak, and Andrea Lockerd Thomaz. Transparent active learning for robots","authors":"Chao","year":"2010","venue":"ACM/IEEE Int. Conf. on Human-Robot Interaction","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chernova and Veloso, 2009] Sonia Chernova and Manuela Veloso. Interactive policy learning through confidence- based autonomy. Journal of Artificial Intelligence Re- search, 34(1):1, 2009.","title":"Interactive policy learning through confidencebased autonomy","authors":"Sonia Chernova, Manuela Veloso","year":"2009","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Daniel et al., 2014] Christian Daniel, Malte Viering, Jan Metz, Oliver Kroemer, and Jan Peters. Active reward learning. In Robotics: Science and Systems, 2014.","title":"Active reward learning","authors":"Daniel","year":"2014","venue":"Robotics: Science and Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Fu et al., 2013] Yifan Fu, Xingquan Zhu, and Bin Li. A sur- vey on instance selection for active learning. Knowledge and information systems, pages 1-35, 2013.","title":"A survey on instance selection for active learning","authors":"Yifan Fu, Xingquan Zhu, Bin Li","year":"2013","venue":"Knowledge and information systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gonzalez-Pacheco et al., 2018] Víctor Gonzalez-Pacheco, María Malfaz, A Castro-Gonzalez, José Carlos Castillo, F Alonso, and Miguel Angel Salichs. Analyzing the im- pact of different feature queries in active learning for social robots. International Journal of Social Robotics, pages 1- 14, 2018.","title":"Analyzing the impact of different feature queries in active learning for social robots","authors":"Gonzalez-Pacheco","year":"2018","venue":"International Journal of Social Robotics","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Harnad, 1990] Stevan Harnad. The symbol grounding prob- lem. Physica D: Nonlinear Phenomena, 42(1):335-346, 1990.","title":"The symbol grounding problem","authors":"Stevan Harnad","year":"1990","venue":"Physica D: Nonlinear Phenomena","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hayes and Scassellati, 2014] Bradley Hayes and Brian Scassellati. Discovering task constraints through observa- tion and active learning. In 2014 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, pages 4442-4449, 2014.","title":"Discovering task constraints through observation and active learning","authors":"Scassellati Hayes, Bradley Hayes, Brian Scassellati","year":"2014","venue":"IEEE/RSJ Int. Conf. on Intelligent Robots and Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Knox et al., 2013] W Bradley Knox, Peter Stone, and Cyn- thia Breazeal. Training a robot via human feedback: A case study. In International Conference on Social Robotics, pages 460-470. Springer, 2013.","title":"Training a robot via human feedback: A case study","authors":"Knox Bradley, Peter Stone, Cynthia Breazeal","year":"2013","venue":"International Conference on Social Robotics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kroemer et al., 2010] OB Kroemer, Renaud Detry, Justus Piater, and Jan Peters. Combining active learning and reac- tive control for robot grasping. Robotics and Autonomous Systems, 58(9):1105-1116, 2010.","title":"Combining active learning and reactive control for robot grasping","authors":"Kroemer","year":"2010","venue":"Robotics and Autonomous Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kulick et al., 2013] Johannes Kulick, Marc Toussaint, To- bias Lang, and Manuel Lopes. Active learning for teach- ing a robot grounded relational symbols. In Proceedings of the Twenty-Third international joint conference on Arti- ficial Intelligence, pages 1451-1457. AAAI Press, 2013.","title":"Active learning for teaching a robot grounded relational symbols","authors":"Johannes Kulick, Marc Toussaint, Tobias Lang, Manuel Lopes","year":"2013","venue":"Proceedings of the Twenty-Third international joint conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2013/217"},{"position":17,"raw":"[Lai et al., 2011] Kevin Lai, Liefeng Bo, Xiaofeng Ren, and Dieter Fox. A large-scale hierarchical multi-view rgb-d object dataset. In IEEE Int. Conf. on Robotics and Au- tomation, pages 1817-1824, 2011.","title":"A large-scale hierarchical multi-view rgb-d object dataset","authors":"Kevin Lai, Liefeng Bo, Xiaofeng Ren, Dieter Fox","year":"2011","venue":"IEEE Int. Conf. on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lopes et al., 2009] Manuel Lopes, Francisco Melo, and Luis Montesano. Active learning for reward estimation in inverse reinforcement learning. In Machine Learn- ing and Knowledge Discovery in Databases, pages 31-46. Springer, 2009.","title":"Active learning for reward estimation in inverse reinforcement learning","authors":"Manuel Lopes, Francisco Melo, Luis Montesano","year":"2009","venue":"Machine Learning and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ng et al., 2000] Andrew Y Ng, Stuart J Russell, et al. Al- gorithms for inverse reinforcement learning. In Icml, vol- ume 1, page 2, 2000.","title":"Algorithms for inverse reinforcement learning","authors":"Y Andrew, Stuart J Ng, Russell","year":"2000","venue":"Icml","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Osa et al., 2018] Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, Jan Peters, et al. An algorithmic perspective on imitation learn- ing. Foundations and Trends R in Robotics, 7(1-2):1-179, 2018.","title":"An algorithmic perspective on imitation learning","authors":"Takayuki Osa, Joni Pajarinen, Gerhard Neumann, Andrew Bagnell, Pieter Abbeel, Jan Peters","year":"2018","venue":"Foundations and Trends R in Robotics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Racca and Kyrki, 2018] Mattia Racca and Ville Kyrki. Ac- tive robot learning for temporal task models. In Proceed- ings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction, pages 123-131. ACM, 2018.","title":"Active robot learning for temporal task models","authors":"Mattia Racca, Ville Kyrki","year":"2018","venue":"Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rosenthal and Veloso, 2011] Stephanie Rosenthal and Manuela Veloso. Modeling humans as observation providers using pomdps. In RO-MAN, 2011 IEEE, pages 53-58. IEEE, 2011.","title":"Modeling humans as observation providers using pomdps","authors":"Stephanie Rosenthal, Manuela Veloso","year":"2011","venue":"RO-MAN","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Settles, 2012] Burr Settles. Active learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 6(1):1-114, 2012.","title":"Burr Settles. Active learning","authors":"Settles","year":"2012","venue":"Synthesis Lectures on Artificial Intelligence and Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Thomason et al., 2017] Jesse Thomason, Aishwarya Pad- makumar, Jivko Sinapov, Justin Hart, Peter Stone, and Raymond J Mooney. Opportunistic active learning for grounding natural language descriptions. In Conference on Robot Learning, pages 67-76, 2017.","title":"Opportunistic active learning for grounding natural language descriptions","authors":"Jesse Thomason, Aishwarya Padmakumar, Jivko Sinapov, Justin Hart, Peter Stone, Raymond J Mooney","year":"2017","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ziebart et al., 2008] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In AAAI, volume 8, pages 1433-1438. Chicago, IL, USA, 2008.","title":"Maximum entropy inverse reinforcement learning","authors":"Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey","year":"2008","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}