{"type":"ScholarlyArticle","title":"Understanding the Power and Limitations of Teaching with Imperfect Knowledge","authors":[{"name":"Rati Devidze","slug":"rati-devidze","given":"Rati","family":"Devidze","affiliation":"Max Planck Institute for Software Systems (MPI-SWS)"},{"name":"Farnam Mansouri","slug":"farnam-mansouri","given":"Farnam","family":"Mansouri","affiliation":"Max Planck Institute for Software Systems (MPI-SWS)"},{"name":"Luis Haug","slug":"luis-haug","given":"Luis","family":"Haug","affiliation":"ETH Zurich"},{"name":"Yuxin Chen","slug":"yuxin-chen","given":"Yuxin","family":"Chen","affiliation":"University of Chicago {rdevidze ,"},{"name":"Adish Singla","slug":"adish-singla","given":"Adish","family":"Singla","affiliation":"Max Planck Institute for Software Systems (MPI-SWS)"}],"abstract":"Machine teaching studies the interaction between a teacher and a student/learner where the teacher selects training examples for the learner to learn a specific task. The typical assumption is that the teacher has perfect knowledge of the task---this knowledge comprises knowing the desired learning target, having the exact task representation used by the learner, and knowing the parameters capturing the learning dynamics of the learner. Inspired by real-world applications of machine teaching in education, we consider the setting where teacher's knowledge is limited and noisy, and the key research question we study is the following: When does a teacher succeed or fail in effectively teaching a learner using its imperfect knowledge? We answer this question by showing connections to how imperfect knowledge affects the teacher's solution of the corresponding machine teaching problem when constructing optimal teaching sets. Our results have important implications for designing robust teaching algorithms for real-world applications.","doi":"10.24963/ijcai.2020/367","url":"https://ijcai.org/proceedings/2020/367","pdf":"https://ijcai.org/proceedings/2020/0367.pdf","markdown":"https://ijcai.org/proceedings/2020/367.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/367","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":367,"pages":{"first":2647,"last":2654},"keywords":["Machine Learning: Other","Humans and AI: Computer-Aided Education"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24262/file_24262.m3u8","length_sec":263,"provider":"hls"}],"references":[{"position":1,"raw":"[Archambault et al., 2009] Isabelle Archambault, Michel Janosz, Jean-Sébastien Fallu, and Linda S Pagani. Stu- dent engagement and its relationship with early high school dropout. Journal of adolescence, 32(3):651-670, 2009.","title":"Student engagement and its relationship with early high school dropout","authors":"Isabelle Archambault, Michel Janosz, Jean-Sébastien Fallu, Linda S Pagani","year":"2009","venue":"Journal of adolescence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2018a] Yuxin Chen, Oisin Mac Aodha, Shihan Su, Pietro Perona, and Yisong Yue. Near-optimal machine teaching via explanatory teaching sets. In AISTATS, 2018.","title":"Near-optimal machine teaching via explanatory teaching sets","authors":"Chen","year":"2018","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2018b] Yuxin Chen, Adish Singla, Oisin Mac Aodha, Pietro Perona, and Yisong Yue. Understanding the role of adaptivity in machine teaching: The case of version space learners. In NeurIPS, pages 1483-1493, 2018.","title":"Understanding the role of adaptivity in machine teaching: The case of version space learners","authors":"Chen","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dasgupta et al., 2019] Sanjoy Dasgupta, Daniel Hsu, Ste- fanos Poulis, and Xiaojin Zhu. Teaching a black-box learner. In ICML, pages 1547-1555, 2019.","title":"Teaching a black-box learner","authors":"Sanjoy Dasgupta, Daniel Hsu, Stefanos Poulis, Xiaojin Zhu","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Devidze et al., 2020] Rati Devidze, Farnam Mansouri, Luis Haug, Yuxin Chen, and Adish Singla. Understanding the power and limitations of teaching with imperfect knowl- edge. CoRR, abs/2003.09712, 2020.","title":"Understanding the power and limitations of teaching with imperfect knowledge","authors":"Rati Devidze, Farnam Mansouri, Luis Haug, Yuxin Chen, Adish Singla","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Goldman and Kearns, 1995] Sally A Goldman and Michael J Kearns. On the complexity of teaching. Journal of Computer and System Sciences, 50(1):20-31, 1995.","title":"On the complexity of teaching","authors":"A Sally, Michael J Goldman, Kearns","year":"1995","venue":"Journal of Computer and System Sciences","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Haug et al., 2018] Luis Haug, Sebastian Tschiatschek, and Adish Singla. Teaching inverse reinforcement learners via features and demonstrations. In NeurIPS, 2018.","title":"Teaching inverse reinforcement learners via features and demonstrations","authors":"Luis Haug, Sebastian Tschiatschek, Adish Singla","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hunziker et al., 2019] Anette Hunziker, Yuxin Chen, Oisin Mac Aodha, Manuel Gomez Rodriguez, Andreas Krause, Pietro Perona, Yisong Yue, and Adish Singla. Teaching multiple concepts to a forgetful learner. In NeurIPS, 2019.","title":"Teaching multiple concepts to a forgetful learner","authors":"Anette Hunziker, Yuxin Chen, Oisin Mac Aodha, Manuel Gomez Rodriguez, Andreas Krause, Pietro Perona, Yisong Yue, Adish Singla","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kamalaruban et al., 2019] Parameswaran Kamalaruban, Rati Devidze, Volkan Cevher, and Adish Singla. Interactive teaching algorithms for inverse reinforcement learning. In IJCAI, pages 2692-2700, 2019.","title":"Interactive teaching algorithms for inverse reinforcement learning","authors":"Parameswaran Kamalaruban, Rati Devidze, Volkan Cevher, Adish Singla","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/374"},{"position":10,"raw":"[Khajah et al., 2016] Mohammad Khajah, Robert V. Lindsey, and Michael Mozer. How deep is knowledge tracing? In EDM, 2016.","title":"How deep is knowledge tracing? In EDM","authors":"Mohammad Khajah, Robert V Lindsey, Michael Mozer","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Klingler et al., 2015] Severin Klingler, Tanja Käser, Barbara Solenthaler, and Markus Gross. On the performance char- acteristics of latent-factor and knowledge tracing models. International Educational Data Mining Society, 2015.","title":"On the performance characteristics of latent-factor and knowledge tracing models","authors":"Severin Klingler, Tanja Käser, Barbara Solenthaler, Markus Gross","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2018] Weiyang Liu, Bo Dai, Xingguo Li, Zhen Liu, James M. Rehg, and Le Song. Towards black-box iterative machine teaching. In ICML, 2018.","title":"Towards black-box iterative machine teaching","authors":"Weiyang Liu, Bo Dai, Xingguo Li, Zhen Liu, James M Rehg, Le Song","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ma et al., 2019] Yuzhe Ma, Xiaojin Zhu, and Justin Hsu. Data poisoning against differentially-private learners: At- tacks and defenses. In IJCAI, pages 4732-4738, 2019.","title":"Data poisoning against differentially-private learners: Attacks and defenses","authors":"Yuzhe Ma, Xiaojin Zhu, Justin Hsu","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/657"},{"position":14,"raw":"[Mac Aodha et al., 2018] Oisin Mac Aodha, Shihan Su, Yuxin Chen, Pietro Perona, and Yisong Yue. Teaching categories to human learners with visual explanations. In CVPR, pages 3820-3828, 2018.","title":"Teaching categories to human learners with visual explanations","authors":"Oisin Mac Aodha, Shihan Su, Yuxin Chen, Pietro Perona, Yisong Yue","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mansouri et al., 2019] Farnam Mansouri, Yuxin Chen, Ara Vartanian, Jerry Zhu, and Adish Singla. Preference-based batch and sequential teaching: Towards a unified view of models. In NeurIPS, pages 9195-9205, 2019.","title":"Preference-based batch and sequential teaching: Towards a unified view of models","authors":"Farnam Mansouri, Yuxin Chen, Ara Vartanian, Jerry Zhu, Adish Singla","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mei and Zhu, 2015] Shike Mei and Xiaojin Zhu. Using ma- chine teaching to identify optimal training-set attacks on machine learners. In AAAI, pages 2871-2877, 2015.","title":"Using machine teaching to identify optimal training-set attacks on machine learners","authors":"Zhu ; ] Shike Mei, Xiaojin Mei, Zhu","year":"2015","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Melo et al., 2018] Francisco S. Melo, Carla Guerra, and Manuel Lopes. Interactive optimal teaching with unknown learners. In IJCAI, pages 2567-2573, 2018.","title":"Interactive optimal teaching with unknown learners","authors":"Melo","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/356"},{"position":18,"raw":"[Piech et al., 2015] Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, and Jascha Sohl-Dickstein. Deep knowledge tracing. In Advances in neural information processing systems, 2015.","title":"Deep knowledge tracing","authors":"Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, Jascha Sohl-Dickstein","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rafferty et al., 2016] Anna N Rafferty, Emma Brunskill, Thomas L Griffiths, and Patrick Shafto. Faster teaching via pomdp planning. Cognitive science, 2016.","title":"Faster teaching via pomdp planning","authors":"Anna N Rafferty, Emma Brunskill, Thomas L Griffiths, Patrick Shafto","year":"2016","venue":"Cognitive science","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sen et al., 2018] Ayon Sen, Purav Patel, Martina A. Rau, Blake Mason, Robert Nowak, Timothy T. Rogers, and Xi- aojin Zhu. Machine beats human at sequencing visuals for perceptual-fluency practice. In EDM, 2018.","title":"Machine beats human at sequencing visuals for perceptual-fluency practice","authors":"Ayon Sen, Purav Patel, Martina A Rau, Blake Mason, Robert Nowak, Timothy T Rogers, Xiaojin Zhu","year":"2018","venue":"EDM","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Settles and Meeder, 2016] Burr Settles and Brendan Meeder. A trainable spaced repetition model for language learning. In ACL, pages 1848-1858, 2016.","title":"A trainable spaced repetition model for language learning","authors":"Meeder ; Burr Settles, Brendan Settles, Meeder","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Singla et al., 2013] Adish Singla, Ilija Bogunovic, G Bartók, A Karbasi, and A Krause. On actively teaching the crowd to classify. In NIPS Workshop on Data Driven Education, 2013.","title":"On actively teaching the crowd to classify","authors":"Adish Singla, Ilija Bogunovic, G Bartók, Karbasi, Krause","year":"2013","venue":"NIPS Workshop on Data Driven Education","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Singla et al., 2014] Adish Singla, Ilija Bogunovic, Gábor Bartók, Amin Karbasi, and Andreas Krause. Near- optimally teaching the crowd to classify. In ICML, 2014.","title":"Nearoptimally teaching the crowd to classify","authors":"Adish Singla, Ilija Bogunovic, Gábor Bartók, Amin Karbasi, Andreas Krause","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sullivan et al., 2009] Brian Sullivan, Christopher Wood, Marshall Iliff, Rick Bonney, Daniel Fink, and Steve Kelling. eBird: A citizen-based bird observation network in the bio- logical sciences. Biological Conservation, 2009.","title":"eBird: A citizen-based bird observation network in the biological sciences","authors":"Brian Sullivan, Christopher Wood, Marshall Iliff, Rick Bonney, Daniel Fink, Steve Kelling","year":"2009","venue":"Biological Conservation","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tschiatschek et al., 2019] Sebastian Tschiatschek, Ahana Ghosh, Luis Haug, Rati Devidze, and Adish Singla. Learner-aware teaching: Inverse reinforcement learning with preferences and constraints. In NeurIPS, 2019.","title":"Learner-aware teaching: Inverse reinforcement learning with preferences and constraints","authors":"Sebastian Tschiatschek, Ahana Ghosh, Luis Haug, Rati Devidze, Adish Singla","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Van Horn et al., 2018] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In CVPR, 2018.","title":"The inaturalist species classification and detection dataset","authors":"Van Horn","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yeo et al., 2019] Teresa Yeo, Parameswaran Kamalaruban, Adish Singla, Arpit Merchant, Thibault Asselborn, Louis Faucon, Pierre Dillenbourg, and Volkan Cevher. Iterative classroom teaching. In AAAI, pages 5684-5692, 2019.","title":"Iterative classroom teaching","authors":"Teresa Yeo, Parameswaran Kamalaruban, Adish Singla, Arpit Merchant, Louis Thibault Asselborn, Pierre Faucon, Volkan Dillenbourg, Cevher","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2018] Xuezhou Zhang, Xiaojin Zhu, and Stephen J. Wright. Training set debugging using trusted items. In AAAI, pages 4482-4489, 2018.","title":"Training set debugging using trusted items","authors":"Zhang","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhu et al., 2018] Xiaojin Zhu, Adish Singla, Sandra Zilles, and Anna N. Rafferty. An overview of machine teaching. CoRR, abs/1801.05927, 2018.","title":"An overview of machine teaching","authors":"Xiaojin Zhu, Adish Singla, Sandra Zilles, Anna N Rafferty","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhu, 2015] Xiaojin Zhu. Machine teaching: An inverse prob- lem to machine learning and an approach toward optimal education. In AAAI, pages 4083-4087, 2015.","title":"Machine teaching: An inverse problem to machine learning and an approach toward optimal education","authors":"Xiaojin Zhu","year":"2015","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhu, 2018] Xiaojin Zhu. An optimal control view of adver- sarial machine learning. CoRR, abs/1811.04422, 2018.","title":"An optimal control view of adversarial machine learning","authors":"Xiaojin Zhu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zilles et al., 2011] Sandra Zilles, Steffen Lange, Robert Holte, and Martin Zinkevich. Models of cooperative teach- ing and learning. JMLR, 12(Feb):349-384, 2011.","title":"Models of cooperative teaching and learning","authors":"Sandra Zilles, Steffen Lange, Robert Holte, Martin Zinkevich","year":"2011","venue":"JMLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}