{"type":"ScholarlyArticle","title":"Learning in the Wild with Incremental Skeptical Gaussian Processes","authors":[{"name":"Andrea Bontempelli","slug":"andrea-bontempelli","given":"Andrea","family":"Bontempelli","affiliation":"University of Trento , Italy"},{"name":"Stefano Teso","slug":"stefano-teso","given":"Stefano","family":"Teso","affiliation":"University of Trento , Italy"},{"name":"Fausto Giunchiglia","slug":"fausto-giunchiglia","given":"Fausto","family":"Giunchiglia","affiliation":"University of Trento , Italy; Jilin University , Changchun , China"},{"name":"Andrea Passerini","slug":"andrea-passerini","given":"Andrea","family":"Passerini","affiliation":"University of Trento , Italy"}],"abstract":"The ability to learn from human supervision is fundamental for personal assistants and other interactive applications of AI. Two central challenges for deploying interactive learners in the wild are the unreliable nature of the supervision and the varying complexity of the prediction task. We address a simple but representative setting, incremental classification in the wild, where the supervision is noisy and the number of classes grows over time. In order to tackle this task, we propose a redesign of skeptical learning centered around Gaussian Processes (GPs). Skeptical learning is a recent interactive strategy in which, if the machine is sufficiently confident that an example is mislabeled, it asks the annotator to reconsider her feedback. In many cases, this is often enough to obtain clean supervision. Our redesign, dubbed ISGP , leverages the uncertainty estimates supplied by GPs to better allocate labeling and contradiction queries, especially in the presence of noise. Our experiments on synthetic and real-world data show that, as a result, while the original formulation of skeptical learning produces over-confident models that can fail completely in the wild, ISGP works well at varying levels of noise and as new classes are observed.","doi":"10.24963/ijcai.2020/399","url":"https://ijcai.org/proceedings/2020/399","pdf":"https://ijcai.org/proceedings/2020/0399.pdf","markdown":"https://ijcai.org/proceedings/2020/399.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/399","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":399,"pages":{"first":2886,"last":2892},"keywords":["Machine Learning: Kernel Methods","Machine Learning: Classification","Machine Learning: Active Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25387/file_25387.m3u8","length_sec":284,"provider":"hls"}],"references":[{"position":1,"raw":"[Angluin and Laird, 1988] Dana Angluin and Philip Laird. Learning from noisy examples. Machine Learning, 1988.","title":"Learning from noisy examples","authors":"Angluin, Laird ;, Dana Angluin, Philip Laird","year":"1988","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baxter, 2000] Jonathan Baxter. A model of inductive bias learning. JAIR, 2000.","title":"A model of inductive bias learning","authors":"Jonathan Baxter","year":"2000","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bendale and Boult, 2015] Abhijit Bendale and Terrance Boult. Towards open world recognition. In CVPR, 2015.","title":"Towards open world recognition","authors":"Abhijit Bendale, Terrance Boult","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Beygelzimer et al., 2009] Alina Beygelzimer, Sanjoy Das- gupta, and John Langford. Importance weighted active learning. In ICML, 2009.","title":"Importance weighted active learning","authors":"Alina Beygelzimer, Sanjoy Dasgupta, John Langford","year":"2009","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Boult et al., 2019] TE Boult, S Cruz, AR Dhamija, M Gun- ther, J Henrydoss, and WJ Scheirer. Learning and the un- known: Surveying steps toward open world recognition. In AAAI, 2019.","title":"Learning and the unknown: Surveying steps toward open world recognition","authors":"S Boult, Cruz, M Dhamija, J Gunther, Henrydoss, Scheirer","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cesa-Bianchi et al., 2006] Nicolo Cesa-Bianchi, Claudio Gentile, and Luca Zaniboni. Worst-case analysis of se- lective sampling for linear classification. JMLR, 2006.","title":"Worst-case analysis of selective sampling for linear classification","authors":"Cesa-Bianchi","year":"2006","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[De Rosa et al., 2016] Rocco De Rosa, Thomas Mensink, and Barbara Caputo. Online open world recognition. arXiv preprint arXiv:1604.02275, 2016.","title":"Online open world recognition","authors":"Rocco De, Rosa, Thomas Mensink, Barbara Caputo","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Denevi et al., 2018] G Denevi, C Ciliberto, D Stamos, and M Pontil. Incremental learning-to-learn with statistical guarantees. In UAI, 2018.","title":"Incremental learning-to-learn with statistical guarantees","authors":"Denevi, D Ciliberto, M Stamos, Pontil","year":"2018","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dietterich, 2017] Thomas G Dietterich. Steps toward robust artificial intelligence. AI Magazine, 2017.","title":"Steps toward robust artificial intelligence","authors":"G Thomas, Dietterich","year":"2017","venue":"AI Magazine","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ester et al., 1996] Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In KDD, 1996.","title":"A density-based algorithm for discovering clusters in large spatial databases with noise","authors":"Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu","year":"1996","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Frénay and Verleysen, 2014] Benoît Frénay and Michel Verleysen. Classification in the presence of label noise: a survey. IEEE Trans. Neural Netw. Learn. Syst, 2014.","title":"Classification in the presence of label noise: a survey","authors":"Verleysen Frénay, Benoît Frénay, Michel Verleysen","year":"2014","venue":"IEEE Trans. Neural Netw. Learn. Syst","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Guo et al., 2010] Shengbo Guo, Scott Sanner, and Edwin V Bonilla. Gaussian process preference elicitation. In NeurIPS, 2010.","title":"Gaussian process preference elicitation","authors":"Guo","year":"2010","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kapoor et al., 2007] Ashish Kapoor, Kristen Grauman, Raquel Urtasun, and Trevor Darrell. Active learning with gaussian processes for object categorization. In ICCM, 2007.","title":"Active learning with gaussian processes for object categorization","authors":"Ashish Kapoor, Kristen Grauman, Raquel Urtasun, Trevor Darrell","year":"2007","venue":"ICCM","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lütz et al., 2013] Alexander Lütz, Erik Rodner, and Joachim Denzler. I Want To Know More -Efficient Multi-Class Incremental Learning Using Gaussian Processes. Pattern Recognition and Image Analysis, 2013.","title":"I Want To Know More -Efficient Multi-Class Incremental Learning Using Gaussian Processes","authors":"Alexander Lütz, Erik Rodner, Joachim Denzler","year":"2013","venue":"Pattern Recognition and Image Analysis","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pillonetto et al., 2008] Gianluigi Pillonetto, Francesco Din- uzzo, and Giuseppe De Nicolao. Bayesian online multitask learning of gaussian processes. IEEE Trans. Pattern Anal. Mach. Intell, 2008.","title":"Bayesian online multitask learning of gaussian processes","authors":"Gianluigi Pillonetto, Francesco Dinuzzo, Giuseppe De Nicolao","year":"2008","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Quiñonero-Candela and Rasmussen, 2005] Joaquin Quiñonero-Candela and Carl Edward Rasmussen. A unifying view of sparse approximate gaussian process regression. JMLR, 2005.","title":"Joaquin Quiñonero-Candela and Carl Edward Rasmussen. A unifying view of sparse approximate gaussian process regression","authors":"Quiñonero-Candela, Rasmussen","year":"2005","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rodrigues et al., 2014] Filipe Rodrigues, Francisco Pereira, and Bernardete Ribeiro. Gaussian process classification and active learning with multiple annotators. In ICML, 2014.","title":"Gaussian process classification and active learning with multiple annotators","authors":"Filipe Rodrigues, Francisco Pereira, Bernardete Ribeiro","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rudd et al., 2017] Ethan M Rudd, Lalit P Jain, Walter J Scheirer, and Terrance E Boult. The extreme value ma- chine. IEEE Trans. Pattern Anal. Mach. Intell., 2017.","title":"The extreme value machine","authors":"Ethan M Rudd, Lalit P Jain, Walter J Scheirer, Terrance E Boult","year":"2017","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Scheirer et al., 2013] Walter J Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E Boult. Toward open set recognition. IEEE Trans. Pattern Anal. Mach. Intell., 2013.","title":"Toward open set recognition","authors":"J Walter, Anderson Scheirer, De Rezende, Archana Rocha, Terrance E Sapkota, Boult","year":"2013","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Scheirer et al., 2014] Walter J Scheirer, Lalit P Jain, and Terrance E Boult. Probability models for open set recog- nition. IEEE Trans. Pattern Anal. Mach. Intell., 2014.","title":"Probability models for open set recognition","authors":"J Walter, Lalit P Scheirer, Terrance E Jain, Boult","year":"2014","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Skolidis and Sanguinetti, 2011] Grigorios Skolidis and Guido Sanguinetti. Bayesian multitask classification with gaussian process priors. IEEE Trans. Neural Netw., 2011.","title":"Grigorios Skolidis and Guido Sanguinetti. Bayesian multitask classification with gaussian process priors","authors":"Sanguinetti Skolidis","year":"2011","venue":"IEEE Trans. Neural Netw","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Srinivas et al., 2012] Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias W Seeger. Information- theoretic regret bounds for gaussian process optimization in the bandit setting. IEEE Transactions on Information Theory, 2012.","title":"Informationtheoretic regret bounds for gaussian process optimization in the bandit setting","authors":"Niranjan Srinivas, Andreas Krause, Matthias W Sham M Kakade, Seeger","year":"2012","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Thrun, 1996] Sebastian Thrun. Is learning the n-th thing any easier than learning the first? In NeurIPS, 1996.","title":"Is learning the n-th thing any easier than learning the first?","authors":"Sebastian Thrun, Thrun","year":"1996","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tourangeau et al., 2000] Roger Tourangeau, Lance J Rips, and Kenneth Rasinski. The psychology of survey response. 2000.","title":"The psychology of survey response","authors":"Roger Tourangeau, Lance J Rips, Kenneth Rasinski","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[West and Sinibaldi, 2013] Brady T West and Jennifer Sini- baldi. The quality of paradata: A literature review. Im- proving surveys with paradata, 2013.","title":"The quality of paradata: A literature review. Improving surveys with paradata","authors":"T Brady, Jennifer West, Sinibaldi","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Williams and Rasmussen, 2006] Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for ma- chine learning. 2006.","title":"Gaussian processes for machine learning","authors":"K I Christopher, Carl Edward Williams, Rasmussen","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zeni et al., 2014] Mattia Zeni, Ilya Zaihrayeu, and Fausto Giunchiglia. Multi-device activity logging. In UbiComp, 2014.","title":"Multi-device activity logging","authors":"Mattia Zeni, Ilya Zaihrayeu, Fausto Giunchiglia","year":"2014","venue":"UbiComp","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zeni et al., 2019] Mattia Zeni, Wanyi Zhang, Enrico Big- notti, Andrea Passerini, and Fausto Giunchiglia. Fixing mislabeling by human annotators leveraging conflict reso- lution and prior knowledge. IMWUT, 2019.","title":"Fixing mislabeling by human annotators leveraging conflict resolution and prior knowledge","authors":"Mattia Zeni, Wanyi Zhang, Enrico Bignotti, Andrea Passerini, Fausto Giunchiglia","year":"2019","venue":"IMWUT","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}