{"type":"ScholarlyArticle","title":"Hierarchical Active Learning with Group Proportion Feedback","authors":[{"name":"Zhipeng Luo","slug":"zhipeng-luo","given":"Zhipeng","family":"Luo","affiliation":"Department of Computer Science , University of Pittsburgh , PA , USA"},{"name":"Milos Hauskrecht","slug":"milos-hauskrecht","given":"Milos","family":"Hauskrecht","affiliation":"Department of Computer Science , University of Pittsburgh , PA , USA"}],"abstract":"Learning of classification models in practice often relies on nontrivial human annotation effort in which humans assign class labels to data instances. As this process can be very time consuming and costly, finding effective ways to reduce the annotation cost becomes critical for building such models. In this work we solve this problem by exploring a new approach that actively learns classification models from groups, which are subpopulations of instances, and human feedback on the groups. Each group is labeled with a number in [0,1] interval representing a human estimate of the proportion of instances with one of the class labels in this subpopulation. To form the groups to be annotated, we develop a hierarchical active learning framework that divides the whole population into smaller subpopulations, which allows us to gradually learn more refined models from the subpopulations and their class proportion labels. Our extensive experiments on numerous datasets show that our method is competitive and outperforms existing approaches for reducing the human annotation cost.","doi":"10.24963/ijcai.2018/351","url":"https://ijcai.org/proceedings/2018/351","pdf":"https://ijcai.org/proceedings/2018/0351.pdf","markdown":"https://ijcai.org/proceedings/2018/351.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/351","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":351,"pages":{"first":2532,"last":2538},"keywords":["Machine Learning: Active Learning","Machine Learning: Classification","Machine Learning: Cost-Sensitive Learning"],"videos":[],"references":[{"position":1,"raw":"[Asuncion and Newman, 2007] Arthur Asuncion and David Newman. Uci machine learning repository, 2007.","title":"Uci machine learning repository","authors":"Arthur Asuncion, David Newman","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bickel and Doksum, 2015] Peter J Bickel and Kjell A Dok- sum. Mathematical statistics: basic ideas and selected topics, volume I, volume 117. CRC Press, 2015.","title":"Mathematical statistics: basic ideas and selected topics","authors":"J Peter, Kjell A Bickel, Doksum","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dasgupta and Hsu, 2008] Sanjoy Dasgupta and Daniel Hsu. Hierarchical sampling for active learning. In Proceedings of the 25th international conference on Machine learning, pages 208-215. ACM, 2008.","title":"Hierarchical sampling for active learning","authors":"Hsu Dasgupta, Sanjoy Dasgupta, Daniel Hsu","year":"2008","venue":"Proceedings of the 25th international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Du and Ling, 2010] Jun Du and Charles X Ling. Asking generalized queries to domain experts to improve learning. Knowledge and Data Engineering, IEEE Transactions on, 22(6):812-825, 2010.","title":"Jun Du and Charles X Ling. Asking generalized queries to domain experts to improve learning","authors":"Ling Du","year":"2010","venue":"Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Freytag et al., 2014] A. Freytag, E. Rodner, and J. Denzler. Selecting influential examples: Active learning with ex- pected model output changes. In European Conference on Computer Vision, pages 562-577. Springer, 2014.","title":"Selecting influential examples: Active learning with expected model output changes","authors":"A Freytag, E Rodner, J Denzler","year":"2014","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kück and de Freitas, 2012] Hendrik Kück and Nando de Freitas. Learning about individuals from group statistics. CoRR, abs/1207.1393, 2012.","title":"Learning about individuals from group statistics","authors":"De Kück, Hendrik Freitas, Nando Kück, De Freitas","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Nguyen et al., 2014] Quang Nguyen, Hamed Valizadegan, and Milos Hauskrecht. Learning classification models with soft-label information. Journal of the American Med- ical Informatics Association, 21(3):501-508, 2014.","title":"Learning classification models with soft-label information","authors":"Quang Nguyen, Hamed Valizadegan, Milos Hauskrecht","year":"2014","venue":"Journal of the American Medical Informatics Association","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Patrini et al., 2014] Giorgio Patrini, Richard Nock, Paul Rivera, and Tiberio Caetano. (almost) no label no cry. In Advances in Neural Information Processing Systems, pages 190-198, 2014.","title":"almost) no label no cry","authors":"Giorgio Patrini, Richard Nock, Paul Rivera, Tiberio Caetano","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Quadrianto et al., 2009] Novi Quadrianto, Alex J Smola, Tiberio S Caetano, and Quoc V Le. Estimating labels from label proportions. Journal of Machine Learning Research, 10(Oct):2349-2374, 2009.","title":"Estimating labels from label proportions","authors":"Quadrianto","year":"2009","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Quinlan, 2014] J Ross Quinlan. C4. 5: programs for ma- chine learning. Elsevier, 2014.","title":"C4. 5: programs for machine learning","authors":"Quinlan Ross","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Rashidi and Cook, 2011] Parisa Rashidi and Diane J Cook. Ask me better questions: active learning queries based on rule induction. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discov- ery and data mining, pages 904-912. ACM, 2011.","title":"Ask me better questions: active learning queries based on rule induction","authors":"Parisa Rashidi, Diane J Cook","year":"2011","venue":"Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Roy and McCallum, 2001] Nicholas Roy and Andrew Mc- Callum. Toward optimal active learning through monte carlo estimation of error reduction. ICML, Williamstown, pages 441-448, 2001.","title":"Toward optimal active learning through monte carlo estimation of error reduction","authors":"Roy, Mccallum, Nicholas Roy, Andrew Mc-Callum","year":"2001","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Settles, 2012] Burr Settles. Active learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 6(1):1-114, 2012.","title":"Burr Settles. Active learning","authors":"Settles","year":"2012","venue":"Synthesis Lectures on Artificial Intelligence and Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Valizadegan et al., 2013] Hamed Valizadegan, Quang Nguyen, and Milos Hauskrecht. Learning classification models from multiple experts. Journal of biomedical informatics, 46(6):1125-1135, 2013.","title":"Learning classification models from multiple experts","authors":"Hamed Valizadegan, Quang Nguyen, Milos Hauskrecht","year":"2013","venue":"Journal of biomedical informatics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ward Jr, 1963] Joe H Ward Jr. Hierarchical grouping to op- timize an objective function. Journal of the American sta- tistical association, 58(301):236-244, 1963.","title":"Hierarchical grouping to optimize an objective function","authors":"Joe H Ward Jr","year":"1963","venue":"Journal of the American statistical association","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Xue and Hauskrecht, 2017] Yanbing Xue and Milos Hauskrecht. Active learning of classification models with likert-scale feedback. In SIAM Data Mining Conference, 2071. SIAM, 2017.","title":"Yanbing Xue and Milos Hauskrecht. Active learning of classification models with likert-scale feedback","authors":"Hauskrecht Xue","year":"2017","venue":"SIAM Data Mining Conference","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Yu et al., 2013] Felix Yu, Dong Liu, Sanjiv Kumar, Jebara Tony, and Shih-Fu Chang. \\proptosvm for learning with label proportions. In ICML, pages 504-512, 2013.","title":"\\proptosvm for learning with label proportions","authors":"Felix Yu, Dong Liu, Sanjiv Kumar, Jebara Tony, Shih-Fu Chang","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}