{"type":"ScholarlyArticle","title":"Cost-Effective Active Learning from Diverse Labelers","authors":[{"name":"Sheng-Jun Huang","slug":"sheng-jun-huang","given":"Sheng-Jun","family":"Huang","affiliation":"College of Computer Science & Technology , Nanjing University of Aeronautics & Astronautics; Collaborative Innovation Center of Novel Software Technology and Industrialization"},{"name":"Jia-Lve Chen","slug":"jia-lve-chen","given":"Jia-Lve","family":"Chen","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University; Collaborative Innovation Center of Novel Software Technology and Industrialization"},{"name":"Xin Mu","slug":"xin-mu","given":"Xin","family":"Mu","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University; Collaborative Innovation Center of Novel Software Technology and Industrialization"},{"name":"Zhi-Hua Zhou","slug":"zhi-hua-zhou","given":"Zhi-Hua","family":"Zhou","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University; Collaborative Innovation Center of Novel Software Technology and Industrialization"}],"abstract":"In traditional active learning, there is only one labeler that always returns the ground truth of queried labels. However, in many applications, multiple labelers are available to offer diverse qualities of labeling with different costs. In this paper, we perform active selection on both instances and labelers, aiming to improve the classification model most with the lowest cost. While the cost of a labeler is proportional to its overall labeling quality, we also observe that different labelers usually have diverse expertise, and thus it is likely that labelers with a low overall quality can provide accurate labels on some specific instances. Based on this fact, we propose a novel active selection criterion to evaluate the cost-effectiveness of instance-labeler pairs, which ensures that the selected instance is helpful for improving the classification model, and meanwhile the selected labeler can provide an accurate label for the instance with a relative low cost. Experiments on both UCI and real crowdsourcing data sets demonstrate the superiority of our proposed approach on selecting cost-effective queries.","doi":"10.24963/ijcai.2017/261","url":"https://ijcai.org/proceedings/2017/261","pdf":"https://ijcai.org/proceedings/2017/0261.pdf","markdown":"https://ijcai.org/proceedings/2017/261.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/261","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":261,"pages":{"first":1879,"last":1885},"keywords":["Machine Learning: Semi-Supervised Learning","Machine Learning: Machine Learning","Machine Learning: Active Learning"],"videos":[],"references":[{"position":1,"raw":"[Ambati et al., 2010] V. Ambati, S. Vogel, and J. G. Carbonell. Ac- tive learning and crowd-sourcing for machine translation. In Pro- ceedings of the International Conference on Language Resources and Evaluation, 2010.","title":"Active learning and crowd-sourcing for machine translation","authors":"V Ambati, S Vogel, J G Carbonell","year":"2010","venue":"Proceedings of the International Conference on Language Resources and Evaluation","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bache and Lichman, 2013] K. Bache and M. Lichman. UCI ma- chine learning repository, 2013.","title":"UCI machine learning repository","authors":"K Bache, M Lichman","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Balcan et al., 2007] M. Balcan, A. Broder, and T. Zhang. Margin based active learning. In Proceedings of the 20th Annual Confer- ence on Learning Theory, pages 35-50, 2007.","title":"Margin based active learning","authors":"M Balcan, A Broder, T Zhang","year":"2007","venue":"Proceedings of the 20th Annual Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chattopadhyay et al., 2012] R. Chattopadhyay, Z. Wang, W. Fan, I. Davidson, S. Panchanathan, and J. Ye. Batch mode active sampling based on marginal probability distribution matching. In Proceedings of the 18th ACM SIGKDD International Confer- ence on Knowledge Discovery and Data Mining, pages 741-749, 2012.","title":"Batch mode active sampling based on marginal probability distribution matching","authors":"R Chattopadhyay, Z Wang, W Fan, I Davidson, S Panchanathan, J Ye","year":"2012","venue":"Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dasgupta and Hsu, 2008] S. Dasgupta and D. Hsu. Hierarchical sampling for active learning. In Proceedings of the 25th Interna- tional Conference on Machine Learning, pages 208-215, 2008.","title":"Hierarchical sampling for active learning","authors":"S Dasgupta, D Hsu","year":"2008","venue":"Proceedings of the 25th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dekel et al., 2012] O. Dekel, C. Gentile, and K. Sridharan. Se- lective sampling and active learning from single and multiple teachers. Journal of Machine Learning Research, 13:2655-2697, 2012.","title":"Selective sampling and active learning from single and multiple teachers","authors":"O Dekel, C Gentile, K Sridharan","year":"2012","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Donmez and Carbonell, 2008] P. Donmez and J. G. Carbonell. Proactive learning: cost-sensitive active learning with multiple imperfect oracles. In Proceedings of the 17th ACM Conference on Information and Knowledge Management, pages 619-628, 2008.","title":"Proactive learning: cost-sensitive active learning with multiple imperfect oracles","authors":"P Donmez, J G Carbonell","year":"2008","venue":"Proceedings of the 17th ACM Conference on Information and Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Donmez et al., 2009] P. Donmez, J. G. Carbonell, and J. G. Schneider. Efficiently learning the accuracy of labeling sources for selective sampling. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 259-268, 2009.","title":"Efficiently learning the accuracy of labeling sources for selective sampling","authors":"P Donmez, J G Carbonell, J G Schneider","year":"2009","venue":"Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Donmez et al., 2010] P. Donmez, J. G. Carbonell, and J. G. Schneider. A probabilistic framework to learn from multiple an- notators with time-varying accuracy. In Proceedings of the SIAM International Conference on Data Mining, pages 826-837, 2010.","title":"A probabilistic framework to learn from multiple annotators with time-varying accuracy","authors":"P Donmez, J G Carbonell, J G Schneider","year":"2010","venue":"Proceedings of the SIAM International Conference on Data Mining","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Fan et al., 2008] R. Fan, K. Chang, C. Hsieh, X. Wang, and C. Lin. Liblinear: A library for large linear classification. JMLR, 9:1871- 1874, 2008.","title":"Liblinear: A library for large linear classification","authors":"R Fan, K Chang, C Hsieh, X Wang, C Lin","year":"2008","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Fang et al., 2014] M. Fang, J. Yin, and D. Tao. Active learning for crowdsourcing using knowledge transfer. In Proceedings of the 28th AAAI Conference on Artificial Intelligence, pages 1809- 1815, 2014.","title":"Active learning for crowdsourcing using knowledge transfer","authors":"M Fang, J Yin, D Tao","year":"2014","venue":"Proceedings of the 28th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Giacinto and Roli, 2000] G. Giacinto and F. Roli. Dynamic clas- sifier selection. In Proceedings of the 1st Workshop on Multiple Classifier Systems, pages 177-189, 2000.","title":"Dynamic classifier selection","authors":"G Giacinto, F Roli","year":"2000","venue":"Proceedings of the 1st Workshop on Multiple Classifier Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Guo and Schuurmans, 2008] Y. Guo and D. Schuurmans. Discrim- inative batch mode active learning. In Advances in Neural Infor- mation Processing Systems 21, pages 593-600. 2008.","title":"Discriminative batch mode active learning","authors":"Y Guo, D Schuurmans","year":"2008","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Huang et al., 2014] S.-J. Huang, R. Jin, and Z.-H. Zhou. Ac- tive learning by querying informative and representative exam- ples. IEEE Transactions on Pattern Analysis and Machine Intel- ligence, (10):1936-1949, 2014.","title":"Active learning by querying informative and representative examples","authors":"S.-J Huang, R Jin, Z.-H Zhou","year":"2014","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ipeirotis et al., 2014] Panagiotis G Ipeirotis, Foster Provost, Vic- tor S Sheng, and Jing Wang. Repeated labeling using mul- tiple noisy labelers. Data Mining and Knowledge Discovery, 28(2):402-441, 2014.","title":"Repeated labeling using multiple noisy labelers","authors":"Foster Panagiotis G Ipeirotis, Victor S Provost, Jing Sheng, Wang","year":"2014","venue":"Data Mining and Knowledge Discovery","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin et al., 2014] Christopher H Lin, Daniel S Weld, et al. To re (la- bel), or not to re (label). In Second AAAI Conference on Human Computation and Crowdsourcing, 2014.","title":"To re (label), or not to re (label)","authors":"Daniel S Christopher H Lin, Weld","year":"2014","venue":"Second AAAI Conference on Human Computation and Crowdsourcing","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Raykar et al., 2010] V. C. Raykar, S. Yu, L. H. Zhao, G. H. Valadez, C. Florin, L. Bogoni, and L. Moy. Learning from crowds. Journal of Machine Learning Research, 11:1297-1322, 2010.","title":"Learning from crowds","authors":"V C Raykar, S Yu, L H Zhao, G H Valadez, C Florin, L Bogoni, L Moy","year":"2010","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Roy and Mccallum, 2001] N. Roy and A. Mccallum. Toward op- timal active learning through sampling estimation of error reduc- tion. In Proceedings of the 18th International Conference on Ma- chine Learning, pages 441-448, 2001.","title":"Toward optimal active learning through sampling estimation of error reduction","authors":"N Roy, A Mccallum","year":"2001","venue":"Proceedings of the 18th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Settles, 2009] B. Settles. Active learning literature survey. Tech- nical report, University of Wisconsin-Madison, 2009.","title":"Active learning literature survey","authors":"B Settles","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sheng et al., 2008] V. Sheng, F. Provost, and P. Ipeirotis. Get an- other label? improving data quality and data mining using multi- ple, noisy labelers. In Proceedings of the 14th ACM SIGKDD In- ternational Conference on Knowledge Discovery and Data Min- ing, pages 614-622, 2008.","title":"Get another label? improving data quality and data mining using multiple, noisy labelers","authors":"V Sheng, F Provost, P Ipeirotis","year":"2008","venue":"Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang and Ye, 2013] Z. Wang and J. Ye. Querying discriminative and representative samples for batch mode active learning. In Proceedings of the 19th ACM SIGKDD International Confer- ence on Knowledge Discovery and Data Mining, pages 158-166, 2013.","title":"Querying discriminative and representative samples for batch mode active learning","authors":"Z Wang, J Ye","year":"2013","venue":"Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Whitehill et al., 2009] J. Whitehill, T. Wu, J. Bergsma, J. R Movel- lan, and P. Ruvolo. Whose vote should count more: Optimal inte- gration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems 22, pages 2035-2043. 2009.","title":"Whose vote should count more: Optimal integration of labels from labelers of unknown expertise","authors":"J Whitehill, T Wu, J Bergsma, J Movellan, P Ruvolo","year":"2009","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yan et al., 2011] Y. Yan, G. M Fung, R. Rosales, and J. Dy. Active learning from crowds. In Proceedings of the 28th International Conference on Machine Learning, pages 1161-1168, 2011.","title":"Active learning from crowds","authors":"Y Yan, G Fung, R Rosales, J Dy","year":"2011","venue":"Proceedings of the 28th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yan et al., 2012] Y. Yan, R. Rosales, G. Fung, F. Farooq, B. Rao, and J. Dy. Active learning from multiple knowledge sources. In Proceedings of the 15th International Conference on Artificial Intelligence and Statistics, pages 1350-1357, 2012.","title":"Active learning from multiple knowledge sources","authors":"Y Yan, R Rosales, G Fung, F Farooq, B Rao, J Dy","year":"2012","venue":"Proceedings of the 15th International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yan et al., 2014] Y. Yan, R. Rosales, G. Fung, S. Ramanathan, and J. Dy. Learning from multiple annotators with varying expertise. Machine Learning, 95(3):291-327, 2014.","title":"Learning from multiple annotators with varying expertise","authors":"Y Yan, R Rosales, G Fung, S Ramanathan, J Dy","year":"2014","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang and Chaudhuri, 2015] Chicheng Zhang and Kamalika Chaudhuri. Active learning from weak and strong labelers. In Advances in Neural Information Processing Systems, pages 703-711. 2015.","title":"Active learning from weak and strong labelers","authors":"Chaudhuri Zhang, Chicheng Zhang, Kamalika Chaudhuri","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2015] J. Zhang, Z. Wu, and V. Shengs. Active learn- ing with imbalanced multiple noisy labeling. IEEE Transactions on Cybernetics, 45(5):1081-1093, 2015.","title":"Active learning with imbalanced multiple noisy labeling","authors":"J Zhang, Z Wu, V Shengs","year":"2015","venue":"IEEE Transactions on Cybernetics","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhao et al., 2011] L. Zhao, G. Sukthankar, and R. Sukthankar. In- cremental relabeling for active learning with noisy crowdsourced annotations. In Proceedings of IEEE 3rd International Confer- ence on Social Computing, pages 728-733, 2011.","title":"Incremental relabeling for active learning with noisy crowdsourced annotations","authors":"L Zhao, G Sukthankar, R Sukthankar","year":"2011","venue":"Proceedings of IEEE 3rd International Conference on Social Computing","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zheng et al., 2010] Y. Zheng, S. Scott, and K. Deng. Active learn- ing from multiple noisy labelers with varied costs. In IEEE 10th International Conference on Data Mining, pages 639-648, 2010.","title":"Active learning from multiple noisy labelers with varied costs","authors":"Y Zheng, S Scott, K Deng","year":"2010","venue":"IEEE 10th International Conference on Data Mining","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhu et al., 2010] J. Zhu, H. Wang, B. Tsou, and M. Ma. Active learning with sampling by uncertainty and density for data an- notations. IEEE Transactions on Audio, Speech, and Language Processing, 18(6):1323-1331, 2010.","title":"Active learning with sampling by uncertainty and density for data annotations","authors":"J Zhu, H Wang, B Tsou, M Ma","year":"2010","venue":"IEEE Transactions on Audio, Speech, and Language Processing","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}