{"type":"ScholarlyArticle","title":"Obtaining High-Quality Label by Distinguishing between Easy and Hard Items in Crowdsourcing","authors":[{"name":"Wei Wang","slug":"wei-wang","given":"Wei","family":"Wang","affiliation":"National Key Laboratory for Novel Software Technology Nanjing University , Nanjing 210023 , China"},{"name":"Xiang-Yu Guo","slug":"xiang-yu-guo","given":"Xiang-Yu","family":"Guo","affiliation":"National Key Laboratory for Novel Software Technology Nanjing University , Nanjing 210023 , China"},{"name":"Shao-Yuan Li","slug":"shao-yuan-li","given":"Shao-Yuan","family":"Li","affiliation":"National Key Laboratory for Novel Software Technology Nanjing University , Nanjing 210023 , China"},{"name":"Yuan Jiang","slug":"yuan-jiang","given":"Yuan","family":"Jiang","affiliation":"National Key Laboratory for Novel Software Technology Nanjing University , Nanjing 210023 , China"},{"name":"Zhi-Hua Zhou","slug":"zhi-hua-zhou","given":"Zhi-Hua","family":"Zhou","affiliation":"National Key Laboratory for Novel Software Technology Nanjing University , Nanjing 210023 , China"}],"abstract":"Crowdsourcing systems make it possible to hire voluntary workers to label large-scale data by offering them small monetary payments. Usually, the taskmaster requires to collect high-quality labels, while the quality of labels obtained from the crowd may not satisfy this requirement. In this paper, we study the problem of obtaining high-quality labels from the crowd and present an approach of learning the difficulty of items in crowdsourcing, in which we construct a small training set of items with estimated difficulty and then learn a model to predict the difficulty of future items. With the predicted difficulty, we can distinguish between easy and hard items to obtain high-quality labels. For easy items, the quality of their labels inferred from the crowd could be high enough to satisfy the requirement; while for hard items, the crowd could not provide high-quality labels, it is better to choose a more knowledgable crowd or employ specialized workers to label them. The experimental results demonstrate that the proposed approach by learning to distinguish between easy and hard items can significantly improve the label quality.","doi":"10.24963/ijcai.2017/413","url":"https://ijcai.org/proceedings/2017/413","pdf":"https://ijcai.org/proceedings/2017/0413.pdf","markdown":"https://ijcai.org/proceedings/2017/413.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/413","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":413,"pages":{"first":2964,"last":2970},"keywords":["Machine Learning: Machine Learning","Machine Learning: Multi-instance/Multi-label/Multi-view learning"],"videos":[],"references":[{"position":1,"raw":"[Buchbinder and Naor, 2005] Niv Buchbinder and Joseph Naor. Online primal-dual algorithms for covering and packing problems. In ESA, pages 689-701, Palma de Mal- lorca, Spain, 2005.","title":"Online primal-dual algorithms for covering and packing problems","authors":"Naor Buchbinder, Niv Buchbinder, Joseph Naor","year":"2005","venue":"ESA","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2013] Xi Chen, Qihang Lin, and Dengyong Zhou. Optimistic knowledge gradient policy for optimal budget allocation in crowdsourcing. In ICML, pages 64- 72, Atlanta, GA, 2013.","title":"Optimistic knowledge gradient policy for optimal budget allocation in crowdsourcing","authors":"Xi Chen, Qihang Lin, Dengyong Zhou","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2015] Xi Chen, Qihang Lin, and Dengyong Zhou. Statistical decision making for optimal budget al- location in crowd labeling. Journal of Machine Learning Research, 16:1-46, 2015.","title":"Statistical decision making for optimal budget allocation in crowd labeling","authors":"Xi Chen, Qihang Lin, Dengyong Zhou","year":"2015","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gadiraju et al., 2015] Ujwal Gadiraju, Ricardo Kawase, Stefan Dietze, and Gianluca Demartini. Understanding malicious behavior in crowdsourcing platforms: The case of online surveys. In ACM CHI, pages 1631-1640, Seoul, Republic of Korea, 2015.","title":"Understanding malicious behavior in crowdsourcing platforms: The case of online surveys","authors":"Ujwal Gadiraju, Ricardo Kawase, Stefan Dietze, Gianluca Demartini","year":"2015","venue":"ACM CHI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ho and Vaughan, 2012] Chien-Ju Ho and Jennifer W. Vaughan. Online task assignment in crowdsourcing mar- kets. In AAAI, Toronto, Canada, 2012.","title":"Online task assignment in crowdsourcing markets","authors":"Chien-Ju Ho, Jennifer W Vaughan","year":"2012","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ho et al., 2013] Chien-Ju Ho, Shahin Jabbari, and Jen- nifer W. Vaughan. Adaptive task assignment for crowd- sourced classification. In ICML, pages 534-542, Atlanta, GA, 2013.","title":"Adaptive task assignment for crowdsourced classification","authors":"Chien-Ju Ho, Shahin Jabbari, Jennifer W Vaughan","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hoeffding, 1963] Wassily Hoeffding. Probability inequali- ties for sums of bounded random variables. Journal of the American Statistical Association, 58:13-30, 1963.","title":"Wassily Hoeffding. Probability inequalities for sums of bounded random variables","authors":"Hoeffding","year":"1963","venue":"Journal of the American Statistical Association","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Howe, 2006] Jeff Howe. The rise of crowdsourcing. Wired, 2006.","title":"The rise of crowdsourcing","authors":"Jeff Howe","year":"2006","venue":"Wired","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Howe, 2008] Jeff Howe. Why the power of the crowd is driving the future of business. Crown Business, 2008.","title":"Why the power of the crowd is driving the future of business","authors":"Jeff Howe","year":"2008","venue":"Crown Business","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Karger et al., 2011] David R. Karger, Sewoong Oh, and De- vavrat Shah. Iterative learning for reliable crowdsourcing systems. In NIPS, pages 1953-1961. MIT Press, Cam- bridge, MA, 2011.","title":"Iterative learning for reliable crowdsourcing systems","authors":"David R Karger, Sewoong Oh, Devavrat Shah","year":"2011","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Karger et al., 2014] David R. Karger, Sewoong Oh, and De- vavrat Shah. Budget-optimal task allocation for reliable crowdsourcing systems. Operations Research, 62(1):1- 24, 2014.","title":"Budget-optimal task allocation for reliable crowdsourcing systems","authors":"David R Karger, Sewoong Oh, Devavrat Shah","year":"2014","venue":"Operations Research","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kazai et al., 2011] Gabriella Kazai, Jaap Kamps, Marijn Koolen, and Natasa Milic-Frayling. Crowdsourcing for book search evaluation: impact of hit design on compara- tive system ranking. In ACM SIGIR, pages 205-214, Bei- jing, China, 2011.","title":"Crowdsourcing for book search evaluation: impact of hit design on comparative system ranking","authors":"Gabriella Kazai, Jaap Kamps, Marijn Koolen, Natasa Milic-Frayling","year":"2011","venue":"ACM SIGIR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2012] Qiang Liu, Jian Peng, and Alexander T. Ih- ler. Variational inference for crowdsourcing. In NIPS, pages 701-709. MIT Press, Cambridge, MA, 2012.","title":"Variational inference for crowdsourcing","authors":"Qiang Liu, Jian Peng, Alexander T Ihler","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2013] Qiang Liu, Alexander T. Ihler, and Mark Steyvers. Scoring workers in crowdsourcing: How many control questions are enough? In NIPS, pages 1914-1922.","title":"Scoring workers in crowdsourcing: How many control questions are enough?","authors":"Qiang Liu, Alexander T Ihler, Mark Steyvers","year":null,"venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Raykar and Agrawal, 2014] Vikas C. Raykar and Priyanka Agrawal. Sequential crowdsourced labeling as an epsilon- greedy exploration in a markov decision process. In AIS- TATS, pages 832-840, Reykjavik, Iceland, 2014.","title":"Sequential crowdsourced labeling as an epsilongreedy exploration in a markov decision process","authors":"C Vikas, Priyanka Raykar, Agrawal","year":"2014","venue":"AIS-TATS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Raykar and Yu, 2012] Vikas C. Raykar and Shipeng Yu. E- liminating spammers and ranking annotators for crowd- sourced labeling tasks. Journal of Machine Learning Re- search, 13:491-518, 2012.","title":"Eliminating spammers and ranking annotators for crowdsourced labeling tasks","authors":"C Vikas, Shipeng Raykar, Yu","year":"2012","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Raykar et al., 2009] Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Anna K. Jerebko, Charles Florin, Gerardo Her- mosillo Valadez, Luca Bogoni, and Linda Moy. Super- vised learning from multiple experts: whom to trust when everyone lies a bit. In ICML, Quebec, Canada, 2009.","title":"Supervised learning from multiple experts: whom to trust when everyone lies a bit","authors":"C Vikas, Shipeng Raykar, Linda H Yu, Anna K Zhao, Charles Jerebko, Gerardo Hermosillo Florin, Luca Valadez, Linda Bogoni, Moy","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Raykar et al., 2010] Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. Learning from crowds. Journal of Machine Learning Research, 11:1297-1322, 2010.","title":"Learning from crowds","authors":"C Vikas, Shipeng Raykar, Linda H Yu, Gerardo Hermosillo Zhao, Charles Valadez, Luca Florin, Linda Bogoni, Moy","year":"2010","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shah and Zhou, 2015] Nihar B. Shah and Dengyong Zhou. Double or nothing: Multiplicative incentive mechanisms for crowdsourcing. In NIPS, pages 1-9. MIT Press, Cam- bridge, MA, 2015.","title":"Double or nothing: Multiplicative incentive mechanisms for crowdsourcing","authors":"B Nihar, Dengyong Shah, Zhou","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shah and Zhou, 2016] Nihar B. Shah and Dengyong Zhou. No oops, you won't do it again: Mechanisms for self- correction in crowdsourcing. In ICML, pages 1-10, New York, NY, 2016.","title":"No oops, you won't do it again: Mechanisms for selfcorrection in crowdsourcing","authors":"B Nihar, Dengyong Shah, Zhou","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shah et al., 2015] Nihar B. Shah, Dengyong Zhou, and Yu- val Peres. Approval voting and incentives in crowdsourc- ing. In ICML, pages 10-19, Lille, France, 2015.","title":"Approval voting and incentives in crowdsourcing","authors":"B Nihar, Dengyong Shah, Yuval Zhou, Peres","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sheng et al., 2008] Victor S. Sheng, Foster J. Provost, and Panagiotis G. Ipeirotis. Get another label? improving data quality and data mining using multiple, noisy labelers. In ACM SIGKDD, pages 614-622, Las Vegas, NV, 2008.","title":"Get another label? improving data quality and data mining using multiple, noisy labelers","authors":"S Victor, Foster J Sheng, Panagiotis G Provost, Ipeirotis","year":"2008","venue":"ACM SIGKDD","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Snow et al., 2008] Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. Cheap and fast -but is it good? evaluating non-expert annotations for natural lan- guage tasks. In EMNLP, pages 254-263, Honolulu, HI, 2008.","title":"Cheap and fast -but is it good? evaluating non-expert annotations for natural language tasks","authors":"Rion Snow, O' Brendan, Daniel Connor, Andrew Y Jurafsky, Ng","year":"2008","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sorokin and Forsyth, 2008] Alexander Sorokin and David A. Forsyth. Utility data annotation with ama- zon mechanical turk. In CVPR Workshop on Internet Vision, pages 1-8, Anchorage, AK, 2008.","title":"Utility data annotation with amazon mechanical turk","authors":"Alexander Sorokin, David A Forsyth","year":"2008","venue":"CVPR Workshop on Internet Vision","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tian and Zhu, 2015] Tian Tian and Jun Zhu. Max-margin majority voting for learning from crowds. In NIPS, pages 1612-1620. MIT Press, Cambridge, MA, 2015.","title":"Max-margin majority voting for learning from crowds","authors":"Zhu ; ] Tian Tian, Jun Tian, Zhu","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Vuurens et al., 2011] Jeroen Vuurens, Arjen P. de Vries, and Carsten Eickhoff. How much spam can you take? an anal- ysis of crowdsourcing results to increase accuracy. In ACM SIGIR Workshop on Crowdsourcing for Information Re- trieval, pages 21-26, Beijing, China, 2011.","title":"How much spam can you take? an analysis of crowdsourcing results to increase accuracy","authors":"Jeroen Vuurens, P Arjen, Carsten De Vries, Eickhoff","year":"2011","venue":"ACM SIGIR Workshop on Crowdsourcing for Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang and Zhou, 2016] Lu Wang and Zhi-Hua Zhou. Cost- saving effect of crowdsourcing learning. In IJCAI, pages 2111-2117, New York, NY, 2016.","title":"Costsaving effect of crowdsourcing learning","authors":"Lu Wang, Zhi-Hua Zhou","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wauthier and Jordan, 2011] Fabian L. Wauthier and Michael I. Jordan. Bayesian bias mitigation for crowd- sourcing. In NIPS, pages 1800-1808. MIT Press, Cambridge, MA, 2011.","title":"Bayesian bias mitigation for crowdsourcing","authors":"Fabian L Wauthier, Michael I Jordan","year":"2011","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Welinder et al., 2010] Peter Welinder, Steve Branson, Serge J. Belongie, and Pietro Perona. The multidimen- sional wisdom of crowds. In NIPS, pages 2424-2432.","title":"The multidimensional wisdom of crowds","authors":"Peter Welinder, Steve Branson, Serge J Belongie, Pietro Perona","year":null,"venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Whitehill et al., 2009] Jacob Whitehill, Paul Ruvolo, T- ingfan Wu, Jacob Bergsma, and Javier R. Movellan. Whose vote should count more: Optimal integration of la- bels from labelers of unknown expertise. In NIPS, pages 2035-2043. MIT Press, Cambridge, MA, 2009.","title":"Whose vote should count more: Optimal integration of labels from labelers of unknown expertise","authors":"Jacob Whitehill, Paul Ruvolo, Tingfan Wu, Jacob Bergsma, Javier R Movellan","year":"2009","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yan et al., 2011] Yan Yan, Róomer Rosales, Glenn Fung, and Jennifer G. Dy. Active learning from crowds. In ICM- L, pages 1161-1168, Bellevue, WA, 2011.","title":"Active learning from crowds","authors":"Yan Yan, Róomer Rosales, Glenn Fung, Jennifer G Dy","year":"2011","venue":"ICM-L","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yuen et al., 2011] Man-Ching Yuen, Irwin King, and Kwong-Sak Leung. A survey of crowdsourcing system- s. In SocialCom, pages 766-773, Boston, MA, 2011.","title":"A survey of crowdsourcing systems","authors":"Yuen","year":"2011","venue":"SocialCom","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhou et al., 2012] Dengyong Zhou, John C. Platt, Sumit Basu, and Yi Mao. Learning from the wisdom of crowd- s by minimax entropy. In NIPS, pages 2204-2212. MIT Press, Cambridge, MA, 2012.","title":"Learning from the wisdom of crowds by minimax entropy","authors":"Dengyong Zhou, John C Platt, Sumit Basu, Yi Mao","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhou et al., 2014] Dengyong Zhou, Qiang Liu, John C. Platt, and Christopher Meek. Aggregating ordinal labels from crowds by minimax conditional entropy. In ICML, pages 262-270, Beijing, China, 2014.","title":"Aggregating ordinal labels from crowds by minimax conditional entropy","authors":"Dengyong Zhou, Qiang Liu, John C Platt, Christopher Meek","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}