{"type":"ScholarlyArticle","title":"On the Efficiency of Data Collection for Crowdsourced Classification","authors":[{"name":"Edoardo Manino","slug":"edoardo-manino","given":"Edoardo","family":"Manino","affiliation":"University of Southampton"},{"name":"Long Tran-Thanh","slug":"long-tran-thanh","given":"Long","family":"Tran-Thanh","affiliation":"University of Southampton"},{"name":"Nicholas R. Jennings","slug":"nicholas-r-jennings","given":"Nicholas R.","family":"Jennings","affiliation":"Imperial College , London"}],"abstract":"The quality of crowdsourced data is often highly variable. For this reason, it is common to collect redundant data and use statistical methods to aggregate it. Empirical studies show that the policies we use to collect such data have a strong impact on the accuracy of the system. However, there is little theoretical understanding of this phenomenon. In this paper we provide the first theoretical explanation of the accuracy gap between the most popular collection policies: the non-adaptive uniform allocation, and the adaptive uncertainty sampling and information gain maximisation. To do so, we propose a novel representation of the collection process in terms of random walks. Then, we use this tool to derive lower and upper bounds on the accuracy of the policies. With these bounds, we are able to quantify the advantage that the two adaptive policies have over the non-adaptive one for the first time.","doi":"10.24963/ijcai.2018/217","url":"https://ijcai.org/proceedings/2018/217","pdf":"https://ijcai.org/proceedings/2018/0217.pdf","markdown":"https://ijcai.org/proceedings/2018/217.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/217","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Humans and AI","track":"Main track","number":217,"pages":{"first":1568,"last":1575},"keywords":["Machine Learning: Classification","Machine Learning: Probabilistic Machine Learning","Humans and AI: Human Computation and Crowdsourcing","Machine Learning: Active Learning"],"videos":[],"references":[{"position":1,"raw":"[Augustin et al., 2017] Alexandry Augustin, Matteo Venanzi, Alex Rogers, and Nicholas R. Jennings. Bayesian Aggregation of Cat- egorical Distributions with Applications in Crowdsourcing. In Proceedings of the 26th International Joint Conference on Artifi- cial Intelligence, pages 1411-1417, 2017.","title":"Bayesian Aggregation of Categorical Distributions with Applications in Crowdsourcing","authors":"Augustin","year":"2017","venue":"Proceedings of the 26th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2017/195"},{"position":2,"raw":"[Barowy et al., 2012] Daniel W. Barowy, Charlie Curtsinger, Emery D. Berger, and Andrew McGregor. AutoMan: A Plat- form for Integrating Human-based and Digital Computation. In Proceedings of the ACM International Conference on Object Ori- ented Programming Systems Languages and Applications, pages 639-654, 2012.","title":"AutoMan: A Platform for Integrating Human-based and Digital Computation","authors":"Daniel W Barowy, Charlie Curtsinger, Emery D Berger, Andrew Mcgregor","year":"2012","venue":"Proceedings of the ACM International Conference on Object Oriented Programming Systems Languages and Applications","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Berend and Kontorovich, 2014] Daniel Berend and Aryeh Kon- torovich. Consistency of Weighted Majority Votes. In Proceed- ings of the 27th International Conference on Neural Information Processing Systems, pages 3446-3454, 2014.","title":"Consistency of Weighted Majority Votes","authors":"Kontorovich Berend","year":"2014","venue":"Proceedings of the 27th International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bonald and Combes, 2017] Thomas Bonald and Richard Combes. A Minimax Optimal Algorithm for Crowdsourcing. In Proceed- ings of the 30th International Conference on Neural Information Processing Systems, pages 4355-4363, 2017.","title":"A Minimax Optimal Algorithm for Crowdsourcing","authors":"Thomas Bonald, Richard Combes","year":"2017","venue":"Proceedings of the 30th International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Buhrmester et al., 2011] Michael Buhrmester, Tracy Kwang, and Samuel D. Gosling. Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data? Perspectives on Psycho- logical Science, 6(1):3-5, 2011.","title":"Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality","authors":"Michael Buhrmester, Tracy Kwang, Samuel D Gosling","year":"2011","venue":"Data? Perspectives on Psychological Science","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2013] Xi Chen, Qihang Lin, and Dengyong Zhou. Op- timistic Knowledge Gradient Policy for Optimal Budget Alloca- tion in Crowdsourcing. In Proceedings of the 30th International Conference on Machine Learning, pages 64-72, 2013.","title":"Optimistic Knowledge Gradient Policy for Optimal Budget Allocation in Crowdsourcing","authors":"Xi Chen, Qihang Lin, Dengyong Zhou","year":"2013","venue":"Proceedings of the 30th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dawid and Skene, 1979] Alexander P. Dawid and Allan M. Skene. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics), 28(1):20-28, 1979.","title":"Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm","authors":"Alexander P Dawid, Allan M Skene","year":"1979","venue":"Journal of the Royal Statistical Society. Series C (Applied Statistics)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Downs et al., 2010] Julie S. Downs, Mandy B. Holbrook, Steve Sheng, and Lorrie F. Cranor. Are Your Participants Gaming the System? Screening Mechanical Turk Workers. In Proceedings of the SIGCHI Conference on Human Factors in Computing Sys- tems, pages 2399-2402, 2010.","title":"Are Your Participants Gaming the System? Screening Mechanical Turk Workers","authors":"Julie S Downs, Mandy B Holbrook, Steve Sheng, Lorrie F Cranor","year":"2010","venue":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Feller, 1968] William Feller. An Introduction to Probability The- ory and Its Applications. John Wiley and Sons, 1968.","title":"An Introduction to Probability Theory and Its Applications","authors":"William Feller","year":"1968","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gao et al., 2016] Chao Gao, Yu Lu, and Dengyong Zhou. Exact Exponent in Optimal Rates for Crowdsourcing. In Proceedings of the 33rd International Conference on Machine Learning, pages 603-611, 2016.","title":"Exact Exponent in Optimal Rates for Crowdsourcing","authors":"Chao Gao, Yu Lu, Dengyong Zhou","year":"2016","venue":"Proceedings of the 33rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ho et al., 2013] Chien-ju Ho, Shahin Jabbari, and Jennifer Wort- man Vaughan. Adaptive Task Assignment for Crowdsourced Classification. In Proceedings of the 30th International Confer- ence on Machine Learning, pages 534-542, 2013.","title":"Adaptive Task Assignment for Crowdsourced Classification","authors":"Chien-Ju Ho, Shahin Jabbari, Jennifer Wortman Vaughan","year":"2013","venue":"Proceedings of the 30th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Karger et al., 2014] David R. Karger, Sewoong Oh, and Devavrat Shah. Budget-Optimal Task Allocation for Reliable Crowdsourc- ing Systems. Operations Research, 62(1):1-24, 2014.","title":"Budget-Optimal Task Allocation for Reliable Crowdsourcing Systems","authors":"David R Karger, Sewoong Oh, Devavrat Shah","year":"2014","venue":"Operations Research","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Khetan and Oh, 2016] Ashish Khetan and Sewoong Oh. Achiev- ing Budget-Optimality with Adaptive Schemes in Crowdsourc- ing. In Proceedings of the 29th International Conference on Neu- ral Information Processing Systems, pages 4844-4852, 2016.","title":"Achieving Budget-Optimality with Adaptive Schemes in Crowdsourcing","authors":"Ashish Khetan, Sewoong Oh","year":"2016","venue":"Proceedings of the 29th International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lasecki et al., 2013] Walter S. Lasecki, Christopher D. Miller, and Jeffrey P. Bigham. Warping Time for More Effective Real-time Crowdsourcing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 2033-2036, 2013.","title":"Warping Time for More Effective Real-time Crowdsourcing","authors":"Walter S Lasecki, Christopher D Miller, Jeffrey P Bigham","year":"2013","venue":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lewis and Gale, 1994] David D. Lewis and William A. Gale. A Sequential Algorithm for Training Text Classifiers. In Proceed- ings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 3-12, 1994.","title":"A Sequential Algorithm for Training Text Classifiers","authors":"David D Lewis, William A Gale","year":"1994","venue":"Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lintott et al., 2011] Chris Lintott, Kevin Schawinski, Steven Bam- ford, Anže Slosar, Kate Land, Daniel Thomas, Edd Edmondson, Karen Masters, Robert C. Nichol, M. Jordan Raddick, Alex Sza- lay, Dan Andreescu, Phil Murray, and Jan Vandenberg. Galaxy Zoo 1: data release of morphological classifications for nearly 900 000 galaxies. Monthly Notices of the Royal Astronomical So- ciety, 410(1):166-178, 2011.","title":"Galaxy Zoo 1: data release of morphological classifications for nearly 900 000 galaxies","authors":"Chris Lintott, Kevin Schawinski, Steven Bamford, Anže Slosar, Kate Land, Daniel Thomas, Edd Edmondson, Karen Masters, Robert C Nichol, M Jordan Raddick, Alex Szalay, Dan Andreescu, Phil Murray, Jan Vandenberg","year":"2011","venue":"Monthly Notices of the Royal Astronomical Society","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2012] Qiang Liu, Jian Peng, and Alexander Ihler. Vari- ational Inference for Crowdsourcing. In Proceedings of the 25th International Conference on Neural Information Processing Sys- tems, pages 692-700, 2012.","title":"Variational Inference for Crowdsourcing","authors":"Qiang Liu, Jian Peng, Alexander Ihler","year":"2012","venue":"Proceedings of the 25th International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[MacKay, 1992] David J. C. MacKay. Information-Based Objec- tive Functions for Active Data Selection. Neural Computation, 4(4):590-604, 1992.","title":"Information-Based Objective Functions for Active Data Selection","authors":"J C David, Mackay","year":"1992","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nitzan and Paroush, 1982] Shmuel Nitzan and Jacob Paroush. Op- timal Decision Rules in Uncertain Dichotomous Choice Situa- tions. International Economic Review, 23(2):289-297, 1982.","title":"Optimal Decision Rules in Uncertain Dichotomous Choice Situations","authors":"Paroush Nitzan, Shmuel Nitzan, Jacob Paroush","year":"1982","venue":"International Economic Review","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ross et al., 2010] Joel Ross, Lilly Irani, M. Six Silberman, An- drew Zaldivar, and Bill Tomlinson. Who Are the Crowdwork- ers?: Shifting Demographics in Mechanical Turk. In Proceedings of the 28th International Conference on Human Factors in Com- puting Systems -Extended Abstracts, pages 2863-2872, 2010.","title":"Who Are the Crowdworkers?: Shifting Demographics in Mechanical Turk","authors":"Joel Ross, Lilly Irani, M Six Silberman, Andrew Zaldivar, Bill Tomlinson","year":"2010","venue":"Proceedings of the 28th International Conference on Human Factors in Computing Systems -Extended Abstracts","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Simpson and Roberts, 2014] Edwin Simpson and Stephen Roberts. Bayesian Methods for Intelligent Task Assignment in Crowdsourcing Systems. In Scalable Decision Making: Uncertainty, Imperfection, Deliberation, pages 1-32. Springer, 2014.","title":"Edwin Simpson and Stephen Roberts. Bayesian Methods for Intelligent Task Assignment in Crowdsourcing Systems","authors":"Roberts Simpson","year":"2014","venue":"Scalable Decision Making: Uncertainty, Imperfection, Deliberation","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Snow et al., 2008] Rion Snow, Brendan O'Connor, Daniel Juraf- sky, and Andrew Y. Ng. Cheap and Fast -but is It Good?: Eval- uating Non-expert Annotations for Natural Language Tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 254-263, 2008.","title":"Cheap and Fast -but is It Good?: Evaluating Non-expert Annotations for Natural Language Tasks","authors":"Rion Snow, O' Brendan, Daniel Connor, Andrew Y Jurafsky, Ng","year":"2008","venue":"Proceedings of the Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vondrick et al., 2013] Carl Vondrick, Donald Patterson, and Deva Ramanan. Efficiently Scaling Up Crowdsourced Video Annota- tion. International Journal of Computer Vision, 101(1):184-204, 2013.","title":"Efficiently Scaling Up Crowdsourced Video Annotation","authors":"Carl Vondrick, Donald Patterson, Deva Ramanan","year":"2013","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wald, 1944] Abraham Wald. On Cumulative Sums of Random Variables. The Annals of Mathematical Statistics, 15(3):283-296, 1944.","title":"Abraham Wald. On Cumulative Sums of Random Variables","authors":"Wald","year":"1944","venue":"The Annals of Mathematical Statistics","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Welinder and Perona, 2010] Peter Welinder and Pietro Perona. Online Crowdsourcing: Rating Annotators and Obtaining Cost- Effective Labels. In Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition -Workshops, pages 25-32, 2010.","title":"Online Crowdsourcing: Rating Annotators and Obtaining Cost-Effective Labels","authors":"Perona Welinder, Peter Welinder, Pietro Perona","year":"2010","venue":"Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition -Workshops","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Whitehill et al., 2009] Jacob Whitehill, Paul Ruvolo, Tingfan Wu, Jacob Bergsma, and Javier Movellan. Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise. In Proceedings of the 22nd International Conference on Neural Information Processing Systems, pages 2035-2043, 2009.","title":"Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise","authors":"Jacob Whitehill, Paul Ruvolo, Tingfan Wu, Jacob Bergsma, Javier Movellan","year":"2009","venue":"Proceedings of the 22nd International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2014] Yuchen Zhang, Xi Chen, Denny Zhou, and Michael I. Jordan. Spectral methods meet EM: A provably op- timal algorithm for crowdsourcing. In Proceedings of the 27th International Conference on Neural Information Processing Sys- tems, pages 1260-1268, 2014.","title":"Spectral methods meet EM: A provably optimal algorithm for crowdsourcing","authors":"Yuchen Zhang, Xi Chen, Denny Zhou, Michael I Jordan","year":"2014","venue":"Proceedings of the 27th International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}