{"type":"ScholarlyArticle","title":"On the Cost Complexity of Crowdsourcing","authors":[{"name":"Yili Fang","slug":"yili-fang","given":"Yili","family":"Fang","affiliation":"Beijing Advanced Innovation Center for Big Data and Brain Computing , Beijing , China"},{"name":"Hailong Sun","slug":"hailong-sun","given":"Hailong","family":"Sun","affiliation":"Beijing Advanced Innovation Center for Big Data and Brain Computing , Beijing , China"},{"name":"Pengpeng Chen","slug":"pengpeng-chen","given":"Pengpeng","family":"Chen","affiliation":"School of Computer Science and Engineering , Beihang University , Beijing , China"},{"name":"Jinpeng Huai","slug":"jinpeng-huai","given":"Jinpeng","family":"Huai","affiliation":"School of Computer Science and Engineering , Beihang University , Beijing , China"}],"abstract":"Existing efforts mainly use empirical analysis to evaluate the effectiveness of crowdsourcing methods, which is often unreliable across experimental settings. Consequently, it is of great importance to study theoretical methods. This work, for the first time, defines the cost complexity of crowdsourcing, and presents two theorems to compute the cost complexity. Our theorems provide a general theoretical method to model the trade-off between costs and quality, which can be used to evaluate and design crowdsourcing algorithms, and characterize the complexity of crowdsourcing problems. Moreover, following our theorems, we prove a set of corollaries that can obtain existing theoretical results for special cases. We have verified our work theoretically and empirically.","doi":"10.24963/ijcai.2018/212","url":"https://ijcai.org/proceedings/2018/212","pdf":"https://ijcai.org/proceedings/2018/0212.pdf","markdown":"https://ijcai.org/proceedings/2018/212.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/212","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Humans and AI","track":"Main track","number":212,"pages":{"first":1531,"last":1537},"keywords":["Humans and AI: Human Computation and Crowdsourcing","Machine Learning: Learning Theory"],"videos":[],"references":[{"position":1,"raw":"[Ambati et al., 2011] Vamshi Ambati, Stephan Vogel, and Jaime Carbonell. Towards task recommendation in micro- task. In HCOMP, pages 80-83, 2011.","title":"Towards task recommendation in microtask","authors":"Ambati","year":"2011","venue":"HCOMP","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[AMT, 2017] AMT developer guide. http://docs.aws.amaz on.com/AWSMechTurk/latest/AWSMechanicalTurkRequ ester/amt-dg.pdf, 2017.","title":"AMT developer guide","authors":null,"year":"2017","venue":null,"doi":null,"url":"http://docs.aws.amazon.com/AWSMechTurk/latest/AWSMechanicalTurkRequester/amt-dg.pdf","internal_url":null},{"position":3,"raw":"[Angluin and Laird, 1988] Dana Angluin and Philip Laird. Learning from noisy examples. Machine Learning, 2(4):343-370, 1988.","title":"Dana Angluin and Philip Laird. Learning from noisy examples","authors":"Laird Angluin","year":"1988","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Balcan et al., 2010] Maria Florina Balcan, Steve Hanneke, and Jennifer Wortman Vaughan. The true sample complex- ity of active learning. Machine Learning, 80(2):111-139, 2010.","title":"The true sample complexity of active learning","authors":"Maria Florina Balcan, Steve Hanneke, Jennifer Wortman Vaughan","year":"2010","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dawid and Skene, 1979] Alexander Philip Dawid and Al- lan M Skene. Maximum likelihood estimation of ob- server error-rates using the em algorithm. Applied statis- tics, pages 20-28, 1979.","title":"Maximum likelihood estimation of observer error-rates using the em algorithm","authors":"Alexander Philip, Dawid, Allan M Skene","year":"1979","venue":"Applied statistics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gao and Zhou, 2016] Chao Gao and Dengyong Zhou. Min- imax optimal convergence rates for estimating ground truth from crowdsourced labels. Statistics, 2016.","title":"Minimax optimal convergence rates for estimating ground truth from crowdsourced labels","authors":"Zhou Gao, Chao Gao, Dengyong Zhou","year":"2016","venue":"Statistics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gao et al., 2016] Chao Gao, Yu Lu, and Dengyong Zhou. Exact exponent in optimal rates for crowdsourcing. In ICML, pages 603-611, 2016.","title":"Exact exponent in optimal rates for crowdsourcing","authors":"Chao Gao, Yu Lu, Dengyong Zhou","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Han et al., 2016] Tao Han, Hailong Sun, Yangqiu Song, Yili Fang, and Xudong Liu. Incorporating external knowledge into crowd intelligence for more specific knowledge acqui- sition. In IJCAI, pages 1541-1547, 2016.","title":"Incorporating external knowledge into crowd intelligence for more specific knowledge acquisition","authors":"Tao Han, Hailong Sun, Yangqiu Song, Yili Fang, Xudong Liu","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/221"},{"position":9,"raw":"[Ho et al., 2013] Chien-Ju Ho, Shahin Jabbari, and Jen- nifer Wortman Vaughan. Adaptive task assignment for crowdsourced classification. In ICML, pages 534-542, 2013.","title":"Adaptive task assignment for crowdsourced classification","authors":"Chien-Ju Ho, Shahin Jabbari, Jennifer Wortman Vaughan","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ipeirotis and Gabrilovich, 2014] Panagiotis G. Ipeirotis and Evgeniy Gabrilovich. Quizz: targeted crowdsourcing with a billion (potential) users. In WWW, pages 143-154, 2014.","title":"Quizz: targeted crowdsourcing with a billion (potential) users","authors":"G Panagiotis, Evgeniy Ipeirotis, Gabrilovich","year":"2014","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jung, 2014] Hyun Joon Jung. Quality assurance in crowd- sourcing via matrix factorization based task routing. In WWW, pages 3-8, 2014.","title":"Quality assurance in crowdsourcing via matrix factorization based task routing","authors":"Hyun Joon, Jung","year":"2014","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kulkarni, 2011] Anand Kulkarni. The complexity of crowd- sourcing: Theoretical problems in human computation. In CHI Workshop on Crowdsourcing and Human Computa- tion, 2011.","title":"The complexity of crowdsourcing: Theoretical problems in human computation","authors":"Anand Kulkarni","year":"2011","venue":"CHI Workshop on Crowdsourcing and Human Computation","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li and Liu, 2015] Hongwei Li and Qiang Liu. Cheaper and better: Selecting good workers for crowdsourcing. Eprint Arxiv, 2015.","title":"Cheaper and better: Selecting good workers for crowdsourcing","authors":"Hongwei Li, Qiang Liu","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2012] Qiang Liu, Jian Peng, and Alexander Ihler. Variational inference for crowdsourcing. In NIPS, pages 692-700, 2012.","title":"Variational inference for crowdsourcing","authors":"Qiang Liu, Jian Peng, Alexander Ihler","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Marcus et al., 2015] Adam Marcus, Adam Marcus, Adam Marcus, and Adam Marcus. Argonaut: macrotask crowd- sourcing for complex data processing. Proceedings of the VLDB Endowment, 8(12):1642-1653, 2015.","title":"Argonaut: macrotask crowdsourcing for complex data processing","authors":"Adam Marcus, Adam Marcus, Adam Marcus, Adam Marcus","year":"2015","venue":"Proceedings of the VLDB Endowment","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[McAllester, 1998] David A. McAllester. Some pac- bayesian theorems. In COLT, pages 230-234, 1998.","title":"Some pacbayesian theorems","authors":"David A Mcallester","year":"1998","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nushi et al., 2015] Besmira Nushi, Adish Singla, Anja Gru- enheid, Erfan Zamanian, Andreas Krause, and Donald Kossmann. Crowd access path optimization: Diversity matters. In HCOMP, pages 130-139, 2015.","title":"Crowd access path optimization: Diversity matters","authors":"Nushi","year":"2015","venue":"HCOMP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Raykar et al., 2010] Vikas C Raykar, Shipeng Yu, Linda H Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. Learning from crowds. Machine Learning, 11(Apr):1297-1322, 2010.","title":"Learning from crowds","authors":"Shipeng Vikas C Raykar, Linda H Yu, Gerardo Hermosillo Zhao, Charles Valadez, Luca Florin, Linda Bogoni, Moy","year":"2010","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Roy et al., 2015] Senjuti Basu Roy, Ioanna Lykourentzou, Saravanan Thirumuruganathan, Sihem Amer-Yahia, and Gautam Das. Task assignment optimization in knowledge- intensive crowdsourcing. The VLDB Journal, 24(4):467- 491, 2015.","title":"Senjuti Basu Roy, Ioanna Lykourentzou, Saravanan Thirumuruganathan, Sihem Amer","authors":"Roy","year":"2015","venue":"The VLDB Journal","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Salek et al., 2013] M Salek, Y Bachrach, and P Key. Hotspotting -a probabilistic graphical model for image ob- ject localization through crowdsourcing. In AAAI, pages 1156-1162, 2013.","title":"Hotspotting -a probabilistic graphical model for image object localization through crowdsourcing","authors":"Salek, Bachrach, Key","year":"2013","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shahaf and Amir, 2007] Dafna Shahaf and Eyal Amir. To- wards a theory of ai completeness. In AAAI Spring Sym- posium: Logical Formalizations of Commonsense Reason- ing, pages 150-155, 2007.","title":"Towards a theory of ai completeness","authors":"Amir Shahaf, Dafna Shahaf, Eyal Amir","year":"2007","venue":"AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Snow et al., 2008] Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. Cheap and fast-but is it good?: Evaluating non-expert annotations for natural lan- guage tasks. In EMNLP, pages 254-263, 2008.","title":"Cheap and fast-but is it good?: Evaluating non-expert annotations for natural language tasks","authors":"Rion Snow, O' Brendan, Daniel Connor, Andrew Y Jurafsky, Ng","year":"2008","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tran-Thanh et al., 2013] Long Tran-Thanh, Matteo Ve- nanzi, Alex Rogers, and Nicholas R Jennings. Efficient budget allocation with accuracy guarantees for crowd- sourcing classification tasks. In AAMAS, pages 901-908, 2013.","title":"Efficient budget allocation with accuracy guarantees for crowdsourcing classification tasks","authors":"Tran-Thanh","year":"2013","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Venanzi et al., 2014] Matteo Venanzi, John Guiver, Gabriella Kazai, Pushmeet Kohli, and Milad Shokouhi. Community-based bayesian aggregation models for crowdsourcing. In WWW, pages 155-164, 2014.","title":"Community-based bayesian aggregation models for crowdsourcing","authors":"Matteo Venanzi, John Guiver, Gabriella Kazai, Pushmeet Kohli, Milad Shokouhi","year":"2014","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang and Zhou, 2016] Lu Wang and Zhihua Zhou. Cost- saving effect of crowdsourcing learning. In IJCAI, pages 2111-2117, 2016.","title":"Costsaving effect of crowdsourcing learning","authors":"Lu Wang, Zhihua Zhou","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wing, 2008] Jeannette M Wing. Five deep questions in computing. CACM, 51(1):58-60, 2008.","title":"Five deep questions in computing","authors":"Jeannette M Wing","year":"2008","venue":"CACM","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yu et al., 2017] Han Yu, Chunyan Miao, Yiqiang Chen, Si- mon Fauvel, Xiaoming Li, and Victor R Lesser. Algorith- mic management for improving collective productivity in crowdsourcing. Scientific Reports, 7(1):12541, 2017.","title":"Algorithmic management for improving collective productivity in crowdsourcing","authors":"Han Yu, Chunyan Miao, Yiqiang Chen, Simon Fauvel, Xiaoming Li, Victor R Lesser","year":"2017","venue":"Scientific Reports","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhou et al., 2012] Dengyong Zhou, John C. Platt, Sumit Basu, and Yi Mao. Learning from the wisdom of crowds by minimax entropy. NIPS, 3:2195-2203, 2012.","title":"Learning from the wisdom of crowds by minimax entropy","authors":"Dengyong Zhou, John C Platt, Sumit Basu, Yi Mao","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhou et al., 2015] Dengyong Zhou, Qiang Liu, John C. Platt, Christopher Meek, and Nihar B. Shah. Regularized minimax conditional entropy for crowdsourcing. Eprint Arxiv, 2015.","title":"Regularized minimax conditional entropy for crowdsourcing","authors":"Dengyong Zhou, Qiang Liu, John C Platt, Christopher Meek, Nihar B Shah","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}