{"type":"ScholarlyArticle","title":"Statistical Quality Control for Human Computation and Crowdsourcing","authors":[{"name":"Yukino Baba","slug":"yukino-baba","given":"Yukino","family":"Baba","affiliation":"University of Tsukuba"}],"abstract":"Human computation is a method for solving difficult problems by combining humans and computers. Quality control is a critical issue in human computation because it relies on a large number of participants (i.e., crowds) and there is an uncertainty about their reliability. A solution for this issue is to leverage the power of the \"wisdom of crowds\"; for example, we can aggregate the outputs of multiple participants or ask a participant to check the output of another participant to improve its quality. In this paper, we review several statistical approaches for controlling the quality of outputs from crowds.","doi":"10.24963/ijcai.2018/806","url":"https://ijcai.org/proceedings/2018/806","pdf":"https://ijcai.org/proceedings/2018/0806.pdf","markdown":"https://ijcai.org/proceedings/2018/806.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/806","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Early Career","track":"Early Career","number":806,"pages":{"first":5667,"last":5671},"keywords":["Humans and AI: Human Computation and Crowdsourcing"],"videos":[],"references":[{"position":1,"raw":"[Dawid and Skene, 1979] Alexander P. Dawid and Allan M. Skene. Maximum likelihood estimation of observer error- rates using the EM algorithm. Journal of the Royal Sta- tistical Society. Series C (Applied Statistics), 28(1):20-28, 1979.","title":"Maximum likelihood estimation of observer errorrates using the EM algorithm","authors":"Alexander P Dawid, Allan M Skene","year":"1979","venue":"Journal of the Royal Statistical Society. Series C (Applied Statistics)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ehara et al., 2016] Yo Ehara, Yukino Baba, Masao Utiyama, and Eiichiro Sumita. Assessing translation abil- ity through vocabulary ability assessment. In Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI), pages 3712-3718, 2016.","title":"Assessing translation ability through vocabulary ability assessment","authors":"Ehara","year":"2016","venue":"Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2016/522"},{"position":3,"raw":"[Kajimura et al., 2015] Shunsuke Kajimura, Yukino Baba, Hiroshi Kajino, and Hisashi Kashima. Quality control for crowdsourced POI collection. In Proceedings of the 19th Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD), pages 255-267, 2015.","title":"Quality control for crowdsourced POI collection","authors":"Shunsuke Kajimura, Yukino Baba, Hiroshi Kajino, Hisashi Kashima","year":"2015","venue":"Proceedings of the 19th Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Kleinberg, 1999] Jon M Kleinberg. Authoritative sources in a hyperlinked environment. Journal of the ACM, 46(5):604-632, 1999.","title":"Authoritative sources in a hyperlinked environment","authors":"Jon M Kleinberg","year":"1999","venue":"Journal of the ACM","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Li et al., 2017] Jiyi Li, Yukino Baba, and Hisashi Kashima. Hyper questions: Unsupervised targeting of a few experts in crowdsourcing. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management (CIKM), pages 1069-1078, 2017.","title":"Hyper questions: Unsupervised targeting of a few experts in crowdsourcing","authors":"Li","year":"2017","venue":"Proceedings of the 2017 ACM on Conference on Information and Knowledge Management (CIKM)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Li et al., 2018] Jiyi Li, Yukino Baba, and Hisashi Kashima. Simultaneous clustering and ranking from pairwise com- parisons. In Proceedings of 27th International Joint Con- ference on Artificial Intelligence and the 23rd European Conference on Artificial Intelligence (IJCAI-ECAI), 2018.","title":"Simultaneous clustering and ranking from pairwise comparisons","authors":"Li","year":"2018","venue":"Proceedings of 27th International Joint Conference on Artificial Intelligence and the 23rd European Conference on Artificial Intelligence (IJCAI-ECAI)","doi":null,"url":null,"internal_url":"/proceedings/2018/215"},{"position":7,"raw":"[Lin et al., 2012] Christopher H. Lin, Mausam, and Daniel S. Weld. Crowdsourcing control: moving beyond multiple choice. In Proceedings of the 28th Conference on Uncertainty in Artificial Intelligence (UAI), pages 491-500, 2012.","title":"Crowdsourcing control: moving beyond multiple choice","authors":"Christopher H Lin, Daniel S Mausam, Weld","year":"2012","venue":"Proceedings of the 28th Conference on Uncertainty in Artificial Intelligence (UAI)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lorenz et al., 2011] Jan Lorenz, Heiko Rauhut, Frank Schweitzer, and Dirk Helbing. How social influence can undermine the wisdom of crowd effect. Proceedings of the National Academy of Sciences, 108(22):9020-9025, 2011.","title":"How social influence can undermine the wisdom of crowd effect","authors":"Lorenz","year":"2011","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Matsui et al., 2014] Toshiko Matsui, Yukino Baba, Toshi- hiro Kamishima, and Hisashi Kashima. Crowdordering. In Proceedings of the 18th Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD), pages 336-347, 2014.","title":"Crowdordering","authors":"Toshiko Matsui, Yukino Baba, Toshihiro Kamishima, Hisashi Kashima","year":"2014","venue":"Proceedings of the 18th Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Otani et al., 2015] Naoki Otani, Yukino Baba, and Hisashi Kashima. Quality control for crowdsourced hierarchical classification. In Proceedings of the 2015 IEEE Interna- tional Conference on Data Mining (ICDM), pages 937- 942, 2015.","title":"Quality control for crowdsourced hierarchical classification","authors":"Naoki Otani, Yukino Baba, Hisashi Kashima","year":"2015","venue":"Proceedings of the 2015 IEEE International Conference on Data Mining (ICDM)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Oyama et al., 2013] Satoshi Oyama, Yukino Baba, Yuko Sakurai, and Hisashi Kashima. Accurate integration of crowdsourced labels using workers' self-reported confi- dence scores. In Proceedings of the 23rd International Joint Conference on Artificial Intelligence (IJCAI), pages 2554-2560, 2013.","title":"Accurate integration of crowdsourced labels using workers' self-reported confidence scores","authors":"Satoshi Oyama, Yukino Baba, Yuko Sakurai, Hisashi Kashima","year":"2013","venue":"Proceedings of the 23rd International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2013/376"},{"position":12,"raw":"[Rasch, 1960] G. Rasch. Probabilistic models for some in- telligence and attainment tests. Denmarks Paedagogiske, 1960.","title":"Probabilistic models for some intelligence and attainment tests","authors":"G Rasch","year":"1960","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Raykar and Yu, 2011] Vikas C Raykar and Shipeng Yu. Ranking annotators for crowdsourced labeling tasks. In Advances in Neural Information Processing Systems 24 (NIPS), pages 1809-1817. 2011.","title":"Ranking annotators for crowdsourced labeling tasks","authors":"C Vikas, Shipeng Raykar, Yu","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ribeiro et al., 2016] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. \"why should I trust you\": Explain- ing the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowl- edge Discovery and Data Mining (KDD), pages 1135- 1144, 2016.","title":"why should I trust you\": Explaining the predictions of any classifier","authors":"Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin","year":"2016","venue":"Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Samejima, 1969] Fumi Samejima. Estimation of latent abil- ity using a response pattern of graded scores. Psychome- trika Monograph Supplement, 1969.","title":"Estimation of latent ability using a response pattern of graded scores","authors":"Fumi Samejima, Samejima","year":"1969","venue":"Psychometrika Monograph Supplement","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sunahase et al., 2017] Takeru Sunahase, Yukino Baba, and Hisashi Kashima. Pairwise HITS: Quality estimation from pairwise comparisons in creator-evaluator crowdsourcing process. In Proceedings of the 31st AAAI Conference on Artificial Intelligence (AAAI), pages 977-983, 2017.","title":"Pairwise HITS: Quality estimation from pairwise comparisons in creator-evaluator crowdsourcing process","authors":"Sunahase","year":"2017","venue":"Proceedings of the 31st AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Takahama et al., 2018] Ryusuke Takahama, Yukino Baba, Nobuyuki Shimizu, Sumio Fujita, and Hisashi Kashima. AdaFlock: Adaptive feature discovery for human-in-the- loop predictive modeling. In Proceedings of the 31st AAAI Conference on Artificial Intelligence (AAAI), pages 1619- 1626, 2018.","title":"AdaFlock: Adaptive feature discovery for human-in-theloop predictive modeling","authors":"Ryusuke Takahama, Yukino Baba, Nobuyuki Shimizu, Sumio Fujita, Hisashi Kashima","year":"2018","venue":"Proceedings of the 31st AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Venanzi et al., 2014] Matteo Venanzi, John Guiver, Gabriella Kazai, Pushmeet Kohli, and Milad Shokouhi. Community-based bayesian aggregation models for crowdsourcing. In Proceedings of the 23rd International Conference on World Wide Web (WWW), pages 155-164, 2014.","title":"Community-based bayesian aggregation models for crowdsourcing","authors":"Matteo Venanzi, John Guiver, Gabriella Kazai, Pushmeet Kohli, Milad Shokouhi","year":"2014","venue":"Proceedings of the 23rd International Conference on World Wide Web (WWW)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[von Ahn et al., 2008] Luis von Ahn, Benjamin Maurer, Colin McMillen, David Abraham, and Manuel Blum. re- CAPTCHA: human-based character recognition via web security measures. Science, 321(5895):1465-1468, 2008.","title":"re-CAPTCHA: human-based character recognition via web security measures","authors":"Von Ahn","year":"2008","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Welinder et al., 2010] Peter Welinder, Steve Branson, Pietro Perona, and Serge J. Belongie. The multidi- mensional wisdom of crowds. In Advances in Neural Information Processing Systems 23 (NIPS), pages 2424-2432. 2010.","title":"The multidimensional wisdom of crowds","authors":"Peter Welinder, Steve Branson, Pietro Perona, Serge J Belongie","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Whitehill et al., 2009] Jacob Whitehill, Ting fan Wu, Jacob Bergsma, Javier R. Movellan, and Paul L. Ruvolo. Whose vote should count more: optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems 22 (NIPS), pages 2035- 2043. 2009.","title":"Whose vote should count more: optimal integration of labels from labelers of unknown expertise","authors":"Jacob Whitehill, Jacob Ting Fan Wu, Javier R Bergsma, Paul L Movellan, Ruvolo","year":"2009","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wu et al., 2012] Xian Wu, Wei Fan, and Yong Yu. Sem- bler: Ensembling crowd sequential labeling for improved quality. In Proceedings of the 26th AAAI Conference on Artificial Intelligence (AAAI), pages 1713-1719, 2012.","title":"Sembler: Ensembling crowd sequential labeling for improved quality","authors":"Xian Wu, Wei Fan, Yong Yu","year":"2012","venue":"Proceedings of the 26th AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}