{"type":"ScholarlyArticle","title":"Performance as a Constraint: An Improved Wisdom of Crowds Using Performance Regularization","authors":[{"name":"Jiyi Li","slug":"jiyi-li","given":"Jiyi","family":"Li","affiliation":"University of Yamanashi"},{"name":"Yasushi Kawase","slug":"yasushi-kawase","given":"Yasushi","family":"Kawase","affiliation":"Tokyo Institute of Technology"},{"name":"Yukino Baba","slug":"yukino-baba","given":"Yukino","family":"Baba","affiliation":"University of Tsukuba"},{"name":"Hisashi Kashima","slug":"hisashi-kashima","given":"Hisashi","family":"Kashima","affiliation":"Kyoto University"}],"abstract":"Quality assurance is one of the most important problems in crowdsourcing and human computation, and it has been extensively studied from various aspects. Typical approaches for quality assurance include unsupervised approaches such as introducing task redundancy (i.e., asking the same question to multiple workers and aggregating their answers) and supervised approaches such as using worker performance on past tasks or injecting qualification questions into tasks in order to estimate the worker performance. In this paper, we propose to utilize the worker performance as a global constraint for inferring the true answers. The existing semi-supervised approaches do not consider such use of qualification questions. We also propose to utilize the constraint as a regularizer combined with existing statistical aggregation methods. The experiments using heterogeneous multiple-choice questions demonstrate that the performance constraint not only has the power to estimate the ground truths when used by itself, but also boosts the existing aggregation methods when used as a regularizer.","doi":"10.24963/ijcai.2020/213","url":"https://ijcai.org/proceedings/2020/213","pdf":"https://ijcai.org/proceedings/2020/0213.pdf","markdown":"https://ijcai.org/proceedings/2020/213.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/213","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Humans and AI","track":"Main track","number":213,"pages":{"first":1534,"last":1541},"keywords":["Humans and AI: Human Computation and Crowdsourcing"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25404/file_25404.m3u8","length_sec":295,"provider":"hls"}],"references":[{"position":1,"raw":"[Bachrach et al., 2012] Yoram Bachrach, Tom Minka, John Guiver, and Thore Graepel. How to grade a test with- out knowing the answers: A bayesian graphical model for adaptive crowdsourcing and aptitude testing. In ICML, pages 819-826, 2012.","title":"How to grade a test without knowing the answers: A bayesian graphical model for adaptive crowdsourcing and aptitude testing","authors":"Bachrach","year":"2012","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baker et al., 2004] Kevin Baker, Anthony Esgate, David Groome, David Heathcote, Richard Kemp, Moira Maguire, and Corriene Reed. An introduction to applied cognitive psychology. Psychology Press, 2004.","title":"An introduction to applied cognitive psychology","authors":"Kevin Baker, Anthony Esgate, David Groome, David Heathcote, Richard Kemp, Moira Maguire, Corriene Reed","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bowman et al., 2016] Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. In CoNLL, pages 10-21, 2016.","title":"Generating sentences from a continuous space","authors":"R Samuel, Luke Bowman, Oriol Vilnis, Andrew Vinyals, Rafal Dai, Samy Jozefowicz, Bengio","year":"2016","venue":"CoNLL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dawid and Skene, 1979] Alexander P. Dawid and Allan M. Skene. Maximum likelihood estimation of observer error- rates using the EM algorithm. Journal of the Royal Statis- tical Society. Series C (Applied Statistics), 28(1), 1979.","title":"Maximum likelihood estimation of observer errorrates using the EM algorithm","authors":"Alexander P Dawid, Allan M Skene","year":"1979","venue":"Journal of the Royal Statistical Society. Series C (Applied Statistics)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Downs et al., 2010] Julie S. Downs, Mandy B. Holbrook, Steve Sheng, and Lorrie Faith Cranor. Are your partic- ipants gaming the system?: Screening mechanical turk workers. In CHI, pages 2399-2402, 2010.","title":"Are your participants gaming the system?: Screening mechanical turk workers","authors":"Julie S Downs, Mandy B Holbrook, Steve Sheng, Lorrie Faith Cranor","year":"2010","venue":"CHI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ipeirotis and Gabrilovich, 2014] Panagiotis G. Ipeirotis and Evgeniy Gabrilovich. Quizz: Targeted crowdsourcing with a billion (potential) users. In WWW, pages 143-154, 2014.","title":"Quizz: Targeted crowdsourcing with a billion (potential) users","authors":"G Panagiotis, Evgeniy Ipeirotis, Gabrilovich","year":"2014","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ipeirotis et al., 2010] Panagiotis G. Ipeirotis, Foster Provost, and Jing Wang. Quality management on amazon mechanical turk. In HCOMP, pages 64-67, 2010.","title":"Quality management on amazon mechanical turk","authors":"G Panagiotis, Foster Ipeirotis, Jing Provost, Wang","year":"2010","venue":"HCOMP","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Karger et al., 2011] David R. Karger, Sewoong Oh, and De- vavrat Shah. Iterative learning for reliable crowdsourcing systems. In NIPS, pages 1953-1961, 2011.","title":"Iterative learning for reliable crowdsourcing systems","authors":"David R Karger, Sewoong Oh, Devavrat Shah","year":"2011","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kawase et al., 2019] Yasushi Kawase, Yuko Kuroki, and Atsushi Miyauchi. Graph mining meets crowdsourcing: Extracting experts for answer aggregation. In IJCAI, pages 1272-1279, 2019.","title":"Graph mining meets crowdsourcing: Extracting experts for answer aggregation","authors":"Yasushi Kawase, Yuko Kuroki, Atsushi Miyauchi","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/177"},{"position":10,"raw":"[Kazai et al., 2012] Gabriella Kazai, Jaap Kamps, and Natasa Milic-Frayling. The face of quality in crowdsourc- ing relevance labels: Demographics, personality and label- ing accuracy. In CIKM, pages 2583-2586, 2012.","title":"The face of quality in crowdsourcing relevance labels: Demographics, personality and labeling accuracy","authors":"Gabriella Kazai, Jaap Kamps, Natasa Milic-Frayling","year":"2012","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kuncheva and Rodríguez, 2014] Ludmila I Kuncheva and Juan J Rodríguez. A weighted voting framework for clas- sifiers ensembles. Knowledge and Information Systems, 38(2):259-275, 2014.","title":"A weighted voting framework for classifiers ensembles","authors":"I Ludmila, Juan J Kuncheva, Rodríguez","year":"2014","venue":"Knowledge and Information Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Law and Ahn, 2011] Edith Law and Luis von Ahn. Human computation. Synthesis lectures on artificial intelligence and machine learning, 5(3):1-121, 2011.","title":"Edith Law and Luis von Ahn. Human computation","authors":"Ahn Law","year":"2011","venue":"Synthesis lectures on artificial intelligence and machine learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2014] Hongwei Li, Bo Zhao, and Ariel Fuxman. The wisdom of minority: Discovering and targeting the right group of workers for crowdsourcing. In WWW, pages 165-176, 2014.","title":"The wisdom of minority: Discovering and targeting the right group of workers for crowdsourcing","authors":"Hongwei Li, Bo Zhao, Ariel Fuxman","year":"2014","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2017] Jiyi Li, Yukino Baba, and Hisashi Kashima. Hyper questions: Unsupervised targeting of a few experts in crowdsourcing. In CIKM, pages 1069-1078, 2017.","title":"Hyper questions: Unsupervised targeting of a few experts in crowdsourcing","authors":"Li","year":"2017","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2012] Qiang Liu, Jian Peng, and Alexander Ihler. Variational inference for crowdsourcing. In NIPS, pages 692-700, 2012.","title":"Variational inference for crowdsourcing","authors":"Qiang Liu, Jian Peng, Alexander Ihler","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ma et al., 2015] Fenglong Ma, Yaliang Li, Qi Li, Minghui Qiu, Jing Gao, Shi Zhi, Lu Su, Bo Zhao, Heng Ji, and Jiawei Han. Faitcrowd: Fine grained truth discovery for crowdsourced data aggregation. In KDD, pages 745-754, 2015.","title":"Faitcrowd: Fine grained truth discovery for crowdsourced data aggregation","authors":"Fenglong Ma, Yaliang Li, Qi Li, Minghui Qiu, Jing Gao, Shi Zhi, Lu Su, Bo Zhao, Ji Heng, Jiawei Han","year":"2015","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mortensen et al., 2013] Jonathan M Mortensen, Mark A Musen, and Natalya F Noy. Crowdsourcing the ver- ification of relationships in biomedical ontologies. In AMIA Annual Symposium Proceedings (AMIA), page 1020, 2013.","title":"Crowdsourcing the verification of relationships in biomedical ontologies","authors":"Jonathan M Mortensen, Mark A Musen, Natalya F Noy","year":"2013","venue":"AMIA Annual Symposium Proceedings (AMIA)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Motwani and Raghavan, 1995] Rajeev Motwani and Prab- hakar Raghavan. Randomized Algorithms. Cambridge University Press, 1995.","title":"Rajeev Motwani and Prabhakar Raghavan","authors":"Raghavan Motwani","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Prelec et al., 2017] Dražen Prelec, H Sebastian Seung, and John McCoy. A solution to the single-question crowd wis- dom problem. Nature, 541(7638):532-535, 2017.","title":"A solution to the single-question crowd wisdom problem","authors":"Dražen Prelec, Sebastian Seung, John Mccoy","year":"2017","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Venanzi et al., 2014] Matteo Venanzi, John Guiver, Gabriella Kazai, Pushmeet Kohli, and Milad Shokouhi. Community-based bayesian aggregation models for crowdsourcing. In WWW, pages 155-164, 2014.","title":"Community-based bayesian aggregation models for crowdsourcing","authors":"Matteo Venanzi, John Guiver, Gabriella Kazai, Pushmeet Kohli, Milad Shokouhi","year":"2014","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wauthier and Jordan, 2011] Fabian L. Wauthier and Michael I. Jordan. Bayesian bias mitigation for crowd- sourcing. In NIPS, pages 1800-1808, 2011.","title":"Bayesian bias mitigation for crowdsourcing","authors":"Fabian L Wauthier, Michael I Jordan","year":"2011","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Welinder et al., 2010] Peter Welinder, Steve Branson, Pietro Perona, and Serge J. Belongie. The multidimen- sional wisdom of crowds. In NIPS, pages 2424-2432, 2010.","title":"The multidimensional wisdom of crowds","authors":"Peter Welinder, Steve Branson, Pietro Perona, Serge J Belongie","year":"2010","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Whitehill et al., 2009] Jacob Whitehill, Paul Ruvolo, Tingfan Wu, Jacob Bergsma, and Javier Movellan. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In NIPS, pages 2035-2043, 2009.","title":"Whose vote should count more: Optimal integration of labels from labelers of unknown expertise","authors":"Jacob Whitehill, Paul Ruvolo, Tingfan Wu, Jacob Bergsma, Javier Movellan","year":"2009","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Whitehill, 2019] Jacob Whitehill. How does knowledge of the auc constrain the set of possible ground-truth label- ings? In AAAI, pages 5425-5432, 2019.","title":"How does knowledge of the auc constrain the set of possible ground-truth labelings?","authors":"Jacob Whitehill, Whitehill","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yin et al., 2017] Li'ang Yin, Jianhua Han, Weinan Zhang, and Yong Yu. Aggregating crowd wisdoms with label- aware autoencoders. In IJCAI, pages 1325-1331, 2017.","title":"Aggregating crowd wisdoms with labelaware autoencoders","authors":"Jianhua Li'ang Yin, Weinan Han, Yong Zhang, Yu","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zheng et al., 2017] Yudian Zheng, Guoliang Li, Yuanbing Li, Caihua Shan, and Reynold Cheng. Truth inference in crowdsourcing: Is the problem solved? Proceedings of the VLDB Endowment, 10(5):541-552, 2017.","title":"Truth inference in crowdsourcing: Is the problem solved?","authors":"Zheng","year":"2017","venue":"Proceedings of the VLDB Endowment","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhou and He, 2016] Yao Zhou and Jingrui He. Crowdsourc- ing via tensor augmentation and completion. In IJCAI, pages 2435-2441, 2016.","title":"Crowdsourcing via tensor augmentation and completion","authors":"Yao Zhou, Jingrui He","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/347"},{"position":28,"raw":"[Zhou et al., 2012] Dengyong Zhou, John C. Platt, Sumit Basu, and Yi Mao. Learning from the wisdom of crowds by minimax entropy. In NIPS, pages 2195-2203, 2012.","title":"Learning from the wisdom of crowds by minimax entropy","authors":"Dengyong Zhou, John C Platt, Sumit Basu, Yi Mao","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}