Statistical Quality Control for Human Computation and Crowdsourcing
Statistical Quality Control for Human Computation and Crowdsourcing
Yukino Baba
Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence
Early Career. Pages 5667-5671.
https://doi.org/10.24963/ijcai.2018/806
Human computation is a method for solving difficult problems by combining humans and computers. Quality control is a critical issue in human computation because it relies on a large number of participants (i.e., crowds) and there is an uncertainty about their reliability. A solution for this issue is to leverage the power of the "wisdom of crowds"; for example, we can aggregate the outputs of multiple participants or ask a participant to check the output of another participant to improve its quality. In this paper, we review several statistical approaches for controlling the quality of outputs from crowds.
Keywords:
Humans and AI: Human Computation and Crowdsourcing