Statistical Quality Control for Human Computation and Crowdsourcing

Statistical Quality Control for Human Computation and Crowdsourcing

Yukino Baba

Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence
Early Career. Pages 5667-5671. https://doi.org/10.24963/ijcai.2018/806

Human computation is a method for solving difficult problems by combining humans and computers. Quality control is a critical issue in human computation because it relies on a large number of participants (i.e., crowds) and there is an uncertainty about their reliability. A solution for this issue is to leverage the power of the "wisdom of crowds"; for example, we can aggregate the outputs of multiple participants or ask a participant to check the output of another participant to improve its quality. In this paper, we review several statistical approaches for controlling the quality of outputs from crowds.
Keywords:
Humans and AI: Human Computation and Crowdsourcing