A Novel Strategy for Active Task Assignment in Crowd Labeling

A Novel Strategy for Active Task Assignment in Crowd Labeling

Zehong Hu, Jie Zhang

Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence
Main track. Pages 1538-1545. https://doi.org/10.24963/ijcai.2018/213

Active learning strategies are often used in crowd labeling to improve task assignment. However, these strategies require prohibitive computation time yet still cannot improve the assignment to the utmost, because they simply evaluate each possible assignment and then greedily select the optimal one. In this paper, we first derive an efficient algorithm for assignment evaluation. Then, to overcome the uncertainty of labels, we develop a novel strategy that modulates the scope of the greedy task assignment with posterior uncertainty and keeps the evaluation optimistic. The experiments on two popular worker models and four MTurk datasets show that our strategy achieves the best performance and highest computation efficiency.
Keywords:
Machine Learning: Active Learning
Humans and AI: Human Computation and Crowdsourcing