{"type":"ScholarlyArticle","title":"A Novel Strategy for Active Task Assignment in Crowd Labeling","authors":[{"name":"Zehong Hu","slug":"zehong-hu","given":"Zehong","family":"Hu","affiliation":"NTU Corporate Lab , School of Computer Science and Engineering Nanyang Technological University , Singapore"},{"name":"Jie Zhang","slug":"jie-zhang","given":"Jie","family":"Zhang","affiliation":"NTU Corporate Lab , School of Computer Science and Engineering Nanyang Technological University , Singapore"}],"abstract":"Active learning strategies are often used in crowd labeling to improve task assignment. However, these strategies require prohibitive computation time yet still cannot improve the assignment to the utmost, because they simply evaluate each possible assignment and then greedily select the optimal one. In this paper, we first derive an efficient algorithm for assignment evaluation. Then, to overcome the uncertainty of labels, we develop a novel strategy that modulates the scope of the greedy task assignment with posterior uncertainty and keeps the evaluation optimistic. The experiments on two popular worker models and four MTurk datasets show that our strategy achieves the best performance and highest computation efficiency.","doi":"10.24963/ijcai.2018/213","url":"https://ijcai.org/proceedings/2018/213","pdf":"https://ijcai.org/proceedings/2018/0213.pdf","markdown":"https://ijcai.org/proceedings/2018/213.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/213","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Humans and AI","track":"Main track","number":213,"pages":{"first":1538,"last":1545},"keywords":["Humans and AI: Human Computation and Crowdsourcing","Machine Learning: Active Learning"],"videos":[],"references":[{"position":1,"raw":"[Buckley et al., 2010] Chris Buckley, Matthew Lease, and Mark D. Smucker. Overview of the trec 2010 relevance feedback track. In TREC Notebook, 2010.","title":"Overview of the trec 2010 relevance feedback track","authors":"Chris Buckley, Matthew Lease, Mark D Smucker","year":"2010","venue":"TREC Notebook","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Burden and Faires, 2004] R. Burden and J. Faires. Numeri- cal Analysis. Cengage Learning, 2004.","title":"Numerical Analysis","authors":"R Burden, J Faires","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2015] Xi Chen, Qihang Lin, and Dengyong Zhou. Statistical decision making for optimal budget al- location in crowd labeling. Journal of Machine Learning Research, 16:1-46, 2015.","title":"Statistical decision making for optimal budget allocation in crowd labeling","authors":"Xi Chen, Qihang Lin, Dengyong Zhou","year":"2015","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proc. of CVPR, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fan et al., 2015] Ju Fan, Guoliang Li, Beng Chin Ooi, Kian- lee Tan, and Jianhua Feng. icrowd: An adaptive crowd- sourcing framework. In Proc. of SIGMOD, 2015.","title":"icrowd: An adaptive crowdsourcing framework","authors":"Ju Fan, Guoliang Li, Beng, Chin Ooi, Kianlee Tan, Jianhua Feng","year":"2015","venue":"Proc. of SIGMOD","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ho et al., 2013] Chien-Ju Ho, Shahin Jabbari, and Jen- nifer Wortman Vaughan. Adaptive task assignment for crowdsourced classification. In Proc. of ICML, 2013.","title":"Adaptive task assignment for crowdsourced classification","authors":"Chien-Ju Ho, Shahin Jabbari, Jennifer Wortman Vaughan","year":"2013","venue":"Proc. of ICML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu and Zhang, 2017] Zehong Hu and Jie Zhang. Optimal posted-price mechanism in microtask crowdsourcing. In Proc. of IJCAI, 2017.","title":"Optimal posted-price mechanism in microtask crowdsourcing","authors":"Zhang ; ] Zehong Hu, Jie Hu, Zhang","year":"2017","venue":"Proc. of IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/33"},{"position":8,"raw":"[Hua et al., 2013] Gang Hua, Chengjiang Long, Ming Yang, and Yan Gao. Collaborative active learning of a kernel ma- chine ensemble for recognition. In Proc. of ICCV, 2013.","title":"Collaborative active learning of a kernel machine ensemble for recognition","authors":"Gang Hua, Chengjiang Long, Ming Yang, Yan Gao","year":"2013","venue":"Proc. of ICCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ipeirotis et al., 2010] Panagiotis G Ipeirotis, Foster Provost, and Jing Wang. Quality management on amazon mechan- ical turk. In Proc. of the ACM SIGKDD workshop on hu- man computation, 2010.","title":"Quality management on amazon mechanical turk","authors":"Foster Panagiotis G Ipeirotis, Jing Provost, Wang","year":"2010","venue":"Proc. of the ACM SIGKDD workshop on human computation","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kamar et al., 2012] Ece Kamar, Severin Hacker, and Eric Horvitz. Combining human and machine intelligence in large-scale crowdsourcing. In Proc. of AAMAS, pages 467-474, 2012.","title":"Combining human and machine intelligence in large-scale crowdsourcing","authors":"Ece Kamar, Severin Hacker, Eric Horvitz","year":"2012","venue":"Proc. of AAMAS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lewis and Catlett, 1994] David D Lewis and Jason Catlett. Heterogeneous uncertainty sampling for supervised learn- ing. In Proc. of ICML, 1994.","title":"Heterogeneous uncertainty sampling for supervised learning","authors":"D David, Jason Lewis, Catlett","year":"1994","venue":"Proc. of ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lin et al., 2016] Christopher H Lin, M Mausam, and Daniel S Weld. Re-active learning: Active learning with relabeling. In Proc. of AAAI, 2016.","title":"Re-active learning: Active learning with relabeling","authors":"M Christopher H Lin, Daniel S Mausam, Weld","year":"2016","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2012] Qiang Liu, Jian Peng, and Alexander T Ih- ler. Variational inference for crowdsourcing. In Proc. of NIPS, 2012.","title":"Variational inference for crowdsourcing","authors":"Qiang Liu, Jian Peng, Alexander T Ihler","year":"2012","venue":"Proc. of NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mező, 2013] István Mező. Some infinite sums arising from the weierstrass product theorem. Applied Mathematics and Computation, 219(18):9838-9846, 2013.","title":"István Mező. Some infinite sums arising from the weierstrass product theorem","authors":"Mező","year":"2013","venue":"Applied Mathematics and Computation","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Muhammadi et al., 2015] Jafar Muhammadi, Hamid R Ra- biee, and Abbas Hosseini. A unified statistical framework for crowd labeling. Knowledge and Information Systems, 45(2):271-294, 2015.","title":"A unified statistical framework for crowd labeling","authors":"Jafar Muhammadi, Abbas Hamid R Rabiee, Hosseini","year":"2015","venue":"Knowledge and Information Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pang and Lee, 2005] Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proc. of ACL, 2005.","title":"Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales","authors":"Lee ; Pang, Bo Pang, Lillian Lee","year":"2005","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Parameswaran et al., 2014] Aditya Parameswaran, Stephen Boyd, Hector Garcia-Molina, Ashish Gupta, Neoklis Poly- zotis, and Jennifer Widom. Optimal crowd-powered rating and filtering algorithms. Proceedings of the VLDB Endow- ment, 7(9):685-696, 2014.","title":"Optimal crowd-powered rating and filtering algorithms","authors":"Aditya Parameswaran, Stephen Boyd, Hector Garcia-Molina, Ashish Gupta, Neoklis Polyzotis, Jennifer Widom","year":"2014","venue":"Proceedings of the VLDB Endowment","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Raykar and Agrawal, 2014] Vikas Raykar and Priyanka Agrawal. Sequential crowdsourced labeling as an epsilon- greedy exploration in a markov decision process. In Proc. of AISTATS, 2014.","title":"Vikas Raykar and Priyanka Agrawal. Sequential crowdsourced labeling as an epsilongreedy exploration in a markov decision process","authors":"Agrawal Raykar","year":"2014","venue":"Proc. of AISTATS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rockafellar and Uryasev, 2002] R Tyrrell Rockafellar and Stanislav Uryasev. Conditional value-at-risk for gen- eral loss distributions. Journal of banking & finance, 26(7):1443-1471, 2002.","title":"Conditional value-at-risk for general loss distributions","authors":"Rockafellar Tyrrell, Stanislav Uryasev","year":"2002","venue":"Journal of banking & finance","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rodway et al., 2012] Paul Rodway, Astrid Schepman, and Jordana Lambert. Preferring the one in the middle: Fur- ther evidence for the centre-stage effect. Applied Cognitive Psychology, 26(2):215-222, 2012.","title":"Preferring the one in the middle: Further evidence for the centre-stage effect","authors":"Rodway","year":"2012","venue":"Applied Cognitive Psychology","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Settles, 2012] Burr Settles. Active learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 6(1):1-114, 2012.","title":"Burr Settles. Active learning","authors":"Settles","year":"2012","venue":"Synthesis Lectures on Artificial Intelligence and Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sheng et al., 2008] Victor S Sheng, Foster Provost, and Panagiotis G Ipeirotis. Get another label? improving data quality and data mining using multiple, noisy labelers. In Proc. of SIGKDD, 2008.","title":"Get another label? improving data quality and data mining using multiple, noisy labelers","authors":"Foster Victor S Sheng, Panagiotis G Provost, Ipeirotis","year":"2008","venue":"Proc. of SIGKDD","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sheshadri and Lease, 2013] Aashish Sheshadri and Matthew Lease. SQUARE: A Benchmark for Re- search on Computing Crowd Consensus. In Proc. of HCOMP, 2013.","title":"SQUARE: A Benchmark for Research on Computing Crowd Consensus","authors":"Aashish Sheshadri, Matthew Lease","year":"2013","venue":"Proc. of HCOMP","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Simpson and Roberts, 2015] Edwin Simpson and Stephen Roberts. Bayesian methods for intelligent task assignment in crowdsourcing systems. In Decision Making: Uncer- tainty, Imperfection, Deliberation and Scalability, pages 1-32. Springer, 2015.","title":"Edwin Simpson and Stephen Roberts. Bayesian methods for intelligent task assignment in crowdsourcing systems","authors":"Roberts Simpson","year":"2015","venue":"Decision Making: Uncertainty, Imperfection, Deliberation and Scalability","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Slivkins and Vaughan, 2014] Aleksandrs Slivkins and Jen- nifer Wortman Vaughan. Online decision making in crowdsourcing markets: Theoretical challenges. ACM SIGecom Exchanges, 12(2):4-23, 2014.","title":"Online decision making in crowdsourcing markets: Theoretical challenges","authors":"Vaughan ; Slivkins, Aleksandrs Slivkins, Jennifer Wortman Vaughan","year":"2014","venue":"ACM SIGecom Exchanges","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Snow et al., 2008] Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y Ng. Cheap and fast-but is it good?: evaluating non-expert annotations for natural lan- guage tasks. In Proc. of EMNLP, 2008.","title":"Cheap and fast-but is it good?: evaluating non-expert annotations for natural language tasks","authors":"Rion Snow, O' Brendan, Daniel Connor, Andrew Y Jurafsky, Ng","year":"2008","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tarable et al., 2017] Alberto Tarable, Alessandro Nordio, Emilio Leonardi, and Marco Ajmone Marsan. The impor- tance of worker reputation information in microtask-based crowd work systems. IEEE Transactions on Parallel and Distributed Systems, 28(2):558-571, 2017.","title":"The importance of worker reputation information in microtask-based crowd work systems","authors":"Alberto Tarable, Alessandro Nordio, Emilio Leonardi, Marco Ajmone Marsan","year":"2017","venue":"IEEE Transactions on Parallel and Distributed Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Welinder and Perona, 2010] Peter Welinder and Pietro Per- ona. Online crowdsourcing: rating annotators and obtain- ing cost-effective labels. In CVPR Workshops, 2010.","title":"Online crowdsourcing: rating annotators and obtaining cost-effective labels","authors":"Perona Welinder, Peter Welinder, Pietro Perona","year":"2010","venue":"CVPR Workshops","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zheng et al., 2015] Yudian Zheng, Jiannan Wang, Guoliang Li, Reynold Cheng, and Jianhua Feng. Qasca: A quality- aware task assignment system for crowdsourcing applica- tions. In Proc. of SIGMOD, 2015.","title":"Qasca: A qualityaware task assignment system for crowdsourcing applications","authors":"Zheng","year":"2015","venue":"Proc. of SIGMOD","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zheng et al., 2017] Yudian Zheng, Guoliang Li, Yuanbing Li, Caihua Shan, and Reynold Cheng. Truth inference in crowdsourcing: is the problem solved? Proc. of the VLDB Endowment, 10(5):541-552, 2017.","title":"Truth inference in crowdsourcing: is the problem solved?","authors":"Zheng","year":"2017","venue":"Proc. of the VLDB Endowment","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}