{"type":"ScholarlyArticle","title":"TDG4Crowd:Test Data Generation for Evaluation of Aggregation Algorithms in Crowdsourcing","authors":[{"name":"Yili Fang","slug":"yili-fang","given":"Yili","family":"Fang","affiliation":"School of Computer Science and Technology , Zhejiang Gongshang University , Hangzhou 310018 , China"},{"name":"Chaojie Shen","slug":"chaojie-shen","given":"Chaojie","family":"Shen","affiliation":"School of Computer Science and Technology , Zhejiang Gongshang University , Hangzhou 310018 , China"},{"name":"Huamao Gu","slug":"huamao-gu","given":"Huamao","family":"Gu","affiliation":"School of Computer Science and Technology , Zhejiang Gongshang University , Hangzhou 310018 , China"},{"name":"Tao Han","slug":"tao-han","given":"Tao","family":"Han","affiliation":"School of Computer Science and Technology , Zhejiang Gongshang University , Hangzhou 310018 , China"},{"name":"Xinyi Ding","slug":"xinyi-ding","given":"Xinyi","family":"Ding","affiliation":"School of Computer Science and Technology , Zhejiang Gongshang University , Hangzhou 310018 , China"}],"abstract":"In crowdsourcing, existing efforts mainly use real datasets collected from crowdsourcing as test datasets to evaluate the effectiveness of aggregation algorithms. However, these work ignore the fact that the datasets obtained by crowdsourcing are usually sparse and imbalanced due to limited budget. As a result, applying the same aggregation algorithm on different datasets often show contradicting conclusions. For example, on the RTE dataset, Dawid and Skene model performs significantly better than Majority Voting, while on the LableMe dataset, the experiments give the opposite conclusion. It is challenging to obtain comprehensive and balanced datasets at a low cost. To our best knowledge, little effort have been made to the fair evaluation of aggregation algorithms. To fill in this gap, we propose a novel method named TDG4Crowd  that can automatically generate comprehensive and balanced datasets. Using Kullback Leibler divergence and Kolmogorov–Smirnov test, the experiment results show the superior of our method compared with others. Aggregation algorithms also perform more consistently on the synthetic datasets generated using our method.","doi":"10.24963/ijcai.2023/333","url":"https://ijcai.org/proceedings/2023/333","pdf":"https://ijcai.org/proceedings/2023/0333.pdf","markdown":"https://ijcai.org/proceedings/2023/333.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/333","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Humans and AI","track":"Main Track","number":333,"pages":{"first":2984,"last":2992},"keywords":["Machine Learning: ML: Cost-sensitive learning","Machine Learning: ML: Autoencoders","Humans and AI: HAI: Human computation and crowdsourcing"],"videos":[],"references":[{"position":1,"raw":"[Albarqouni et al., 2016] Shadi Albarqouni, Christoph Baur, Felix Achilles, Vasileios Belagiannis, Stefanie Demirci, and Nassir Navab. Aggnet: deep learning from crowds for mitosis detection in breast cancer histology images. IEEE transactions on medical imaging, pages 1313-1321, 2016.","title":"Aggnet: deep learning from crowds for mitosis detection in breast cancer histology images","authors":"Shadi Albarqouni, Christoph Baur, Felix Achilles, Vasileios Belagiannis, Stefanie Demirci, Nassir Navab","year":"2016","venue":"IEEE transactions on medical imaging","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baker et al., 2017] Frank B Baker, Seock-Ho Kim, et al. The basics of item response theory using R. Springer, 2017.","title":"The basics of item response theory using R","authors":"B Frank, Seock-Ho Baker, Kim","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dawid and Skene, 1979] Alexander Philip Dawid and Al- lan M Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics), pages 20- 28, 1979.","title":"Maximum likelihood estimation of observer error-rates using the em algorithm","authors":"Skene Dawid, Alexander Philip, Dawid, Allan M Skene","year":"1979","venue":"Journal of the Royal Statistical Society: Series C (Applied Statistics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fang et al., 2018] Yili Fang, Hailong Sun, Pengpeng Chen, and Jinpeng Huai. On the cost complexity of crowd- sourcing. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI), pages 1531- 1537, 2018.","title":"On the cost complexity of crowdsourcing","authors":"Yili Fang, Hailong Sun, Pengpeng Chen, Jinpeng Huai","year":"2018","venue":"Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2018/212"},{"position":5,"raw":"[Galal and El-Sharkawi, 2019] Sh Galal and Mohamed E El- Sharkawi. Trr: Reducing crowdsourcing task redundancy. In Database and Expert Systems Applications -30th Inter- national Conference (DEXA), pages 102-117, 2019.","title":"Trr: Reducing crowdsourcing task redundancy","authors":"El-Sharkawi ; Galal, Sh Galal, Mohamed E El-Sharkawi","year":"2019","venue":"Database and Expert Systems Applications -30th International Conference (DEXA)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gao et al., 2016] Chao Gao, Yu Lu, and Dengyong Zhou. Exact exponent in optimal rates for crowdsourcing. In Pro- ceedings of the 33nd International Conference on Machine Learning (ICML), pages 603-611, 2016.","title":"Exact exponent in optimal rates for crowdsourcing","authors":"Chao Gao, Yu Lu, Dengyong Zhou","year":"2016","venue":"Proceedings of the 33nd International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Heinecke and Reyzin, 2019] Shelby Heinecke and Lev Reyzin. Crowdsourced pac learning under classification noise. In Proceedings of the 7th AAAI Conference on Hu- man Computation and Crowdsourcing (HCOMP), pages 41-49, 2019.","title":"Crowdsourced pac learning under classification noise","authors":"Reyzin Heinecke, Shelby Heinecke, Lev Reyzin","year":"2019","venue":"Proceedings of the 7th AAAI Conference on Human Computation and Crowdsourcing (HCOMP)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ho et al., 2013] Chien-Ju Ho, Shahin Jabbari, and Jen- nifer Wortman Vaughan. Adaptive task assignment for crowdsourced classification. In Proceedings of the 30th International Conference on Machine Learning (ICML), pages 534-542, 2013.","title":"Adaptive task assignment for crowdsourced classification","authors":"Chien-Ju Ho, Shahin Jabbari, Jennifer Wortman Vaughan","year":"2013","venue":"Proceedings of the 30th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Imamura et al., 2018] Hideaki Imamura, Issei Sato, and Masashi Sugiyama. Analysis of minimax error rate for crowdsourcing and its application to worker clustering model. In Proceedings of the 35th International Con- ference on Machine Learning (ICML), pages 2152-2161, 2018.","title":"Analysis of minimax error rate for crowdsourcing and its application to worker clustering model","authors":"Hideaki Imamura, Issei Sato, Masashi Sugiyama","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jing et al., 2018] Jing, Zhang, Victor, S, Sheng, Tao, Li, Xindong, and Wu. Improving crowdsourced label quality using noise correction. IEEE Transactions on Neural Networks Learning Systems, pages 1675-1688, 2018.","title":"Improving crowdsourced label quality using noise correction","authors":"Jing, Zhang, S Victor, Sheng, Tao, Xindong Li, Wu","year":"2018","venue":"IEEE Transactions on Neural Networks Learning Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kawase et al., 2019] Yasushi Kawase, Yuko Kuroki, and Atsushi Miyauchi. Graph mining meets crowdsourcing: Extracting experts for answer aggregation. In Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI), pages 1272-1279, 2019.","title":"Graph mining meets crowdsourcing: Extracting experts for answer aggregation","authors":"Yasushi Kawase, Yuko Kuroki, Atsushi Miyauchi","year":"2019","venue":"Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2019/177"},{"position":12,"raw":"[Li and Yu, 2014] Hongwei Li and Bin Yu. Error rate bounds and iterative weighted majority voting for crowdsourcing. CoRR, abs/1411.4086, 2014.","title":"Error rate bounds and iterative weighted majority voting for crowdsourcing","authors":"Hongwei Li, Bin Yu","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2013] Hongwei Li, Bin Yu, and Dengyong Zhou. Error rate bounds in crowdsourcing models. CoRR, abs/1307.2674, 2013.","title":"Error rate bounds in crowdsourcing models","authors":"Hongwei Li, Bin Yu, Dengyong Zhou","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2012] Qiang Liu, Jian Peng, and Alexander Ihler. Variational inference for crowdsourcing. In Proceedings of the 26th Annual Conference on Neural Information Processing Systems 2012 (NIPS), pages 701-709, 2012.","title":"Variational inference for crowdsourcing","authors":"Qiang Liu, Jian Peng, Alexander Ihler","year":"2012","venue":"Proceedings of the 26th Annual Conference on Neural Information Processing Systems 2012 (NIPS)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Long and Hua, 2016] Chengjiang Long and Gang Hua. Multi-class multi-annotator active learning with robust gaussian process for visual recognition. International Conference on Computer Vision (ICCV), 2016.","title":"Chengjiang Long and Gang Hua. Multi-class multi-annotator active learning with robust gaussian process for visual recognition","authors":"Hua Long","year":"2016","venue":"International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mescheder et al., 2017] Lars M. Mescheder, Sebastian Nowozin, and Andreas Geiger. Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks. In Proceedings of the 34th Inter- national Conference on Machine Learning (ICML), pages 2391-2400, 2017.","title":"Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks","authors":"Lars M Mescheder, Sebastian Nowozin, Andreas Geiger","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pan et al., 2023] Yigong Pan, Ke Tang, and Guangzhong Sun. Theoretical guarantee for crowdsourcing learning with unsure option. Pattern Recognition, page 109316, 2023.","title":"Theoretical guarantee for crowdsourcing learning with unsure option","authors":"Pan","year":"2023","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rodrigues and Pereira, 2018] Filipe Rodrigues and Fran- cisco C. Pereira. Deep learning from crowds. In AAAI conference on artificial intelligence, pages 1611-1618, 2018.","title":"Deep learning from crowds","authors":"Filipe Rodrigues, Francisco C Pereira","year":"2018","venue":"AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Roy et al., 2015] Senjuti Basu Roy, Ioanna Lykourentzou, Saravanan Thirumuruganathan, Sihem Amer-Yahia, and Gautam Das. Task assignment optimization in knowledge- intensive crowdsourcing. The VLDB Journal, pages 467- 491, 2015.","title":"Senjuti Basu Roy, Ioanna Lykourentzou, Saravanan Thirumuruganathan","authors":"Roy","year":"2015","venue":"The VLDB Journal","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Russell et al., 2008] Bryan C. Russell, Antonio Torralba, Kevin P. Murphy, and William T. Freeman. Labelme: A database and web-based tool for image annotation. Inter- national journal of computer vision, pages 157-173, 2008.","title":"Labelme: A database and web-based tool for image annotation","authors":"Bryan C Russell, Antonio Torralba, Kevin P Murphy, William T Freeman","year":"2008","venue":"International journal of computer vision","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sheng et al., 2008] Victor S. Sheng, Foster J. Provost, and Panagiotis G. Ipeirotis. Get another label? improving data quality and data mining using multiple, noisy labelers. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (SIGKDD), pages 614-622, 2008.","title":"Get another label? improving data quality and data mining using multiple, noisy labelers","authors":"S Victor, Foster J Sheng, Panagiotis G Provost, Ipeirotis","year":"2008","venue":"Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (SIGKDD)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sheng, 2011] Victor S Sheng. Simple multiple noisy label utilization strategies. In Proceedings of the 11th IEEE International Conference on Data Mining (ICDM), pages 635-644, 2011.","title":"Simple multiple noisy label utilization strategies","authors":"S Victor, Sheng","year":"2011","venue":"Proceedings of the 11th IEEE International Conference on Data Mining (ICDM)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sinha et al., 2018] Vaibhav B Sinha, Sukrut Rao, and Vi- neeth N Balasubramanian. Fast dawid-skene: A fast vote aggregation scheme for sentiment classification. arXiv preprint arXiv:1803.02781, 2018.","title":"Fast dawid-skene: A fast vote aggregation scheme for sentiment classification","authors":"B Vaibhav, Sukrut Sinha, Rao, Vineeth N Balasubramanian","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Snow et al., 2008] Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. Cheap and fast -but is it good? evaluating non-expert annotations for natural lan- guage tasks. In Proceedings of the 2008 conference on em- pirical methods in natural language processing (EMNLP), pages 254-263, 2008.","title":"Cheap and fast -but is it good? evaluating non-expert annotations for natural language tasks","authors":"Rion Snow, O' Brendan, Daniel Connor, Andrew Y Jurafsky, Ng","year":"2008","venue":"Proceedings of the 2008 conference on empirical methods in natural language processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang and Zhou, 2015] Wei Wang and Zhi-Hua Zhou. Crowdsourcing label quality: a theoretical analysis. Sci- ence China Information Sciences, pages 1-12, 2015.","title":"Crowdsourcing label quality: a theoretical analysis","authors":"Wei Wang, Zhi-Hua Zhou","year":"2015","venue":"Science China Information Sciences","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2013] Jiannan Wang, Guoliang Li, Tim Kraska, Michael J. Franklin, and Jianhua Feng. Leveraging tran- sitive relations for crowdsourced joins. In Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, pages 229-240, 2013.","title":"Leveraging transitive relations for crowdsourced joins","authors":"Wang","year":"2013","venue":"Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wauthier and Jordan, 2011] Fabian L. Wauthier and Michael I. Jordan. Bayesian bias mitigation for crowdsourcing. In Proceedings of the 25th Annual Conference on Neural Information Processing Systems 2011 (NIPS), pages 1800-1808, 2011.","title":"Bayesian bias mitigation for crowdsourcing","authors":"Fabian L Wauthier, Michael I Jordan","year":"2011","venue":"Proceedings of the 25th Annual Conference on Neural Information Processing Systems 2011 (NIPS)","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Whitehill et al., 2009] Jacob Whitehill, Paul Ruvolo, Tingfan Wu, Jacob Bergsma, and Javier R. Movellan. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Proceedings of the 23rd Annual Conference on Neural Information Processing Systems 2009 (NIPS), pages 2035-2043, 2009.","title":"Whose vote should count more: Optimal integration of labels from labelers of unknown expertise","authors":"Jacob Whitehill, Paul Ruvolo, Tingfan Wu, Jacob Bergsma, Javier R Movellan","year":"2009","venue":"Proceedings of the 23rd Annual Conference on Neural Information Processing Systems 2009 (NIPS)","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yu et al., 2017] Han Yu, Chunyan Miao, Yiqiang Chen, Si- mon Fauvel, Xiaoming Li, and Victor R Lesser. Algorith- mic management for improving collective productivity in crowdsourcing. Scientific reports, page 12541, 2017.","title":"Algorithmic management for improving collective productivity in crowdsourcing","authors":"Han Yu, Chunyan Miao, Yiqiang Chen, Simon Fauvel, Xiaoming Li, Victor R Lesser","year":"2017","venue":"Scientific reports","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2016a] Jing Zhang, Xindong Wu, and Vic- tor S. Sheng. Learning from crowdsourced labeled data: a survey. Artificial Intelligence Review, pages 543-576, 2016.","title":"Learning from crowdsourced labeled data: a survey","authors":"Zhang","year":"2016","venue":"Artificial Intelligence Review","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2016b] Yuchen Zhang, Xi Chen, Dengyong Zhou, and Michael I. Jordan. Spectral methods meet em: A provably optimal algorithm for crowdsourcing. Journal of Machine Learning Research, pages 1-44, 2016.","title":"Spectral methods meet em: A provably optimal algorithm for crowdsourcing","authors":"Zhang","year":"2016","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2019] Hao Zhang, Liangxiao Jiang, and Wen- qiang Xu. Multiple noisy label distribution propagation for crowdsourcing. In Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI), pages 1473-1479, 2019.","title":"Multiple noisy label distribution propagation for crowdsourcing","authors":"Hao Zhang, Liangxiao Jiang, Wenqiang Xu","year":"2019","venue":"Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2019/204"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}