{"type":"ScholarlyArticle","title":"Bayesian Decision Process for Budget-efficient Crowdsourced Clustering","authors":[{"name":"Xiaozhou Wang","slug":"xiaozhou-wang","given":"Xiaozhou","family":"Wang","affiliation":"School of Mathematical Sciences , Shanghai Jiao Tong University , China"},{"name":"Xi Chen","slug":"xi-chen","given":"Xi","family":"Chen","affiliation":"Stern School of Business , New York University , USA"},{"name":"Qihang Lin","slug":"qihang-lin","given":"Qihang","family":"Lin","affiliation":"Tippie College of Business , University of Iowa , USA"},{"name":"Weidong Liu","slug":"weidong-liu","given":"Weidong","family":"Liu","affiliation":"School of Mathematical Sciences , Shanghai Jiao Tong University , China; MoE Key Lab of Artificial Intelligence , Shanghai Jiao Tong University , China"}],"abstract":"The performance of clustering depends on an appropriately defined similarity between two items. When the similarity is measured based on human perception, human workers are often employed to estimate a similarity score between items in order to support clustering, leading to a procedure called crowdsourced clustering. Assuming a monetary reward is paid to a worker for each similarity score and assuming the similarities between pairs and workers' reliability have a large diversity, when the budget is limited, it is critical to wisely assign pairs of items to different workers to optimize the clustering result. We model this budget allocation problem as a Markov decision process where item pairs are dynamically assigned to workers based on the historical similarity scores they provided. We propose an optimistic knowledge gradient policy where the assignment of items in each stage is based on the minimum-weight K-cut defined on a similarity graph. We provide simulation studies and real data analysis to demonstrate the performance of the proposed method.","doi":"10.24963/ijcai.2020/283","url":"https://ijcai.org/proceedings/2020/283","pdf":"https://ijcai.org/proceedings/2020/0283.pdf","markdown":"https://ijcai.org/proceedings/2020/283.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/283","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":283,"pages":{"first":2044,"last":2050},"keywords":["Machine Learning Applications: Applications of Unsupervised Learning","Machine Learning: Unsupervised Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24748/file_24748.m3u8","length_sec":281,"provider":"hls"}],"references":[{"position":1,"raw":"[Bagnall et al., 2019] Anthony Bagnall, Jason Lines, William Vickers, and Eamonn Keogh. The UEA & UCR time series classification repository, 2019.","title":"The UEA & UCR time series classification repository","authors":"Anthony Bagnall, Jason Lines, William Vickers, Eamonn Keogh","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bragg et al., 2013] Jonathan Bragg, Mausam, and Daniel S Weld. Crowdsourcing multi-label classification for taxon- omy creation. In First AAAI Conference on Human Com- putation and Crowdsourcing, pages 25-33, 2013.","title":"Crowdsourcing multi-label classification for taxonomy creation","authors":"Jonathan Bragg, Daniel S Mausam, Weld","year":"2013","venue":"First AAAI Conference on Human Computation and Crowdsourcing","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2013] Xi Chen, Paul N Bennett, Kevyn Collins- Thompson, and Eric Horvitz. Pairwise ranking aggrega- tion in a crowdsourced setting. In Proceedings of the sixth ACM International Conference on Web Search and Data Mining, pages 193-202. ACM, 2013.","title":"Pairwise ranking aggregation in a crowdsourced setting","authors":"Xi Chen, Paul N Bennett, Kevyn Collins-Thompson, Eric Horvitz","year":"2013","venue":"Proceedings of the sixth ACM International Conference on Web Search and Data Mining","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2015] Xi Chen, Qihang Lin, and Dengyong Zhou. Statistical decision making for optimal budget allo- cation in crowd labeling. The Journal of Machine Learn- ing Research, 16(1):1-46, 2015.","title":"Statistical decision making for optimal budget allocation in crowd labeling","authors":"Xi Chen, Qihang Lin, Dengyong Zhou","year":"2015","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2016] Xi Chen, Kevin Jiao, and Qihang Lin. Bayesian decision process for cost-efficient dynamic rank- ing via crowdsourcing. The Journal of Machine Learning Research, 17(1):7617-7656, 2016.","title":"Bayesian decision process for cost-efficient dynamic ranking via crowdsourcing","authors":"Xi Chen, Kevin Jiao, Qihang Lin","year":"2016","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dua and Graff, 2017] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.","title":"Dheeru Dua and Casey Graff. UCI machine learning repository","authors":"Graff Dua","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Frazier et al., 2008] Peter I Frazier, Warren B Powell, and Savas Dayanik. A knowledge-gradient policy for sequen- tial information collection. SIAM Journal on Control and Optimization, 47(5):2410-2439, 2008.","title":"A knowledge-gradient policy for sequential information collection","authors":"Warren B Peter I Frazier, Savas Powell, Dayanik","year":"2008","venue":"SIAM Journal on Control and Optimization","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Goldschmidt and Hochbaum, 1988] Olivier Goldschmidt and Dorit S Hochbaum. Polynomial algorithm for the k-cut problem. In Proceedings of 29th Annual Symposium on Foundations of Computer Science, pages 444-451. IEEE, 1988.","title":"Polynomial algorithm for the k-cut problem","authors":"Hochbaum Goldschmidt, Olivier Goldschmidt, Dorit S Hochbaum","year":"1988","venue":"Proceedings of 29th Annual Symposium on Foundations of Computer Science","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gupta and Miescke, 1996] Shanti S Gupta and Klaus J Mi- escke. Bayesian look ahead one-stage sampling alloca- tions for selection of the best population. Journal of Sta- tistical Planning and Inference, 54(2):229-244, 1996.","title":"Bayesian look ahead one-stage sampling allocations for selection of the best population","authors":"S Shanti, Klaus J Gupta, Miescke","year":"1996","venue":"Journal of Statistical Planning and Inference","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hespanha, 2004] Joao P Hespanha. An efficient matlab al- gorithm for graph partitioning. Santa Barbara, CA, USA: University of California, 2004.","title":"An efficient matlab algorithm for graph partitioning","authors":"Joao P Hespanha","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Karger et al., 2011] David R Karger, Sewoong Oh, and De- vavrat Shah. Budget-optimal crowdsourcing using low- rank matrix approximations. In 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 284-291. IEEE, 2011.","title":"Budget-optimal crowdsourcing using lowrank matrix approximations","authors":"Sewoong David R Karger, Devavrat Oh, Shah","year":"2011","venue":"49th Annual Allerton Conference on Communication, Control, and Computing (Allerton)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Luo et al., 2018] Yucen Luo, Tian Tian, Jiaxin Shi, Jun Zhu, and Bo Zhang. Semi-crowdsourced clustering with deep generative models. In Advances in Neural Information Processing Systems, pages 3212-3222, 2018.","title":"Semi-crowdsourced clustering with deep generative models","authors":"Yucen Luo, Tian Tian, Jiaxin Shi, Jun Zhu, Bo Zhang","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mazumdar and Saha, 2016] Arya Mazumdar and Barna Saha. Clustering via crowdsourcing. arXiv preprint arXiv:1604.01839, 2016.","title":"Clustering via crowdsourcing","authors":"Saha Mazumdar, Arya Mazumdar, Barna Saha","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Nocetti et al., 2003] Fabian Garcia Nocetti, Julio Solano Gonzalez, and Ivan Stojmenovic. Connectivity based k- hop clustering in wireless networks. Telecommunication Systems, 22(1-4):205-220, 2003.","title":"Connectivity based khop clustering in wireless networks","authors":"Fabian Garcia Nocetti, Julio Solano Gonzalez, Ivan Stojmenovic","year":"2003","venue":"Telecommunication Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Puterman, 2014] Martin L Puterman. Markov Decision Pro- cesses.: Discrete Stochastic Dynamic Programming. John Wiley & Sons, 2014.","title":"Markov Decision Processes","authors":"Martin L Puterman","year":"2014","venue":"Discrete Stochastic Dynamic Programming","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Tao et al., 2019] D. Tao, J. Cheng, Z. Yu, K. Yue, and L. Wang. Domain-weighted majority voting for crowd- sourcing. IEEE Transactions on Neural Networks and Learning Systems, 30(1):163-174, 2019.","title":"Domain-weighted majority voting for crowdsourcing","authors":"D Tao, J Cheng, Z Yu, K Yue, L Wang","year":"2019","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tran-Thanh et al., 2013] Long Tran-Thanh, Matteo Ve- nanzi, Alex Rogers, and Nicholas R Jennings. Ef- ficient budget allocation with accuracy guarantees for crowdsourcing classification tasks. In Proceedings of the 2013 International Conference on Autonomous Agents and Multi-Agent Systems, pages 901-908. International Foun- dation for Autonomous Agents and Multiagent Systems, 2013.","title":"Efficient budget allocation with accuracy guarantees for crowdsourcing classification tasks","authors":"Tran-Thanh","year":"2013","venue":"Proceedings of the 2013 International Conference on Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Vesdapunt et al., 2014] Norases Vesdapunt, Kedar Bellare, and Nilesh Dalvi. Crowdsourcing algorithms for en- tity resolution. Proceedings of the VLDB Endowment, 7(12):1071-1082, 2014.","title":"Crowdsourcing algorithms for entity resolution","authors":"Vesdapunt","year":"2014","venue":"Proceedings of the VLDB Endowment","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Whang et al., 2013] Steven Euijong Whang, Peter Lofgren, and Hector Garcia-Molina. Question selection for crowd entity resolution. Proceedings of the VLDB Endowment, 6(6):349-360, 2013.","title":"Question selection for crowd entity resolution","authors":"Steven Euijong Whang, Peter Lofgren, Hector Garcia-Molina","year":"2013","venue":"Proceedings of the VLDB Endowment","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xu et al., 1998] Xiaowei Xu, Martin Ester, H-P Kriegel, and Jörg Sander. A distribution-based clustering algo- rithm for mining in large spatial databases. In Proceed- ings 14th International Conference on Data Engineering, pages 324-331. IEEE, 1998.","title":"A distribution-based clustering algorithm for mining in large spatial databases","authors":"Xiaowei Xu, Martin Ester, H-P Kriegel, Jörg Sander","year":"1998","venue":"Proceedings 14th International Conference on Data Engineering","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhang and Wu, 2018] Jing Zhang and Xindong Wu. Multi- label inference for crowdsourcing. In Proceedings of the 24th ACM SIGKDD International Conference on Knowl- edge Discovery & Data Mining, pages 2738-2747. ACM, 2018.","title":"Multilabel inference for crowdsourcing","authors":"Wu ; ] Jing Zhang, Xindong Zhang, Wu","year":"2018","venue":"Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhong, 2005] Shi Zhong. Efficient online spherical k-means clustering. In Proceedings of International Joint Confer- ence on Neural Networks, volume 5, pages 3180-3185. IEEE, 2005.","title":"Efficient online spherical k-means clustering","authors":"Shi Zhong","year":"2005","venue":"Proceedings of International Joint Conference on Neural Networks","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}