{"type":"ScholarlyArticle","title":"Multiple Noisy Label Distribution Propagation for Crowdsourcing","authors":[{"name":"Hao Zhang","slug":"hao-zhang","given":"Hao","family":"Zhang","affiliation":"Department of Computer Science , China University of Geosciences , Wuhan 430074 , China"},{"name":"Liangxiao Jiang","slug":"liangxiao-jiang","given":"Liangxiao","family":"Jiang","affiliation":"Department of Computer Science , China University of Geosciences , Wuhan 430074 , China"},{"name":"Wenqiang Xu","slug":"wenqiang-xu","given":"Wenqiang","family":"Xu","affiliation":"Department of Computer Science , China University of Geosciences , Wuhan 430074 , China"}],"abstract":"Crowdsourcing services provide a fast, efficient, and cost-effective means of obtaining large labeled data for supervised learning. Ground truth inference, also called label integration, designs proper aggregation strategies to infer the unknown true label of each instance from the multiple noisy label set provided by ordinary crowd workers. However, to the best of our knowledge, nearly all existing label integration methods focus solely on the multiple noisy label set itself of the individual instance while totally ignoring the intercorrelation among multiple noisy label sets of different instances. To solve this problem, a multiple noisy label distribution propagation (MNLDP) method is proposed in this study. MNLDP first transforms the multiple noisy label set of each instance into its multiple noisy label distribution and then propagates its multiple noisy label distribution to its nearest neighbors. Consequently, each instance absorbs a fraction of the multiple noisy label distributions from its nearest neighbors and yet simultaneously maintains a fraction of its own original multiple noisy label distribution. Promising experimental results on simulated and real-world datasets validate the effectiveness of our proposed method.","doi":"10.24963/ijcai.2019/204","url":"https://ijcai.org/proceedings/2019/204","pdf":"https://ijcai.org/proceedings/2019/0204.pdf","markdown":"https://ijcai.org/proceedings/2019/204.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/204","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Humans and AI","track":"Main track","number":204,"pages":{"first":1473,"last":1479},"keywords":["Humans and AI: Human Computation and Crowdsourcing","Machine Learning: Multi-instance;Multi-label;Multi-view learning"],"videos":[],"references":[{"position":1,"raw":"[Aucouturier and Pachet, 2003] Jean-Julien Aucouturier and Francois Pachet. Representing musical genre: A state of the art. Journal of New Music Research, 32(1):83-93, 2003.","title":"Representing musical genre: A state of the art","authors":"Pachet Aucouturier","year":"2003","venue":"Journal of New Music Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dawid and Skene, 1979] Alexander Philip Dawid and Al- lan M. Skene. Maximum likelihood estimation of observ- er error-rates using the EM algorithm. Applied statistics, 28(1):20-28, 1979.","title":"Maximum likelihood estimation of observer error-rates using the EM algorithm","authors":"Alexander Philip Dawid, Allan M Skene","year":"1979","venue":"Applied statistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Demartini et al., 2012] Gianluca Demartini, Djellel Eddine Difallah, and Philippe Cudré-Mauroux. Zencrowd: lever- aging probabilistic reasoning and crowdsourcing tech- niques for large-scale entity linking. In Proceedings of the 21st international conference on World Wide Web, pages 469-478, 2012.","title":"Zencrowd: leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking","authors":"Gianluca Demartini, Djellel Eddine Difallah, Philippe Cudré-Mauroux","year":"2012","venue":"Proceedings of the 21st international conference on World Wide Web","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Geng, 2016] Xin Geng. Label distribution learning. IEEE Transactions on Knowledge and Data Engineering, 28(7):1734-1748, 2016.","title":"Label distribution learning","authors":"Xin Geng, Geng","year":"2016","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Guan et al., 2017] Melody Y. Guan, Varun Gulshan, An- drew M. Dai, and Geoffrey E. Hinton. Who said what: Modeling individual labelers improves classification. arX- iv preprint arXiv:1703.08774, 2017.","title":"Who said what: Modeling individual labelers improves classification","authors":"Melody Y Guan, Varun Gulshan, Andrew M Dai, Geoffrey E Hinton","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hou et al., 2016] Peng Hou, Xin Geng, and Min-Ling Zhang. Multi-label manifold learning. In Association for the Advancement of Artificial Intelligence (AAAI), pages 1680-1686, 2016.","title":"Multi-label manifold learning","authors":"Peng Hou, Xin Geng, Min-Ling Zhang","year":"2016","venue":"Association for the Advancement of Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Karger et al., 2014] David R. Karger, Sewoong Oh, and De- vavrat Shah. Budget-optimal task allocation for reliable crowdsourcing systems. Operations Research, 62(1):1- 24, 2014.","title":"Budget-optimal task allocation for reliable crowdsourcing systems","authors":"David R Karger, Sewoong Oh, Devavrat Shah","year":"2014","venue":"Operations Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Li et al., 2016] Chaoqun Li, Victor S. Sheng, Liangxiao Jiang, and Hongwei Li. Noise filtering to improve data and model quality for crowdsourcing. Knowledge-Based Systems, 107:96-103, 2016.","title":"Noise filtering to improve data and model quality for crowdsourcing","authors":"Li","year":"2016","venue":"Knowledge-Based Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2019] Chaoqun Li, Liangxiao Jiang, and Wen- qiang Xu. Noise correction to improve data and model quality for crowdsourcing. Engineering Applications of Artificial Intelligence, 82:184-191, 2019.","title":"Noise correction to improve data and model quality for crowdsourcing","authors":"Li","year":"2019","venue":"Engineering Applications of Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liu et al., 2012] Qiang Liu, Jian Peng, and Alexander Ih- ler. Variational inference for crowdsourcing. In Advances in Neural Information Processing Systems 25, pages 692- 700, 2012.","title":"Variational inference for crowdsourcing","authors":"Qiang Liu, Jian Peng, Alexander Ihler","year":"2012","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Oliva and Torralba, 2001] Aude Oliva and Antonio Torral- ba. Modeling the shape of the scene: A holistic repre- sentation of the spatial envelope. International journal of computer vision, 42(3):145-175, 2001.","title":"Aude Oliva and Antonio Torralba. Modeling the shape of the scene: A holistic representation of the spatial envelope","authors":"Torralba Oliva","year":"2001","venue":"International journal of computer vision","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Qiu et al., 2018] Chen Qiu, Liangxiao Jiang, and Zhihua Cai. Using differential evolution to estimate labeler qual- ity for crowdsourcing. In Pacific Rim International Con- ference on Artificial Intelligence, pages 165-173, 2018.","title":"Using differential evolution to estimate labeler quality for crowdsourcing","authors":"Chen Qiu, Liangxiao Jiang, Zhihua Cai","year":"2018","venue":"Pacific Rim International Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Raykar et al., 2010] Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. Learning from crowds. Journal of Machine Learning Research, 11(Apr):1297-1322, 2010.","title":"Learning from crowds","authors":"C Vikas, Shipeng Raykar, Linda H Yu, Gerardo Hermosillo Zhao, Charles Valadez, Luca Florin, Linda Bogoni, Moy","year":"2010","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Rodrigues and Pereira, 2017] Filipe Rodrigues and Francis- co C. Pereira. Deep learning from crowds. arXiv preprint arXiv:1709.01779, 2017.","title":"Filipe Rodrigues and Francisco","authors":"Pereira Rodrigues, C Pereira","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Rodrigues et al., 2013] Filipe Rodrigues, Francisco Pereira, and Bernardete Ribeiro. Learning from multiple annota- tors: distinguishing good from random labelers. Pattern Recognition Letters, 34(12):1428-1436, 2013.","title":"Learning from multiple annotators: distinguishing good from random labelers","authors":"Filipe Rodrigues, Francisco Pereira, Bernardete Ribeiro","year":"2013","venue":"Pattern Recognition Letters","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Roweis and Saul, 2000] Sam T. Roweis and Lawrence K. Saul. Nonlinear dimensionality reduction by locally lin- ear embedding. Science, 290(5500):2323-2326, 2000.","title":"Nonlinear dimensionality reduction by locally linear embedding","authors":"Sam T Roweis, Lawrence K Saul","year":"2000","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Russell et al., 2008] Bryan C. Russell, Antonio Torralba, Kevin P. Murphy, and William T. Freeman. Labelme: a database and web-based tool for image annotation. In- ternational journal of computer vision, 77(1-3):157-173, 2008.","title":"Labelme: a database and web-based tool for image annotation","authors":"Bryan C Russell, Antonio Torralba, Kevin P Murphy, William T Freeman","year":"2008","venue":"International journal of computer vision","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sheng et al., 2008] Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. Get another label? improving data quality and data mining using multiple noisy labelers. In Proceedings of the 14th ACM SIGKDD international con- ference on Knowledge discovery and data mining, pages 614-622, 2008.","title":"Get another label? improving data quality and data mining using multiple noisy labelers","authors":"S Victor, Foster Sheng, Panagiotis G Provost, Ipeirotis","year":"2008","venue":"Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sheshadri and Lease, 2013] Aashish Sheshadri and Matthew Lease. Square: A benchmark for research on computing crowd consensus. In First AAAI Conference on Human Computation and Crowdsourcing, pages 156-164, 2013.","title":"Square: A benchmark for research on computing crowd consensus","authors":"Aashish Sheshadri, Matthew Lease","year":"2013","venue":"First AAAI Conference on Human Computation and Crowdsourcing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tian and Zhu, 2015] Tian Tian and Jun Zhu. Max-margin majority voting for learning from crowds. In Advances in Neural Information Processing Systems, pages 1612- 1620, 2015.","title":"Max-margin majority voting for learning from crowds","authors":"Zhu ; ] Tian Tian, Jun Tian, Zhu","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang and Zhang, 2008] Fei Wang and Changshui Zhang. Label propagation through linear neighborhoods. IEEE Transactions on Knowledge and Data Engineering, 20(1):55-67, 2008.","title":"Label propagation through linear neighborhoods","authors":"Wang, Zhang ; ] Fei Wang, Changshui Zhang","year":"2008","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Whitehill et al., 2009] Jacob Whitehill, Ting fan Wu, Jacob Bergsma, Javier R. Movellan, and Paul L. Ruvolo. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in neural information processing systems, pages 2035-2043, 2009.","title":"Whose vote should count more: Optimal integration of labels from labelers of unknown expertise","authors":"Jacob Whitehill, Jacob Ting Fan Wu, Javier R Bergsma, Paul L Movellan, Ruvolo","year":"2009","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang and Wu, 2018] Jing Zhang and Xindong Wu. Multi- label inference for crowdsourcing. In Proceedings of the 24th ACM SIGKDD International Conference on Knowl- edge Discovery & Data Mining, pages 2738-2747, 2018.","title":"Multilabel inference for crowdsourcing","authors":"Wu ; ] Jing Zhang, Xindong Zhang, Wu","year":"2018","venue":"Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2016a] Jing Zhang, Victor S. Sheng, Jian Wu, and Xindong Wu. Multi-class ground truth inference in crowdsourcing with clustering. IEEE Transactions on Knowledge and Data Engineering, 28(4):1080-1085, 2016.","title":"Multi-class ground truth inference in crowdsourcing with clustering","authors":"Zhang","year":"2016","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2016b] Jing Zhang, Xindong Wu, and Vic- tor S. Sheng. Learning from crowdsourced labeled data: a survey. Artificial Intelligence Review, 46(4):1-34, 2016.","title":"Learning from crowdsourced labeled data: a survey","authors":"Zhang","year":"2016","venue":"Artificial Intelligence Review","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhu et al., 2005] Xiaojin Zhu, John Lafferty, and Ronald Rosenfeld. Semi-supervised learning with graphs. PhD thesis, Carnegie Mellon University, language technologies institute, school of computer science, 2005.","title":"Semi-supervised learning with graphs","authors":"Xiaojin Zhu, John Lafferty, Ronald Rosenfeld","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}