{"type":"ScholarlyArticle","title":"Structured Probabilistic End-to-End Learning from Crowds","authors":[{"name":"Zhijun Chen","slug":"zhijun-chen","given":"Zhijun","family":"Chen","affiliation":"SKLSDE Lab , School of Computer Science and Engineering , Beihang University , China; Beijing Advanced Innovation Center for Big Data and Brain Computing , Beihang University , China"},{"name":"Huimin Wang","slug":"huimin-wang","given":"Huimin","family":"Wang","affiliation":"SKLSDE Lab , School of Computer Science and Engineering , Beihang University , China; Beijing Advanced Innovation Center for Big Data and Brain Computing , Beihang University , China"},{"name":"Hailong Sun","slug":"hailong-sun","given":"Hailong","family":"Sun","affiliation":"SKLSDE Lab , School of Computer Science and Engineering , Beihang University , China; Beijing Advanced Innovation Center for Big Data and Brain Computing , Beihang University , China"},{"name":"Pengpeng Chen","slug":"pengpeng-chen","given":"Pengpeng","family":"Chen","affiliation":"SKLSDE Lab , School of Computer Science and Engineering , Beihang University , China; Beijing Advanced Innovation Center for Big Data and Brain Computing , Beihang University , China"},{"name":"Tao Han","slug":"tao-han","given":"Tao","family":"Han","affiliation":"SKLSDE Lab , School of Computer Science and Engineering , Beihang University , China; Beijing Advanced Innovation Center for Big Data and Brain Computing , Beihang University , China"},{"name":"Xudong Liu","slug":"xudong-liu","given":"Xudong","family":"Liu","affiliation":"SKLSDE Lab , School of Computer Science and Engineering , Beihang University , China; Beijing Advanced Innovation Center for Big Data and Brain Computing , Beihang University , China"},{"name":"Jie Yang","slug":"jie-yang","given":"Jie","family":"Yang","affiliation":"Web Information Systems , Delft University of Technology , Netherlands"}],"abstract":"End-to-end learning from crowds has recently been introduced as an EM-free approach to training deep neural networks directly from noisy crowdsourced annotations. It models the relationship between true labels and annotations with a specific type of neural layer, termed as the crowd layer, which can be trained using pure backpropagation. Parameters of the crowd layer, however, can hardly be interpreted as annotator reliability, as compared with the more principled probabilistic approach. The lack of probabilistic interpretation further prevents extensions of the approach to account for important factors of annotation processes, e.g., instance difficulty. This paper presents SpeeLFC, a structured probabilistic model that incorporates the constraints of probability axioms for parameters of the crowd layer, which allows to explicitly model annotator reliability while benefiting from the end-to-end training of neural networks. Moreover, we propose SpeeLFC-D, which further takes into account instance difficulty. Extensive validation on real-world datasets shows that our methods improve the state-of-the-art.","doi":"10.24963/ijcai.2020/210","url":"https://ijcai.org/proceedings/2020/210","pdf":"https://ijcai.org/proceedings/2020/0210.pdf","markdown":"https://ijcai.org/proceedings/2020/210.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/210","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Humans and AI","track":"Main track","number":210,"pages":{"first":1512,"last":1518},"keywords":["Humans and AI: Human-AI Collaboration","Machine Learning: Probabilistic Machine Learning","Humans and AI: Human Computation and Crowdsourcing"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24749/file_24749.m3u8","length_sec":277,"provider":"hls"}],"references":[{"position":1,"raw":"[Albarqouni et al., 2016] Shadi Albarqouni, Christoph Baur, Felix Achilles, Vasileios Belagiannis, Stefanie Demirci, and Nassir Navab. Aggnet: deep learning from crowds for mitosis detection in breast cancer histology images. IEEE transactions on medical imaging, 35(5):1313-1321, 2016.","title":"Aggnet: deep learning from crowds for mitosis detection in breast cancer histology images","authors":"Shadi Albarqouni, Christoph Baur, Felix Achilles, Vasileios Belagiannis, Stefanie Demirci, Nassir Navab","year":"2016","venue":"IEEE transactions on medical imaging","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Atarashi et al., 2018] Kyohei Atarashi, Satoshi Oyama, and Masahito Kurihara. Semi-supervised learning from crowds using deep generative models. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.","title":"Semi-supervised learning from crowds using deep generative models","authors":"Kyohei Atarashi, Satoshi Oyama, Masahito Kurihara","year":"2018","venue":"Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bi et al., 2014] Wei Bi, Liwei Wang, James T Kwok, and Zhuowen Tu. Learning to predict from crowdsourced data. In UAI, pages 82-91, 2014.","title":"Learning to predict from crowdsourced data","authors":"Wei Bi, Liwei Wang, James T Kwok, Zhuowen Tu","year":"2014","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bishop, 2006] Christopher M Bishop. Pattern recognition and machine learning. Springer Science+ Business Media, 2006.","title":"Pattern recognition and machine learning","authors":"M Christopher, Bishop","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dawid and Skene, 1979] Alexander Philip Dawid and Al- lan M Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics), 28(1):20- 28, 1979.","title":"Maximum likelihood estimation of observer error-rates using the em algorithm","authors":"Skene Dawid, Alexander Philip, Dawid, Allan M Skene","year":"1979","venue":"Journal of the Royal Statistical Society: Series C (Applied Statistics)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Han et al., 2016] Tao Han, Hailong Sun, Yangqiu Song, Yili Fang, and Xudong Liu. Incorporating external knowledge into crowd intelligence for more specific knowledge acqui- sition. In IJCAI, volume 2016, pages 1541-1547, 2016.","title":"Incorporating external knowledge into crowd intelligence for more specific knowledge acquisition","authors":"Tao Han, Hailong Sun, Yangqiu Song, Yili Fang, Xudong Liu","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/221"},{"position":7,"raw":"[Kajino et al., 2012] Hiroshi Kajino, Yuta Tsuboi, and Hisashi Kashima. A convex formulation for learning from crowds. In Twenty-Sixth AAAI Conference on Artificial In- telligence, 2012.","title":"A convex formulation for learning from crowds","authors":"Kajino","year":"2012","venue":"Twenty-Sixth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kingma and Welling, 2013] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Raykar et al., 2010] Vikas C Raykar, Shipeng Yu, Linda H Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. Learning from crowds. Journal of Machine Learning Research, 11(Apr):1297-1322, 2010.","title":"Learning from crowds","authors":"Shipeng Vikas C Raykar, Linda H Yu, Gerardo Hermosillo Zhao, Charles Valadez, Luca Florin, Linda Bogoni, Moy","year":"2010","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Rodrigues and Pereira, 2018] Filipe Rodrigues and Fran- cisco C Pereira. Deep learning from crowds. In Thirty- Second AAAI Conference on Artificial Intelligence, 2018.","title":"Deep learning from crowds","authors":"Filipe Rodrigues, Francisco C Pereira","year":"2018","venue":"Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Rodrigues et al., 2013] Filipe Rodrigues, Francisco Pereira, and Bernardete Ribeiro. Learning from multiple annota- tors: distinguishing good from random labelers. Pattern Recognition Letters, 34(12):1428-1436, 2013.","title":"Learning from multiple annotators: distinguishing good from random labelers","authors":"Filipe Rodrigues, Francisco Pereira, Bernardete Ribeiro","year":"2013","venue":"Pattern Recognition Letters","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Rodrigues et al., 2017] Filipe Rodrigues, Mariana Lourenc ¸o, Bernardete Ribeiro, and Francisco C Pereira. Learning supervised topic models for classification and regression from crowds. IEEE transactions on pattern analysis and machine intelligence, 39(12):2409-2422, 2017.","title":"Learning supervised topic models for classification and regression from crowds","authors":"Filipe Rodrigues, Mariana Lourenc ¸o, Bernardete Ribeiro, Francisco C Pereira","year":"2017","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Welinder et al., 2010] Peter Welinder, Steve Branson, Pietro Perona, and Serge J Belongie. The multidi- mensional wisdom of crowds. In Advances in neural information processing systems, pages 2424-2432, 2010.","title":"The multidimensional wisdom of crowds","authors":"Welinder","year":"2010","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Whitehill et al., 2009] Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier R Movellan, and Paul L Ruvolo. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in neural information processing systems, pages 2035-2043, 2009.","title":"Whose vote should count more: Optimal integration of labels from labelers of unknown expertise","authors":"Jacob Whitehill, Ting-Fan, Jacob Wu, Javier R Bergsma, Paul L Movellan, Ruvolo","year":"2009","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Yan et al., 2012] Yan Yan, Rómer Rosales, Glenn Fung, and Jennifer Dy. Modeling multiple annotator expertise in the semi-supervised learning scenario. arXiv preprint arXiv:1203.3529, 2012.","title":"Modeling multiple annotator expertise in the semi-supervised learning scenario","authors":"Yan Yan, Rómer Rosales, Glenn Fung, Jennifer Dy","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Yang et al., 2019] Jie Yang, Alisa Smirnova, Dingqi Yang, Gianluca Demartini, Yuan Lu, and Philippe Cudré- Mauroux. Scalpel-cd: leveraging crowdsourcing and deep probabilistic modeling for debugging noisy training data. In The World Wide Web Conference, pages 2158-2168. ACM, 2019.","title":"Scalpel-cd: leveraging crowdsourcing and deep probabilistic modeling for debugging noisy training data","authors":"Jie Yang, Alisa Smirnova, Dingqi Yang, Gianluca Demartini, Yuan Lu, Philippe Cudré-Mauroux","year":"2019","venue":"The World Wide Web Conference","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Zhang et al., 2016] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understand- ing deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016.","title":"Understanding deep learning requires rethinking generalization","authors":"Zhang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Zheng et al., 2017] Yudian Zheng, Guoliang Li, Yuanbing Li, Caihua Shan, and Reynold Cheng. Truth inference in crowdsourcing: Is the problem solved? Proceedings of the VLDB Endowment, 10(5):541-552, 2017.","title":"Truth inference in crowdsourcing: Is the problem solved?","authors":"Zheng","year":"2017","venue":"Proceedings of the VLDB Endowment","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}