{"type":"ScholarlyArticle","title":"Aggregating Crowd Wisdom with Side Information via a Clustering-based Label-aware Autoencoder","authors":[{"name":"Li'ang Yin","slug":"li-ang-yin","given":"Li'ang","family":"Yin","affiliation":"Shanghai Jiao Tong University , No.800 Dongchuan Road , Shanghai , China"},{"name":"Yunfei Liu","slug":"yunfei-liu","given":"Yunfei","family":"Liu","affiliation":"Shanghai Jiao Tong University , No.800 Dongchuan Road , Shanghai , China"},{"name":"Weinan Zhang","slug":"weinan-zhang","given":"Weinan","family":"Zhang","affiliation":"Shanghai Jiao Tong University , No.800 Dongchuan Road , Shanghai , China"},{"name":"Yong Yu","slug":"yong-yu","given":"Yong","family":"Yu","affiliation":"Shanghai Jiao Tong University , No.800 Dongchuan Road , Shanghai , China"}],"abstract":"Aggregating crowd wisdom infers true labels for objects, from multiple noisy labels provided by various sources. Besides labels from sources, side information such as object features is also introduced to achieve higher inference accuracy. Usually, the learning-from-crowds framework is adopted. However, the framework considers each object in isolation and does not make full use of object features to overcome label noise. In this paper, we propose a clustering-based label-aware autoencoder (CLA) to alleviate label noise.  CLA utilizes clusters to gather objects with similar features and exploits clustering to infer true labels, by constructing a novel deep generative process to simultaneously generate object features and source labels from clusters. For model inference, CLA extends the framework of variational autoencoders and utilizes maximizing a posteriori (MAP) estimation, which prevents the model from overfitting and trivial solutions. Experiments on real-world tasks demonstrate the significant improvement of CLA compared with the state-of-the-art aggregation algorithms.","doi":"10.24963/ijcai.2020/214","url":"https://ijcai.org/proceedings/2020/214","pdf":"https://ijcai.org/proceedings/2020/0214.pdf","markdown":"https://ijcai.org/proceedings/2020/214.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/214","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Humans and AI","track":"Main track","number":214,"pages":{"first":1542,"last":1548},"keywords":["Machine Learning: Deep Generative Models","Humans and AI: Human Computation and Crowdsourcing","Machine Learning: Unsupervised Learning","Machine Learning: Clustering"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24301/file_24301.m3u8","length_sec":262,"provider":"hls"}],"references":[{"position":1,"raw":"[Albarqouni et al., 2016] Shadi Albarqouni, Christoph Baur, Felix Achilles, Vasileios Belagiannis, Stefanie Demirci, and Nassir Navab. Aggnet: deep learning from crowds for mitosis detection in breast cancer histology images. IEEE transactions on medical imaging, 35(5):1313-1321, 2016.","title":"Aggnet: deep learning from crowds for mitosis detection in breast cancer histology images","authors":"Shadi Albarqouni, Christoph Baur, Felix Achilles, Vasileios Belagiannis, Stefanie Demirci, Nassir Navab","year":"2016","venue":"IEEE transactions on medical imaging","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Aydin et al., 2014] Bahadir Ismail Aydin, Yavuz Selim Yil- maz, Yaliang Li, Qi Li, Jing Gao, and Murat Demirbas. Crowdsourcing for multiple-choice question answering. In AAAI, pages 2946-2953, 2014.","title":"Crowdsourcing for multiple-choice question answering","authors":"Bahadir Ismail Aydin, Yavuz Selim Yilmaz, Yaliang Li, Qi Li, Jing Gao, Murat Demirbas","year":"2014","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bachrach et al., 2012] Yoram Bachrach, Thore Graepel, Tom Minka, and John Guiver. How to grade a test with- out knowing the answers-a bayesian graphical model for adaptive crowdsourcing and aptitude testing. In ICML-12, pages 1183-1190, 2012.","title":"How to grade a test without knowing the answers-a bayesian graphical model for adaptive crowdsourcing and aptitude testing","authors":"Yoram Bachrach, Thore Graepel, Tom Minka, John Guiver","year":"2012","venue":"ICML-12","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Blei et al., 2003] David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation. the Journal of machine Learning research, 3:993-1022, 2003.","title":"Latent dirichlet allocation","authors":"Andrew Y David M Blei, Michael I Ng, Jordan","year":"2003","venue":"the Journal of machine Learning research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cao et al., 2019] Peng Cao, Yilun Xu, Yuqing Kong, and Yizhou Wang. Max-mig: an information theoretic ap- proach for joint learning from crowds. arXiv preprint arXiv:1905.13436, 2019.","title":"Max-mig: an information theoretic approach for joint learning from crowds","authors":"Peng Cao, Yilun Xu, Yuqing Kong, Yizhou Wang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dawid and Skene, 1979] Alexander Philip Dawid and Al- lan M Skene. Maximum likelihood estimation of ob- server error-rates using the em algorithm. Applied statis- tics, pages 20-28, 1979.","title":"Maximum likelihood estimation of observer error-rates using the em algorithm","authors":"Skene Dawid, Alexander Philip, Dawid, Allan M Skene","year":"1979","venue":"Applied statistics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dilokthanakul et al., 2016] Nat Dilokthanakul, Pedro A. M. Mediano, Marta Garnelo, Matthew C. H. Lee, Hugh Sal- imbeni, Kai Arulkumaran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture variational autoencoders. arXiv preprint, (arXiv:1611.02648), 2016.","title":"Deep unsupervised clustering with gaussian mixture variational autoencoders","authors":"Dilokthanakul","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dizaji and Huang, 2018] Kamran Ghasedi Dizaji and Heng Huang. Sentiment analysis via deep hybrid textual-crowd learning model. In AAAI, 2018.","title":"Sentiment analysis via deep hybrid textual-crowd learning model","authors":"Kamran Ghasedi, Dizaji, Heng Huang","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Felt et al., 2014] Paul Felt, Robbie Haertel, Eric K. Ring- ger, and Kevin D. Seppi. Momresp: A bayesian model for multi-annotator document labeling. LREC, 2014.","title":"Momresp: A bayesian model for multi-annotator document labeling","authors":"Paul Felt, Robbie Haertel, Eric K Ringger, Kevin D Seppi","year":"2014","venue":"LREC","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jiang et al., 2016] Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep embedding: An unsupervised and generative approach to clustering. arXiv preprint, (arXiv:1611.05148), 2016.","title":"Variational deep embedding: An unsupervised and generative approach to clustering","authors":"Jiang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2017/273"},{"position":11,"raw":"[Kim and Ghahramani, 2012] Hyun-Chul Kim and Zoubin Ghahramani. Bayesian classifier combination. In AIS- TATS, pages 619-627, 2012.","title":"Bayesian classifier combination","authors":"Hyun-Chul Kim, Zoubin Ghahramani","year":"2012","venue":"AIS-TATS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kingma and Welling, 2013] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kingma et al., 2014] Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi- supervised learning with deep generative models. In NIPS, pages 3581-3589, 2014.","title":"Semisupervised learning with deep generative models","authors":"Shakir Diederik P Kingma, Danilo Mohamed, Max Jimenez Rezende, Welling","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2016] Yaliang Li, Jing Gao, Chuishi Meng, Qi Li, Lu Su, Bo Zhao, Wei Fan, and Jiawei Han. A survey on truth discovery. ACM SIGKDD Explorations Newsletter, 17(2):1-16, 2016.","title":"A survey on truth discovery","authors":"Yaliang Li, Jing Gao, Chuishi Meng, Qi Li, Lu Su, Bo Zhao, Wei Fan, Jiawei Han","year":"2016","venue":"ACM SIGKDD Explorations Newsletter","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2012] Qiang Liu, Jian Peng, and Alexander T Ih- ler. Variational inference for crowdsourcing. In NIPS, pages 692-700, 2012.","title":"Variational inference for crowdsourcing","authors":"Qiang Liu, Jian Peng, Alexander T Ihler","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Maaten and Hinton, 2008] Laurens van der Maaten and Ge- offrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579-2605, 2008.","title":"Laurens van der Maaten and Geoffrey Hinton","authors":"Hinton Maaten","year":"2008","venue":"Journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Raykar et al., 2010] Vikas C Raykar, Shipeng Yu, Linda H Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. Learning from crowds. The Jour- nal of Machine Learning Research, 11:1297-1322, 2010.","title":"Learning from crowds","authors":"Shipeng Vikas C Raykar, Linda H Yu, Gerardo Hermosillo Zhao, Charles Valadez, Luca Florin, Linda Bogoni, Moy","year":"2010","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rodrigues and Pereira, 2018] Filipe Rodrigues and Fran- cisco C. Pereira. Deep learning from crowds. Thirty- Second AAAI Conference on Artificial Intelligence, 2018.","title":"Deep learning from crowds","authors":"Filipe Rodrigues, Francisco C Pereira","year":"2018","venue":"Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rodrigues et al., 2013] Filipe Rodrigues, Francisco Pereira, and Bernardete Ribeiro. Learning from multiple annota- tors: distinguishing good from random labelers. Pattern Recognition Letters, 34(12):1428-1436, 2013.","title":"Learning from multiple annotators: distinguishing good from random labelers","authors":"Filipe Rodrigues, Francisco Pereira, Bernardete Ribeiro","year":"2013","venue":"Pattern Recognition Letters","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rodrigues et al., 2017] Filipe Rodrigues, Mariana Lourenc ¸o, Bernardete Ribeiro, and Francisco C Pereira. Learning supervised topic models for classification and regression from crowds. IEEE PAMI, 39(12):2409-2422, 2017.","title":"Learning supervised topic models for classification and regression from crowds","authors":"Filipe Rodrigues, Mariana Lourenc ¸o, Bernardete Ribeiro, Francisco C Pereira","year":"2017","venue":"IEEE PAMI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Simpson et al., 2013] Edwin Simpson, Stephen Roberts, Ioannis Psorakis, and Arfon Smith. Dynamic bayesian combination of multiple imperfect classifiers. In Decision Making and Imperfection, pages 1-35. Springer, 2013.","title":"Dynamic bayesian combination of multiple imperfect classifiers","authors":"Edwin Simpson, Stephen Roberts, Ioannis Psorakis, Arfon Smith","year":"2013","venue":"Decision Making and Imperfection","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tian and Zhu, 2015] Tian Tian and Jun Zhu. Max-margin majority voting for learning from crowds. In NIPS, pages 1621-1629, 2015.","title":"Max-margin majority voting for learning from crowds","authors":"Zhu ; ] Tian Tian, Jun Tian, Zhu","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Venanzi et al., 2014] Matteo Venanzi, John Guiver, Gabriella Kazai, Pushmeet Kohli, and Milad Shokouhi. Community-based bayesian aggregation models for crowdsourcing. In Proceedings of the 23rd international conference on World wide web, pages 155-164, 2014.","title":"Community-based bayesian aggregation models for crowdsourcing","authors":"Matteo Venanzi, John Guiver, Gabriella Kazai, Pushmeet Kohli, Milad Shokouhi","year":"2014","venue":"Proceedings of the 23rd international conference on World wide web","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Welinder et al., 2010] Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-ucsd birds 200. (CNS-TR-2010-001), 2010.","title":"Caltech-ucsd birds 200","authors":"Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, Pietro Perona","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yan et al., 2014] Yan Yan, Rómer Rosales, Glenn Fung, Ramanathan Subramanian, and Jennifer Dy. Learning from multiple annotators with varying expertise. Machine learning, 95(3):291-327, 2014.","title":"Learning from multiple annotators with varying expertise","authors":"Yan Yan, Rómer Rosales, Glenn Fung, Ramanathan Subramanian, Jennifer Dy","year":"2014","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yin et al., 2008] Xiaoxin Yin, Jiawei Han, and Philip S Yu. Truth discovery with multiple conflicting information providers on the web. Knowledge and Data Engineering, IEEE Transactions on, 20(6):796-808, 2008.","title":"Truth discovery with multiple conflicting information providers on the web","authors":"Xiaoxin Yin, Jiawei Han, Philip S Yu","year":"2008","venue":"Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yin et al., 2017] Li'ang Yin, Jianhua Han, Weinan Zhang, and Yong Yu. Aggregating crowd wisdoms with label- aware autoencoders. In IJCAI, pages 1325-1331. AAAI Press, 2017.","title":"Aggregating crowd wisdoms with labelaware autoencoders","authors":"Jianhua Li'ang Yin, Weinan Han, Yong Zhang, Yu","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zheng et al., 2017] Yudian Zheng, Guoliang Li, Yuanbing Li, Caihua Shan, and Reynold Cheng. Truth inference in crowdsourcing: Is the problem solved? Proceedings of the VLDB Endowment, 10(5):541-552, 2017.","title":"Truth inference in crowdsourcing: Is the problem solved?","authors":"Zheng","year":"2017","venue":"Proceedings of the VLDB Endowment","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}