{"type":"ScholarlyArticle","title":"Aggregating Crowd Wisdoms with Label-aware Autoencoders","authors":[{"name":"Li'ang Yin","slug":"li-ang-yin","given":"Li'ang","family":"Yin","affiliation":"Shanghai Jiao Tong University No. 800 Dongchuan Road Shanghai , 200240 , China"},{"name":"Jianhua Han","slug":"jianhua-han","given":"Jianhua","family":"Han","affiliation":"Shanghai Jiao Tong University No. 800 Dongchuan Road Shanghai , 200240 , China"},{"name":"Weinan Zhang","slug":"weinan-zhang","given":"Weinan","family":"Zhang","affiliation":"Shanghai Jiao Tong University No. 800 Dongchuan Road Shanghai , 200240 , China"},{"name":"Yong Yu","slug":"yong-yu","given":"Yong","family":"Yu","affiliation":"Shanghai Jiao Tong University No. 800 Dongchuan Road Shanghai , 200240 , China"}],"abstract":"Aggregating crowd wisdoms takes multiple labels from various sources and infers true labels for objects. Recent research work makes progress by learning source credibility from data and roughly form three kinds of modeling frameworks: weighted majority voting, trust propagation, and generative models. In this paper, we propose a novel framework named Label-Aware Autoencoders (LAA) to aggregate crowd wisdoms. LAA integrates a classifier and a reconstructor into a unified model to infer labels in an unsupervised manner. Analogizing classical autoencoders, we can regard the classifier as an encoder, the reconstructor as a decoder, and inferred labels as latent features. To the best of our knowledge, it is the first trial to combine label aggregation with autoencoders. We adopt networks to implement the classifier and the reconstructor which have the potential to automatically learn underlying patterns of source credibility. To further improve inference accuracy, we introduce object ambiguity and latent aspects into LAA. Experiments on three real-world datasets show that proposed models achieve impressive inference accuracy improvement over state-of-the-art models.","doi":"10.24963/ijcai.2017/184","url":"https://ijcai.org/proceedings/2017/184","pdf":"https://ijcai.org/proceedings/2017/0184.pdf","markdown":"https://ijcai.org/proceedings/2017/184.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/184","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Knowledge Representation, Reasoning, and Logic","track":"Main track","number":184,"pages":{"first":1325,"last":1331},"keywords":["Knowledge Representation, Reasoning, and Logic: Judgement Aggregation and Information Fusion","Machine Learning: Unsupervised Learning","Machine Learning: Neural Networks"],"videos":[],"references":[{"position":1,"raw":"[Aydin et al., 2014] Bahadir Ismail Aydin, Yavuz Selim Yil- maz, Yaliang Li, Qi Li, Jing Gao, and Murat Demirbas. Crowdsourcing for multiple-choice question answering. In AAAI, pages 2946-2953, 2014.","title":"Crowdsourcing for multiple-choice question answering","authors":"Bahadir Ismail Aydin, Yavuz Selim Yilmaz, Yaliang Li, Qi Li, Jing Gao, Murat Demirbas","year":"2014","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bachrach et al., 2012] Yoram Bachrach, Thore Graepel, Tom Minka, and John Guiver. How to grade a test with- out knowing the answers-a bayesian graphical model for adaptive crowdsourcing and aptitude testing. In Pro- ceedings of the 29th International Conference on Machine Learning (ICML-12), pages 1183-1190, 2012.","title":"How to grade a test without knowing the answers-a bayesian graphical model for adaptive crowdsourcing and aptitude testing","authors":"Yoram Bachrach, Thore Graepel, Tom Minka, John Guiver","year":"2012","venue":"Proceedings of the 29th International Conference on Machine Learning (ICML-12)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dawid and Skene, 1979] Alexander Philip Dawid and Al- lan M Skene. Maximum likelihood estimation of ob- server error-rates using the em algorithm. Applied statis- tics, pages 20-28, 1979.","title":"Maximum likelihood estimation of observer error-rates using the em algorithm","authors":"Alexander Philip, Dawid, Allan M Skene","year":"1979","venue":"Applied statistics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[De Winter et al., 2015] JCF De Winter, Miltos Kyriakidis, Dimitra Dodou, and Riender Happee. Using crowdflower to study the relationship between self-reported violations and traffic accidents. Procedia Manufacturing, 3:2518- 2525, 2015.","title":"Using crowdflower to study the relationship between self-reported violations and traffic accidents","authors":"De Winter","year":"2015","venue":"Procedia Manufacturing","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Galland et al., 2010] Alban Galland, Serge Abiteboul, Amélie Marian, and Pierre Senellart. Corroborating information from disagreeing views. In Proceedings of the third ACM international conference on Web search and data mining, pages 131-140. ACM, 2010.","title":"Corroborating information from disagreeing views","authors":"Alban Galland, Serge Abiteboul, Amélie Marian, Pierre Senellart","year":"2010","venue":"Proceedings of the third ACM international conference on Web search and data mining","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gomes et al., 2011] Ryan G Gomes, Peter Welinder, An- dreas Krause, and Pietro Perona. Crowdclustering. In Ad- vances in Neural Information Processing Systems, pages 558-566, 2011.","title":"Crowdclustering","authors":"Peter Ryan G Gomes, Andreas Welinder, Pietro Krause, Perona","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ipeirotis, 2010] Panagiotis G Ipeirotis. Analyzing the ama- zon mechanical turk marketplace. XRDS: Crossroads, The ACM Magazine for Students, 17(2):16-21, 2010.","title":"Analyzing the amazon mechanical turk marketplace","authors":"G Panagiotis, Ipeirotis","year":"2010","venue":"The ACM Magazine for Students","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kim and Ghahramani, 2012] Hyun-Chul Kim and Zoubin Ghahramani. Bayesian classifier combination. In AIS- TATS, pages 619-627, 2012.","title":"Bayesian classifier combination","authors":"Hyun-Chul Kim, Zoubin Ghahramani","year":"2012","venue":"AIS-TATS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kingma and Welling, 2013] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2014] Qi Li, Yaliang Li, Jing Gao, Lu Su, Bo Zhao, Murat Demirbas, Wei Fan, and Jiawei Han. A confidence- aware approach for truth discovery on long-tail data. Pro- ceedings of the VLDB Endowment, 8(4):425-436, 2014.","title":"A confidenceaware approach for truth discovery on long-tail data","authors":"Qi Li, Yaliang Li, Jing Gao, Lu Su, Bo Zhao, Murat Demirbas, Wei Fan, Jiawei Han","year":"2014","venue":"Proceedings of the VLDB Endowment","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2016] Yaliang Li, Jing Gao, Chuishi Meng, Qi Li, Lu Su, Bo Zhao, Wei Fan, and Jiawei Han. A survey on truth discovery. ACM SIGKDD Explorations Newsletter, 17(2):1-16, 2016.","title":"A survey on truth discovery","authors":"Yaliang Li, Jing Gao, Chuishi Meng, Qi Li, Lu Su, Bo Zhao, Wei Fan, Jiawei Han","year":"2016","venue":"ACM SIGKDD Explorations Newsletter","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Metrikov et al., 2015] Pavel Metrikov, Virgil Pavlu, and Javed A Aslam. Aggregation of crowdsourced ordinal as- sessments and integration with learning to rank: A latent trait model. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Manage- ment, pages 1391-1400. ACM, 2015.","title":"Aggregation of crowdsourced ordinal assessments and integration with learning to rank: A latent trait model","authors":"Metrikov","year":"2015","venue":"Proceedings of the 24th ACM International on Conference on Information and Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Pasternack and Roth, 2010] Jeff Pasternack and Dan Roth. Knowing what to believe (when you already know some- thing). In Proceedings of the 23rd International Confer- ence on Computational Linguistics, pages 877-885. Asso- ciation for Computational Linguistics, 2010.","title":"Knowing what to believe (when you already know some-thing)","authors":"Roth ; Pasternack, Jeff Pasternack, Dan Roth","year":"2010","venue":"Proceedings of the 23rd International Conference on Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Qi et al., 2013] Guo-Jun Qi, Charu C Aggarwal, Jiawei Han, and Thomas Huang. Mining collective intelligence in diverse groups. In Proceedings of the 22nd international conference on World Wide Web, pages 1041-1052. Inter- national World Wide Web Conferences Steering Commit- tee, 2013.","title":"Mining collective intelligence in diverse groups","authors":"Guo-Jun Qi, C Charu, Jiawei Aggarwal, Thomas Han, Huang","year":"2013","venue":"Proceedings of the 22nd international conference on World Wide Web","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Simpson et al., 2013] Edwin Simpson, Stephen Roberts, Ioannis Psorakis, and Arfon Smith. Dynamic bayesian combination of multiple imperfect classifiers. In Decision Making and Imperfection, pages 1-35. Springer, 2013.","title":"Dynamic bayesian combination of multiple imperfect classifiers","authors":"Edwin Simpson, Stephen Roberts, Ioannis Psorakis, Arfon Smith","year":"2013","venue":"Decision Making and Imperfection","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Tian and Zhu, 2015a] Tian Tian and Jun Zhu. Max-margin majority voting for learning from crowds. In Advances in Neural Information Processing Systems, pages 1621- 1629, 2015.","title":"Max-margin majority voting for learning from crowds","authors":"Zhu Tian, Tian Tian, Jun Zhu","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tian and Zhu, 2015b] Tian Tian and Jun Zhu. Uncovering the latent structures of crowd labeling. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 392-404. Springer, 2015.","title":"Uncovering the latent structures of crowd labeling","authors":"Tian, Jun Zhu ; B] Tian Tian, Zhu","year":"2015","venue":"Pacific-Asia Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Vincent et al., 2008] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extract- ing and composing robust features with denoising autoen- coders. In Proceedings of the 25th international confer- ence on Machine learning, pages 1096-1103. ACM, 2008.","title":"Extracting and composing robust features with denoising autoencoders","authors":"Pascal Vincent, Hugo Larochelle, Yoshua Bengio, Pierre-Antoine Manzagol","year":"2008","venue":"Proceedings of the 25th international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Vincent et al., 2010] Pascal Vincent, Hugo Larochelle, Is- abelle Lajoie, Yoshua Bengio, and Pierre-Antoine Man- zagol. Stacked denoising autoencoders: Learning use- ful representations in a deep network with a local de- noising criterion. Journal of Machine Learning Research, 11(Dec):3371-3408, 2010.","title":"Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion","authors":"Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol","year":"2010","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Welinder et al., 2010] Peter Welinder, Steve Branson, Pietro Perona, and Serge J Belongie. The multidi- mensional wisdom of crowds. In Advances in neural information processing systems, pages 2424-2432, 2010.","title":"The multidimensional wisdom of crowds","authors":"Peter Welinder, Steve Branson, Pietro Perona, Serge J Belongie","year":"2010","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Whitehill et al., 2009] Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier R Movellan, and Paul L Ruvolo. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in neural information processing systems, pages 2035-2043, 2009.","title":"Whose vote should count more: Optimal integration of labels from labelers of unknown expertise","authors":"Jacob Whitehill, Ting-Fan, Jacob Wu, Javier R Bergsma, Paul L Movellan, Ruvolo","year":"2009","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yin et al., 2008] Xiaoxin Yin, Jiawei Han, and Philip S Yu. Truth discovery with multiple conflicting information providers on the web. Knowledge and Data Engineering, IEEE Transactions on, 20(6):796-808, 2008.","title":"Truth discovery with multiple conflicting information providers on the web","authors":"Xiaoxin Yin, Jiawei Han, Philip S Yu","year":"2008","venue":"Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhi et al., 2015] Shi Zhi, Bo Zhao, Wenzhu Tong, Jing Gao, Dian Yu, Heng Ji, and Jiawei Han. Modeling truth exis- tence in truth discovery. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discov- ery and Data Mining, pages 1543-1552. ACM, 2015.","title":"Modeling truth existence in truth discovery","authors":"Bo Shi Zhi, Wenzhu Zhao, Jing Tong, Dian Gao, Yu, Ji Heng, Jiawei Han","year":"2015","venue":"Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhou et al., 2012] Dengyong Zhou, Sumit Basu, Yi Mao, and John C Platt. Learning from the wisdom of crowds by minimax entropy. In Advances in Neural Information Processing Systems, pages 2195-2203, 2012.","title":"Learning from the wisdom of crowds by minimax entropy","authors":"Dengyong Zhou, Sumit Basu, Yi Mao, John C Platt","year":"2012","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}