{"type":"ScholarlyArticle","title":"Improving Learning-from-Crowds through Expert Validation","authors":[{"name":"Mengchen Liu","slug":"mengchen-liu","given":"Mengchen","family":"Liu","affiliation":"School of Software , Tsinghua University , Beijing , P.R. China; Tsinghua National Lab for Information Science and Technology {liumc13 , jiangl16 , liujl12 ,"},{"name":"Liu Jiang","slug":"liu-jiang","given":"Liu","family":"Jiang","affiliation":"School of Software , Tsinghua University , Beijing , P.R. China; Tsinghua National Lab for Information Science and Technology {liumc13 , jiangl16 , liujl12 ,"},{"name":"Junlin Liu","slug":"junlin-liu","given":"Junlin","family":"Liu","affiliation":"School of Software , Tsinghua University , Beijing , P.R. China; Tsinghua National Lab for Information Science and Technology {liumc13 , jiangl16 , liujl12 ,"},{"name":"Xiting Wang","slug":"xiting-wang","given":"Xiting","family":"Wang","affiliation":"Microsoft Research , Beijing , P.R. China"},{"name":"Jun Zhu","slug":"jun-zhu","given":"Jun","family":"Zhu","affiliation":"Department of Computer Science and Technology , Tsinghua University , Beijing , P.R. China; Tsinghua National Lab for Information Science and Technology {liumc13 , jiangl16 , liujl12 ,"},{"name":"Shixia Liu","slug":"shixia-liu","given":"Shixia","family":"Liu","affiliation":"School of Software , Tsinghua University , Beijing , P.R. China; Tsinghua National Lab for Information Science and Technology {liumc13 , jiangl16 , liujl12 ,"}],"abstract":"Although several effective learning-from-crowd methods have been developed to infer correct labels from noisy crowdsourced labels, a method for post-processed expert validation is still needed. This paper introduces a semi-supervised learning algorithm that is capable of selecting the most informative instances and maximizing the influence of expert labels. Specifically, we have developed a complete uncertainty assessment to facilitate the selection of the most informative instances. The expert labels are then propagated to similar instances via regularized Bayesian inference. Experiments on both real-world and simulated datasets indicate that given a specific accuracy goal (e.g., 95%) our method reduces expert effort from 39% to 60% compared with the state-of-the-art method.","doi":"10.24963/ijcai.2017/324","url":"https://ijcai.org/proceedings/2017/324","pdf":"https://ijcai.org/proceedings/2017/0324.pdf","markdown":"https://ijcai.org/proceedings/2017/324.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/324","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":324,"pages":{"first":2329,"last":2336},"keywords":["Machine Learning: Semi-Supervised Learning","Machine Learning: Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Bachrach et al., 2012] Yoram Bachrach, Thore Graepel, Tom Minka, and John Guiver. How to grade a test with- out knowing the answers -a bayesian graphical model for adaptive crowdsourcing and aptitude testing. In ICML, pages 1183-1190, 2012.","title":"How to grade a test without knowing the answers -a bayesian graphical model for adaptive crowdsourcing and aptitude testing","authors":"Bachrach","year":"2012","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cohn et al., 1994] David Cohn, Les Atlas, and Richard Lad- ner. Improving generalization with active learning. Ma- chine learning, 15(2):201-221, 1994.","title":"Improving generalization with active learning","authors":"David Cohn, Les Atlas, Richard Ladner","year":"1994","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Crammer and Singer, 2001] Koby Crammer and Yoram Singer. On the algorithmic implementation of multiclass kernel-based vector machines. JMLR, 2(Dec):265-292, 2001.","title":"On the algorithmic implementation of multiclass kernel-based vector machines","authors":"Koby Crammer, Yoram Singer","year":"2001","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dawid and Skene, 1979] Alexander Philip Dawid and Al- lan M Skene. Maximum likelihood estimation of ob- server error-rates using the em algorithm. Applied statis- tics, 28(1):20-28, 1979.","title":"Maximum likelihood estimation of observer error-rates using the em algorithm","authors":"Alexander Philip, Dawid, Allan M Skene","year":"1979","venue":"Applied statistics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248-255, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Goldberg, 1989] David E. Goldberg. Genetic Algorithms in Search, Optimization and Machine Learning. Addison- Wesley Longman Publishing Co., Inc., 1989.","title":"Genetic Algorithms in Search, Optimization and Machine Learning","authors":"David E Goldberg","year":"1989","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Han et al., 2015] Hu Han, Charles Otto, Xiaoming Liu, and Anil K Jain. Demographic estimation from face im- ages: Human vs. machine performance. IEEE PAMI, 37(6):1148-1161, 2015.","title":"Demographic estimation from face images: Human vs. machine performance","authors":"Charles Hu Han, Xiaoming Otto, Anil K Liu, Jain","year":"2015","venue":"IEEE PAMI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hung et al., 2013] Nguyen Quoc Viet Hung, Nguyen Thanh Tam, Lam Ngoc Tran, and Karl Aberer. An evaluation of aggregation techniques in crowdsourcing. In ICWISE, pages 1-15, 2013.","title":"An evaluation of aggregation techniques in crowdsourcing","authors":"Nguyen Quoc, Viet Hung, Thanh Nguyen, Lam Tam, Karl Ngoc Tran, Aberer","year":"2013","venue":"ICWISE","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hung et al., 2015] Nguyen Quoc Viet Hung, Duong Chi Thang, Matthias Weidlich, and Karl Aberer. Minimizing efforts in validating crowd answers. In SIGMOD, pages 999-1014, 2015.","title":"Minimizing efforts in validating crowd answers","authors":"Nguyen Quoc, Viet Hung, Chi Duong, Matthias Thang, Karl Weidlich, Aberer","year":"2015","venue":"SIGMOD","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Karger et al., 2011] David R. Karger, Sewoong Oh, and De- vavrat Shah. Iterative learning for reliable crowdsourcing systems. In NIPS, pages 1953-1961, 2011.","title":"Iterative learning for reliable crowdsourcing systems","authors":"David R Karger, Sewoong Oh, Devavrat Shah","year":"2011","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lang, 1995] Ken Lang. Newsweeder: Learning to filter net- news. In ICML, pages 331-339, 1995.","title":"Newsweeder: Learning to filter netnews","authors":"Lang ;, Ken Lang","year":"1995","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2012] Qiang Liu, Jian Peng, and Alexander T Ih- ler. Variational inference for crowdsourcing. In NIPS, pages 692-700. 2012.","title":"Variational inference for crowdsourcing","authors":"Qiang Liu, Jian Peng, Alexander T Ihler","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2016] Mengchen Liu, Shixia Liu, Xizhou Zhu, Qinying Liao, Furu Wei, and Shimei Pan. An uncertainty- aware approach for exploratory microblog retrieval. IEEE TVCG, 22(1):250-259, 2016.","title":"An uncertaintyaware approach for exploratory microblog retrieval","authors":"Liu","year":"2016","venue":"IEEE TVCG","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Settles, 2010] Burr Settles. Active learning literature sur- vey. 52(55-66):11, 2010.","title":"Burr Settles","authors":"Settles","year":"2010","venue":"Active learning literature survey","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Shannon, 2001] C. E. Shannon. A mathematical theory of communication. SIGMOBILE MC2R, 5(1):3-55, 2001.","title":"A mathematical theory of communication","authors":"C E Shannon","year":"2001","venue":"SIGMOBILE MC2R","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Simonyan and Zisserman, 2014] K. Simonyan and A. Zis- serman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"K Simonyan, A Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Snow et al., 2008] Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. Cheap and fast-but is it good?: Evaluating non-expert annotations for natural lan- guage tasks. In EMNLP, pages 254-263, 2008.","title":"Cheap and fast-but is it good?: Evaluating non-expert annotations for natural language tasks","authors":"Rion Snow, O' Brendan, Daniel Connor, Andrew Y Jurafsky, Ng","year":"2008","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sun and Zhou, 2012] Tengyu Sun and Jie Zhou. Batch- mode active learning by using misclassified data. In ALRA Workshop at ECML-PKDD, 2012.","title":"Tengyu Sun and Jie Zhou. Batchmode active learning by using misclassified data","authors":"Zhou Sun","year":"2012","venue":"ALRA Workshop at ECML-PKDD","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tian and Zhu, 2015] Tian Tian and Jun Zhu. Max-margin majority voting for learning from crowds. In NIPS, pages 1621-1629, 2015.","title":"Max-margin majority voting for learning from crowds","authors":"Zhu ; ] Tian Tian, Jun Tian, Zhu","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang and Zhai, 2016] Xizhao Wang and Junhai Zhai. Learning with Uncertainty. CRC Press, 2016.","title":"Xizhao Wang and Junhai Zhai","authors":"Zhai Wang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang and Zhou, 2016] Lu Wang and Zhi-Hua Zhou. Cost- saving effect of crowdsourcing learning. In IJCAI, pages 2111-2117, 2016.","title":"Costsaving effect of crowdsourcing learning","authors":"Lu Wang, Zhi-Hua Zhou","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wikipedia, 2017] Exponential growth model. https: //en.wikipedia.org/wiki/Exponential_ growth, February 2017.","title":"Exponential growth model","authors":null,"year":"2017","venue":null,"doi":null,"url":"https://en.wikipedia.org/wiki/Exponential_growth","internal_url":null},{"position":23,"raw":"[Yan et al., 2015] Rui Yan, Yiping Song, Cheng-Te Li, Ming Zhang, and Xiaohua Hu. Opportunities or risks to reduce labor in crowdsourcing translation? characterizing cost versus quality via a pagerank-hits hybrid model. In IJCAI, pages 1025-1032, 2015.","title":"Opportunities or risks to reduce labor in crowdsourcing translation? characterizing cost versus quality via a pagerank-hits hybrid model","authors":"Rui Yan, Yiping Song, Cheng-Te Li, Ming Zhang, Xiaohua Hu","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/149"},{"position":24,"raw":"[Zhang et al., 2016] Jing Zhang, Xindong Wu, and Victor S Sheng. Learning from crowdsourced labeled data: a sur- vey. Artificial Intelligence Review, 46(4):1-34, 2016.","title":"Learning from crowdsourced labeled data: a survey","authors":"Jing Zhang, Xindong Wu, Victor S Sheng","year":"2016","venue":"Artificial Intelligence Review","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhou and He, 2016] Yao Zhou and Jingrui He. Crowdsourc- ing via tensor augmentation and completion. In IJCAI, pages 2435-2441, 2016.","title":"Crowdsourcing via tensor augmentation and completion","authors":"Yao Zhou, Jingrui He","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/347"},{"position":26,"raw":"[Zhou et al., 2012] Denny Zhou, Sumit Basu, Yi Mao, and John C Platt. Learning from the wisdom of crowds by minimax entropy. In NIPS, pages 2195-2203, 2012.","title":"Learning from the wisdom of crowds by minimax entropy","authors":"Denny Zhou, Sumit Basu, Yi Mao, John C Platt","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhou et al., 2014] Dengyong Zhou, Qiang Liu, John C Platt, and Christopher Meek. Aggregating ordinal labels from crowds by minimax conditional entropy. In ICML, pages 262-270, 2014.","title":"Aggregating ordinal labels from crowds by minimax conditional entropy","authors":"Dengyong Zhou, Qiang Liu, John C Platt, Christopher Meek","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhu et al., 2014] Jun Zhu, Ning Chen, and Eric P. Xing. Bayesian inference with posterior regularization and ap- plications to infinite latent svms. JMLR, 15(1):1799-1847, 2014.","title":"Bayesian inference with posterior regularization and applications to infinite latent svms","authors":"Jun Zhu, Ning Chen, Eric P Xing","year":"2014","venue":"JMLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}