{"type":"ScholarlyArticle","title":"Graph Mining Meets Crowdsourcing: Extracting Experts for Answer Aggregation","authors":[{"name":"Yasushi Kawase","slug":"yasushi-kawase","given":"Yasushi","family":"Kawase","affiliation":"Tokyo Institute of Technology; RIKEN AIP"},{"name":"Yuko Kuroki","slug":"yuko-kuroki","given":"Yuko","family":"Kuroki","affiliation":"The University of Tokyo; RIKEN AIP"},{"name":"Atsushi Miyauchi","slug":"atsushi-miyauchi","given":"Atsushi","family":"Miyauchi","affiliation":"RIKEN AIP"}],"abstract":"Aggregating responses from crowd workers is a fundamental task in the process of crowdsourcing. In cases where a few experts are overwhelmed by a large number of non-experts, most answer aggregation algorithms such as the majority voting fail to identify the correct answers. Therefore, it is crucial to extract reliable experts from the crowd workers. In this study, we introduce the notion of \"expert core\", which is a set of workers that is very unlikely to contain a non-expert. We design a graph-mining-based efficient algorithm that exactly computes the expert core. To answer the aggregation task, we propose two types of algorithms. The first one incorporates the expert core into existing answer aggregation algorithms such as the majority voting, whereas the second one utilizes information provided by the expert core extraction algorithm pertaining to the reliability of workers. We then give a theoretical justification for the first type of algorithm. Computational experiments using synthetic and real-world datasets demonstrate that our proposed answer aggregation algorithms outperform state-of-the-art algorithms.","doi":"10.24963/ijcai.2019/177","url":"https://ijcai.org/proceedings/2019/177","pdf":"https://ijcai.org/proceedings/2019/0177.pdf","markdown":"https://ijcai.org/proceedings/2019/177.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/177","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Heuristic Search and Game Playing","track":"Main track","number":177,"pages":{"first":1272,"last":1279},"keywords":["Machine Learning Applications: Networks","Machine Learning: Clustering","Humans and AI: Human Computation and Crowdsourcing","Heuristic Search and Game Playing: Combinatorial Search and Optimisation","Machine Learning: Data Mining"],"videos":[],"references":[{"position":1,"raw":"[Andersen and Chellapilla, 2009] Reid Andersen and Kumar Chellapilla. Finding dense subgraphs with size bounds. In WAW, pages 25-37, 2009.","title":"Finding dense subgraphs with size bounds","authors":"Reid Andersen, Kumar Chellapilla","year":"2009","venue":"WAW","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Asahiro et al., 2000] Yuichi Asahiro, Kazuo Iwama, Hisao Tamaki, and Takeshi Tokuyama. Greedily finding a dense subgraph. J. Algorithms, 34(2):203-221, 2000.","title":"Greedily finding a dense subgraph","authors":"Yuichi Asahiro, Kazuo Iwama, Hisao Tamaki, Takeshi Tokuyama","year":"2000","venue":"J. Algorithms","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Aydin et al., 2014] Bahadir Ismail Aydin, Yavuz Selim Yil- maz, Yaliang Li, Qi Li, Jing Gao, and Murat Demirbas. Crowdsourcing for multiple-choice question answering. In AAAI, pages 2946-2953, 2014.","title":"Crowdsourcing for multiple-choice question answering","authors":"Bahadir Ismail Aydin, Yavuz Selim Yilmaz, Yaliang Li, Qi Li, Jing Gao, Murat Demirbas","year":"2014","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bachrach et al., 2012] Yoram Bachrach, Tom Minka, John Guiver, and Thore Graepel. How to grade a test with- out knowing the answers: A Bayesian graphical model for adaptive crowdsourcing and aptitude testing. In ICML, pages 819-826, 2012.","title":"How to grade a test without knowing the answers: A Bayesian graphical model for adaptive crowdsourcing and aptitude testing","authors":"Bachrach","year":"2012","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bader and Hogue, 2003] Gary D. Bader and Christopher W. V. Hogue. An automated method for finding molecu- lar complexes in large protein interaction networks. BMC Bioinform., 4(1):1-27, 2003.","title":"An automated method for finding molecular complexes in large protein interaction networks","authors":"D Gary, Christopher W V Bader, Hogue","year":"2003","venue":"BMC Bioinform","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Charikar, 2000] Moses Charikar. Greedy approximation al- gorithms for finding dense components in a graph. In AP- PROX, pages 84-95, 2000.","title":"Moses Charikar. Greedy approximation algorithms for finding dense components in a graph","authors":"Charikar","year":"2000","venue":"AP-PROX","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dawid and Skene, 1979] A. P. Dawid and A. M. Skene. Maximum likelihood estimation of observer error-rates us- ing the EM algorithm. Appl. Stat., pages 20-28, 1979.","title":"Maximum likelihood estimation of observer error-rates using the EM algorithm","authors":"A P Dawid, A M Skene","year":"1979","venue":"Appl. Stat","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Demartini et al., 2012] Gianluca Demartini, Djellel Eddine Difallah, and Philippe Cudré-Mauroux. ZenCrowd: Lever- aging probabilistic reasoning and crowdsourcing tech- niques for large-scale entity linking. In WWW, pages 469- 478, 2012.","title":"ZenCrowd: Leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking","authors":"Gianluca Demartini, Djellel Eddine Difallah, Philippe Cudré-Mauroux","year":"2012","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dourisboure et al., 2007] Yon Dourisboure, Filippo Geraci, and Marco Pellegrini. Extraction and classification of dense communities in the web. In WWW, pages 461-470, 2007.","title":"Extraction and classification of dense communities in the web","authors":"Yon Dourisboure, Filippo Geraci, Marco Pellegrini","year":"2007","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Finin et al., 2010] Tim Finin, Will Murnane, Anand Karandikar, Nicholas Keller, and Justin Martineau. Anno- tating named entities in Twitter data with crowdsourcing. In NAACL HLT Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, pages 80-88, 2010.","title":"Annotating named entities in Twitter data with crowdsourcing","authors":"Tim Finin, Will Murnane, Anand Karandikar, Nicholas Keller, Justin Martineau","year":"2010","venue":"NAACL HLT Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gibson et al., 2005] David Gibson, Ravi Kumar, and An- drew Tomkins. Discovering large dense subgraphs in mas- sive graphs. In VLDB, pages 721-732, 2005.","title":"Discovering large dense subgraphs in massive graphs","authors":"David Gibson, Ravi Kumar, Andrew Tomkins","year":"2005","venue":"VLDB","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hsueh et al., 2009] Pei-Yun Hsueh, Prem Melville, and Vikas Sindhwani. Data quality from crowdsourcing: A study of annotation selection criteria. In NAACL HLT Workshop on Active Learning for Natural Language Pro- cessing, pages 27-35, 2009.","title":"Data quality from crowdsourcing: A study of annotation selection criteria","authors":"Pei-Yun Hsueh, Prem Melville, Vikas Sindhwani","year":"2009","venue":"NAACL HLT Workshop on Active Learning for Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Karger et al., 2011] David R. Karger, Sewoong Oh, and De- vavrat Shah. Iterative learning for reliable crowdsourcing systems. In NIPS, pages 1953-1961, 2011.","title":"Iterative learning for reliable crowdsourcing systems","authors":"David R Karger, Sewoong Oh, Devavrat Shah","year":"2011","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kawase and Miyauchi, 2018] Yasushi Kawase and Atsushi Miyauchi. The densest subgraph problem with a con- vex/concave size function. Algorithmica, 80(12):3461- 3480, 2018.","title":"The densest subgraph problem with a convex/concave size function","authors":"Yasushi Kawase, Atsushi Miyauchi","year":"2018","venue":"Algorithmica","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kawase et al., 2019] Yasushi Kawase, Yuko Kuroki, and Atsushi Miyauchi. Graph mining meets crowd- sourcing: Extracting experts for answer aggregation. arXiv:1905.08088, 2019.","title":"Graph mining meets crowdsourcing: Extracting experts for answer aggregation","authors":"Yasushi Kawase, Yuko Kuroki, Atsushi Miyauchi","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Khuller and Saha, 2009] Samir Khuller and Barna Saha. On finding dense subgraphs. In ICALP, pages 597-608, 2009.","title":"On finding dense subgraphs","authors":"Samir Khuller, Barna Saha","year":"2009","venue":"ICALP","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kim and Ghahramani, 2012] Hyun-Chul Kim and Zoubin Ghahramani. Bayesian classifier combination. In AIS- TATS, pages 619-627, 2012.","title":"Bayesian classifier combination","authors":"Hyun-Chul Kim, Zoubin Ghahramani","year":"2012","venue":"AIS-TATS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2014a] Hongwei Li, Bo Zhao, and Ariel Fuxman. The wisdom of minority: Discovering and targeting the right group of workers for crowdsourcing. In WWW, pages 165-176, 2014.","title":"The wisdom of minority: Discovering and targeting the right group of workers for crowdsourcing","authors":"Li","year":"2014","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2014b] Qi Li, Yaliang Li, Jing Gao, Lu Su, Bo Zhao, Murat Demirbas, Wei Fan, and Jiawei Han. A confidence-aware approach for truth discovery on long-tail data. In VLDB, pages 425-436, 2014.","title":"A confidence-aware approach for truth discovery on long-tail data","authors":"Li","year":"2014","venue":"VLDB","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Li et al., 2017] Jiyi Li, Yukino Baba, and Hisashi Kashima. Hyper questions: Unsupervised targeting of a few experts in crowdsourcing. In CIKM, pages 1069-1078, 2017.","title":"Hyper questions: Unsupervised targeting of a few experts in crowdsourcing","authors":"Li","year":"2017","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Li et al., 2018] Jiyi Li, Yukino Baba, and Hisashi Kashima. Incorporating worker similarity for label aggregation in crowdsourcing. In ICANN, pages 596-606, 2018.","title":"Incorporating worker similarity for label aggregation in crowdsourcing","authors":"Li","year":"2018","venue":"ICANN","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lin et al., 2015] Yi-ling Lin, Christoph Trattner, Peter Brusilovsky, and Daqing He. The impact of image de- scriptions on user tagging behavior: A study of the nature and functionality of crowdsourced tags. J. Assoc. Inform. Sci. Tech., 66(9):1785-1798, 2015.","title":"The impact of image descriptions on user tagging behavior: A study of the nature and functionality of crowdsourced tags","authors":"Yi-Ling Lin, Christoph Trattner, Peter Brusilovsky, Daqing He","year":"2015","venue":"J. Assoc. Inform. Sci. Tech","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Liu et al., 2012a] Qiang Liu, Jian Peng, and Alexander T. Ihler. Variational inference for crowdsourcing. In NIPS, pages 692-700, 2012.","title":"Variational inference for crowdsourcing","authors":"Liu","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Liu et al., 2012b] Xuan Liu, Meiyu Lu, Beng Chin Ooi, Yanyan Shen, Sai Wu, and Meihui Zhang. CDAS: A crowdsourcing data analytics system. In VLDB, pages 1040-1051, 2012.","title":"CDAS: A crowdsourcing data analytics system","authors":"Liu","year":"2012","venue":"VLDB","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Liu et al., 2013] Qiang Liu, Alexander T. Ihler, and Mark Steyvers. Scoring workers in crowdsourcing: How many control questions are enough? In NIPS, pages 1914-1922, 2013.","title":"Scoring workers in crowdsourcing: How many control questions are enough?","authors":"Qiang Liu, Alexander T Ihler, Mark Steyvers","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Ma et al., 2015] Fenglong Ma, Yaliang Li, Qi Li, Minghui Qiu, Jing Gao, Shi Zhi, Lu Su, Bo Zhao, Heng Ji, and Jiawei Han. Faitcrowd: Fine grained truth discovery for crowdsourced data aggregation. In KDD, pages 745-754, 2015.","title":"Faitcrowd: Fine grained truth discovery for crowdsourced data aggregation","authors":"Ma","year":"2015","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Miyauchi and Kakimura, 2018] Atsushi Miyauchi and Naonori Kakimura. Finding a dense subgraph with sparse cut. In CIKM, pages 547-556, 2018.","title":"Finding a dense subgraph with sparse cut","authors":"Kakimura Miyauchi, Atsushi Miyauchi, Naonori Kakimura","year":"2018","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Miyauchi and Kawase, 2015] Atsushi Miyauchi and Ya- sushi Kawase. What is a network community? A novel quality function and detection algorithms. In CIKM, pages 1471-1480, 2015.","title":"What is a network community? A novel quality function and detection algorithms","authors":"Kawase Miyauchi, Atsushi Miyauchi, Yasushi Kawase","year":"2015","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Raykar et al., 2010] Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. Learning from crowds. Journal of Machine Learning Research, 11:1297-1322, 2010.","title":"Learning from crowds","authors":"C Vikas, Shipeng Raykar, Linda H Yu, Gerardo Hermosillo Zhao, Charles Valadez, Luca Florin, Linda Bogoni, Moy","year":"2010","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Snow et al., 2008] Rion Snow, Brendan O'Conno, Daniel Jurafsky, and Andrew Y. Ng. Cheap and fast-but is it good? evaluating non-expert annotations for natural lan- guage tasks. In EMNLP, pages 254-263, 2008.","title":"Cheap and fast-but is it good? evaluating non-expert annotations for natural language tasks","authors":"Rion Snow, O' Brendan, Daniel Conno, Andrew Y Jurafsky, Ng","year":"2008","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Tsourakakis et al., 2013] Charalampos E. Tsourakakis, Francesco Bonchi, Aristides Gionis, Francesco Gullo, and Maria A. Tsiarli. Denser than the densest subgraph: Extracting optimal quasi-cliques with quality guarantees. In KDD, pages 104-112, 2013.","title":"Denser than the densest subgraph: Extracting optimal quasi-cliques with quality guarantees","authors":"Charalampos E Tsourakakis, Francesco Bonchi, Aristides Gionis, Francesco Gullo, Maria A Tsiarli","year":"2013","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Venanzi et al., 2014] Matteo Venanzi, John Guiver, Gabriella Kazai, Pushmeet Kohli, and Milad Shokouhi. Community-based Bayesian aggregation models for crowdsourcing. In WWW, pages 155-164, 2014.","title":"Community-based Bayesian aggregation models for crowdsourcing","authors":"Matteo Venanzi, John Guiver, Gabriella Kazai, Pushmeet Kohli, Milad Shokouhi","year":"2014","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wauthier and Jordan, 2011] Fabian L. Wauthier and Michael I. Jordan. Bayesian bias mitigation for crowd- sourcing. In NIPS, pages 1800-1808, 2011.","title":"Bayesian bias mitigation for crowdsourcing","authors":"Fabian L Wauthier, Michael I Jordan","year":"2011","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Welinder et al., 2010] Peter Welinder, Steve Branson, Pietro Perona, and Serge J. Belongie. The multidimen- sional wisdom of crowds. In NIPS, pages 2424-2432, 2010.","title":"The multidimensional wisdom of crowds","authors":"Peter Welinder, Steve Branson, Pietro Perona, Serge J Belongie","year":"2010","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Whitehill et al., 2009] Jacob Whitehill, Paul Ruvolo, Tingfan Wu, and Jacob Bergsma. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In NIPS, pages 2035-2043, 2009.","title":"Whose vote should count more: Optimal integration of labels from labelers of unknown expertise","authors":"Jacob Whitehill, Paul Ruvolo, Tingfan Wu, Jacob Bergsma","year":"2009","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zheng et al., 2017] Yudian Zheng, Guoliang Li, Yuanbing Li, Caihua Shan, and Reynold Cheng. Truth inference in crowdsourcing: Is the problem solved? In VLDB, pages 541-552, 2017.","title":"Truth inference in crowdsourcing: Is the problem solved?","authors":"Zheng","year":"2017","venue":"VLDB","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhou et al., 2012] Dengyong Zhou, John C. Platt, Sumit Basu, and Yi Mao. Learning from the wisdom of crowds by minimax entropy. In NIPS, pages 2195-2203, 2012.","title":"Learning from the wisdom of crowds by minimax entropy","authors":"Dengyong Zhou, John C Platt, Sumit Basu, Yi Mao","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}