{"type":"ScholarlyArticle","title":"Crowdsourcing with Multiple-Source Knowledge Transfer","authors":[{"name":"Guangyang Han","slug":"guangyang-han","given":"Guangyang","family":"Han","affiliation":"College of Computer and Information Sciences , Southwest University , Chongqing , China"},{"name":"Jinzheng Tu","slug":"jinzheng-tu","given":"Jinzheng","family":"Tu","affiliation":"College of Computer and Information Sciences , Southwest University , Chongqing , China"},{"name":"Guoxian Yu","slug":"guoxian-yu","given":"Guoxian","family":"Yu","affiliation":"College of Computer and Information Sciences , Southwest University , Chongqing , China"},{"name":"Jun Wang","slug":"jun-wang","given":"Jun","family":"Wang","affiliation":"College of Computer and Information Sciences , Southwest University , Chongqing , China"},{"name":"Carlotta Domeniconi","slug":"carlotta-domeniconi","given":"Carlotta","family":"Domeniconi","affiliation":"Department of Computer Science , George Mason University , VA , USA"}],"abstract":"Crowdsourcing is a new computing paradigm that harnesses human effort to solve computer-hard problems. Budget and quality are two fundamental factors in crowdsourcing, but they are antagonistic and their balance is crucially important. Induction and inference are principled ways for humans to acquire knowledge. Transfer learning can also enable induction and inference processes. When a new task comes, we may not know how to go about approaching it. On the other hand, we may have easy access to relevant knowledge that can help us with the new task. As such, via appropriate knowledge transfer, for example, an improved annotation can be achieved for the task at a small cost. To make this idea concrete, we introduce the Crowdsourcing with Multiple-source Knowledge Transfer (CrowdMKT)approach to transfer knowledge from multiple, similar, but different domains for a new task, and to reduce the negative impact of irrelevant sources. CrwodMKT first learns a set of concentrated high-level feature vectors of tasks using knowledge transfer from multiple sources, and then introduces a probabilistic graphical model to jointly model the tasks with high-level features, workers, and their annotations. Finally, it adopts an EM algorithm to estimatethe workers strengths and consensus. Experimental results on real-world image and text datasets prove the effectiveness of CrowdMKT in improving quality and reducing the budget.","doi":"10.24963/ijcai.2020/402","url":"https://ijcai.org/proceedings/2020/402","pdf":"https://ijcai.org/proceedings/2020/0402.pdf","markdown":"https://ijcai.org/proceedings/2020/402.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/402","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":402,"pages":{"first":2908,"last":2914},"keywords":["Humans and AI: Human Computation and Crowdsourcing","Machine Learning: Transfer, Adaptation, Multi-task Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25026/file_25026.m3u8","length_sec":301,"provider":"hls"}],"references":[{"position":1,"raw":"[Cooper et al., 2010] Seth Cooper, Firas Khatib, Adrien Treuille, Janos Barbero, Jeehyung Lee, Michael Beenen, Andrew Leaver-Fay, David Baker, Zoran Popović, et al. Pre- dicting protein structures with a multiplayer online game. Nature, 466(7307):756, 2010.","title":"Predicting protein structures with a multiplayer online game","authors":"Seth Cooper, Firas Khatib, Adrien Treuille, Janos Barbero, Jeehyung Lee, Michael Beenen, Andrew Leaver-Fay, David Baker, Zoran Popović","year":"2010","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Daniel et al., 2018] Florian Daniel, Pavel Kucherbaev, Cinzi- a Cappiello, Boualem Benatallah, and Mohammad Allah- bakhsh. Quality control in crowdsourcing: A survey of quality attributes, assessment techniques, and assurance actions. ACM Computing Surveys, 51(1):7, 2018.","title":"Quality control in crowdsourcing: A survey of quality attributes, assessment techniques, and assurance actions","authors":"Florian Daniel, Pavel Kucherbaev, Cinzia Cappiello, Boualem Benatallah, Mohammad Allahbakhsh","year":"2018","venue":"ACM Computing Surveys","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Demartini et al., 2012] Gianluca Demartini, Djellel Eddine Difallah, and Philippe Cudré-Mauroux. Zencrowd: leverag- ing probabilistic reasoning and crowdsourcing techniques for large-scale entity linking. In WWW, pages 469-478, 2012.","title":"Zencrowd: leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking","authors":"Gianluca Demartini, Djellel Eddine Difallah, Philippe Cudré-Mauroux","year":"2012","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ding et al., 2018] Zhengming Ding, Ming Shao, and Yun Fu. Incomplete multisource transfer learning. TNNLS, 29(2):310-323, 2018.","title":"Incomplete multisource transfer learning","authors":"Zhengming Ding, Ming Shao, Yun Fu","year":"2018","venue":"TNNLS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fang et al., 2014] Meng Fang, Jie Yin, and Dacheng Tao. Active learning for crowdsourcing using knowledge trans- fer. In AAAI, pages 1809-1815, 2014.","title":"Active learning for crowdsourcing using knowledge transfer","authors":"Meng Fang, Jie Yin, Dacheng Tao","year":"2014","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fletcher, 2013] Roger Fletcher. Practical methods of opti- mization. John Wiley & Sons, 2013.","title":"Practical methods of optimization","authors":"Roger Fletcher","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gao et al., 2013] Jinyang Gao, Xuan Liu, Beng Chin Ooi, Haixun Wang, and Gang Chen. An online cost sensitive decision-making method in crowdsourcing systems. In SIGMOD, pages 217-228, 2013.","title":"An online cost sensitive decision-making method in crowdsourcing systems","authors":"Jinyang Gao, Xuan Liu, Beng, Chin Ooi, Haixun Wang, Gang Chen","year":"2013","venue":"SIGMOD","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu and Zhang, 2018] Zehong Hu and Jie Zhang. A novel strategy for active task assignment in crowd labeling. In IJCAI, pages 1538-1545, 2018.","title":"A novel strategy for active task assignment in crowd labeling","authors":"Zhang ; ] Zehong Hu, Jie Hu, Zhang","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/213"},{"position":9,"raw":"[Kazai et al., 2011] Gabriella Kazai, Jaap Kamps, and Natasa Milic-Frayling. Worker types and personality traits in crowdsourcing relevance labels. In CIKM, pages 1941- 1944, 2011.","title":"Worker types and personality traits in crowdsourcing relevance labels","authors":"Gabriella Kazai, Jaap Kamps, Natasa Milic-Frayling","year":"2011","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Koller and Friedman, 2009] Daphne Koller and Nir Fried- man. Probabilistic graphical models: principles and tech- niques. MIT press, 2009.","title":"Probabilistic graphical models: principles and techniques","authors":"Friedman Koller","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kurve et al., 2015] Aditya Kurve, David J. Miller, and George Kesidis. Multicategory crowdsourcing account- ing for variable task difficulty, worker skill, and worker intention. TKDE, 27(3):794-809, 2015.","title":"Multicategory crowdsourcing accounting for variable task difficulty, worker skill, and worker intention","authors":"Aditya Kurve, David J Miller, George Kesidis","year":"2015","venue":"TKDE","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lee et al., 2007] Honglak Lee, Alexis Battle, Rajat Raina, and Andrew Y Ng. Efficient sparse coding algorithms. In NeurIPS, pages 801-808, 2007.","title":"Efficient sparse coding algorithms","authors":"Lee","year":"2007","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2017] Guoliang Li, Yudian Zheng, Ju Fan, Jiannan Wang, and Reynold Cheng. Crowdsourced data manage- ment: Overview and challenges. In SIGMOD, pages 1711- 1716, 2017.","title":"Crowdsourced data management: Overview and challenges","authors":"Guoliang Li, Yudian Zheng, Ju Fan, Jiannan Wang, Reynold Cheng","year":"2017","venue":"SIGMOD","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2012] Xuan Liu, Meiyu Lu, Beng Chin Ooi, Yanyan Shen, Sai Wu, and Meihui Zhang. Cdas: a crowd- sourcing data analytics system. VLDB, 5(10):1040-1051, 2012.","title":"Cdas: a crowdsourcing data analytics system","authors":"Xuan Liu, Meiyu Lu, Beng, Chin Ooi, Yanyan Shen, Sai Wu, Meihui Zhang","year":"2012","venue":"VLDB","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mavridis et al., 2016] Panagiotis Mavridis, David Gross- Amblard, and Zoltán Miklós. Using hierarchical skills for optimized task assignment in knowledge-intensive crowd- sourcing. In WWW, pages 843-853, 2016.","title":"Using hierarchical skills for optimized task assignment in knowledge-intensive crowdsourcing","authors":"Panagiotis Mavridis, David Gross-Amblard, Zoltán Miklós","year":"2016","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mo et al., 2013] Kaixiang Mo, Erheng Zhong, and Qiang Yang. Cross-task crowdsourcing. In KDD, pages 677-685, 2013.","title":"Cross-task crowdsourcing","authors":"Mo","year":"2013","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pan and Yang, 2009] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. TKDE, 22(10):1345-1359, 2009.","title":"A survey on transfer learning","authors":"Jialin Sinno, Qiang Pan, Yang","year":"2009","venue":"TKDE","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rosenstein et al., 2005] Michael T Rosenstein, Zvika Marx, Leslie Pack Kaelbling, and Thomas G Dietterich. To trans- fer or not to transfer. In NuerIPS workshop on Transfer Learning, pages 1-4, 2005.","title":"To transfer or not to transfer","authors":"Zvika Michael T Rosenstein, Leslie Marx, Thomas G Pack Kaelbling, Dietterich","year":"2005","venue":"NuerIPS workshop on Transfer Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arX- iv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tu et al., 2020a] Jingzheng Tu, Guoxian Yu, Jun Wang, Car- lotta Domeniconi, and Xiangliang Zhang. Attention-aware answers of the crowd. In SDM, pages 451-459, 2020.","title":"Attention-aware answers of the crowd","authors":"Tu","year":"2020","venue":"SDM","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tu et al., 2020b] Jinzheng Tu, Guoxian Yu, Carlotta Domeniconi, Jun Wang, Guoqiang Xiao, and Maozu Guo. Multi-label crowd consensus via joint matrix factorization. KAIS, 62(4):1341-1369, 2020.","title":"Multi-label crowd consensus via joint matrix factorization","authors":"Tu","year":"2020","venue":"KAIS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wah et al., 2011] C. Wah, S. Branson, P. Welinder, P. Peron- a, and S. Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical report, 2011.","title":"The Caltech-UCSD Birds-200-2011 Dataset","authors":"C Wah, S Branson, P Welinder, P Perona, S Belongie","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Weiss et al., 2016] Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big data, 3(1):1-9, 2016.","title":"A survey of transfer learning","authors":"Karl Weiss, Taghi M Khoshgoftaar, Dingding Wang","year":"2016","venue":"Journal of Big data","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wright et al., 2010] John Wright, Yi Ma, Julien Mairal, Guillermo Sapiro, Thomas S Huang, and Shuicheng Yan. Sparse representation for computer vision and pattern recog- nition. Proc. of IEEE, 98(6):1031-1044, 2010.","title":"Sparse representation for computer vision and pattern recognition","authors":"John Wright, Yi Ma, Julien Mairal, Guillermo Sapiro, Thomas S Huang, Shuicheng Yan","year":"2010","venue":"Proc. of IEEE","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yao and Doretto, 2010] Yi Yao and Gianfranco Doretto. Boosting for transfer learning with multiple sources. In CVPR, pages 1855-1862, 2010.","title":"Boosting for transfer learning with multiple sources","authors":"Yi Yao, Gianfranco Doretto","year":"2010","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yu et al., 2020] Guoxian Yu, Jinzheng Tu, Jun Wang, Carlot- ta Domeniconi, and Xiangliang Zhang. Active multi-label crowd consensus. TNNLS, 99(1):1-12, 2020.","title":"Active multi-label crowd consensus","authors":"Yu","year":"2020","venue":"TNNLS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2017] Jing Zhang, Victor S Sheng, Qianmu Li, Jian Wu, and Xindong Wu. Consensus algorithms for biased labeling in crowdsourcing. Information Sciences, 382:254-273, 2017.","title":"Consensus algorithms for biased labeling in crowdsourcing","authors":"Jing Zhang, S Victor, Qianmu Sheng, Jian Li, Xindong Wu, Wu","year":"2017","venue":"Information Sciences","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhao et al., 2013] Zhou Zhao, Da Yan, Wilfred Ng, and Shi Gao. A transfer learning based framework of crowd- selection on twitter. In KDD, pages 1514-1517, 2013.","title":"A transfer learning based framework of crowdselection on twitter","authors":"Zhou Zhao, Da Yan, Wilfred Ng, Shi Gao","year":"2013","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zheng et al., 2016] Yudian Zheng, Guoliang Li, and Reynold Cheng. Docs: a domain-aware crowdsourcing system using knowledge bases. PVLDB, 10(4):361-372, 2016.","title":"Docs: a domain-aware crowdsourcing system using knowledge bases","authors":"Zheng","year":"2016","venue":"PVLDB","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}