{"type":"ScholarlyArticle","title":"MiSC: Mixed Strategies Crowdsourcing","authors":[{"name":"Ching Yun Ko","slug":"ching-yun-ko","given":"Ching Yun","family":"Ko","affiliation":"The University of Hong Kong , Hong Kong"},{"name":"Rui Lin","slug":"rui-lin","given":"Rui","family":"Lin","affiliation":"The University of Hong Kong , Hong Kong"},{"name":"Shu Li","slug":"shu-li","given":"Shu","family":"Li","affiliation":"Nanjing University , Nanjing 210023 , China"},{"name":"Ngai Wong","slug":"ngai-wong","given":"Ngai","family":"Wong","affiliation":"The University of Hong Kong , Hong Kong"}],"abstract":"Popular crowdsourcing techniques mostly focus on evaluating workers' labeling quality before adjusting their weights during label aggregation. Recently, another cohort of models regard crowdsourced annotations as incomplete tensors and recover unfilled labels by tensor completion. However, mixed strategies of the two methodologies have never been comprehensively investigated, leaving them as rather independent approaches. In this work, we propose MiSC ( Mixed Strategies Crowdsourcing), a versatile framework integrating arbitrary conventional crowdsourcing and tensor completion techniques. In particular, we propose a novel iterative Tucker label aggregation algorithm that outperforms state-of-the-art methods in extensive experiments.","doi":"10.24963/ijcai.2019/193","url":"https://ijcai.org/proceedings/2019/193","pdf":"https://ijcai.org/proceedings/2019/0193.pdf","markdown":"https://ijcai.org/proceedings/2019/193.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/193","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Humans and AI","track":"Main track","number":193,"pages":{"first":1394,"last":1400},"keywords":["Humans and AI: Human-AI Collaboration","Humans and AI: Human Computation and Crowdsourcing"],"videos":[],"references":[{"position":1,"raw":"[Candès and Recht, 2009] Emmanuel J Candès and Ben- jamin Recht. Exact matrix completion via convex op- timization. Foundations of Computational mathematics, 9(6):717, 2009.","title":"Exact matrix completion via convex optimization","authors":"J Emmanuel, Benjamin Candès, Recht","year":"2009","venue":"Foundations of Computational mathematics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen, 2015] Yudong Chen. Incoherence-optimal matrix completion. IEEE Transactions on Information Theory, 61(5):2909-2923, 2015.","title":"Incoherence-optimal matrix completion","authors":"Yudong Chen","year":"2015","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cichocki et al., 2015] Andrzej Cichocki, Danilo Mandic, Lieven De Lathauwer, Guoxu Zhou, Qibin Zhao, Cesar Caiafa, and Anh-Huy Phan. Tensor decompositions for signal processing applications: From two-way to multiway component analysis. IEEE Signal Processing Magazine, 32(2):145-163, 2015.","title":"Tensor decompositions for signal processing applications: From two-way to multiway component analysis","authors":"Andrzej Cichocki, Danilo Mandic, Lieven De Lathauwer, Guoxu Zhou, Qibin Zhao, Cesar Caiafa, Anh-Huy Phan","year":"2015","venue":"IEEE Signal Processing Magazine","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Collins and Cohen, 2012] Michael Collins and Shay B. Co- hen. Tensor decomposition for fast parsing with latent- variable PCFGs. In NIPS, 2012.","title":"Tensor decomposition for fast parsing with latentvariable PCFGs","authors":"Michael Collins, Shay B Cohen","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dawid and Skene, 1979] Alexander P. Dawid and Allan M. Skene. Maximum likelihood estimation of observer error- rates using the em algorithm. Applied statistics, pages 20- 28, 1979.","title":"Maximum likelihood estimation of observer errorrates using the em algorithm","authors":"Alexander P Dawid, Allan M Skene","year":"1979","venue":"Applied statistics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[De Lathauwer et al., 2000a] Lieven De Lathauwer, Bart De Moor, and Joos Vandewalle. A multilinear singular value decomposition. SIAM journal on Matrix Analysis and Applications, 21(4):1253-1278, 2000.","title":"A multilinear singular value decomposition","authors":"De Lathauwer","year":"2000","venue":"SIAM journal on Matrix Analysis and Applications","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[De Lathauwer et al., 2000b] Lieven De Lathauwer, Bart De Moor, and Joos Vandewalle. On the best rank-1 and rank-(r 1, r 2,..., rn) approximation of higher-order ten- sors. SIAM journal on Matrix Analysis and Applications, 21(4):1324-1342, 2000.","title":"On the best rank-1 and rank-(r 1, r 2,..., rn) approximation of higher-order tensors","authors":"De Lathauwer","year":"2000","venue":"SIAM journal on Matrix Analysis and Applications","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Denton et al., 2014] Emily L. Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In NIPS, 2014.","title":"Exploiting linear structure within convolutional networks for efficient evaluation","authors":"Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann Lecun, Rob Fergus","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Imaizumi et al., 2017] Masaaki Imaizumi, Takanori Mae- hara, and Kohei Hayashi. On tensor train rank minimiza- tion: Statistical efficiency and scalable algorithm. In NIPS, 2017.","title":"On tensor train rank minimization: Statistical efficiency and scalable algorithm","authors":"Masaaki Imaizumi, Takanori Maehara, Kohei Hayashi","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jain and Oh, 2014] Prateek Jain and Sewoong Oh. Provable tensor factorization with missing data. In NIPS, 2014.","title":"Provable tensor factorization with missing data","authors":"Prateek Jain, Sewoong Oh","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Karger et al., 2011] David R. Karger, Sewoong Oh, and De- vavrat Shah. Iterative learning for reliable crowdsourcing systems. In NIPS, 2011.","title":"Iterative learning for reliable crowdsourcing systems","authors":"David R Karger, Sewoong Oh, Devavrat Shah","year":"2011","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kolda and Bader, 2009] Tamara G. Kolda and Brett W. Bader. Tensor decompositions and applications. SIAM Review, 51(3):455-500, 2009.","title":"Tensor decompositions and applications","authors":"Tamara G Kolda, Brett W Bader","year":"2009","venue":"SIAM Review","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Levin, 1963] Joseph Levin. Three-Mode Factor Analysis. PhD thesis, University of Illinois at Urbana-Champaign, 1963.","title":"Three-Mode Factor Analysis","authors":"Joseph Levin","year":"1963","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li and Jiang, 2018] Shao-Yuan Li and Yuan Jiang. Multi- label crowdsourcing learning with incomplete annotations. In PRICAI, 2018.","title":"Multilabel crowdsourcing learning with incomplete annotations","authors":"Shao-Yuan Li, Yuan Jiang","year":"2018","venue":"PRICAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2012] Qiang Liu, Jian Peng, and Alexander T. Ih- ler. Variational inference for crowdsourcing. In NIPS, 2012.","title":"Variational inference for crowdsourcing","authors":"Qiang Liu, Jian Peng, Alexander T Ihler","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2013] Ji Liu, Przemyslaw Musialski, Peter Wonka, and Jieping Ye. Tensor completion for estimating missing values in visual data. IEEE Transactions on Pat- tern Analysis and Machine Intelligence, 35(1):208-220, 2013.","title":"Tensor completion for estimating missing values in visual data","authors":"Ji Liu, Przemyslaw Musialski, Peter Wonka, Jieping Ye","year":"2013","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2015] Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Learning context-sensitive word embeddings with neural tensor skip-gram model. In IJCAI, 2015.","title":"Learning context-sensitive word embeddings with neural tensor skip-gram model","authors":"Pengfei Liu, Xipeng Qiu, Xuanjing Huang","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/185"},{"position":19,"raw":"[Mozafari et al., 2014] Barzan Mozafari, Purna Sarkar, Michael Franklin, Michael Jordan, and Samuel Madden. Scaling up crowd-sourcing to very large datasets: A case for active learning. PVLDB, 8(2):125-136, 2014.","title":"Scaling up crowd-sourcing to very large datasets: A case for active learning","authors":"Barzan Mozafari, Purna Sarkar, Michael Franklin, Michael Jordan, Samuel Madden","year":"2014","venue":"PVLDB","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Novikov et al., 2015] Alexander Novikov, Dmitry Podoprikhin, Anton Osokin, and Dmitry Vetrov. Tensoriz- ing neural networks. In NIPS, 2015.","title":"Tensorizing neural networks","authors":"Alexander Novikov, Dmitry Podoprikhin, Anton Osokin, Dmitry Vetrov","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Raykar et al., 2009] Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Anna Jerebko, Charles Florin, Gerardo H. Valadez, Luca Bogoni, and Linda Moy. Supervised learning from multiple experts: whom to trust when everyone lies a bit. In ICML, 2009.","title":"Supervised learning from multiple experts: whom to trust when everyone lies a bit","authors":"C Vikas, Shipeng Raykar, Linda H Yu, Anna Zhao, Charles Jerebko, Gerardo H Florin, Luca Valadez, Linda Bogoni, Moy","year":"2009","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Signoretto et al., 2014] Marco Signoretto, Quoc Tran Dinh, Lieven De Lathauwer, and Johan A. K. Suykens. Learning with tensors: a framework based on convex optimization and spectral regularization. Machine Learning, 94(3):303- 351, 2014.","title":"Learning with tensors: a framework based on convex optimization and spectral regularization","authors":"Marco Signoretto, Tran Quoc, Lieven Dinh, Johan A K De Lathauwer, Suykens","year":"2014","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Snow et al., 2008] Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. Cheap and fast-but is it good?: evaluating non-expert annotations for natural lan- guage tasks. In EMNLP, 2008.","title":"Cheap and fast-but is it good?: evaluating non-expert annotations for natural language tasks","authors":"Rion Snow, O' Brendan, Daniel Connor, Andrew Y Jurafsky, Ng","year":"2008","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Suzuki, 2015] Taiji Suzuki. Convergence rate of Bayesian tensor estimator and its minimax optimality. In ICML, 2015.","title":"Convergence rate of Bayesian tensor estimator and its minimax optimality","authors":"Taiji Suzuki, Suzuki","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tucker, 1963] Ledyard R. Tucker. Implications of factor analysis of three-way matrices for measurement of change. Problems in measuring change, 15:122-137, 1963.","title":"Implications of factor analysis of three-way matrices for measurement of change","authors":"R Ledyard, Tucker","year":"1963","venue":"Problems in measuring change","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Welinder et al., 2010] Peter Welinder, Steve Branson, Pietro Perona, and Serge J. Belongie. The multidimen- sional wisdom of crowds. In NIPS, 2010.","title":"The multidimensional wisdom of crowds","authors":"Peter Welinder, Steve Branson, Pietro Perona, Serge J Belongie","year":"2010","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Whitehill et al., 2009] Jacob Whitehill, Ting fan Wu, Jacob Bergsma, Javier R. Movellan, and Paul L. Ruvolo. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In NIPS, 2009.","title":"Whose vote should count more: Optimal integration of labels from labelers of unknown expertise","authors":"Jacob Whitehill, Jacob Ting Fan Wu, Javier R Bergsma, Paul L Movellan, Ruvolo","year":"2009","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2016] Yuchen Zhang, Xi Chen, Dengyong Zhou, and Michael I. Jordan. Spectral methods meet em: A provably optimal algorithm for crowdsourcing. The Journal of Machine Learning Research, 17(1):3537-3580, 2016.","title":"Spectral methods meet em: A provably optimal algorithm for crowdsourcing","authors":"Yuchen Zhang, Xi Chen, Dengyong Zhou, Michael I Jordan","year":"2016","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhao et al., 2015] Q. Zhao, L. Zhang, and A. Cichocki. Bayesian CP factorization of incomplete tensors with au- tomatic rank determination. IEEE Transactions on Pat- tern Analysis and Machine Intelligence, 37(9):1751-1763, 2015.","title":"Bayesian CP factorization of incomplete tensors with automatic rank determination","authors":"Q Zhao, L Zhang, A Cichocki","year":"2015","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhou and He, 2016] Yao Zhou and Jingrui He. Crowdsourc- ing via tensor augmentation and completion. In IJCAI, 2016.","title":"Crowdsourcing via tensor augmentation and completion","authors":"Yao Zhou, Jingrui He","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/347"},{"position":31,"raw":"[Zhou et al., 2012] Denny Zhou, Sumit Basu, Yi Mao, and John C. Platt. Learning from the wisdom of crowds by minimax entropy. In NIPS, 2012.","title":"Learning from the wisdom of crowds by minimax entropy","authors":"Denny Zhou, Sumit Basu, Yi Mao, John C Platt","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}