{"type":"ScholarlyArticle","title":"Bayesian Aggregation of Categorical Distributions with Applications in Crowdsourcing","authors":[{"name":"Alexandry Augustin","slug":"alexandry-augustin","given":"Alexandry","family":"Augustin","affiliation":"Southampton University Southampton , UK"},{"name":"Matteo Venanzi","slug":"matteo-venanzi","given":"Matteo","family":"Venanzi","affiliation":"Microsoft London , UK"},{"name":"Alex Rogers","slug":"alex-rogers","given":"Alex","family":"Rogers","affiliation":"Oxford University Oxford , UK"},{"name":"Nicholas R. Jennings","slug":"nicholas-r-jennings","given":"Nicholas R.","family":"Jennings","affiliation":"Imperial College London , UK"}],"abstract":"A key problem in crowdsourcing is the aggregation of judgments of proportions. For example, workers might be presented with a news article or an image, and be asked to identify the proportion of each topic, sentiment, object, or colour present in it. These varying judgments then need to be aggregated to form a consensus view of the document’s or image’s contents. Often, however, these judgments are skewed by workers who provide judgments randomly. Such spammers make the cost of acquiring judgments more expensive and degrade the accuracy of the aggregation. For such cases, we provide a new Bayesian framework for aggregating these responses (expressed in the form of categorical distributions) that for the first time accounts for spammers. We elicit 796 judgments about proportions of objects and coloursin images. Experimental results show comparable aggregation accuracy when 60% of the workers are spammers, as other state of the art approaches do when there are no spammers.","doi":"10.24963/ijcai.2017/195","url":"https://ijcai.org/proceedings/2017/195","pdf":"https://ijcai.org/proceedings/2017/0195.pdf","markdown":"https://ijcai.org/proceedings/2017/195.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/195","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":195,"pages":{"first":1411,"last":1417},"keywords":["Machine Learning: Ensemble Methods","Machine Learning: Machine Learning","Machine Learning: Unsupervised Learning"],"videos":[],"references":[{"position":1,"raw":"[Attias, 2000] H. Attias. A Variational Bayesian Frame- work for Graphical Models. In Advances in Neural In- formation Processing Systems 12, pages 209-215. MIT Press, 2000.","title":"A Variational Bayesian Framework for Graphical Models","authors":"H Attias","year":"2000","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Augustin and Venanzi, 2017] A. Augustin and M. Ve- nanzi. MBCC, 2017. https://github.com/alexandry- augustin/mbcc/.","title":"MBCC","authors":"A Augustin, M Venanzi","year":"2017","venue":null,"doi":null,"url":"https://github.com/alexandry-augustin/mbcc/","internal_url":null},{"position":3,"raw":"[Bacharach, 1979] M. Bacharach. Normal Bayesian Di- alogues. Journal of the American Statistical Associa- tion, 74(368):837, 1979.","title":"Normal Bayesian Dialogues","authors":"M Bacharach","year":"1979","venue":"Journal of the American Statistical Association","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Blei et al., 2003] D.M. Blei, A.Y. Ng, and M.I. Jor- dan. Latent Dirichlet Allocation. Journal of Machine Learning Research, 3:993-1022, 2003.","title":"Latent Dirichlet Allocation","authors":"D M Blei, A Y Ng, M I Jordan","year":"2003","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Devroye, 1983] L. Devroye. The Equivalence of Weak, Strong and Complete Convergence in L1 for Kernel Density Estimates. The Annals of Statistics, pages 896-904, 1983.","title":"The Equivalence of Weak, Strong and Complete Convergence in L1 for Kernel Density Estimates","authors":"L Devroye","year":"1983","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dietz, 2010] L. Dietz. Directed factor graph notation for generative models. Technical report, Max Planck Institute for Informatics, Tech. Rep, 2010.","title":"Directed factor graph notation for generative models","authors":"L Dietz","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Difallah et al., 2012] D.E. Difallah, G. Demartini, and P. Cudre-Mauroux. Mechanical Cheat: Spamming Schemes and Adversarial Techniques on Crowdsourc- ing Platforms. In CrowdSearch, pages 26-30. Citeseer, 2012.","title":"Mechanical Cheat: Spamming Schemes and Adversarial Techniques on Crowdsourcing Platforms","authors":"D E Difallah, G Demartini, P Cudre-Mauroux","year":"2012","venue":"CrowdSearch","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Escalante et al., 2010] H.J. Escalante, C.A. Hernandez, J.A. Gonzalez, A. Lopez-Lopez, M. Montes, Ed- uardo F Morales, L Enrique Sucar, L. Villasenor, and M. Grubinger. The segmented and annotated IAPR TC-12 benchmark. Computer Vision and Image Un- derstanding, 114(4):419-428, 2010.","title":"The segmented and annotated IAPR TC-12 benchmark","authors":"H J Escalante, C A Hernandez, J A Gonzalez, A Lopez-Lopez, M Montes, Eduardo F Morales, L Enrique Sucar, L Villasenor, M Grubinger","year":"2010","venue":"Computer Vision and Image Understanding","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Geman and Geman, 1984] S. Geman and D. Geman. Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images. Pattern Analysis and Machine Intelligence, IEEE Transactions on, PAMI- 6(6):721-741, 1984.","title":"Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images. Pattern Analysis and Machine Intelligence","authors":"S Geman, D Geman","year":"1984","venue":"IEEE Transactions on","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Geng, 2016] X. Geng. Label distribution learning. IEEE Transactions on Knowledge and Data Engineer- ing, 28(7):1734-1748, 2016.","title":"Label distribution learning","authors":"X Geng","year":"2016","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ho et al., 2016] C. J. Ho, R. Frongillo, and Y. Chen. Eliciting Categorical Data for Optimal Aggregation. In Advances In Neural Information Processing Sys- tems, pages 2442-2450, 2016.","title":"Eliciting Categorical Data for Optimal Aggregation","authors":"C J Ho, R Frongillo, Y Chen","year":"2016","venue":"Advances In Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hollands and Spence, 1992] J.G. Hollands and I. Spence. Judgments of Change and Propor- tion in Graphical Perception. Human Factors: The Journal of the Human Factors and Ergonomics Society, 34(3):313-334, 1992.","title":"Judgments of Change and Proportion in Graphical Perception","authors":"Spence ; ] J G Hollands, I Hollands, Spence","year":"1992","venue":"Human Factors: The Journal of the Human Factors and Ergonomics Society","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ipeirotis et al., 2010] P.G. Ipeirotis, F. Provost, and J. Wang. Quality Management on Amazon Mechan- ical Turk. In Proceedings of the ACM SIGKDD workshop on human computation, pages 64-67. ACM, 2010.","title":"Quality Management on Amazon Mechanical Turk","authors":"P G Ipeirotis, F Provost, J Wang","year":"2010","venue":"Proceedings of the ACM SIGKDD workshop on human computation","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jordan et al., 1999] M.I. Jordan, Z. Ghahramani, T.S. Jaakkola, and L.K. Saul. An Introduction to Varia- tional Methods for Graphical Models. Machine learn- ing, 37(2):183-233, 1999.","title":"An Introduction to Variational Methods for Graphical Models","authors":"M I Jordan, Z Ghahramani, T S Jaakkola, L K Saul","year":"1999","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kim and Ghahramani, 2012] H.C. Kim and Z. Ghahra- mani. Bayesian Classifier Combination. In Proceedings of the 15th International Conference on Artificial In- telligence and Statistics (AISTATS), 2012.","title":"Bayesian Classifier Combination","authors":"H C Kim, Z Ghahramani","year":"2012","venue":"Proceedings of the 15th International Conference on Artificial Intelligence and Statistics (AISTATS)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Laplace, 1986] P.S. Laplace. Memoir on the Probability of the Causes of Events. Statistical Science, 1(3):364- 378, 1986.","title":"Memoir on the Probability of the Causes of Events","authors":"P S Laplace","year":"1986","venue":"Statistical Science","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Minka, 2013] T.P. Minka. Expectation Propagation for Approximate Bayesian Inference. In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, volume abs/1301.2294, 2013.","title":"Expectation Propagation for Approximate Bayesian Inference","authors":"T P Minka","year":"2013","venue":"Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Oakley, 2010] J. Oakley. Eliciting Univariate Probabil- ity Distributions. Rethinking risk measurement and reporting, 1, 2010.","title":"Eliciting Univariate Probability Distributions","authors":"J Oakley","year":"2010","venue":"Rethinking risk measurement and reporting","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Snow et al., 2008] R. Snow, B. O'Connor, D. Jurafsky, and A.Y. Ng. Cheap and Fast -But is it Good?: Eval- uating Non-expert Annotations for Natural Language Tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 254- 263. Association for Computational Linguistics, 2008.","title":"Cheap and Fast -But is it Good?: Evaluating Non-expert Annotations for Natural Language Tasks","authors":"R Snow, B O'connor, D Jurafsky, A Y Ng","year":"2008","venue":"Proceedings of the Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Strapparava and Mihalcea, 2007] C. Strapparava and R. Mihalcea. Semeval-2007 Task 14: Affective Text. In Proceedings of the 4th International Workshop on Semantic Evaluations, pages 70-74. Association for Computational Linguistics, 2007.","title":"Semeval-2007 Task 14: Affective Text","authors":"C Strapparava, R Mihalcea","year":"2007","venue":"Proceedings of the 4th International Workshop on Semantic Evaluations","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Varey et al., 1990] C.A. Varey, B.A. Mellers, and M.H. Birnbaum. Judgments of Proportions. Journal of Ex- perimental Psychology: Human Perception and Per- formance, 16(3):613, 1990.","title":"Judgments of Proportions","authors":"C A Varey, B A Mellers, M H Birnbaum","year":"1990","venue":"Journal of Experimental Psychology: Human Perception and Performance","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Venanzi et al., 2014] M. Venanzi, J. Guiver, G. Kazai, P. Kohli, and M. Shokouhi. Community-based Bayesian Aggregation Models for Crowdsourcing. In Proceedings of the 23rd International Conference on World Wide Web, pages 155-164. ACM Press, 2014.","title":"Community-based Bayesian Aggregation Models for Crowdsourcing","authors":"M Venanzi, J Guiver, G Kazai, P Kohli, M Shokouhi","year":"2014","venue":"Proceedings of the 23rd International Conference on World Wide Web","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vuurens et al., 2011] J. Vuurens, A.P. de Vries, and C. Eickhoff. How Much Spam Can You Take? An Analysis of Crowdsourcing Results to Increase Accu- racy. In Proceedings of the ACM SIGIR Workshop on Crowdsourcing for Information Retrieval (CIR'11), pages 21-26, 2011.","title":"How Much Spam Can You Take? An Analysis of Crowdsourcing Results to Increase Accuracy","authors":"J Vuurens, A P De Vries, C Eickhoff","year":"2011","venue":"Proceedings of the ACM SIGIR Workshop on Crowdsourcing for Information Retrieval (CIR'11)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Weerahandi and Zidek, 1981] S. Weerahandi and J.V. Zidek. Multi-Bayesian Statistical Decision Theory. Journal of the Royal Statistical Society. Series A (General), 144(1):85, 1981.","title":"Multi-Bayesian Statistical Decision Theory","authors":"S Weerahandi, J V Zidek","year":"1981","venue":"Journal of the Royal Statistical Society. Series A (General)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Winn and Bishop, 2005] J. Winn and C.M. Bishop. Variational Message Passing. Journal of Machine Learning Research, 6:661-694, 2005.","title":"Variational Message Passing","authors":"J Winn, C M Bishop","year":"2005","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}