{"type":"ScholarlyArticle","title":"Crowd Learning: Improving Online Decision Making Using Crowdsourced Data","authors":[{"name":"Yang Liu","slug":"yang-liu","given":"Yang","family":"Liu","affiliation":"$ University of Michigan , Ann Arbor MI , USA"},{"name":"Mingyan Liu","slug":"mingyan-liu","given":"Mingyan","family":"Liu","affiliation":"# Harvard University , Cambridge MA , USA"}],"abstract":"We analyze an online learning problem that arises in crowdsourcing systems for users facing crowdsourced data: a user at each discrete time step t can choose K out of a total of N options (bandits), and receives randomly generated rewards dependent on user-specific and option-specific statistics unknown to the user. Each user aims to maximize her expected total rewards over a certain time horizon through a sequence of exploration and exploitation steps.  Different from the typical regret/bandit learning setting, in this case a user may also exploit crowdsourced information to augment her learning process, i.e., other users' choices or rewards from using these options.  We consider two scenarios, one in which only their choices are shared, and the other in which users share full information including their choices and subsequent rewards. In both cases we derive bounds on the weak regret,  the difference between the user's expected total reward and the reward from a user-specific best single-action policy; and show how they improve over their individual-learning counterpart. We also evaluate the performance of our algorithms using simulated data as well as the real-world movie ratings dataset MovieLens.","doi":"10.24963/ijcai.2017/45","url":"https://ijcai.org/proceedings/2017/45","pdf":"https://ijcai.org/proceedings/2017/0045.pdf","markdown":"https://ijcai.org/proceedings/2017/45.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/45","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra, IIIA-CSIC","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Agent-based and Multi-agent Systems","track":"Main track","number":45,"pages":{"first":317,"last":323},"keywords":["Machine Learning: Online Learning","Agent-based and Multi-agent Systems: Multi-agent Learning","Multidisciplinary Topics and Applications: Multidisciplinary Topics and Applications"],"videos":[],"references":[{"position":1,"raw":"[Anantharam et al., 1986] Venkat Anantharam, Pravin Varaiya, and Jean Walrand. Asymptotically Efficient Allocation Rules for the Multiarmed Bandit Problem with Multiple Plays Part I: I.I.D. Rewards, Part II: Markovian Rewards. Technical Report UCB/ERL M86/62, EECS Department, University of California, Berkeley, 1986.","title":"Asymptotically Efficient Allocation Rules for the Multiarmed Bandit Problem with Multiple Plays Part I: I.I.D. Rewards, Part II: Markovian Rewards","authors":"Venkat Anantharam, Pravin Varaiya, Jean Walrand","year":"1986","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Auer et al., 2002] Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time Analysis of the Multiarmed Ban- dit Problem. Mach. Learn., 47:235-256, May 2002.","title":"Finite-time Analysis of the Multiarmed Bandit Problem","authors":"Auer","year":"2002","venue":"Mach. Learn","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bandura and Walters, 1963] Albert Bandura and Richard H Walters. Social learning and personality development, volume 14. JSTOR, 1963.","title":"Social learning and personality development","authors":"Walters Bandura, Albert Bandura, Richard H Walters","year":"1963","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bresler et al., 2015] Guy Bresler, Devavrat Shah, and Luis F Voloch. Collaborative filtering with low regret. arXiv preprint arXiv:1507.05371, 2015.","title":"Collaborative filtering with low regret","authors":"Guy Bresler, Devavrat Shah, Luis F Voloch","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[chun Wang et al., 2005] Chih chun Wang, Student Member, Sanjeev R. Kulkarni, and H. Vincent Poor. Bandit prob- lems with side observations. IEEE Transactions on Auto- matic Control, 50:338-355, 2005.","title":"Bandit problems with side observations","authors":"Wang Chun","year":"2005","venue":"IEEE Transactions on Automatic Control","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Donmez et al., 2009] Pinar Donmez, Jaime G Carbonell, and Jeff Schneider. Efficiently learning the accuracy of labeling sources for selective sampling. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 259-268. ACM, 2009.","title":"Efficiently learning the accuracy of labeling sources for selective sampling","authors":"Pinar Donmez, Jaime G Carbonell, Jeff Schneider","year":"2009","venue":"Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ertekin et al., 2012] Seyda Ertekin, Haym Hirsh, and Cyn- thia Rudin. Learning to predict the wisdom of crowds. arXiv preprint arXiv:1204.3611, 2012.","title":"Learning to predict the wisdom of crowds","authors":"Seyda Ertekin, Haym Hirsh, Cynthia Rudin","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gentile and Li, 2014] Claudio Gentile and Shuai Li. Online clustering of bandits. In ICML, 2014.","title":"Online clustering of bandits","authors":"Claudio Gentile, Shuai Li","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Goldstein et al., 2014] Daniel G Goldstein, Randolph Pre- ston McAfee, and Siddharth Suri. The wisdom of smaller, smarter crowds. In Proceedings of the fifteenth ACM con- ference on Economics and computation, pages 471-488. ACM, 2014.","title":"The wisdom of smaller, smarter crowds","authors":"Randolph Daniel G Goldstein, Siddharth Preston Mcafee, Suri","year":"2014","venue":"Proceedings of the fifteenth ACM conference on Economics and computation","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ho et al., 2013] Chien-Ju Ho, Shahin Jabbari, and Jen- nifer W Vaughan. Adaptive task assignment for crowd- sourced classification. In Proceedings of the 30th Interna- tional Conference on Machine Learning (ICML-13), pages 534-542, 2013.","title":"Adaptive task assignment for crowdsourced classification","authors":"Chien-Ju Ho, Shahin Jabbari, Jennifer W Vaughan","year":"2013","venue":"Proceedings of the 30th International Conference on Machine Learning (ICML-13)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[KONECT, 2014] KONECT. Movielens 1m network dataset -KONECT, August 2014.","title":"KONECT. Movielens 1m network dataset -KONECT","authors":"Konect","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lai and Robbins, 1985] T. L. Lai and H. Robbins. Asymp- totically Efficient Adaptive Allocation Rules. Advances in Applied Mathematics, 6:4-22, 1985.","title":"Asymptotically Efficient Adaptive Allocation Rules","authors":"T L Lai, H Robbins","year":"1985","venue":"Advances in Applied Mathematics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Langford and Zhang, 2007] John Langford and Tong Zhang. The Epoch-Greedy Algorithm for Multi-armed Bandits with Side Information. In NIPS, 2007.","title":"The Epoch-Greedy Algorithm for Multi-armed Bandits with Side Information","authors":"Zhang Langford, John Langford, Tong Zhang","year":"2007","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu and Liu, 2017] Yang Liu and Mingyan Liu. Crowd learning: Improving online decision making using crowd- sourced data. full version, IJCAI, 2017.","title":"Crowd learning: Improving online decision making using crowdsourced data","authors":"Liu ; ] Liu, Yang Liu, Mingyan Liu","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lu et al., 2010] Tyler Lu, Dvid Pl, and Martin Pal. Contex- tual multi-armed bandits. Journal of Machine Learning Research, 9:485-492, 2010.","title":"Contextual multi-armed bandits","authors":"Lu","year":"2010","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rosenberg et al., 2007] Dinah Rosenberg, Eilon Solan, and Nicolas Vieille. Social learning in one-arm bandit prob- lems. Econometrica, 75(6):1591-1611, 2007.","title":"Social learning in one-arm bandit problems","authors":"Rosenberg","year":"2007","venue":"Econometrica","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Smith and Sørensen, 2000] Lones Smith and Peter Sørensen. Pathological outcomes of observational learning. Econometrica, 68(2):371-398, 2000.","title":"Pathological outcomes of observational learning","authors":"Sørensen ; Smith, Lones Smith, Peter Sørensen","year":"2000","venue":"Econometrica","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tekin and Liu, 2012] Cem Tekin and Mingyan Liu. Online Learning of Rested and Restless Bandits. Information The- ory, IEEE Transactions on, 58(8):5588-5611, 2012.","title":"Online Learning of Rested and Restless Bandits. Information Theory","authors":"Liu Tekin, Cem Tekin, Mingyan Liu","year":"2012","venue":"IEEE Transactions on","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"Valko et al., ] Michal Valko, Rémi Munos, Branislav Kve- ton, and Tomas Kocak. Spectral bandits for smooth graph functions.","title":"Spectral bandits for smooth graph functions","authors":"Valko","year":null,"venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}