{"type":"ScholarlyArticle","title":"Efficient Techniques for Crowdsourced Top-k Lists","authors":[{"name":"Luca de Alfaro","slug":"luca-de-alfaro","given":"Luca","family":"de Alfaro","affiliation":""},{"name":"Vassilis Polychronopoulos","slug":"vassilis-polychronopoulos","given":"Vassilis","family":"Polychronopoulos","affiliation":"UC Santa Cruz"},{"name":"Neoklis Polyzotis","slug":"neoklis-polyzotis","given":"Neoklis","family":"Polyzotis","affiliation":"Google Inc."}],"abstract":"We focus on the problem of obtaining top-k lists of items from larger itemsets, using human workers for doing comparisons among items.An example application is short-listing a large set of college applications using advanced students as workers. We describe novel efficient techniques and explore their tolerance to adversarial behavior and the tradeoffs among different measures of performance (latency, expense and quality of results). We empirically evaluate the proposed techniques against prior art using simulations as well as real crowds in Amazon Mechanical Turk. A randomized variant of the proposed algorithms achieves significant budget saves, especially for very large itemsets and large top-k lists, with negligible risk of lowering the quality of the output.","doi":"10.24963/ijcai.2017/670","url":"https://ijcai.org/proceedings/2017/670","pdf":"https://ijcai.org/proceedings/2017/0670.pdf","markdown":"https://ijcai.org/proceedings/2017/670.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/670","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Best Sister Conferences","track":"Best Sister Conferences","number":670,"pages":{"first":4801,"last":4805},"keywords":["Artificial Intelligence: human computer interaction","Artificial Intelligence: other"],"videos":[],"references":[{"position":1,"raw":"[Ailon, 2012] Nir Ailon. An active learning algorithm for ranking from pairwise preferences with an almost optimal query complexity. JMLR, 13(Jan):137-164, 2012.","title":"An active learning algorithm for ranking from pairwise preferences with an almost optimal query complexity","authors":"Nir Ailon, Ailon","year":"2012","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ciceri et al., 2016] Eleonora Ciceri, Piero Fraternali, Da- vide Martinenghi, and Marco Tagliasacchi. Crowd- sourcing for top-k query processing over uncertain data. IEEE Transactions on Knowledge and Data Engineering, 28(1):41-53, 2016.","title":"Crowdsourcing for top-k query processing over uncertain data","authors":"Eleonora Ciceri, Piero Fraternali, Davide Martinenghi, Marco Tagliasacchi","year":"2016","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Davidson et al., 2013] Susan B. Davidson, Sanjeev Khanna, Tova Milo, and Sudeepa Roy. Using the crowd for top-k and group-by queries. In ICDT, pages 225-236, 2013.","title":"Using the crowd for top-k and group-by queries","authors":"Susan B Davidson, Sanjeev Khanna, Tova Milo, Sudeepa Roy","year":"2013","venue":"ICDT","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[de Alfaro et al., 2016] Luca de Alfaro, Vassilis Poly- chronopoulos, and Neoklis Polyzotis. Efficient techniques for crowdsourced top-k lists. In Proceedings of the 4th AAAI Conference on Human Computation and Crowd- sourcing (HCOMP), pages 22-31. AAAI, 2016.","title":"Luca de Alfaro, Vassilis Polychronopoulos, and Neoklis Polyzotis. Efficient techniques for crowdsourced top-k lists","authors":"De Alfaro","year":"2016","venue":"Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fagin et al., 2003] Ronald Fagin, Ravi Kumar, and D. Sivakumar. Efficient similarity search and classifica- tion via rank aggregation. In SIGMOD, pages 301-312, New York, NY, USA, 2003. ACM.","title":"Efficient similarity search and classification via rank aggregation","authors":"Ronald Fagin, Ravi Kumar, D Sivakumar","year":"2003","venue":"SIGMOD","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fagin et al., 2004] Ronald Fagin, Ravi Kumar, Mohammad Mahdian, D. Sivakumar, and Erik Vee. Comparing and aggregating rankings with ties. In PODS, pages 47-58. ACM, 2004.","title":"Comparing and aggregating rankings with ties","authors":"Ronald Fagin, Ravi Kumar, D Mohammad Mahdian, Erik Sivakumar, Vee","year":"2004","venue":"PODS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Feige et al., 1994] Uriel Feige, Prabhakar Raghavan, David Peleg, and Eli Upfal. Computing with noisy information. SIAM J. Comput., 23(5):1001-1018, October 1994.","title":"Computing with noisy information","authors":"Uriel Feige, Prabhakar Raghavan, David Peleg, Eli Upfal","year":"1994","venue":"SIAM J. Comput","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Guo et al., 2012] Stephen Guo, Aditya Parameswaran, and Hector Garcia-Molina. So who won?: Dynamic max dis- covery with the crowd. In SIGMOD, pages 385-396, 2012.","title":"So who won?: Dynamic max discovery with the crowd","authors":"Stephen Guo, Aditya Parameswaran, Hector Garcia-Molina","year":"2012","venue":"SIGMOD","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ipeirotis, 2010] Panos Ipeirotis. Mechanical turk: Now with 40.92% spam. Behind Enemy Lines blog, 2010.","title":"Panos Ipeirotis","authors":"Ipeirotis","year":"2010","venue":"Mechanical turk: Now with 40.92% spam","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Marcus et al., 2011] Adam Marcus, Eugene Wu, David Karger, Samuel Madden, and Robert Miller. Human- powered sorts and joins. VLDB, 5(1):13-24, September 2011.","title":"Humanpowered sorts and joins","authors":"Adam Marcus, Eugene Wu, David Karger, Samuel Madden, Robert Miller","year":"2011","venue":"VLDB","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Polychronopoulos et al., 2013] Vassilis Polychronopoulos, Luca de Alfaro, James Davis, Hector Garcia-Molina, and Neoklis Polyzotis. Human-powered top-k lists. In WebDB, pages 25-30, 2013.","title":"Human-powered top-k lists","authors":"Vassilis Polychronopoulos, James Luca De Alfaro, Hector Davis, Neoklis Garcia-Molina, Polyzotis","year":"2013","venue":"WebDB","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Thurstone, 1927] Louis L Thurstone. A law of comparative judgment. Psychological review, 34(4):273, 1927.","title":"A law of comparative judgment","authors":"Louis L Thurstone","year":"1927","venue":"Psychological review","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Venetis and Garcia-Molina, 2012a] Petros Venetis and Hec- tor Garcia-Molina. Dynamic max algorithms in crowd- sourcing environments. Technical report, Stanford Uni- versity, August 2012.","title":"Dynamic max algorithms in crowdsourcing environments","authors":"Garcia-Molina ; Venetis, Garcia-Molina","year":"2012","venue":"Petros Venetis and Hector","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Venetis and Garcia-Molina, 2012b] Petros Venetis and Hec- tor Garcia-Molina. Quality control for comparison micro- tasks. In CrowdKDD, pages 15-21. ACM, 2012.","title":"Quality control for comparison microtasks","authors":"Garcia-Molina ; Venetis, Garcia-Molina","year":"2012","venue":"CrowdKDD","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Wauthier et al., 2013] Fabian L. Wauthier, Michael I. Jor- dan, and Nebojsa Jojic. Efficient ranking from pairwise comparisons. In ICML (3), pages 109-117, 2013.","title":"Efficient ranking from pairwise comparisons","authors":"Fabian L Wauthier, Michael I Jordan, Nebojsa Jojic","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}