{"type":"ScholarlyArticle","title":"Learning a Ground Truth Ranking Using Noisy Approval Votes","authors":[{"name":"Ioannis Caragiannis","slug":"ioannis-caragiannis","given":"Ioannis","family":"Caragiannis","affiliation":"University of Patras"},{"name":"Evi Micha","slug":"evi-micha","given":"Evi","family":"Micha","affiliation":"University of Patras"}],"abstract":"We consider a voting scenario where agents have opinions that are estimates of an underlying common ground truth ranking of the available alternatives, and each agent is asked to approve a set with her most preferred alternatives. We assume that estimates are implicitly formed using the well-known Mallows model for generating random rankings. We show that k-approval voting --- where all agents are asked to approve the same number k of alternatives and the outcome is obtained by sorting the alternatives in terms of their number of approvals --- has exponential sample complexity for all values of k. This negative result suggests that an exponential (in terms of the number of alternatives m) number of agents is always necessary in order to recover the ground truth ranking with high probability. In contrast, by just asking each agent to approve a random number of alternatives, the sample complexity improves dramatically: it now depends only polynomially on m. Our results may have implications on the effectiveness of crowdsourcing applications that ask workers to provide their input by approving sets of available alternatives.","doi":"10.24963/ijcai.2017/22","url":"https://ijcai.org/proceedings/2017/22","pdf":"https://ijcai.org/proceedings/2017/0022.pdf","markdown":"https://ijcai.org/proceedings/2017/22.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/22","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Agent-based and Multi-agent Systems","track":"Main track","number":22,"pages":{"first":149,"last":155},"keywords":["Agent-based and Multi-agent Systems: Social Choice Theory","Agent-based and Multi-agent Systems: Economic paradigms, auctions and market-based systems"],"videos":[],"references":[{"position":1,"raw":"Arrow, 1953","title":null,"authors":"Arrow","year":"1953","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"Kenneth J. Arrow. Social Choice and Individ- ual Values. Cowles Foundation, 1953.","title":"Social Choice and Individual Values","authors":"J Kenneth, Arrow","year":"1953","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brams and Fishburn, 2007] Steven J. Brams and Peter C. Fishburn. Approval Voting. Springer, 2007.","title":"Approval Voting","authors":"Steven J Brams, Peter C Fishburn","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Braverman and Mossel, 2008] Mark Braverman and Elchanan Mossel. Noisy sorting without resampling. In Proceedings of the 19th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 268-276, 2008.","title":"Noisy sorting without resampling","authors":"Mark Braverman, Elchanan Mossel","year":"2008","venue":"Proceedings of the 19th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Caragiannis et al., 2016] Ioannis Caragiannis, Ariel D. Pro- caccia, and Nisarg Shah. When do noisy votes reveal the truth? ACM Transactions on Economics and Computation, 4(3):15, 2016.","title":"When do noisy votes reveal the truth?","authors":"Ioannis Caragiannis, Ariel D Procaccia, Nisarg Shah","year":"2016","venue":"ACM Transactions on Economics and Computation","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Caragiannis et al., 2017] Ioannis Caragiannis, Xenophon Chatzigeorgiou, George A. Krimpas, and Alexandros A. Voudouris. Optimizing positional scoring rules for rank aggregation. In Proceedings of the 31st AAAI Conference on Artificial Intelligence (AAAI), 2017.","title":"Optimizing positional scoring rules for rank aggregation","authors":"Ioannis Caragiannis, Xenophon Chatzigeorgiou, George A Krimpas, Alexandros A Voudouris","year":"2017","venue":"Proceedings of the 31st AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chierichetti and Kleinberg, 2014] Flavio Chierichetti and Jon M. Kleinberg. Voting with limited information and many alternatives. SIAM Journal on Computing, 43(5):1615-1653, 2014.","title":"Voting with limited information and many alternatives","authors":"Flavio Chierichetti, Jon M Kleinberg","year":"2014","venue":"SIAM Journal on Computing","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chierichetti et al., 2014] Flavio Chierichetti, Anirban Das- gupta, Ravi Kumar, and Silvio Lattanzi. On reconstruct- ing a hidden permutation. In Approximation, Random- ization, and Combinatorial Optimization. Algorithms and Techniques, APPROX/RANDOM, pages 604-617, 2014.","title":"On reconstructing a hidden permutation","authors":"Flavio Chierichetti, Anirban Dasgupta, Ravi Kumar, Silvio Lattanzi","year":"2014","venue":"Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques, APPROX/RANDOM","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Conitzer and Sandholm, 2005] Vincent Conitzer and Tuo- mas Sandholm. Common voting rules as maximum like- lihood estimators. In Proceedings of the 21st Conference on Uncertainty in Artificial Intelligence (UAI), pages 145- 152, 2005.","title":"Common voting rules as maximum likelihood estimators","authors":"Sandholm ; Conitzer, Tuomas Conitzer, Sandholm","year":"2005","venue":"Proceedings of the 21st Conference on Uncertainty in Artificial Intelligence (UAI)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Critchlow et al., 1991] Douglas E. Critchlow, Michael A. Fligner, and Joseph S. Verducci. Probability mod- els on rankings. Journal of Mathematical Psychology, 35(3):294-318, 1991.","title":"Probability models on rankings","authors":"Douglas E Critchlow, Michael A Fligner, Joseph S Verducci","year":"1991","venue":"Journal of Mathematical Psychology","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[de Weerdt et al., 2016] Mathijs M. de Weerdt, Emrico H. Gerding, and Sebastian Stein. Minimising the rank ag- gregation error. In Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems, pages 1375-1376, 2016.","title":"Minimising the rank aggregation error","authors":"M Mathijs, Emrico H De Weerdt, Sebastian Gerding, Stein","year":"2016","venue":"Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Dey and Bhattacharyya, 2015] Palash Dey and Arnab Bhat- tacharyya. Sample complexity for winner prediction in elections. In Proceedings of the 14th International Confer- ence on Autonomous Agents and Multiagent Systems (AA- MAS), pages 1421-1430, 2015.","title":"Sample complexity for winner prediction in elections","authors":"Palash Dey, Arnab Bhattacharyya","year":"2015","venue":"Proceedings of the 14th International Conference on Autonomous Agents and Multiagent Systems (AA-MAS)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Doignon et al., 2004] Jean-Paul Doignon, Aleksandar Pakeč, and Michel Regenwetter. The repeated insertion model for rankings: Missing link between two subset choice models. Psychometrika, 69(1):33-54, 2004.","title":"The repeated insertion model for rankings: Missing link between two subset choice models","authors":"Doignon","year":"2004","venue":"Psychometrika","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Fligner and Verducci, 1988] Michael A. Fligner and Joseph S. Verducci. Multi-stage ranking models. Journal of the American Statistical Association, 83:892-901, 1988.","title":"Multi-stage ranking models","authors":"Michael A Fligner, Joseph S Verducci","year":"1988","venue":"Journal of the American Statistical Association","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Gibbard, 1973] Allan Gibbard. Manipulation of voting schemes: A general result. Econometrica, 41:587-601, 1973.","title":"Manipulation of voting schemes: A general result","authors":"Allan Gibbard, Gibbard","year":"1973","venue":"Econometrica","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hoeffding, 1963] Wassily Hoeffding. Probability inequali- ties for sums of bounded random variables. Journal of the American Statistical Association, 58(301):13-30, 1963.","title":"Wassily Hoeffding. Probability inequalities for sums of bounded random variables","authors":"Hoeffding","year":"1963","venue":"Journal of the American Statistical Association","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Laslier and Sanver, 2010] Jean-Francois Laslier and M. Remzi Sanver, editors. Handbook on Approval Voting. Springer, 2010.","title":"Handbook on Approval Voting","authors":null,"year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lebanon and Lafferty, 2002] Guy Lebanon and John Laf- ferty. Cranking: Combining rankings using conditional probability models on permutations. In Proceedings of the 19nd International Conference on Machine Learning (ICML), pages 363-370, 2002.","title":"Guy Lebanon and John Lafferty. Cranking: Combining rankings using conditional probability models on permutations","authors":"Lafferty Lebanon","year":"2002","venue":"Proceedings of the 19nd International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lee et al., 2014] Jeehyung Lee, Wipapat Kladwang, Min- jae Lee, Daniel Cantu, Martin Azizyan, Hanjoo Kim, Alex Limpaecher, Snehal Gaikwad, Sungroh Yoon, Adrien Treuille, Rhiju Das, and EteRNA participants. RNA de- sign rules from a massive open laboratory. Proceedings of the National Academy of Sciences, 111(6):2122-2127, 2014.","title":"RNA design rules from a massive open laboratory","authors":"Lee","year":"2014","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lu and Boutilier, 2014] Tyler Lu and Craig Boutilier. Ef- fective sampling and learning for Mallows models with pairwise-preference data. Journal of Machine Learning Research, 15:3963-4009, 2014.","title":"Effective sampling and learning for Mallows models with pairwise-preference data","authors":"Lu, Boutilier, Tyler Lu, Craig Boutilier","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Mallows, 1957] C. L. Mallows. Non-null ranking models. Biometrika, 44:114-130, 1957.","title":"Non-null ranking models","authors":"C L Mallows","year":"1957","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Mitzenmacher and Upfal, 2005] Michael Mitzenmacher and Eli Upfal. Probability and Computing -Randomized Algorithms and Probabilistic Analysis. Cambridge University Press, 2005.","title":"Probability and Computing -Randomized Algorithms and Probabilistic Analysis","authors":"Michael Mitzenmacher, Eli Upfal","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Procaccia and Shah, 2015] Ariel D. Procaccia and Nisarg Shah. Is approval voting optimal given approval votes? In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Sys- tems, pages 1801-1809, 2015.","title":"Is approval voting optimal given approval votes?","authors":"Ariel D Procaccia, Nisarg Shah","year":"2015","venue":"Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Rubinstein and Vardi, 2017] Aviad Rubinstein and Shai Vardi. Sorting from noisier samples. In Proceedings of the 28th Annual ACM-SIAM Symposium on Discrete Al- gorithms (SODA), pages 960-972, 2017.","title":"Sorting from noisier samples","authors":"Vardi ; ] Aviad Rubinstein, Shai Rubinstein, Vardi","year":"2017","venue":"Proceedings of the 28th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Satterthwaite, 1975] Mark A. Satterthwaite. Strategy proof- ness and Arrow's conditions: Existence and correspon- dence theorems for voting procedures and social welfare functions. Journal of Economic Theory, 10:187-217, 1975.","title":"Strategy proofness and Arrow's conditions: Existence and correspondence theorems for voting procedures and social welfare functions","authors":"Mark A Satterthwaite","year":"1975","venue":"Journal of Economic Theory","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Shah et al., 2015] Nihar B. Shah, Dengyong Zhou, and Yu- val Peres. Approval voting and incentives in crowdsourc- ing. In Proceedings of the 32nd International Conference on Machine Learning (ICML), pages 10-19, 2015.","title":"Approval voting and incentives in crowdsourcing","authors":"B Nihar, Dengyong Shah, Yuval Zhou, Peres","year":"2015","venue":"Proceedings of the 32nd International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Young, 1988] H. Peyton Young. Condorcet's theory of vot- ing. American Political Science Review, 82(4):1231-1244, 1988.","title":"Condorcet's theory of voting","authors":"H, Peyton Young","year":"1988","venue":"American Political Science Review","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}