{"type":"ScholarlyArticle","title":"PeerNomination: Relaxing Exactness for Increased Accuracy in Peer Selection","authors":[{"name":"Nicholas Mattei","slug":"nicholas-mattei","given":"Nicholas","family":"Mattei","affiliation":"Department of Computer Science , Tulane University"},{"name":"Paolo Turrini","slug":"paolo-turrini","given":"Paolo","family":"Turrini","affiliation":"Department of Computer Science , University of Warwick"},{"name":"Stanislav Zhydkov","slug":"stanislav-zhydkov","given":"Stanislav","family":"Zhydkov","affiliation":"Department of Computer Science , University of Warwick"}],"abstract":"In peer selection agents must choose a subset of themselves for an award or a prize. As agents are self-interested, we want to design algorithms that are impartial, so that an individual agent cannot affect their own chance of being selected. This problem has broad application in resource allocation and mechanism design and has received substantial attention in the artificial intelligence literature. Here, we present a novel algorithm for impartial peer selection, PeerNomination, and provide a theoretical analysis of its accuracy.  Our algorithm possesses various desirable features. In particular, it does not require an explicit partitioning of the agents, as previous algorithms in the literature. We show empirically that it achieves higher accuracy than the exiting algorithms over several metrics.","doi":"10.24963/ijcai.2020/55","url":"https://ijcai.org/proceedings/2020/55","pdf":"https://ijcai.org/proceedings/2020/0055.pdf","markdown":"https://ijcai.org/proceedings/2020/55.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/55","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Agent-based and Multi-agent Systems","track":"Main track","number":55,"pages":{"first":393,"last":399},"keywords":["Agent-based and Multi-agent Systems: Noncooperative Games","Agent-based and Multi-agent Systems: Algorithmic Game Theory","Agent-based and Multi-agent Systems: Computational Social Choice"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25233/file_25233.m3u8","length_sec":305,"provider":"hls"}],"references":[{"position":1,"raw":"[Alon et al., 2011] Noga Alon, Felix Fischer, Ariel Procac- cia, and Moshe Tennenholtz. Sum of us: Strategyproof se- lection from the selectors. In Proceedings of the 13th Con- ference on Theoretical Aspects of Rationality and Knowl- edge (TARK), pages 101-110, 2011.","title":"Sum of us: Strategyproof selection from the selectors","authors":"Noga Alon, Felix Fischer, Ariel Procaccia, Moshe Tennenholtz","year":"2011","venue":"Proceedings of the 13th Conference on Theoretical Aspects of Rationality and Knowledge (TARK)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Aziz et al., 2016] Haris Aziz, Omer Lev, Nicholas Mattei, Jeffrey S. Rosenschein, and Toby Walsh. Strategyproof peer selection: Mechanisms, analyses, and experiments. In Dale Schuurmans and Michael P. Wellman, editors, AAAI, pages 397-403. AAAI Press, 2016.","title":"Strategyproof peer selection: Mechanisms, analyses, and experiments","authors":"Haris Aziz, Omer Lev, Nicholas Mattei, Jeffrey S Rosenschein, Toby Walsh","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Aziz et al., 2019] Haris Aziz, Omer Lev, Nicholas Mattei, Jeffrey S. Rosenschein, and Toby Walsh. Strategyproof peer selection using randomization, partitioning, and ap- portionment. Artificial Intelligence, 275:295-309, 2019.","title":"Strategyproof peer selection using randomization, partitioning, and apportionment","authors":"Haris Aziz, Omer Lev, Nicholas Mattei, Jeffrey S Rosenschein, Toby Walsh","year":"2019","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Berger and Müller-Hannemann, 2010] Annabell Berger and Matthias Müller-Hannemann. Uniform sampling of di- graphs with a fixed degree sequence. In International Workshop on Graph-Theoretic Concepts in Computer Sci- ence, pages 220-231. Springer, 2010.","title":"Uniform sampling of digraphs with a fixed degree sequence","authors":"Annabell Berger, Matthias Müller-Hannemann","year":"2010","venue":"International Workshop on Graph-Theoretic Concepts in Computer Science","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bishop, 2006] Christopher M Bishop. Pattern recognition and machine learning. Springer, 2006.","title":"Pattern recognition and machine learning","authors":"M Christopher, Bishop","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bjelde et al., 2017] Antje Bjelde, Felix Fischer, and Max Klimm. Impartial selection and the power of up to two choices. ACM Transactions on Economics and Computa- tion, 5(4):1-20, 2017.","title":"Impartial selection and the power of up to two choices","authors":"Antje Bjelde, Felix Fischer, Max Klimm","year":"2017","venue":"ACM Transactions on Economics and Computation","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bousquet et al., 2014] N. Bousquet, S. Norin, and A. Vetta. A near-optimal mechanism for impartial selection. In Pro- ceedings of the 10th International Workshop on Internet and Network Economics (WINE), Lecture Notes in Com- puter Science (LNCS), pages 133-146, 2014.","title":"A near-optimal mechanism for impartial selection","authors":"N Bousquet, S Norin, A Vetta","year":"2014","venue":"Proceedings of the 10th International Workshop on Internet and Network Economics (WINE)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Caragiannis et al., 2016] Ioannis Caragiannis, George A. Krimpas, and Alexandros A. Voudouris. How effective can simple ordinal peer grading be? Proceedings of the 2016 ACM Conference on Economics and Computation - EC 16, 2016.","title":"How effective can simple ordinal peer grading be","authors":"Ioannis Caragiannis, George A Krimpas, Alexandros A Voudouris","year":"2016","venue":"Proceedings of the 2016 ACM Conference on Economics and Computation -EC 16","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[de Clippel et al., 2008] Geoffroy de Clippel, Hervé Moulin, and Nicolaus Tideman. Impartial division of a dollar. Jour- nal of Economic Theory, 139:176-191, 2008.","title":"Impartial division of a dollar","authors":"Hervé Geoffroy De Clippel, Nicolaus Moulin, Tideman","year":"2008","venue":"Journal of Economic Theory","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Douceur, 2009] John R. Douceur. Paper rating vs. paper ranking. SIGOPS Oper. Syst. Rev., 43(2):117-121, April 2009.","title":"Paper rating vs. paper ranking","authors":"John R Douceur","year":"2009","venue":"SIGOPS Oper. Syst. Rev","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Fischer and Klimm, 2014] Felix Fischer and Max Klimm. Optimal impartial selection. In Proceedings of the 15th ACM Conference on Economics and Computation (ACM- EC), pages 803-820, 2014.","title":"Optimal impartial selection","authors":"Klimm Fischer, Felix Fischer, Max Klimm","year":"2014","venue":"Proceedings of the 15th ACM Conference on Economics and Computation (ACM-EC)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Flach, 2012] Peter A. Flach. Machine Learning -The Art and Science of Algorithms that Make Sense of Data. Cam- bridge University Press, 2012.","title":"Machine Learning -The Art and Science of Algorithms that Make Sense of Data","authors":"A Peter, Flach","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Holzman and Moulin, 2013] Ron Holzman and Hervé Moulin. Impartial nominations for a prize. Econometrica, 81(1):173-196, 2013.","title":"Impartial nominations for a prize","authors":"Moulin Holzman, Ron Holzman, Hervé Moulin","year":"2013","venue":"Econometrica","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kahng et al., 2018] Anson Kahng, Yasmine Kotturi, Chin- may Kulkarni, David Kurokawa, and Ariel Procaccia. Ranking wily people who rank each other. In Proceed- ings of the 32nd AAAI Conference on Artificial Intelligence (AAAI), 2018.","title":"Ranking wily people who rank each other","authors":"Anson Kahng, Yasmine Kotturi, Chinmay Kulkarni, David Kurokawa, Ariel Procaccia","year":"2018","venue":"Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Keynes, 1936] John Maynard Keynes. The General Theory of Employment, Interest and Money. Palgrave Macmillan, 1936.","title":"The General Theory of Employment, Interest and Money","authors":"John Maynard, Keynes","year":"1936","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kurokawa et al., 2015] David Kurokawa, Omer Lev, Jamie Morgenstern, and Ariel D. Procaccia. Impartial peer re- view. In Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI'15, pages 582-588. AAAI Press, 2015.","title":"Impartial peer review","authors":"David Kurokawa, Omer Lev, Jamie Morgenstern, Ariel D Procaccia","year":"2015","venue":"Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI'15","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Langford, 2015] John Langford. The NIPS experiment, Jan 2015.","title":"The NIPS experiment","authors":"John Langford","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lian et al., 2018] Jing Wu Lian, Nicholas Mattei, Renee Noble, and Toby Walsh. The conference paper assignment problem: Using order weighted averages to assign indivis- ible goods. In Proceedings of the32ndAAAI Conference on Artificial Intelligence (AAAI), pages 1138-1145, 2018.","title":"The conference paper assignment problem: Using order weighted averages to assign indivisible goods","authors":"Jing Wu, Lian, Nicholas Mattei, Renee Noble, Toby Walsh","year":"2018","venue":"Proceedings of the32ndAAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mallows, 1957] Colin Lingwood Mallows. Non-null rank- ing models. I. Biometrika, 44(1-2):114-130, June 1957.","title":"Non-null ranking models","authors":"Colin Lingwood, Mallows","year":"1957","venue":"I. Biometrika","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mattei and Walsh, 2017] Nicholas Mattei and Toby Walsh. A PREFLIB.ORG Retrospective: Lessons Learned and New Directions. In U. Endriss, editor, Trends in Com- putational Social Choice, chapter 15, pages 289-309. AI Access Foundation, 2017.","title":"ORG Retrospective: Lessons Learned and New Directions","authors":"Nicholas Mattei, Toby Walsh A Preflib","year":"2017","venue":"Trends in Computational Social Choice","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Merrifield and Saari, 2009] Michael Merrifield and Donald Saari. Telescope time without tears: a distributed approach to peer review. Astronomy and Geophysics, 50(4):4.16- 4.20, 2009.","title":"Telescope time without tears: a distributed approach to peer review","authors":"Saari Merrifield, Michael Merrifield, Donald Saari","year":"2009","venue":"Astronomy and Geophysics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Naghizadeh and Liu, 2013] Parinaz Naghizadeh and Mingyan Liu. Incentives, quality, and risks: A look into the NSF proposal review pilot. CoRR, abs/1307.6528, 2013.","title":"Parinaz Naghizadeh and Mingyan Liu. Incentives, quality, and risks: A look into the NSF proposal review pilot","authors":"Liu Naghizadeh","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Piech et al., 2013] Chris Piech, Jonathan Huang, Zhenghao Chen, Chuong Do, Andrew Ng, and Daphne Koller. Tuned models of peer assessment in moocs. arXiv preprint arXiv:1307.2579, 2013.","title":"Tuned models of peer assessment in moocs","authors":"Chris Piech, Jonathan Huang, Zhenghao Chen, Chuong Do, Andrew Ng, Daphne Koller","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shah et al., 2018] Nihar B Shah, Behzad Tabibian, Krikamol Muandet, Isabelle Guyon, and Ulrike Von Luxburg. Design and analysis of the NIPS 2016 re- view process. The Journal of Machine Learning Research, 19(1):1913-1946, 2018.","title":"Design and analysis of the NIPS 2016 review process","authors":"B Nihar, Behzad Shah, Krikamol Tabibian, Isabelle Muandet, Ulrike Guyon, Luxburg Von","year":"2018","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang and Shah, 2019] Jingyan Wang and Nihar B Shah. Your 2 is my 1, your 3 is my 9: Handling arbitrary miscali- brations in ratings. In Proceedings of the18th International Joint Conference on Autonomous Agents and Multi-Agent Systems (AAMAS), pages 864-872, 2019.","title":"Your 2 is my 1, your 3 is my 9: Handling arbitrary miscalibrations in ratings","authors":"Wang, Shah, Jingyan Wang, Nihar B Shah","year":"2019","venue":"Proceedings of the18th International Joint Conference on Autonomous Agents and Multi-Agent Systems (AAMAS)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xia, 2019] Lirong Xia. Learning and Decision-Making from Rank Data. Synthesis Lectures on Artificial Intelli- gence and Machine Learning. Morgan and Claypool, Jan- uary 2019.","title":"Learning and Decision-Making from Rank Data","authors":"Lirong Xia","year":"2019","venue":"Synthesis Lectures on Artificial Intelligence and Machine Learning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xu et al., 2019] Yichong Xu, Han Zhao, Xiaofei Shi, and Nihar B. Shah. On strategyproof conference peer review. In Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI), pages 616-622, Macau, August 2019.","title":"On strategyproof conference peer review","authors":"Xu","year":"2019","venue":"Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2019/87"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}