{"type":"ScholarlyArticle","title":"On Strategyproof Conference Peer Review","authors":[{"name":"Yichong Xu","slug":"yichong-xu","given":"Yichong","family":"Xu","affiliation":"Machine Learning Department , Carnegie Mellon University , Pittsburgh , PA , USA"},{"name":"Han Zhao","slug":"han-zhao","given":"Han","family":"Zhao","affiliation":"Machine Learning Department , Carnegie Mellon University , Pittsburgh , PA , USA"},{"name":"Xiaofei Shi","slug":"xiaofei-shi","given":"Xiaofei","family":"Shi","affiliation":"Department of Mathematical Sciences , Carnegie Mellon University , Pittsburgh , PA , USA"},{"name":"Nihar B. Shah","slug":"nihar-b-shah","given":"Nihar B.","family":"Shah","affiliation":"Machine Learning Department , Carnegie Mellon University , Pittsburgh , PA , USA"}],"abstract":"We consider peer review under a conference setting where there are conflicts between the reviewers and the submissions. Under such conflicts, reviewers can manipulate their reviews in a strategic manner to influence the final rankings of their own papers. Present-day peer-review systems are not designed to guard against such strategic behavior, beyond minimal (and insufficient) checks such as not assigning a paper to a conflicted reviewer. In this work, we address this problem through the lens of social choice, and present a theoretical framework for strategyproof and efficient peer review. Given the conflict graph which satisfies a simple property, we first present and analyze a flexible framework for reviewer-assignment and aggregation for the reviews that guarantees not only strategyproofness but also a natural efficiency property (unanimity). Our framework is based on the so-called partitioning method, and can be treated as a generalization of this type of method to conference peer review settings. We then empirically show that the requisite property on the (authorship) conflict graph is indeed satisfied in the ICLR-17 submissions data, and further demonstrate a simple trick to make the partitioning method more practically appealing under conference peer-review settings. Finally, we complement our positive results with negative theoretical results where we prove that under slightly stronger requirements, it is impossible for any algorithm to be both strategyproof and efficient.","doi":"10.24963/ijcai.2019/87","url":"https://ijcai.org/proceedings/2019/87","pdf":"https://ijcai.org/proceedings/2019/0087.pdf","markdown":"https://ijcai.org/proceedings/2019/87.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/87","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Agent-based and Multi-agent Systems","track":"Main track","number":87,"pages":{"first":616,"last":622},"keywords":["Agent-based and Multi-agent Systems: Voting","Agent-based and Multi-agent Systems: Computational Social Choice"],"videos":[],"references":[{"position":1,"raw":"[Alon et al., 2011] Noga Alon, Felix Fischer, Ariel Procac- cia, and Moshe Tennenholtz. Sum of us: Strategyproof selection from the selectors. In TARK. ACM, 2011.","title":"Sum of us: Strategyproof selection from the selectors","authors":"Noga Alon, Felix Fischer, Ariel Procaccia, Moshe Tennenholtz","year":"2011","venue":"TARK","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anderson et al., 2007] Melissa S Anderson, Emily A Ron- ning, Raymond De Vries, and Brian C Martinson. The perverse effects of competition on scientists' work and relationships. Science and engineering ethics, 2007.","title":"The perverse effects of competition on scientists' work and relationships","authors":"Emily A Melissa S Anderson, Raymond De Ronning, Brian C Vries, Martinson","year":"2007","venue":"Science and engineering ethics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Arrow, 1950] Kenneth J Arrow. A difficulty in the concept of social welfare. Journal of political economy, 1950.","title":"A difficulty in the concept of social welfare","authors":"J Kenneth, Arrow","year":"1950","venue":"Journal of political economy","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Aziz et al., 2016] Haris Aziz, Omer Lev, Nicholas Mattei, Jeffrey S Rosenschein, and Toby Walsh. Strategyproof peer selection: Mechanisms, analyses, and experiments. In AAAI, pages 397-403, 2016.","title":"Strategyproof peer selection: Mechanisms, analyses, and experiments","authors":"Haris Aziz, Omer Lev, Nicholas Mattei, Toby Jeffrey S Rosenschein, Walsh","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Barnett, 2003] William Barnett. The modern theory of con- sumer behavior: Ordinal or cardinal? The Quarterly Jour- nal of Austrian Economics, 6(1):41-65, 2003.","title":"The modern theory of consumer behavior: Ordinal or cardinal?","authors":"William Barnett","year":"2003","venue":"The Quarterly Journal of Austrian Economics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Brandt et al., 2016] Felix Brandt, Vincent Conitzer, Ulle En- driss, Ariel D Procaccia, and Jérôme Lang. Handbook of computational social choice. 2016.","title":"Handbook of computational social choice","authors":"Felix Brandt, Vincent Conitzer, Ulle Endriss, Ariel D Procaccia, Jérôme Lang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Caragiannis et al., 2017] Ioannis Caragiannis, Xenophon Chatzigeorgiou, George A Krimpas, and Alexandros A Voudouris. Optimizing positional scoring rules for rank aggregation. In AAAI, pages 430-436, 2017.","title":"Optimizing positional scoring rules for rank aggregation","authors":"Ioannis Caragiannis, Xenophon Chatzigeorgiou, George A Krimpas, Alexandros A Voudouris","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Charlin and Zemel, 2013] L. Charlin and R. S. Zemel. The Toronto Paper Matching System: An automated paper- reviewer assignment system, 2013.","title":"The Toronto Paper Matching System: An automated paperreviewer assignment system","authors":"L Charlin, R S Zemel","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Díez Peláez et al., 2013] Jorge Díez Peláez, Óscar Lu- aces Rodríguez, Amparo Alonso Betanzos, Alicia Tron- coso, and Antonio Bahamonde Rionda. Peer assessment in moocs using preference learning via matrix factorization. In NIPS Workshop on Data Driven Education, 2013.","title":"Peer assessment in moocs using preference learning via matrix factorization","authors":"Díez Peláez","year":"2013","venue":"NIPS Workshop on Data Driven Education","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dörfler et al., 2017] Florian Dörfler, Yuanzhang Xiao, and Mihaela van der Schaar. Incentive design in peer review: Rating and repeated endogenous matching. IEEE Transac- tions on Network Science and Engineering, 2017.","title":"Incentive design in peer review: Rating and repeated endogenous matching","authors":"Florian Dörfler, Yuanzhang Xiao, Mihaela Van Der Schaar","year":"2017","venue":"IEEE Transactions on Network Science and Engineering","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Douceur, 2009] John R Douceur. Paper rating vs. paper rank- ing. ACM SIGOPS Operating Systems Review, 2009.","title":"Paper rating vs. paper ranking","authors":"John R Douceur","year":"2009","venue":"ACM SIGOPS Operating Systems Review","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Emerson, 2013] Peter Emerson. The original Borda count and partial voting. Social Choice and Welfare, 2013.","title":"The original Borda count and partial voting","authors":"Peter Emerson","year":"2013","venue":"Social Choice and Welfare","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Fischer and Klimm, 2015] Felix Fischer and Max Klimm. Optimal impartial selection. SIAM Journal on Computing, 44(5):1263-1285, 2015.","title":"Felix Fischer and Max Klimm. Optimal impartial selection","authors":"Klimm Fischer","year":"2015","venue":"SIAM Journal on Computing","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Fishburn, 2015] Peter C Fishburn. The theory of social choice. Princeton University Press, 2015.","title":"The theory of social choice","authors":"Peter C Fishburn","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Garg et al., 2010] N. Garg, T. Kavitha, A. Kumar, K. Mehlhorn, and J. Mestre. Assigning papers to referees. Algorithmica, 58(1):119-136, Sep 2010.","title":"Assigning papers to referees","authors":"N Garg, T Kavitha, A Kumar, K Mehlhorn, J Mestre","year":"2010","venue":"Algorithmica","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Gibbard, 1973] Allan Gibbard. Manipulation of voting schemes: a general result. Econometrica: journal of the Econometric Society, pages 587-601, 1973.","title":"Manipulation of voting schemes: a general result","authors":"Allan Gibbard","year":"1973","venue":"Econometrica: journal of the Econometric Society","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hajek et al., 2014] Bruce Hajek, Sewoong Oh, and Jiaming Xu. Minimax-optimal inference from partial rankings. In Advances in Neural Information Processing Systems, pages 1475-1483, 2014.","title":"Minimax-optimal inference from partial rankings","authors":"Bruce Hajek, Sewoong Oh, Jiaming Xu","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Hartvigsen et al., 1999] David Hartvigsen, Jerry C. Wei, and Richard Czuchlewski. The conference paper-reviewer as- signment problem. Decision Sciences, 1999.","title":"The conference paper-reviewer assignment problem","authors":"David Hartvigsen, Jerry C Wei, Richard Czuchlewski","year":"1999","venue":"Decision Sciences","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Hazelrigg, 2013] GA Hazelrigg. Dear colleague letter: Infor- mation to principal investigators (PIs) planning to submit proposals to the Sensors and Sensing Systems (SSS) pro- gram October 1, 2013, deadline. 2013.","title":"Dear colleague letter: Information to principal investigators (PIs) planning to submit proposals to the Sensors and Sensing Systems","authors":"; Ga Hazelrigg, Hazelrigg","year":"2013","venue":"SSS) program","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Holzman and Moulin, 2013] Ron Holzman and Hervé Moulin. Impartial nominations for a prize. Econometrica, 81(1):173-196, 2013.","title":"Impartial nominations for a prize","authors":"Moulin Holzman, Ron Holzman, Hervé Moulin","year":"2013","venue":"Econometrica","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Kahng et al., 2018] Anson Kahng, Yasmine Kotturi, Chin- may Kulkarni, David Kurokawa, and Ariel D Procaccia. Ranking wily people who rank each other. In AAAI, 2018.","title":"Ranking wily people who rank each other","authors":"Anson Kahng, Yasmine Kotturi, Chinmay Kulkarni, David Kurokawa, Ariel D Procaccia","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Kurokawa et al., 2015] David Kurokawa, Omer Lev, Jamie Morgenstern, and Ariel D Procaccia. Impartial peer review. In IJCAI, pages 582-588, 2015.","title":"Impartial peer review","authors":"David Kurokawa, Omer Lev, Jamie Morgenstern, Ariel D Procaccia","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Merrifield and Saari, 2009] Michael R Merrifield and Don- ald G Saari. Telescope time without tears: a distributed approach to peer review. Astronomy & Geophysics, 2009.","title":"Telescope time without tears: a distributed approach to peer review","authors":"R Michael, Donald G Merrifield, Saari","year":"2009","venue":"Astronomy & Geophysics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Piech et al., 2013] Chris Piech, Jonathan Huang, Zhenghao Chen, Chuong Do, Andrew Ng, and Daphne Koller. Tuned models of peer assessment in moocs. arXiv preprint arXiv:1307.2579, 2013.","title":"Tuned models of peer assessment in moocs","authors":"Chris Piech, Jonathan Huang, Zhenghao Chen, Chuong Do, Andrew Ng, Daphne Koller","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Satterthwaite, 1975] Mark Allen Satterthwaite. Strategy- proofness and arrow's conditions: Existence and correspon- dence theorems for voting procedures and social welfare functions. Journal of economic theory, 1975.","title":"Strategyproofness and arrow's conditions: Existence and correspondence theorems for voting procedures and social welfare functions","authors":"Mark Allen, Satterthwaite","year":"1975","venue":"Journal of economic theory","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Shah et al., 2013] Nihar B Shah, Joseph K Bradley, Abhay Parekh, Martin Wainwright, and Kannan Ramchandran. A case for ordinal peer-evaluation in moocs. In NIPS Work- shop on Data Driven Education, 2013.","title":"A case for ordinal peer-evaluation in moocs","authors":"B Nihar, Joseph K Shah, Abhay Bradley, Martin Parekh, Kannan Wainwright, Ramchandran","year":"2013","venue":"NIPS Workshop on Data Driven Education","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Shah et al., 2016] Nihar B Shah, Sivaraman Balakrishnan, Joseph Bradley, Abhay Parekh, Kannan Ramchandran, and Martin J Wainwright. Estimation from pairwise compar- isons: Sharp minimax bounds with topology dependence. The Journal of Machine Learning Research, 2016.","title":"Estimation from pairwise comparisons: Sharp minimax bounds with topology dependence","authors":"B Nihar, Sivaraman Shah, Joseph Balakrishnan, Abhay Bradley, Kannan Parekh, Martin J Ramchandran, Wainwright","year":"2016","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Shah et al., 2017] Nihar B Shah, Behzad Tabibian, Krikamol Muandet, Isabelle Guyon, and Ulrike von Luxburg. Design and Analysis of the NIPS 2016 Review Process. arXiv preprint arXiv:1708.09794, 2017.","title":"Design and Analysis of the NIPS 2016 Review Process","authors":"B Nihar, Behzad Shah, Krikamol Tabibian, Isabelle Muandet, Ulrike Guyon, Luxburg Von","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Stelmakh et al., 2018] Ivan Stelmakh, Nihar B Shah, and Aarti Singh. PeerReview4All: Fair and accurate reviewer assignment in peer review. 2018.","title":"PeerReview4All: Fair and accurate reviewer assignment in peer review","authors":"Ivan Stelmakh, B Nihar, Aarti Shah, Singh","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Stewart et al., 2005] Neil Stewart, Gordon DA Brown, and Nick Chater. Absolute identification by relative judgment. Psychological review, 112(4):881, 2005.","title":"Absolute identification by relative judgment","authors":"Neil Stewart, Gordon Da Brown, Nick Chater","year":"2005","venue":"Psychological review","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Thurner and Hanel, 2011] Stefan Thurner and Rudolf Hanel. Peer-review in a world with rational scientists: Toward selection of the average. The European Physical Journal B, 84(4):707-711, 2011.","title":"Peer-review in a world with rational scientists: Toward selection of the average","authors":"Hanel Thurner, Stefan Thurner, Rudolf Hanel","year":"2011","venue":"The European Physical Journal B","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Tsukida and Gupta, 2011] Kristi Tsukida and Maya R Gupta. How to analyze paired comparison data. Technical report, DTIC Document, 2011.","title":"How to analyze paired comparison data","authors":"Kristi Tsukida, Maya R Gupta","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Xu et al., 2018] Yichong Xu, Han Zhao, Xiaofei Shi, and Nihar B Shah. On Strategyproof Conference Peer Review. arXiv preprint arXiv:1806.06266, 2018.","title":"On Strategyproof Conference Peer Review","authors":"Xu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}