{"type":"ScholarlyArticle","title":"Incentivizing Evaluation with Peer Prediction and Limited Access to Ground Truth (Extended Abstract)","authors":[{"name":"Alice Gao","slug":"alice-gao","given":"Alice","family":"Gao","affiliation":"University of Waterloo , Waterloo , ON , Canada"},{"name":"James Wright","slug":"james-wright","given":"James","family":"Wright","affiliation":"University of Alberta , Edmonton , AB , Canada"},{"name":"Kevin Leyton-Brown","slug":"kevin-leyton-brown","given":"Kevin","family":"Leyton-Brown","affiliation":"University of British Columbia , Vancouver , BC , Canada"}],"abstract":"In many settings, an effective way of evaluating objects of interest is to collect evaluations from dispersed individuals and to aggregate these evaluations together.  Some examples are categorizing online content and evaluating student assignments via peer grading. For this data science problem, one challenge is to motivate participants to conduct such evaluations carefully and to report them honestly, particularly when doing so is costly. Existing approaches, notably peer-prediction mechanisms, can incentivize truth telling in equilibrium.  However, they also give rise to equilibria in which agents do not pay the costs required to evaluate accurately, and hence fail to elicit useful information. We show that this problem is unavoidable whenever agents are able to coordinate using low-cost signals about the items being evaluated (e.g., text labels or pictures). We then consider ways of circumventing this problem by comparing agents' reports to ground truth, which is available in practice when there exist trusted evaluators---such as teaching assistants in the peer grading scenario---who can perform a limited number of unbiased (but noisy) evaluations. Of course, when such ground truth is available, a simpler approach is also possible: rewarding each agent based on agreement with ground truth with some probability, and unconditionally rewarding the agent otherwise. Surprisingly, we show that the simpler mechanism achieves stronger incentive guarantees given less access to ground truth than a large set of peer-prediction mechanisms.","doi":"10.24963/ijcai.2020/723","url":"https://ijcai.org/proceedings/2020/723","pdf":"https://ijcai.org/proceedings/2020/0723.pdf","markdown":"https://ijcai.org/proceedings/2020/723.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/723","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Journal track","track":"Journal track","number":723,"pages":{"first":5140,"last":5144},"keywords":["Agent-based and Multi-agent Systems: Algorithmic Game Theory"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/26874/file_26874.m3u8","length_sec":913,"provider":"hls"}],"references":[{"position":1,"raw":"[Dasgupta and Ghosh, 2013] Anirban Dasgupta and Arpita Ghosh. Crowdsourced judgement elicitation with endogenous profi- ciency. In Proceedings of the 22nd International Conference on the World Wide Web, pages 319-330, 2013.","title":"Crowdsourced judgement elicitation with endogenous proficiency","authors":"Ghosh Dasgupta, Anirban Dasgupta, Arpita Ghosh","year":"2013","venue":"Proceedings of the 22nd International Conference on the World Wide Web","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[de Alfaro et al., 2015] Luca de Alfaro, Vassilis Polychronopoulos, and Michael Shavlovsky. Incentives for truthful peer grading. UC Santa Cruz Technical Report, 2015.","title":"Incentives for truthful peer grading","authors":"Vassilis Luca De Alfaro, Michael Polychronopoulos, Shavlovsky","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Faltings et al., 2012] Boi Faltings, Jimmy J Li, and Radu Jurca. Eliciting truthful measurements from a community of sensors. In 3rd International Conference on the Internet of Things (IOT), pages 47-54. IEEE, 2012.","title":"Eliciting truthful measurements from a community of sensors","authors":"Boi Faltings, Jimmy J Li, Radu Jurca","year":"2012","venue":"3rd International Conference on the Internet of Things (IOT)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Faltings et al., 2014] Boi Faltings, Radu Jurca, Pearl Pu, and Bao Duy Tran. Incentives to counter bias in human computa- tion. In Second AAAI Conference on Human Computation and Crowdsourcing, 2014.","title":"Incentives to counter bias in human computation","authors":"Boi Faltings, Radu Jurca, Pearl Pu, Bao Duy, Tran","year":"2014","venue":"Second AAAI Conference on Human Computation and Crowdsourcing","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gao et al., 2014] Xi Alice Gao, Andrew Mao, Yiling Chen, and Ryan Prescott Adams. Trick or treat: putting peer prediction to the test. In Proceedings of the Fifteenth ACM Conference on Economics and Computation, pages 507-524. ACM, 2014.","title":"Trick or treat: putting peer prediction to the test","authors":"Gao","year":"2014","venue":"Proceedings of the Fifteenth ACM Conference on Economics and Computation","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gao et al., 2019] Xi Alice Gao, James R Wright, and Kevin Leyton-Brown. Incentivizing evaluation with peer prediction and limited access to ground truth. Artificial Intelligence, 275:618- 638, 2019.","title":"Incentivizing evaluation with peer prediction and limited access to ground truth","authors":"Gao","year":"2019","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[John et al., 2012] Leslie K John, George Loewenstein, and Drazen Prelec. Measuring the prevalence of questionable research prac- tices with incentives for truth telling. Psychological Science, 23(5):524-532, 2012.","title":"Measuring the prevalence of questionable research practices with incentives for truth telling","authors":"George Leslie K John, Drazen Loewenstein, Prelec","year":"2012","venue":"Psychological Science","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jurca and Faltings, 2005] Radu Jurca and Boi Faltings. Enforc- ing truthful strategies in incentive compatible reputation mech- anisms. Internet and Network Economics, pages 268-277, 2005.","title":"Enforcing truthful strategies in incentive compatible reputation mechanisms","authors":"Radu Jurca, Boi Faltings","year":"2005","venue":"Internet and Network Economics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jurca and Faltings, 2009] Radu Jurca and Boi Faltings. Mecha- nisms for making crowds truthful. Journal of Artificial Intelli- gence Research, 34(1):209-253, 2009.","title":"Mechanisms for making crowds truthful","authors":"Radu Jurca, Boi Faltings","year":"2009","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kamble et al., 2015] Vijay Kamble, Nihar Shah, David Marn, Abhay Parekh, and Kannan Ramachandran. Truth serums for massively crowdsourced evaluation tasks. arXiv preprint, arXiv:1507.07045, 2015.","title":"Truth serums for massively crowdsourced evaluation tasks","authors":"Vijay Kamble, Nihar Shah, David Marn, Abhay Parekh, Kannan Ramachandran","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kong et al., 2016] Yuqing Kong, Katrina Ligett, and Grant Schoenebeck. Putting peer prediction under the mi- cro(economic)scope and making truth-telling focal. arXiv preprint, arXiv:1603.07319, 2016.","title":"Putting peer prediction under the micro(economic)scope and making truth-telling focal","authors":"Yuqing Kong, Katrina Ligett, Grant Schoenebeck","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kulkarni et al., 2014] Chinmay E Kulkarni, Richard Socher, Michael S Bernstein, and Scott R Klemmer. Scaling short-answer grading by combining peer assessment with algorithmic scor- ing. In Proceedings of the First ACM Conference on Learning @Scale, pages 99-108, 2014.","title":"Scaling short-answer grading by combining peer assessment with algorithmic scoring","authors":"E Chinmay, Richard Kulkarni, Socher, Scott R Michael S Bernstein, Klemmer","year":"2014","venue":"Proceedings of the First ACM Conference on Learning @Scale","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Miller et al., 2005] Nolan Miller, Paul Resnick, and Richard Zeck- hauser. Eliciting informative feedback: The peer-prediction method. Management Science, 51(9):1359-1373, 2005.","title":"Eliciting informative feedback: The peer-prediction method","authors":"Nolan Miller, Paul Resnick, Richard Zeckhauser","year":"2005","venue":"Management Science","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Prelec, 2004] Dražen Prelec. A Bayesian truth serum for subjec- tive data. Science, 306(5695):462-466, 2004.","title":"Dražen Prelec. A Bayesian truth serum for subjective data","authors":"Prelec","year":"2004","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Radanovic and Faltings, 2013] Goran Radanovic and Boi Faltings. A robust Bayesian truth serum for non-binary signals. In Pro- ceedings of the 27th AAAI Conference on Artificial Intelligence, AAAI 2013, pages 833-839, 2013.","title":"A robust Bayesian truth serum for non-binary signals","authors":"Faltings Radanovic, Goran Radanovic, Boi Faltings","year":"2013","venue":"Proceedings of the 27th AAAI Conference on Artificial Intelligence, AAAI 2013","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Radanovic and Faltings, 2014] Goran Radanovic and Boi Faltings. Incentives for truthful information elicitation of continuous sig- nals. In Twenty-Eighth AAAI Conference on Artificial Intelli- gence, 2014.","title":"Incentives for truthful information elicitation of continuous signals","authors":"Faltings Radanovic, Goran Radanovic, Boi Faltings","year":"2014","venue":"Twenty-Eighth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Radanovic and Faltings, 2015] Goran Radanovic and Boi Faltings. Incentives for subjective evaluations with private beliefs. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.","title":"Incentives for subjective evaluations with private beliefs","authors":"Faltings Radanovic, Goran Radanovic, Boi Faltings","year":"2015","venue":"Twenty-Ninth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Radanovic et al., 2016] Goran Radanovic, Boi Faltings, and Radu Jurca. Incentives for effort in crowdsourcing using the peer truth serum. ACM Transactions on Intelligent Systems and Technology (TIST), 7(4):1-28, 2016.","title":"Incentives for effort in crowdsourcing using the peer truth serum","authors":"Goran Radanovic, Boi Faltings, Radu Jurca","year":"2016","venue":"ACM Transactions on Intelligent Systems and Technology (TIST)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Raman and Joachims, 2014] Karthik Raman and Thorsten Joachims. Methods for ordinal peer grading. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1037-1046. ACM, 2014.","title":"Methods for ordinal peer grading","authors":"Karthik Raman, Thorsten Joachims","year":"2014","venue":"Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Riley, 2014] Blake Riley. Minimum truth serums with optional predictions. In Proceedings of the 4th Workshop on Social Com- puting and User Generated Content (SC14), 2014.","title":"Minimum truth serums with optional predictions","authors":"Blake Riley","year":"2014","venue":"Proceedings of the 4th Workshop on Social Computing and User Generated Content (SC14)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shnayder et al., 2016] Victor Shnayder, Arpit Agarwal, Rafael Frongillo, and David C Parkes. Informed truthfulness in multi- task peer prediction. In Proceedings of the 2016 ACM Conference on Economics and Computation, pages 179-196. ACM, 2016.","title":"Informed truthfulness in multitask peer prediction","authors":"Victor Shnayder, Arpit Agarwal, Rafael Frongillo, David C Parkes","year":"2016","venue":"Proceedings of the 2016 ACM Conference on Economics and Computation","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Waggoner and Chen, 2014] Bo Waggoner and Yiling Chen. Out- put agreement mechanisms and common knowledge. In Proceed- ings of the 2nd AAAI Conference on Human Computation and Crowdsourcing, 2014.","title":"Output agreement mechanisms and common knowledge","authors":"Chen ; Waggoner, Bo Waggoner, Yiling Chen","year":"2014","venue":"Proceedings of the 2nd AAAI Conference on Human Computation and Crowdsourcing","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wilson, 1987] R. Wilson. Game-theoretic approaches to trad- ing processes. In Advances in Economic Theory: Fifth World Congress, pages 33-77, 1987.","title":"Game-theoretic approaches to trading processes","authors":"R Wilson","year":"1987","venue":"Advances in Economic Theory: Fifth World Congress","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Witkowski and Parkes, 2012] Jens Witkowski and David C Parkes. A robust Bayesian truth serum for small populations. In Pro- ceedings of the 26th AAAI Conference on Artificial Intelligence (AAAI'12). Association for the Advancement of Artificial Intelli- gence, 2012.","title":"A robust Bayesian truth serum for small populations","authors":"Parkes Witkowski, Jens Witkowski, David C Parkes","year":"2012","venue":"Proceedings of the 26th AAAI Conference on Artificial Intelligence (AAAI'12). Association for the Advancement of Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Witkowski and Parkes, 2013] Jens Witkowski and David C Parkes. Learning the prior in minimal peer prediction. In Proceedings of the 3rd Workshop on Social Computing and User Generated Content at the ACM Conference on Electronic Commerce, pages 39:1-39:12, 2013.","title":"Learning the prior in minimal peer prediction","authors":"Parkes Witkowski, Jens Witkowski, David C Parkes","year":"2013","venue":"Proceedings of the 3rd Workshop on Social Computing and User Generated Content at the ACM Conference on Electronic Commerce","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Witkowski et al., 2013] Jens Witkowski, Yoram Bachrach, Peter Key, and David Christopher Parkes. Dwelling on the negative: Incentivizing effort in peer prediction. In Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing, 2013.","title":"Dwelling on the negative: Incentivizing effort in peer prediction","authors":"Witkowski","year":"2013","venue":"Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wright et al., 2015] James R Wright, Chris Thornton, and Kevin Leyton-Brown. Mechanical TA: Partially automated high-stakes peer grading. In Proceedings of the 46th ACM Technical Sympo- sium on Computer Science Education, pages 96-101, 2015.","title":"Mechanical TA: Partially automated high-stakes peer grading","authors":"Chris James R Wright, Kevin Thornton, Leyton-Brown","year":"2015","venue":"Proceedings of the 46th ACM Technical Symposium on Computer Science Education","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zarkoob et al., 2019] Hedayat Zarkoob, Hu Fu, and Kevin Leyton- Brown. Report-sensitive spot-checking in peer grading sys- tems. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 2306-2308. International Foundation for Autonomous Agents and Multiagent Systems, 2019.","title":"Report-sensitive spot-checking in peer grading systems","authors":"Hedayat Zarkoob, Hu Fu, Kevin Leyton-Brown","year":"2019","venue":"Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang and Chen, 2014] Peter Zhang and Yiling Chen. Elicitability and knowledge-free elicitation with peer prediction. In Proceed- ings of the 2014 International Conference on Autonomous Agents and Multiagent Systems, pages 245-252. International Founda- tion for Autonomous Agents and Multiagent Systems, 2014.","title":"Elicitability and knowledge-free elicitation with peer prediction","authors":"Chen ; ] Peter Zhang, Yiling Zhang, Chen","year":"2014","venue":"Proceedings of the 2014 International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}