{"type":"ScholarlyArticle","title":"Measuring Performance of Peer Prediction Mechanisms Using Replicator Dynamics","authors":[{"name":"Victor Shnayder","slug":"victor-shnayder","given":"Victor","family":"Shnayder","affiliation":""},{"name":"Rafael M. Frongillo","slug":"rafael-m-frongillo","given":"Rafael M.","family":"Frongillo","affiliation":""},{"name":"David C. Parkes","slug":"david-c-parkes","given":"David C.","family":"Parkes","affiliation":""}],"abstract":"Peer prediction is the problem of eliciting private, but correlated, information from agents. By rewarding an agent for the amount that their report predicts that of another agent, mechanisms can promote effort and truthful reports. A common concern in peer prediction is the multiplicity of equilibria, perhaps including high-payoff equilibria that reveal no information. Rather than assume agents counter-speculate and compute an equilibrium, we adopt replicator dynamics as a model for population learning. We take the size of the basin of attraction of the truthful equilibrium as a proxy for the robustness of truthful play. We study different mechanism designs, using models estimated from real peer evaluations in several massive on-line courses. Among other observations, we confirm that recent mechanisms present a significant improvement in robustness over earlier approaches.","doi":null,"url":"https://ijcai.org/proceedings/2016/371","pdf":"https://ijcai.org/proceedings/2016/0371.pdf","markdown":"https://ijcai.org/proceedings/2016/371.md","bibtex":"https://ijcai.org/proceedings/2016/bibtex/371","volume":{"slug":"2016","title":"Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2016","url":"https://ijcai.org/proceedings/2016/","year":2016,"location":"New York, NY, USA","dates":"9–15 July 2016","editors":"Subbarao Kambhampati","isbn":"978-1-57735-770-4; 978-1-57735-771-1","publisher":"AAAI Press / International Joint Conferences on Artificial Intelligence","doi_prefix":""},"section":null,"track":"","number":371,"pages":{"first":2611,"last":2616},"keywords":[],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}