Measuring Performance of Peer Prediction Mechanisms Using Replicator Dynamics

Shnayder, Victor; Frongillo, Rafael M.; Parkes, David C.

Measuring Performance of Peer Prediction Mechanisms Using Replicator Dynamics

Victor Shnayder, Rafael M. Frongillo, David C. Parkes

Download BibTeX Cite

Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence. Pages 2611-2616.

Abstract

Peer prediction is the problem of eliciting private, but correlated, information from agents. By rewarding an agent for the amount that their report predicts that of another agent, mechanisms can promote effort and truthful reports. A common concern in peer prediction is the multiplicity of equilibria, perhaps including high-payoff equilibria that reveal no information. Rather than assume agents counter-speculate and compute an equilibrium, we adopt replicator dynamics as a model for population learning. We take the size of the basin of attraction of the truthful equilibrium as a proxy for the robustness of truthful play. We study different mechanism designs, using models estimated from real peer evaluations in several massive on-line courses. Among other observations, we confirm that recent mechanisms present a significant improvement in robustness over earlier approaches.

Victor Shnayder, Rafael M. Frongillo, and David C. Parkes. 2016. Measuring Performance of Peer Prediction Mechanisms Using Replicator Dynamics. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI-16. IJCAI, 2611–2616. https://ijcai.org/proceedings/2016/371

@inproceedings{ijcai2016p371,
  title     = {Measuring Performance of Peer Prediction Mechanisms Using Replicator Dynamics},
  author    = {Shnayder, Victor and Frongillo, Rafael M. and Parkes, David C.},
  booktitle = {Proceedings of the Twenty-Fifth International Joint Conference on
               Artificial Intelligence, {IJCAI-16}},
  publisher = {AAAI Press / International Joint Conferences on Artificial Intelligence},
  editor    = {Subbarao Kambhampati},
  pages     = {2611--2616},
  year      = {2016},
  month     = {7},
  url       = {https://ijcai.org/proceedings/2016/371},
}

TY  - CPAPER
TI  - Measuring Performance of Peer Prediction Mechanisms Using Replicator Dynamics
AU  - Shnayder, Victor
AU  - Frongillo, Rafael M.
AU  - Parkes, David C.
T2  - Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI-16
PB  - AAAI Press / International Joint Conferences on Artificial Intelligence
PY  - 2016
SP  - 2611
EP  - 2616
UR  - https://ijcai.org/proceedings/2016/371
ER  -