Themis: A Fair Evaluation Platform for Computer Vision Competitions

Cai, Zinuo; Yuan, Jianyong; Hua, Yang; Song, Tao; Wang, Hao; Xue, Zhengui; Hu, Ningxin; Ding, Jonathan; Ma, Ruhui; Haghighat, Mohammad Reza; Guan, Haibing

doi:10.24963/ijcai.2021/83

Themis: A Fair Evaluation Platform for Computer Vision Competitions

Zinuo Cai, Jianyong Yuan, Yang Hua, Tao Song, Hao Wang, Zhengui Xue, Ningxin Hu, Jonathan Ding, Ruhui Ma, Mohammad Reza Haghighat, Haibing Guan

Download BibTeX Cite

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence. Main Track. Pages 599-605.

https://doi.org/10.24963/ijcai.2021/83

Keywords

Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation
Machine Learning Applications: Applications of Supervised Learning
AI Ethics, Trust, Fairness: Fairness

Abstract

It has become increasingly thorny for computer vision competitions to preserve fairness when participants intentionally fine-tune their models against the test datasets to improve their performance. To mitigate such unfairness, competition organizers restrict the training and evaluation process of participants' models. However, such restrictions introduce massive computation overheads for organizers and potential intellectual property leakage for participants. Thus, we propose Themis, a framework that trains a noise generator jointly with organizers and participants to prevent intentional fine-tuning by protecting test datasets from surreptitious manual labeling. Specifically, with the carefully designed noise generator, Themis adds noise to perturb test sets without twisting the performance ranking of participants' models. We evaluate the validity of Themis with a wide spectrum of real-world models and datasets. Our experimental results show that Themis effectively enforces competition fairness by precluding manual labeling of test sets and preserving the performance ranking of participants' models.

Zinuo Cai, Jianyong Yuan, Yang Hua, Tao Song, Hao Wang, Zhengui Xue, Ningxin Hu, Jonathan Ding, Ruhui Ma, Mohammad Reza Haghighat, and Haibing Guan. 2021. Themis: A Fair Evaluation Platform for Computer Vision Competitions. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21. IJCAI, 599–605. https://doi.org/10.24963/ijcai.2021/83

@inproceedings{ijcai2021p83,
  title     = {Themis: A Fair Evaluation Platform for Computer Vision Competitions},
  author    = {Cai, Zinuo and Yuan, Jianyong and Hua, Yang and Song, Tao and Wang, Hao and Xue, Zhengui and Hu, Ningxin and Ding, Jonathan and Ma, Ruhui and Haghighat, Mohammad Reza and Guan, Haibing},
  booktitle = {Proceedings of the Thirtieth International Joint Conference on
               Artificial Intelligence, {IJCAI-21}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Prof. Zhi-Hua Zhou},
  pages     = {599--605},
  year      = {2021},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2021/83},
  url       = {https://doi.org/10.24963/ijcai.2021/83},
}

TY  - CPAPER
TI  - Themis: A Fair Evaluation Platform for Computer Vision Competitions
AU  - Cai, Zinuo
AU  - Yuan, Jianyong
AU  - Hua, Yang
AU  - Song, Tao
AU  - Wang, Hao
AU  - Xue, Zhengui
AU  - Hu, Ningxin
AU  - Ding, Jonathan
AU  - Ma, Ruhui
AU  - Haghighat, Mohammad Reza
AU  - Guan, Haibing
T2  - Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2021
SP  - 599
EP  - 605
DO  - 10.24963/ijcai.2021/83
UR  - https://doi.org/10.24963/ijcai.2021/83
ER  -