{"type":"ScholarlyArticle","title":"TuringBox: An Experimental Platform for the Evaluation of AI Systems","authors":[{"name":"Ziv Epstein","slug":"ziv-epstein","given":"Ziv","family":"Epstein","affiliation":"MIT Media Lab"},{"name":"Blakeley H. Payne","slug":"blakeley-h-payne","given":"Blakeley H.","family":"Payne","affiliation":"MIT Media Lab"},{"name":"Judy Hanwen Shen","slug":"judy-hanwen-shen","given":"Judy Hanwen","family":"Shen","affiliation":"MIT Media Lab"},{"name":"Casey Jisoo Hong","slug":"casey-jisoo-hong","given":"Casey Jisoo","family":"Hong","affiliation":"MIT Media Lab"},{"name":"Bjarke Felbo","slug":"bjarke-felbo","given":"Bjarke","family":"Felbo","affiliation":"MIT Media Lab"},{"name":"Abhimanyu Dubey","slug":"abhimanyu-dubey","given":"Abhimanyu","family":"Dubey","affiliation":"MIT Media Lab"},{"name":"Matthew Groh","slug":"matthew-groh","given":"Matthew","family":"Groh","affiliation":"MIT Media Lab"},{"name":"Nick Obradovich","slug":"nick-obradovich","given":"Nick","family":"Obradovich","affiliation":"MIT Media Lab"},{"name":"Manuel Cebrian","slug":"manuel-cebrian","given":"Manuel","family":"Cebrian","affiliation":"MIT Media Lab"},{"name":"Iyad Rahwan","slug":"iyad-rahwan","given":"Iyad","family":"Rahwan","affiliation":"MIT Media Lab"}],"abstract":"We introduce TuringBox, a platform to democratize the study of AI. On one side of the platform, AI contributors upload existing and novel algorithms to be studied scientifically by others. On the other side, AI examiners develop and post machine intelligence tasks to evaluate and characterize the outputs of algorithms. We outline the architecture of such a platform, and describe two interactive case studies of algorithmic auditing on the platform.","doi":"10.24963/ijcai.2018/851","url":"https://ijcai.org/proceedings/2018/851","pdf":"https://ijcai.org/proceedings/2018/0851.pdf","markdown":"https://ijcai.org/proceedings/2018/851.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/851","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Demos","track":"Demos","number":851,"pages":{"first":5826,"last":5828},"keywords":["Natural Language Processing: NLP Applications and Tools","Multidisciplinary Topics and Applications: Human-Computer Interaction","Multidisciplinary Topics and Applications: AI and Social Sciences","Multidisciplinary Topics and Applications: Multidisciplinary Topics and Applications"],"videos":[],"references":[{"position":1,"raw":"[Blodgett et al., 2016] Su Lin Blodgett, Lisa Green, and Brendan T. O'Connor. Demographic dialectal variation in social media: A case study of african-american en- glish. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 1119- 1130, 2016.","title":"Demographic dialectal variation in social media: A case study of african-american english","authors":"Lin Su, Lisa Blodgett, Brendan T Green, O'connor","year":"2016","venue":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bolukbasi et al., 2016] Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman is to home- maker? debiasing word embeddings. In Advances in Neu- ral Information Processing Systems, pages 4349-4357, 2016.","title":"Man is to computer programmer as woman is to homemaker? debiasing word embeddings","authors":"Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, Adam T Kalai","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Buolamwini and Gebru, 2018] Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy dispari- ties in commercial gender classification. In Conference on Fairness, Accountability and Transparency, pages 77-91, 2018.","title":"Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification","authors":"Gebru Buolamwini","year":"2018","venue":"Conference on Fairness, Accountability and Transparency","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Friedman and Nissenbaum, 1996] Batya Friedman and He- len Nissenbaum. Bias in computer systems. ACM Trans. Inf. Syst., 14(3):330-347, July 1996.","title":"Bias in computer systems","authors":"Nissenbaum Friedman, Batya Friedman, Helen Nissenbaum","year":"1996","venue":"ACM Trans. Inf. Syst","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hannak et al., 2014] Aniko Hannak, Gary Soeller, David Lazer, Alan Mislove, and Christo Wilson. Measuring Price Discrimination and Steering on E-commerce Web Sites. In Proceedings of the ACM Internet Measurement Confer- ence (IMC'14), Vancouver, Canada, Nov 2014.","title":"Measuring Price Discrimination and Steering on E-commerce Web Sites","authors":"Aniko Hannak, Gary Soeller, David Lazer, Alan Mislove, Christo Wilson","year":"2014","venue":"Proceedings of the ACM Internet Measurement Conference (IMC'14)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hutson, 2018] Matthew Hutson. Artificial intelligence faces reproducibility crisis, 2018.","title":"Artificial intelligence faces reproducibility crisis","authors":"Matthew Hutson","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Larson et al., 2016] J Larson, S Mattu, L Kirchner, and J Angwin. How we analyzed the compas recidivism al- gorithm. propublica, 2016.","title":"How we analyzed the compas recidivism algorithm","authors":"S Larson, L Mattu, Kirchner, Angwin","year":"2016","venue":"propublica","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Larson et al., 2017] J Larson, J Angwin, L Kirchner, and S Mattu. How we examined racial discrimination in auto insurance prices. propublica, 2017.","title":"How we examined racial discrimination in auto insurance prices","authors":"J Larson, L Angwin, Kirchner, Mattu","year":"2017","venue":"propublica","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[O'Neil, 2017] Cathy O'Neil. Weapons of math destruction: How big data increases inequality and threatens democ- racy. Broadway Books, 2017.","title":"Weapons of math destruction: How big data increases inequality and threatens democracy","authors":"O' Cathy, Neil","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Sweeney, 2013] Latanya Sweeney. Discrimination in online ad delivery. Queue, 11(3):10, 2013.","title":"Discrimination in online ad delivery","authors":"Latanya Sweeney, Sweeney","year":"2013","venue":"Queue","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Zhao et al., 2017] Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Men also like shop- ping: Reducing gender bias amplification using corpus- level constraints. In Proceedings of the 2017 Confer- ence on Empirical Methods in Natural Language Process- ing, pages 2979-2989, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.","title":"Men also like shopping: Reducing gender bias amplification using corpuslevel constraints","authors":"Zhao","year":"2017","venue":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}