{"type":"ScholarlyArticle","title":"Evaluating and Complementing Vision-to-Language Technology for People who are Blind with Conversational Crowdsourcing","authors":[{"name":"Elliot Salisbury","slug":"elliot-salisbury","given":"Elliot","family":"Salisbury","affiliation":"University of Southampton"},{"name":"Ece Kamar","slug":"ece-kamar","given":"Ece","family":"Kamar","affiliation":"Microsoft Research"},{"name":"Meredith Ringel Morris","slug":"meredith-ringel-morris","given":"Meredith Ringel","family":"Morris","affiliation":"Microsoft Research"}],"abstract":"We study how real-time crowdsourcing can be used both for evaluating the value provided by existing automated approaches and for enabling workflows that provide scalable and useful alt text to blind users. We show that the shortcomings of existing AI image captioning systems frequently hinder a user's understanding of an image they cannot see to a degree that even clarifying conversations with sighted assistants cannot correct. Based on analysis of clarifying conversations collected from our studies, we design experiences that can effectively assist users in a scalable way without the need for real-time interaction. Our results provide lessons and guidelines that the designers of future AI captioning systems can use to improve labeling of social media imagery for blind users.","doi":"10.24963/ijcai.2018/751","url":"https://ijcai.org/proceedings/2018/751","pdf":"https://ijcai.org/proceedings/2018/0751.pdf","markdown":"https://ijcai.org/proceedings/2018/751.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/751","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Sister Conferences Best Papers","track":"Best Sister Conferences","number":751,"pages":{"first":5349,"last":5353},"keywords":["Humans and AI: Human Computation and Crowdsourcing","Humans and AI: Human-AI Collaboration","Natural Language Processing: Resources and Evaluation","Humans and AI: Ethical Issues in AI"],"videos":[],"references":[{"position":1,"raw":"[Bigham et al., 2006] Jeffrey P Bigham, Ryan S Kaminsky, Richard E Ladner, Oscar M Danielsson, and Gordon L Hempton. Webinsight:: making web images accessible. In Proceedings of the 8th international ACM SIGACCESS conference on Computers and accessibility, pages 181- 188. ACM, 2006.","title":"Webinsight:: making web images accessible","authors":"Ryan S Jeffrey P Bigham, Richard E Kaminsky, Oscar M Ladner, Gordon L Danielsson, Hempton","year":"2006","venue":"Proceedings of the 8th international ACM SIGACCESS conference on Computers and accessibility","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bigham et al., 2010] Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. Vizwiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM sym- posium on User interface software and technology, pages 333-342. ACM, 2010.","title":"Vizwiz: nearly real-time answers to visual questions","authors":"Chandrika Jeffrey P Bigham, Hanjie Jayant, Greg Ji, Andrew Little, Robert C Miller, Robin Miller, Aubrey Miller, Brandyn Tatarowicz, Samual White, White","year":"2010","venue":"Proceedings of the 23nd annual ACM symposium on User interface software and technology","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brady et al., 2015] Erin Brady, Meredith Ringel Morris, and Jeffrey P Bigham. Gauging receptiveness to social microvolunteering. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Sys- tems, pages 1055-1064. ACM, 2015.","title":"Gauging receptiveness to social microvolunteering","authors":"Erin Brady, Meredith Ringel Morris, Jeffrey P Bigham","year":"2015","venue":"Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Duggan et al., 2015] Maeve Duggan, Nicole B Ellison, Cliff Lampe, Amanda Lenhart, and Mary Madden. Social media update 2014. pew research center, 2015.","title":"Social media update 2014","authors":"Maeve Duggan, Nicole B Ellison, Cliff Lampe, Amanda Lenhart, Mary Madden","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fang et al., 2015] Hao Fang, Saurabh Gupta, Forrest Ian- dola, Rupesh K Srivastava, Li Deng, Piotr Dollár, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C Platt, et al. From captions to visual concepts and back. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1473-1482, 2015.","title":"From captions to visual concepts and back","authors":"Saurabh Hao Fang, Forrest Gupta, Iandola, K Rupesh, Li Srivastava, Piotr Deng, Jianfeng Dollár, Xiaodong Gao, Margaret He, John C Mitchell, Platt","year":"2015","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Goodwin et al., 2011] Morten Goodwin, Deniz Susar, An- nika Nietzio, Mikael Snaprud, and Christian S Jensen. Global web accessibility analysis of national government portals and ministry web sites. Journal of Information Technology & Politics, 8(1):41-67, 2011.","title":"Global web accessibility analysis of national government portals and ministry web sites","authors":"Goodwin","year":"2011","venue":"Journal of Information Technology & Politics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Karpathy and Fei-Fei, 2015] Andrej Karpathy and Li Fei- Fei. Deep visual-semantic alignments for generating im- age descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3128- 3137, 2015.","title":"Deep visual-semantic alignments for generating image descriptions","authors":"Fei-Fei ; Karpathy, Andrej Karpathy, Li Fei-Fei","year":"2015","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kloots, 2016] Todd Kloots. Accessible images for every- one. https://blog.twitter.com/2016/accessible-images-for- everyone, 2016.","title":"Accessible images for everyone","authors":"Todd Kloots","year":null,"venue":null,"doi":null,"url":"https://blog.twitter.com/2016/accessible-images-for-everyone,2016","internal_url":null},{"position":9,"raw":"[Lasecki et al., 2013] Walter S Lasecki, Rachel Wesley, Jef- frey Nichols, Anand Kulkarni, James F Allen, and Jef- frey P Bigham. Chorus: a crowd-powered conversational assistant. In Proceedings of the 26th annual ACM sym- posium on User interface software and technology, pages 151-162. ACM, 2013.","title":"Chorus: a crowd-powered conversational assistant","authors":"Rachel Walter S Lasecki, Jeffrey Wesley, Anand Nichols, James F Kulkarni, Jeffrey P Allen, Bigham","year":"2013","venue":"Proceedings of the 26th annual ACM symposium on User interface software and technology","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[MacLeod et al., 2017] Haley MacLeod, Cynthia L. Bennett, Meredith Ringel Morris, and Edward Cutrell. Understand- ing blind people's experiences with computer-generated captions of social media images. In Proceedings of the 2017 CHI Conference, 2017.","title":"Understanding blind people's experiences with computer-generated captions of social media images","authors":"Haley Macleod, Cynthia L Bennett, Meredith Ringel Morris, Edward Cutrell","year":"2017","venue":"Proceedings of the 2017 CHI Conference","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mao et al., 2012] Andrew Mao, Yiling Chen, Krzysztof Z Gajos, David Parkes, Ariel Procaccia, and Haoqi Zhang. Turkserver: Enabling synchronous and longitudinal online experiments. In Proceedings of the Fourth Workshop on Human Computation (HCOMP'12). AAAI Press, 2012.","title":"Turkserver: Enabling synchronous and longitudinal online experiments","authors":"Andrew Mao, Yiling Chen, Z Krzysztof, David Gajos, Ariel Parkes, Haoqi Procaccia, Zhang","year":"2012","venue":"Proceedings of the Fourth Workshop on Human Computation (HCOMP'12)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Microsoft, 2016] Microsoft. Microsoft cognitive services api. https://www.microsoft.com/cognitive-services/en- us/computer-vision-api, 2016.","title":"Microsoft. Microsoft cognitive services api","authors":null,"year":"2016","venue":null,"doi":null,"url":"https://www.microsoft.com/cognitive-services/en-us/computer-vision-api,2016","internal_url":null},{"position":13,"raw":"[Morris et al., 2016] Meredith Ringel Morris, Annuska Zolyomi, Catherine Yao, Sina Bahram, Jeffrey P Bigham, and Shaun K Kane. With most of it being pictures now, i rarely use it: Understanding twitter's evolving acces- sibility to blind users. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pages 5506-5516. ACM, 2016.","title":"With most of it being pictures now, i rarely use it: Understanding twitter's evolving accessibility to blind users","authors":"Meredith Ringel, Morris, Annuska Zolyomi, Catherine Yao, Sina Bahram, Jeffrey P Bigham, Shaun K Kane","year":"2016","venue":"Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Salisbury et al., 2017] Elliot Salisbury, Ece Kamar, and Meredith Ringel Morris. Toward scalable social alt text: Conversational crowdsourcing as a tool for refining vision- to-language technology for the blind. In Proceedings of the fifth AAAI Conference on Human Computation and Crowdsourcing, 2017.","title":"Toward scalable social alt text: Conversational crowdsourcing as a tool for refining visionto-language technology for the blind","authors":"Salisbury","year":"2017","venue":"Proceedings of the fifth AAAI Conference on Human Computation and Crowdsourcing","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"Wu et al., ] S Wu, H Pique, and J Wieland. Us- ing artificial intelligence to help blind people see facebook. http://newsroom.fb.com/news/2016/04/using- artificial-intelligence-to-help-. [Zyskowski et al., 2015]","title":"Using artificial intelligence to help blind people see facebook","authors":"Wu","year":"2015","venue":null,"doi":null,"url":"http://newsroom.fb.com/news/2016/04/using-artificial-intelligence-to-help-","internal_url":null},{"position":16,"raw":"Kathryn Zyskowski, Mered- ith Ringel Morris, Jeffrey P. Bigham, Mary L. Gray, and Shaun Kane. Accessible crowdwork? understanding the value in and challenge of microtask employment for peo- ple with disabilities. ACM: Association for Computing Machinery, March 2015.","title":"Accessible crowdwork? understanding the value in and challenge of microtask employment for people with disabilities","authors":"Kathryn Zyskowski, Meredith Ringel Morris, Jeffrey P Bigham, Mary L Gray, Shaun Kane","year":"2015","venue":"ACM: Association for Computing Machinery","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}