{"type":"ScholarlyArticle","title":"SaGol: Using MiniGPT-4 to Generate Alt Text for Improving Image Accessibility","authors":[{"name":"Yunseo Moon","slug":"yunseo-moon","given":"Yunseo","family":"Moon","affiliation":"University of Seoul , Seoul , Republic of Korea"},{"name":"Hyunmin Lee","slug":"hyunmin-lee","given":"Hyunmin","family":"Lee","affiliation":"University of Seoul , Seoul , Republic of Korea"},{"name":"SeungYoung Oh","slug":"seungyoung-oh","given":"SeungYoung","family":"Oh","affiliation":"University of Seoul , Seoul , Republic of Korea"},{"name":"Hyunggu Jung","slug":"hyunggu-jung","given":"Hyunggu","family":"Jung","affiliation":"University of Seoul , Seoul , Republic of Korea"}],"abstract":"SaGol is an AI-powered application to improve image accessibility for people with visual impairments (PVI) users. Alternative (alt) text, a general method of web accessibility for PVI users, is text or phrases that describe images on a website in an understandable way. SaGol generates alt text with the images on the user's smartphone using a vision large language model called MiniGPT-4. SaGol searches for similar images based on the generated alt text. We evaluated the length of alt text and the search accuracy. This paper shows a potential opportunity to improve image accessibility for PVI users.","doi":"10.24963/ijcai.2024/1023","url":"https://ijcai.org/proceedings/2024/1023","pdf":"https://ijcai.org/proceedings/2024/1023.pdf","markdown":"https://ijcai.org/proceedings/2024/1023.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/1023","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Demo Track","track":"Demo Track","number":1023,"pages":{"first":8745,"last":8748},"keywords":["Humans and AI: HAI: Human-computer interaction","Humans and AI: HAI: Intelligent user interfaces","Humans and AI: HAI: Applications"],"videos":[],"references":[{"position":1,"raw":"[Aizawa, 2003] Akiko N. Aizawa. An information-theoretic perspective of tf-idf measures. Information Processing Management, 39(1):45-65, 2003.","title":"An information-theoretic perspective of tf-idf measures","authors":"Akiko N Aizawa","year":"2003","venue":"Information Processing Management","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cho and Kim, 2023] Jaemin Cho and Hee Jae Kim. Dimen- sional alt text: Enhancing spatial understanding through dimensional layering of image descriptions for screen reader users. In Extended Abstracts of the 2023 CHI Con- ference on Human Factors in Computing Systems, CHI EA '23, New York, NY, USA, 2023. Association for Comput- ing Machinery. [","title":"Dimensional alt text: Enhancing spatial understanding through dimensional layering of image descriptions for screen reader users","authors":"Jaemin Cho, Hee Jae Kim","year":"2023","venue":"Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, CHI EA '23","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chollet and others, 2015] Franc ¸ois Chollet et al. Keras. https://keras.io, 2015. Accessed: 2024-02-07.","title":"Franc ¸ois Chollet","authors":"Chollet","year":"2015","venue":null,"doi":null,"url":"https://keras.io,2015","internal_url":null},{"position":4,"raw":"[Gleason et al., 2019] Cole Gleason, Amy Pavel, Xingyu Liu, Patrick Carrington, Lydia B. Chilton, and Jeffrey P. Bigham. Making memes accessible. In Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS '19, page 367-376, New York, NY, USA, 2019. Association for Computing Machinery.","title":"Making memes accessible","authors":"Cole Gleason, Amy Pavel, Xingyu Liu, Patrick Carrington, Lydia B Chilton, Jeffrey P Bigham","year":"2019","venue":"Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS '19","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gleason et al., 2020] Cole Gleason, Amy Pavel, Emma Mc- Camey, Christina Low, Patrick Carrington, Kris M. Kitani, and Jeffrey P. Bigham. Twitter a11y: A browser extension to make twitter images accessible. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI '20, page 1-12, New York, NY, USA, 2020. Association for Computing Machinery.","title":"Twitter a11y: A browser extension to make twitter images accessible","authors":"Cole Gleason, Amy Pavel, Emma Mc-Camey, Christina Low, Patrick Carrington, Kris M Kitani, Jeffrey P Bigham","year":"2020","venue":"Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI '20","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Guinness et al., 2018] Darren Guinness, Edward Cutrell, and Meredith Ringel Morris. Caption crawler: Enabling reusable alternative text descriptions using reverse image search. In Proceedings of the 2018 CHI Conference on Hu- man Factors in Computing Systems, CHI '18, page 1-11, New York, NY, USA, 2018. Association for Computing Machinery.","title":"Caption crawler: Enabling reusable alternative text descriptions using reverse image search","authors":"Darren Guinness, Edward Cutrell, Meredith Ringel, Morris","year":"2018","venue":"Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI '18","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), June 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Howard et al., 2017] Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mo- bilenets: Efficient convolutional neural networks for mo- bile vision applications, 2017.","title":"Mobilenets: Efficient convolutional neural networks for mobile vision applications","authors":"Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, Hartwig Adam","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[James Edwards et al., 2021] Emory James Edwards, Kyle Lewis Polster, Isabel Tuason, Emily Blank, Michael Gilbert, and Stacy Branham. \"that's in the eye of the beholder\": Layers of interpretation in image descriptions for fictional representations of people with disabilities. In Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS '21, New York, NY, USA, 2021. Association for Computing Machinery.","title":"that's in the eye of the beholder\": Layers of interpretation in image descriptions for fictional representations of people with disabilities","authors":"James Edwards","year":"2021","venue":"Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS '21","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jobin et al., 2022] K. V. Jobin, Ajoy Mondal, and C. V. Jawahar. Classroom slide narration system. In Bal- asubramanian Raman, Subrahmanyam Murala, Ananda Chowdhury, Abhinav Dhall, and Puneet Goyal, editors, Computer Vision and Image Processing, pages 135-146, Cham, 2022. Springer International Publishing.","title":"Classroom slide narration system","authors":"K V Jobin, Ajoy Mondal, C V Jawahar","year":"2022","venue":"Computer Vision and Image Processing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jung et al., 2022] Crescentia Jung, Shubham Mehta, Atharva Kulkarni, Yuhang Zhao, and Yea-Seul Kim. Communicating visualizations without visuals: Investiga- tion of visualization alternative text for people with visual impairments. IEEE Transactions on Visualization and Computer Graphics, 28(1):1095-1105, 2022.","title":"Communicating visualizations without visuals: Investigation of visualization alternative text for people with visual impairments","authors":"Crescentia Jung, Shubham Mehta, Atharva Kulkarni, Yuhang Zhao, Yea-Seul Kim","year":"2022","venue":"IEEE Transactions on Visualization and Computer Graphics","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kirkpatrick et al., 2018] Andrew Kirkpatrick, Joshue O. Connor, Alastair Campbell, and Michael Cooper. Web content accessibility guidelines (wcag) 2.1. https://www. w3.org/TR/WCAG21/, 2018. Accessed: 2024-02-04.","title":"Web content accessibility guidelines","authors":"Andrew Kirkpatrick, Joshue O Connor, Alastair Campbell, Michael Cooper","year":"2024","venue":null,"doi":null,"url":"https://www.w3.org/TR/WCAG21/,2018","internal_url":null},{"position":13,"raw":"[Li et al., 2022] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Ma- chine Learning Research, pages 12888-12900. PMLR, 17-23 Jul 2022.","title":"BLIP: Bootstrapping language-image pretraining for unified vision-language understanding and generation","authors":"Li","year":"2022","venue":"Proceedings of the 39th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Loper and Bird, 2002] Edward Loper and Steven Bird. Nltk: The natural language toolkit, 2002.","title":"Nltk: The natural language toolkit","authors":"Bird ; Loper, Edward Loper, Steven Bird","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pareddy et al., 2019] Sujeath Pareddy, Anhong Guo, and Jeffrey P. Bigham. X-ray: Screenshot accessibility via em- bedded metadata. In Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessi- bility, ASSETS '19, page 389-395, New York, NY, USA, 2019. Association for Computing Machinery.","title":"X-ray: Screenshot accessibility via embedded metadata","authors":"Pareddy","year":"2019","venue":"Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS '19","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Szegedy et al., 2016] Christian Szegedy, Vincent Van- houcke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.","title":"Rethinking the inception architecture for computer vision","authors":"Szegedy","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Veit et al., 2016] Andreas Veit, Tomas Matera, Lukas Neu- mann, Jiri Matas, and Serge Belongie. Coco-text: Dataset and benchmark for text detection and recognition in natu- ral images, 2016.","title":"Coco-text: Dataset and benchmark for text detection and recognition in natural images","authors":"Andreas Veit, Tomas Matera, Lukas Neumann, Jiri Matas, Serge Belongie","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Zhang et al., 2022] Mingrui Ray Zhang, Mingyuan Zhong, and Jacob O. Wobbrock. Ga11y: An automated gif anno- tation system for visually impaired users. In Proceedings of the 2022 CHI Conference on Human Factors in Com- puting Systems, CHI '22, New York, NY, USA, 2022. As- sociation for Computing Machinery.","title":"Ga11y: An automated gif annotation system for visually impaired users","authors":"Zhang","year":"2022","venue":"Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI '22","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zhu et al., 2023] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xi- ang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large lan- guage models, 2023.","title":"Minigpt-4: Enhancing vision-language understanding with advanced large language models","authors":"Zhu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}