{"type":"ScholarlyArticle","title":"Sign-to-Speech Model for Sign Language Understanding: A Case Study of Nigerian Sign Language","authors":[{"name":"Steven Kolawole","slug":"steven-kolawole","given":"Steven","family":"Kolawole","affiliation":""},{"name":"Opeyemi Osakuade","slug":"opeyemi-osakuade","given":"Opeyemi","family":"Osakuade","affiliation":"Data Science Nigeria"},{"name":"Nayan Saxena","slug":"nayan-saxena","given":"Nayan","family":"Saxena","affiliation":""},{"name":"Babatunde Kazeem Olorisade","slug":"babatunde-kazeem-olorisade","given":"Babatunde Kazeem","family":"Olorisade","affiliation":"Cardiff Metropolitan University"}],"abstract":"Through this paper, we seek to reduce the communication barrier between the hearing-impaired community and the larger society who are usually not familiar with sign language in the sub-Saharan region of Africa with the largest occurrences of hearing disability cases, while using Nigeria as a case study. The dataset is a pioneer dataset for the Nigerian Sign Language and was created in collaboration with relevant stakeholders. We pre-processed the data in readiness for two different object detection models and a classification model and employed diverse evaluation metrics to gauge model performance on sign-language to text conversion tasks. Finally, we convert the predicted sign texts to speech and deploy the best performing model in a lightweight application that works in real-time and achieves impressive results converting sign words/phrases to text and subsequently, into speech.","doi":"10.24963/ijcai.2022/855","url":"https://ijcai.org/proceedings/2022/855","pdf":"https://ijcai.org/proceedings/2022/0855.pdf","markdown":"https://ijcai.org/proceedings/2022/855.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/855","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Demo Track","track":"Demo Track","number":855,"pages":{"first":5924,"last":5927},"keywords":["Computer Vision: Recognition (object detection, categorization)","AI Ethics, Trust, Fairness: Fairness & Diversity","AI Ethics, Trust, Fairness: Societal Impact of AI","Computer Vision: Biometrics, Face, Gesture and Pose Recognition"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987217","length_sec":64,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985360","length_sec":355,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Everingham et al., 2010] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zis- serman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303-338, 2010.","title":"The pascal visual object classes (voc) challenge","authors":"Mark Everingham, Luc Van Gool, K I Christopher, John Williams, Andrew Winn, Zisserman","year":"2010","venue":"International journal of computer vision","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Li et al., 2020] Dongxu Li, Cristian Rodriguez, Xin Yu, and Hongdong Li. Word-level deep sign language recogni- tion from video: A new large-scale dataset and methods comparison. In Proceedings of the IEEE/CVF winter con- ference on applications of computer vision, pages 1459- 1469, 2020.","title":"Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison","authors":"Li","year":"2020","venue":"Proceedings of the IEEE/CVF winter conference on applications of computer vision","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In European conference on com- puter vision, pages 740-755. Springer, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Liu et al., 2016] Wei Liu, Dragomir Anguelov, Dumitru Er- han, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference on computer vision, pages 21-37. Springer, 2016.","title":"Ssd: Single shot multibox detector","authors":"Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, Alexander C Berg","year":"2016","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Odartey et al., 2019] Lamptey K Odartey, Yonfeng Huang, Effah E Asantewaa, and Promise R Agbedanu. Ghanaian sign language recognition using deep learning. In Pro- ceedings of the 2019 the International Conference on Pat- tern Recognition and Artificial Intelligence, pages 81-86, 2019.","title":"Ghanaian sign language recognition using deep learning","authors":"Yonfeng Lamptey K Odartey, Effah E Huang, Promise R Asantewaa, Agbedanu","year":"2019","venue":"Proceedings of the 2019 the International Conference on Pattern Recognition and Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Organization and others, 2018] World Health Organization et al. Addressing the rising prevalence of hearing loss. 2018.","title":"Addressing the rising prevalence of hearing loss","authors":null,"year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Redmon et al., 2016] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 779-788, 2016.","title":"You only look once: Unified, real-time object detection","authors":"Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Sandler et al., 2018] Mark Sandler, Andrew Howard, Men- glong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018.","title":"Mobilenetv2: Inverted residuals and linear bottlenecks","authors":"Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, Liang-Chieh Chen","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}