{"type":"ScholarlyArticle","title":"Sign Language-to-Text Dictionary with Lightweight Transformer Models","authors":[{"name":"Jérôme Fink","slug":"jerome-fink","given":"Jérôme","family":"Fink","affiliation":"Namur Digital Institute (NaDI); Namur Institute of Language , Text and Transmediality (NaLTT; University of Namur"},{"name":"Pierre Poitier","slug":"pierre-poitier","given":"Pierre","family":"Poitier","affiliation":"Namur Digital Institute (NaDI); University of Namur"},{"name":"Maxime André","slug":"maxime-andre","given":"Maxime","family":"André","affiliation":"Namur Digital Institute (NaDI); University of Namur"},{"name":"Loup Meurice","slug":"loup-meurice","given":"Loup","family":"Meurice","affiliation":"Namur Digital Institute (NaDI); University of Namur"},{"name":"Benoît Frénay","slug":"benoit-frenay","given":"Benoît","family":"Frénay","affiliation":"Namur Digital Institute (NaDI); University of Namur"},{"name":"Anthony Cleve","slug":"anthony-cleve","given":"Anthony","family":"Cleve","affiliation":"Namur Digital Institute (NaDI); University of Namur"},{"name":"Bruno Dumas","slug":"bruno-dumas","given":"Bruno","family":"Dumas","affiliation":"Namur Digital Institute (NaDI); University of Namur"},{"name":"Laurence Meurant","slug":"laurence-meurant","given":"Laurence","family":"Meurant","affiliation":"Namur Institute of Language , Text and Transmediality (NaLTT; University of Namur"}],"abstract":"The recent advances in deep learning have been beneficial to automatic sign language recognition (SLR). However, free-to-access, usable, and accessible tools are still not widely available to the deaf community. The need for a sign language-to-text dictionary was raised by a bilingual deaf school in Belgium and linguist experts in sign languages (SL) in order to improve the autonomy of students. To meet that need, an efficient SLR system was built based on a specific transformer model. The proposed system is able to recognize 700 different signs, with a top-10 accuracy of 83%. Those results are competitive with other systems in the literature while using 10 times less parameters than existing solutions. The integration of this model into a usable and accessible web application for the dictionary is also introduced. A user-centered human-computer interaction (HCI) methodology was followed to design and implement the user interface. To the best of our knowledge, this is the first publicly released sign language-to-text dictionary using video captured by a standard camera.","doi":"10.24963/ijcai.2023/662","url":"https://ijcai.org/proceedings/2023/662","pdf":"https://ijcai.org/proceedings/2023/0662.pdf","markdown":"https://ijcai.org/proceedings/2023/662.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/662","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"AI for Good","track":"AI for Good","number":662,"pages":{"first":5968,"last":5976},"keywords":["AI for Good: Multidisciplinary Topics and Applications","AI for Good: Natural Language Processing","AI for Good: Humans and AI","AI for Good: Computer Vision"],"videos":[],"references":[{"position":1,"raw":"[Albanie et al., 2020] Samuel Albanie, Gül Varol, Liliane Momeni, Triantafyllos Afouras, Joon Son Chung, Neil Fox, and Andrew Zisserman. Bsl-1k: Scaling up co- articulated sign language recognition using mouthing cues. In Computer Vision-ECCV 2020: 16th European Con- ference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pages 35-53. Springer, 2020.","title":"Bsl-1k: Scaling up coarticulated sign language recognition using mouthing cues","authors":"Albanie","year":"2020","venue":"Computer Vision-ECCV 2020: 16th European Conference","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[André, 2022] Maxime André. Recommandations pour des interfaces utilisateurs adaptées à la surdité. Master's thesis, Université de Namur, 2022.","title":"Recommandations pour des interfaces utilisateurs adaptées à la surdité","authors":"André","year":"2022","venue":"Maxime André","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ba et al., 2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Ge- offrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.","title":"Layer normalization","authors":"Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E Hinton","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Caldwell et al., 2008] Ben Caldwell, Michael Cooper, Loretta Guarino Reid, Gregg Vanderheiden, Wendy Chisholm, John Slatin, and Jason White. Web content accessibility guidelines (wcag) 2.0. WWW Consortium (W3C), 290:1-34, 2008.","title":"Web content accessibility guidelines (wcag) 2.0","authors":"Ben Caldwell, Michael Cooper, Loretta Guarino Reid, Gregg Vanderheiden, Wendy Chisholm, John Slatin, Jason White","year":"2008","venue":"WWW Consortium (W3C)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cao et al., 2019] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Openpose: Realtime multi-person 2d pose estimation using part affinity fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.","title":"Openpose: Realtime multi-person 2d pose estimation using part affinity fields","authors":"Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh","year":"2019","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[De Coster et al., 2020] Mathieu De Coster, Mieke Van Her- reweghe, and Joni Dambre. Sign language recogni- tion with transformer networks. In 12th international conference on language resources and evaluation, pages 6018-6024. European Language Resources Association (ELRA), 2020.","title":"Sign language recognition with transformer networks","authors":"Mieke Mathieu De Coster, Joni Van Herreweghe, Dambre","year":"2020","venue":"12th international conference on language resources and evaluation","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Trans- formers for image recognition at scale. In International Conference on Learning Representations, 2021.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fink et al., 2021] Jérôme Fink, Benoît Frénay, Laurence Meurant, and Anthony Cleve. Lsfb-cont and lsfb-isol: Two new datasets for vision-based sign language recog- nition. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1-8, 2021.","title":"Lsfb-cont and lsfb-isol: Two new datasets for vision-based sign language recognition","authors":"Fink","year":"2021","venue":"International Joint Conference on Neural Networks (IJCNN)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang and Huang, 1998] Chung-Lin Huang and Wen-Yi Huang. Sign language recognition using model-based tracking and a 3d hopfield neural network. Machine Vi- sion and Applications, 10(5-6):292-307, April 1998.","title":"Sign language recognition using model-based tracking and a 3d hopfield neural network","authors":"Chung-Lin Huang, Wen-Yi Huang","year":"1998","venue":"Machine Vision and Applications","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Izutov, 2020] Evgeny Izutov. Asl recognition with metric- learning based lightweight network. arXiv preprint arXiv:2004.05054, 2020.","title":"Asl recognition with metriclearning based lightweight network","authors":"Evgeny Izutov","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kadir et al., 2004] Timor Kadir, Richard Bowden, Eng-Jon Ong, and Andrew Zisserman. Minimal training, large lex- icon, unconstrained sign language recognition. In BMVC, pages 1-10, 2004.","title":"Minimal training, large lexicon, unconstrained sign language recognition","authors":"Kadir","year":"2004","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Koller, 2020] Oscar Koller. Quantitative survey of the state of the art in sign language recognition. arXiv preprint arXiv:2008.09918, 2020.","title":"Quantitative survey of the state of the art in sign language recognition","authors":"Oscar Koller, Koller","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Konstantinidis et al., 2018] Dimitrios Konstantinidis, Kos- mas Dimitropoulos, and Petros Daras. Sign language recognition based on hand and body skeletal data. In 2018- 3DTV-Conference: The True Vision-Capture, Transmis- sion and Display of 3D Video (3DTV-CON), pages 1-4. IEEE, 2018.","title":"Sign language recognition based on hand and body skeletal data","authors":"Dimitrios Konstantinidis, Kosmas Dimitropoulos, Petros Daras","year":"2018","venue":"2018-3DTV-Conference: The True Vision-Capture, Transmission and Display of 3D Video (3DTV-CON)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lallemand, 2018] Carine Lallemand. Méthodes de De- sign UX. 30 méthodes fondamentales pour concevoir des expériences optimales. (2e edition). 09 2018.","title":"Méthodes de Design UX. 30 méthodes fondamentales pour concevoir des expériences optimales","authors":"Carine Lallemand","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[LeCun et al., 2015] Yann LeCun, Yoshua Bengio, and Ge- offrey Hinton. Deep learning. Nature, 521(7553):436- 444, May 2015.","title":"Deep learning","authors":"Yann Lecun, Yoshua Bengio, Geoffrey Hinton","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2020] Dongxu Li, Cristian Rodriguez, Xin Yu, and Hongdong Li. Word-level deep sign language recognition from video: A new large-scale dataset and methods com- parison. In Proceedings of the IEEE/CVF Winter Confer- ence on Applications of Computer Vision (WACV), March 2020.","title":"Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison","authors":"Li","year":"2020","venue":"Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liao et al., 2019] Yanqiu Liao, Pengwen Xiong, Weidong Min, Weiqiong Min, and Jiahao Lu. Dynamic sign lan- guage recognition based on video sequence with blstm-3d residual networks. IEEE Access, 7:38044-38054, 2019.","title":"Dynamic sign language recognition based on video sequence with blstm-3d residual networks","authors":"Liao","year":"2019","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lugaresi et al., 2019] Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, Wan-Teh Chang, Wei Hua, Manfred Georg, and Matthias Grundmann. Mediapipe: A framework for building perception pipelines. 2019.","title":"Mediapipe: A framework for building perception pipelines","authors":"Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris Mcclanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, Wan-Teh Chang, Wei Hua, Manfred Georg, Matthias Grundmann","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Meurant, 2015] Laurence Meurant. Corpus LSFB. Corpus informatisé en libre acces de vidéo et d'annotations de langue des signes de Belgique francophone. Namur: Lab- oratoire de langue des signes de Belgique francophone (LSFB Lab), FRS-FNRS, Université de Namur, 2015.","title":"Corpus LSFB. Corpus informatisé en libre acces de vidéo et d'annotations de langue des signes de Belgique francophone","authors":"Laurence Meurant, Meurant","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pigou et al., 2016] Lionel Pigou, Mieke Van Herreweghe, and Joni Dambre. Sign classification in sign language corpora with deep neural networks. In Eleni Efthimiou, Stavroula-Evita Fotinea, Thomas Hanke, Julie A. Hochge- sang, Jette Kristoffersen, and Johanna Mesch, editors, Pro- ceedings of the LREC2016 7th Workshop on the Represen- tation and Processing of Sign Languages: Corpus Mining, pages 175-178, Portorož, Slovenia, May 2016. European Language Resources Association (ELRA).","title":"Sign classification in sign language corpora with deep neural networks","authors":"Lionel Pigou, Mieke Van Herreweghe, Joni Dambre","year":"2016","venue":"Proceedings of the LREC2016 7th Workshop on the Representation and Processing of Sign Languages: Corpus Mining","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sandler et al., 2018] Mark Sandler, Andrew Howard, Men- glong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018.","title":"Mobilenetv2: Inverted residuals and linear bottlenecks","authors":"Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, Liang-Chieh Chen","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Savitzky and Golay, 1964] Abraham. Savitzky and M. J. E. Golay. Smoothing and differentiation of data by sim- plified least squares procedures. Analytical Chemistry, 36(8):1627-1639, July 1964.","title":"Smoothing and differentiation of data by simplified least squares procedures","authors":"Golay Savitzky, Abraham Savitzky, M J E Golay","year":"1964","venue":"Analytical Chemistry","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Stokoe, 1972] William C Stokoe. Classification and de- scription of sign languages. Current trends in linguistics, 12:345-371, 1972.","title":"Classification and description of sign languages","authors":"C William, Stokoe","year":"1972","venue":"Current trends in linguistics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[UN, 2015] UN. The 17 goals -sustainable develop- ment. https://sdgs.un.org/goals, 2015. (Accessed on 05/15/2023).","title":"UN. The 17 goals -sustainable development","authors":null,"year":"2015","venue":null,"doi":null,"url":"https://sdgs.un.org/goals,2015","internal_url":null},{"position":25,"raw":"[Vaezi Joze and Koller, 2019] Hamid Vaezi Joze and Oscar Koller. Ms-asl: A large-scale data set and benchmark for understanding american sign language. In The British Ma- chine Vision Conference (BMVC), September 2019.","title":"Ms-asl: A large-scale data set and benchmark for understanding american sign language","authors":"Vaezi Joze, Hamid Koller, Joze Vaezi, Oscar Koller","year":"2019","venue":"The British Machine Vision Conference (BMVC)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Sys- tems, volume 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2012] Haijing Wang, Alexandra Stefan, Saj- jad Moradi, Vassilis Athitsos, Carol Neidle, and Farhad Kamangar. A system for large vocabulary sign search. In Trends and Topics in Computer Vision: ECCV 2010 Workshops, Heraklion, Crete, Greece, September 10-11, 2010, Revised Selected Papers, Part I 11, pages 342-353. Springer, 2012.","title":"A system for large vocabulary sign search","authors":"Haijing Wang, Alexandra Stefan, Sajjad Moradi, Vassilis Athitsos, Carol Neidle, Farhad Kamangar","year":"2010","venue":"Trends and Topics in Computer Vision: ECCV 2010 Workshops","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}