Proceedings

A Gloss-driven Indian Sign Language Production System Using Learned Pose Representations

Download BibTeX Cite

Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence. AI and Social Good. Pages 7399-7407.

https://doi.org/10.24963/ijcai.2026/823

Abstract

Sign Language Production (SLP) system translates spoken or written language into sign language, enabling accessible communication between the deaf/hard-of-hearing and the hearing population. Being one of the most widely used sign languages globally, Indian Sign Language (ISL) is a very low-resource language and lacks such SLP systems. This paper presents a scalable and modular SLP framework based on Sign-Pose-VQ-VAE model, designed for low-resource settings. The model learns discrete pose representations (codes) by disentangling body, left-hand, and right-hand keypoints, enabling efficient pose modeling and co-articulated sign generation. The proposed system is evaluated using a Hindi movie subtitle corpus coupled with an off-the-shelf back-translation model and achieves a gloss BLEU-4 score of 47.20. The system-generated signs are evaluated by certified ISL interpreters with an average rating of 4.33/5, and a BERT precision of 0.7683 on glosses. In addition, the proposed system achieves state-of-the-art performance among keypoint-based methods on the PHOENIX14T benchmark, attaining a BLEU-4 score of 10.03 and surpassing the previous best method by 0.67 points. The system with code is available at https://cs.rkmvu.ac.in/~isl/sl_gen_vqvae.

  1. [Albanie et al., 2021] Samuel Albanie, Gül Varol, Liliane Momeni, Hannah Bull, Triantafyllos Afouras, Himel Chowdhury, Neil Fox, Bencie Woll, Rob Cooper, An- drew McParland, and Andrew Zisserman. BOBSL: BBC- Oxford British Sign Language Dataset. arXiv, 2021.
  2. [Camgoz et al., 2018] Necati Cihan Camgoz, Simon Had- field, Oscar Koller, Hermann Ney, and Richard Bowden. Neural sign language translation. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pages 7784-7793, 2018.
  3. [Camgoz et al., 2020] Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. Sign language transformers: Joint end-to-end sign language recognition and translation. In Proceedings of the Conference on Com- puter Vision and Pattern Recognition, 2020.
  4. [Cox et al., 2002] Stephen Cox, Michael Lincoln, Judy Tryggvason, Melanie Nakisa, Mark Wells, Marcus Tutt, and Sanja Abbott. Tessa, a system to aid communica- tion with deaf people. In Proceedings of the International ACM Conference on Assistive Technologies, pages 205- 212, 2002.
  5. [Duarte et al., 2021] Amanda Duarte, Shruti Palaskar, Lucas Ventura, Deepti Ghadiyaram, Kenneth DeHaan, Florian Metze, Jordi Torres, and Xavier Giro-i Nieto. How2Sign: A Large-scale Multimodal Dataset for Continuous Ameri- can Sign Language. In Proceedings of the Conference on Computer Vision and Pattern Recognition, 2021.
  6. [Fink et al., 2023] Jerome Fink, Pierre Poitier, Maxime André, Loup Meurice, Benoît Frénay, Anthony Cleve, Bruno Dumas, and Laurence Meurant. Sign language-to- text dictionary with lightweight transformer models. In Proceedings of the International Joint Conference on Ar- tificial Intelligence, pages 5968-5976, 2023.
  7. [Glauert et al., 2006] John RW Glauert, Ralph Elliott, Stephen J Cox, Judy Tryggvason, and Mary Sheard. Vanessa-a system for communication between deaf and hearing people. Technology and Disability, 18(4):207- 216, 2006.
  8. [Goodfellow et al., 2014] Ian J Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. Advances in Neural Information Processing Systems, 27, 2014.
  9. [Goyal and Goyal, 2016] Lalit Goyal and Vishal Goyal. Au- tomatic translation of English text to Indian Sign Lan- guage synthetic animations. In Proceedings of the Interna- tional Conference on Natural Language Processing, pages 144-153, 2016.
  10. [Hanke, 2004] Thomas Hanke. Hamnosys-representing sign language data in language resources and language process- ing contexts. In Proceedings of the International Confer- ence on Language Resources and Evaluation, pages 1-6, 2004.
  11. [He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the Conference on Computer Vi- sion and Pattern Recognition, pages 770-778, 2016.
  12. [Hu et al., 2022] Li Hu, Jiahui Li, Jiashuo Zhang, Qi Wang, Bang Zhang, and Ping Tan. A speech-driven sign language avatar animation system for hearing impaired applications. In IJCAI, pages 5912-5915, 2022. IJCAI
  13. [Huang et al., 2021] Wencan Huang, Wenwen Pan, Zhou Zhao, and Qi Tian. Towards fast and high-quality sign language production. In Proceedings of the ACM Interna- tional Conference on Multimedia, page 3172-3181, 2021.
  14. [Inan et al., 2024] Mert Inan, Katherine Atwell, Anthony Si- cilia, Lorna Quandt, and Malihe Alikhani. Generating signed language instructions in large-scale dialogue sys- tems. In Proceedings of the Conference of the North Amer- ican Chapter of the Association for Computational Lin- guistics, pages 140-154, 2024.
  15. [Joshi et al., 2022] Abhinav Joshi, Ashwani Bhat, Pradeep S, Priya Gole, Shashwat Gupta, Shreyansh Agarwal, and Ashutosh Modi. CISLR: Corpus for Indian Sign Language recognition. In Proceedings of the Conference on Em- pirical Methods in Natural Language Processing, pages 10357-10366, 2022.
  16. [Kumar et al., 2021] Parteek Kumar, Sanmeet Kaur, et al. Indian sign language generation system. Computer, 54(3):37-46, 2021.
  17. [Patra et al., 2025a] Suvajit Patra, Arkadip Maitra, and Soumitra Samanta. A continuous indian sign language production system from isolated signs. In Proceedings of the International Conference on Smart Computing and Communications, pages 1-6, 2025.
  18. [Patra et al., 2025b] Suvajit Patra, Arkadip Maitra, Megha Tiwari, K Kumaran, Swathy Prabhu, Swami Punyesh- warananda, and Soumitra Samanta. Hierarchical win- dowed graph attention transformer encoder and a large scale dataset for indian sign language recognition. Pattern Analysis and Applications, 28(3):148, 2025.
  19. [Rastgoo et al., 2024] Razieh Rastgoo, Kourosh Kiani, Ser- gio Escalera, Vassilis Athitsos, and Mohammad Sabokrou. A survey on recent advances in sign language production. Expert Systems with Applications, 243:122846, 2024.
  20. [Razavi et al., 2019] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in Neural Information Process- ing Systems, 32, 2019.
  21. [Saunders et al., 2020] Ben Saunders, Necati Cihan Cam- goz, and Richard Bowden. Progressive transformers for end-to-end sign language production. In Proceedings of the European Conference on Computer Vision, pages 687- 705, 2020.
  22. [Saunders et al., 2022] Ben Saunders, Necati Cihan Cam- goz, and Richard Bowden. Signing at scale: Learning to co-articulate signs for large-scale photo-realistic sign lan- guage production. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pages 5141- 5151, 2022.
  23. [Shan et al., 2023] Wenkang Shan, Zhenhua Liu, Xinfeng Zhang, Zhao Wang, Kai Han, Shanshe Wang, Siwei Ma, and Wen Gao. Diffusion-based 3d human pose estima- tion with multi-hypothesis aggregation. In Proceedings of the International Conference on Computer Vision, pages 14761-14771, 2023.
  24. [Shi et al., 2024] Tongkai Shi, Lianyu Hu, Fanhua Shang, Jichao Feng, Peidong Liu, and Wei Feng. Pose-guided fine-grained sign language video generation. In Proceed- ings of the European Conference on Computer Vision, pages 392-409, 2024.
  25. [Sridhar et al., 2020] Advaith Sridhar, Rohith Gandhi Gane- san, Pratyush Kumar, and Mitesh Khapra. Include: A large scale dataset for indian sign language recognition. In Pro- ceedings of the ACM International Conference on Multi- media, pages 1366-1375, 2020.
  26. [Stoll et al., 2018] Stephanie Stoll, Necati Cihan Camgöz, Simon Hadfield, and Richard Bowden. Sign language pro- duction using neural machine translation and generative adversarial networks. In Proceedings of the British Ma- chine Vision Conference, 2018.
  27. [Sugandhi et al., 2020] Sugandhi, Parteek Kumar, and San- meet Kaur. Sign language generation system based on indian sign language grammar. ACM Transactions on Asian and Low-Resource Language Information Process- ing, 19(4):1-26, 2020.
  28. [Sun et al., 2019] Ke Sun, Bin Xiao, Dong Liu, and Jing- dong Wang. Deep high-resolution representation learning for human pose estimation. In Proceedings of the confer- ence on computer vision and pattern recognition, pages 5693-5703, 2019.
  29. [Tang et al., 2022] Shengeng Tang, Richang Hong, Dan Guo, and Meng Wang. Gloss semantic-enhanced network with online back-translation for sign language production. In Proceedings of the ACM International Conference on Multimedia, pages 5630-5638, 2022.
  30. [Tang et al., 2025a] Shengeng Tang, Jiayi He, Lechao Cheng, Jingjing Wu, Dan Guo, and Richang Hong. Dis- crete to continuous: Generating smooth transition poses from sign language observations. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pages 3481-3491, 2025.
  31. [Tang et al., 2025b] Shengeng Tang, Jiayi He, Dan Guo, Yanyan Wei, Feng Li, and Richang Hong. Sign-idd: Iconicity disentangled diffusion for sign language produc- tion. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7266-7274, 2025.
  32. [Tang et al., 2025c] Shengeng Tang, Feng Xue, Jingjing Wu, Shuo Wang, and Richang Hong. Gloss-driven conditional diffusion models for sign language production. ACM Transactions on Multimedia Computing, Communications and Applications, 21(4):1-17, 2025.
  33. [Van Den Oord et al., 2017] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in Neural Information Processing Systems, 30, 2017.
  34. [Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Sys- tems, 30, 2017.
  35. [Viegas et al., 2023] Carla Viegas, Mert Inan, Lorna Quandt, and Malihe Alikhani. Including facial expressions in con- textual embeddings for sign language generation. In Pro- ceedings of the Joint Conference on Lexical and Compu- tational Semantics, pages 1-10, 2023.
  36. [Walsh et al., 2024] Harry Walsh, Abolfazl Ravanshad, Mariam Rahmani, and Richard Bowden. A data-driven representation for sign language production. In Proceed- ings of the International Conference on Automatic Face and Gesture Recognition, pages 1-10, 2024.
  37. [Walsh et al., 2025] Harry Walsh, Ed Fish, Ozge Mer- canoglu Sincan, Mohamed Ilyes Lakhal, Richard Bowden, Neil Fox, Bencie Woll, Kepeng Wu, Zecheng Li, Weichao Zhao, et al. Slrtp2025 sign language production challenge: Methodology, results and future work. In Proceedings of the Conference on Computer Vision and Pattern Recogni- tion, pages 4109-4119, 2025.
  38. [Wang et al., 2025] Xu Wang, Shengeng Tang, Peipei Song, Shuo Wang, Dan Guo, and Richang Hong. Linguistics- vision monotonic consistent network for sign language production. In Proceedings of the International Confer- ence on Acoustics, Speech and Signal Processing, pages 1-5, 2025.
  39. [Williams et al., 2020] Will Williams, Sam Ringer, Tom Ash, David MacLeod, Jamie Dougherty, and John Hughes. Hierarchical quantized autoencoders. Advances in Neural Information Processing Systems, 33:4524-4535, 2020.
  40. [Xie et al., 2024] Pan Xie, Qipeng Zhang, Peng Taiying, Hao Tang, Yao Du, and Zexian Li. G2p-ddm: Generat- ing sign pose sequence from gloss sequence with discrete diffusion model. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 6234-6242, 2024.
  41. [Yin et al., 2024] Aoxiong Yin, Haoyuan Li, Kai Shen, Sil- iang Tang, and Yueting Zhuang. T2s-gpt: Dynamic vector quantization for autoregressive sign language production from text. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 3345- 3356, 2024.
  42. [Zelinka and Kanis, 2020] Jan Zelinka and Jakub Kanis. Neural sign language synthesis: Words are our glosses. In Proceedings of the Winter Conference on Applications of Computer Vision, pages 3395-3403, 2020.
  43. [Zhang et al., 2019] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019.
  44. [Zhang et al., 2025] Shijun Zhang, Hongkai Zhao, Yimin Zhong, and Haomin Zhou. Why shallow networks strug- gle to approximate and learn high frequencies. Informa- tion and Inference: A Journal of the IMA, 14(3):iaaf022, 2025.
  45. [Zhou et al., 2021] Hao Zhou, Wengang Zhou, Weizhen Qi, Junfu Pu, and Houqiang Li. Improving sign language translation with monolingual data by sign back-translation. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pages 1316-1325, 2021.
  46. [Zuo et al., 2024] Ronglai Zuo, Fangyun Wei, Zenggui Chen, Brian Mak, Jiaolong Yang, and Xin Tong. A simple baseline for spoken language to sign language translation with 3d avatars. In Proceedings of the European Confer- ence on Computer Vision, pages 36-54, 2024.
  47. [Zwitserlood et al., 2004] Inge Zwitserlood, Margriet Ver- linden, Johan Ros, Sanny Van Der Schoot, and T Nether- lands. Synthetic signing for the deaf: Esign. In Proceed- ings of the Conference and Workshop on Assistive Tech- nologies for Vision and Hearing Impairment, 2004.

References were extracted automatically from the PDF and may contain errors.

Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards