{"type":"ScholarlyArticle","title":"A Speech-driven Sign Language Avatar Animation System for Hearing Impaired Applications","authors":[{"name":"Li Hu","slug":"li-hu","given":"Li","family":"Hu","affiliation":"XR Lab , Alibaba Group"},{"name":"Jiahui Li","slug":"jiahui-li","given":"Jiahui","family":"Li","affiliation":"XR Lab , Alibaba Group"},{"name":"Jiashuo Zhang","slug":"jiashuo-zhang","given":"Jiashuo","family":"Zhang","affiliation":"XR Lab , Alibaba Group"},{"name":"Qi Wang","slug":"qi-wang","given":"Qi","family":"Wang","affiliation":"XR Lab , Alibaba Group"},{"name":"Bang Zhang","slug":"bang-zhang","given":"Bang","family":"Zhang","affiliation":"XR Lab , Alibaba Group"},{"name":"Ping Tan","slug":"ping-tan","given":"Ping","family":"Tan","affiliation":"XR Lab , Alibaba Group"}],"abstract":"Sign language is the communication language used in hearing impaired community. Recently, the research of sign language production has made great progress but still need to cope with some critical challenges. In this paper, we propose a system-level scheme and push forward the implementation of sign language production for practical usage. We build a system capable of translating speech into sign language avatar. Different from previous approach only focusing on single technology, we systematically combine algorithms of language translation, body gesture animation and facial avatar generation. We also develop two applications: Sign Language Interpretation APP and Virtual Sign Language Anchor, to facilitate easy and clear communication for hearing impaired people.","doi":"10.24963/ijcai.2022/852","url":"https://ijcai.org/proceedings/2022/852","pdf":"https://ijcai.org/proceedings/2022/0852.pdf","markdown":"https://ijcai.org/proceedings/2022/852.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/852","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Demo Track","track":"Demo Track","number":852,"pages":{"first":5912,"last":5915},"keywords":["Machine Learning: Applications","Humans and AI: Human-Computer Interaction","Humans and AI: Applications","Humans and AI: Cognitive Systems"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987211","length_sec":75,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985354","length_sec":319,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bangham et al., 2000] J Andrew Bangham, SJ Cox, Ralph Elliott, John RW Glauert, Ian Marshall, Sanja Rankov, and Mark Wells. Virtual signing: Capture, animation, storage and transmission-an overview of the visicast project. In IEE Seminar on speech and language processing for dis- abled and elderly people (Ref. No. 2000/025), pages 6-1. IET, 2000.","title":"Virtual signing: Capture, animation, storage and transmission-an overview of the visicast project","authors":"Bangham","year":"2000","venue":"IEE Seminar on speech and language processing for disabled and elderly people","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Camgöz et al., 2018] Necati Cihan Camgöz, Simon Had- field, Oscar Koller, Hermann Ney, and Richard Bowden. Neural sign language translation. In 2018 IEEE Confer- ence on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 7784-7793. Computer Vision Foundation / IEEE Com- puter Society, 2018.","title":"Neural sign language translation","authors":"Simon Necati Cihan Camgöz, Oscar Hadfield, Hermann Koller, Richard Ney, Bowden","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cox et al., 2002] Stephen Cox, Michael Lincoln, Judy Tryggvason, Melanie Nakisa, Mark Wells, Marcus Tutt, and Sanja Abbott. Tessa, a system to aid communica- tion with deaf people. In Vicki L. Hanson and Julie A. Jacko, editors, Proceedings of the ACM Conference on As- sistive Technologies, ASSETS 2002, Edinburgh, Scotland, UK, July 8-10, 2002, pages 205-212. ACM, 2002.","title":"Tessa, a system to aid communication with deaf people","authors":"Stephen Cox, Michael Lincoln, Judy Tryggvason, Melanie Nakisa, Mark Wells, Marcus Tutt, Sanja Abbott","year":"2002","venue":"Proceedings of the ACM Conference on Assistive Technologies, ASSETS 2002","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ebling and Glauert, 2013] Sarah Ebling and John Glauert. Exploiting the full potential of jasigning to build an avatar signing train announcements. In Third International Sym- posium on Sign Language Translation and Avatar Tech- nology, 2013.","title":"Exploiting the full potential of jasigning to build an avatar signing train announcements","authors":"Glauert Ebling, Sarah Ebling, John Glauert","year":"2013","venue":"Third International Symposium on Sign Language Translation and Avatar Technology","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Guo et al., 2018] Dan Guo, Wengang Zhou, Houqiang Li, and Meng Wang. Hierarchical lstm for sign language translation. In Proceedings of the AAAI conference on ar- tificial intelligence, volume 32, 2018.","title":"Hierarchical lstm for sign language translation","authors":"Dan Guo, Wengang Zhou, Houqiang Li, Meng Wang","year":"2018","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Karras et al., 2019] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 4401-4410. Computer Vision Foundation / IEEE, 2019.","title":"A style-based generator architecture for generative adversarial networks","authors":"Karras","year":"2019","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kataoka et al., 2016] Yuusuke Kataoka, Takashi Matsub- ara, and Kuniaki Uehara. Image generation using genera- tive adversarial networks and attention mechanism. In 15th IEEE/ACIS International Conference on Computer and In- formation Science, ICIS 2016, Okayama, Japan, June 26- 29, 2016, pages 1-6. IEEE Computer Society, 2016.","title":"Image generation using generative adversarial networks and attention mechanism","authors":"Yuusuke Kataoka, Takashi Matsubara, Kuniaki Uehara","year":"2016","venue":"15th IEEE/ACIS International Conference on Computer and Information Science, ICIS 2016","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kushalnagar, 2019] Raja Kushalnagar. Deafness and hear- ing loss. In Web Accessibility, pages 35-47. Springer, 2019.","title":"Deafness and hearing loss","authors":"Raja Kushalnagar","year":"2019","venue":"Web Accessibility","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lee and Shin, 1999] Jehee Lee and Sung Yong Shin. A hier- archical approach to interactive motion editing for human- like figures. In Warren N. Waggenspack, editor, Proceed- ings of the 26th Annual Conference on Computer Graph- ics and Interactive Techniques, SIGGRAPH 1999, Los An- geles, CA, USA, August 8-13, 1999, pages 39-48. ACM, 1999.","title":"A hierarchical approach to interactive motion editing for humanlike figures","authors":"Lee, Shin ; ] Jehee Lee, Sung Yong Shin","year":"1999","venue":"Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1999","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lewis et al., 2020] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In","title":"BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension","authors":"Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, Luke Zettlemoyer","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7871-7880. Associ- ation for Computational Linguistics, 2020.","title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020","authors":null,"year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Stoll et al., 2020] Stephanie Stoll, Necati Cihan Camgöz, Simon Hadfield, and Richard Bowden. Text2sign: To- wards sign language production using neural machine translation and generative adversarial networks. Int. J. Comput. Vis., 128(4):891-908, 2020.","title":"Text2sign: Towards sign language production using neural machine translation and generative adversarial networks","authors":"Stephanie Stoll, Cihan Necati, Simon Camgöz, Richard Hadfield, Bowden","year":"2020","venue":"Int. J. Comput. Vis","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[van den Oord et al., 2016] Aäron van den Oord, Nal Kalch- brenner, Lasse Espeholt, Koray Kavukcuoglu, Oriol Vinyals, and Alex Graves. Conditional image genera- tion with pixelcnn decoders. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Ro- man Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 4790-4798, 2016.","title":"Conditional image generation with pixelcnn decoders","authors":"Oord Van Den","year":"2016","venue":"Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Vasani et al., 2020] Neel Vasani, Pratik Autee, Samip Kalyani, and Ruhina Karani. Generation of indian sign language by sentence processing and generative adversar- ial networks. In 2020 3rd International Conference on In- telligent Sustainable Systems (ICISS), pages 1250-1255. IEEE, 2020.","title":"Generation of indian sign language by sentence processing and generative adversarial networks","authors":"Neel Vasani, Pratik Autee, Samip Kalyani, Ruhina Karani","year":"2020","venue":"2020 3rd International Conference on Intelligent Sustainable Systems (ICISS)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vish- wanathan, and Roman Garnett, editors, Advances in Neu- ral Information Processing Systems 30: Annual Confer- ence on Neural Information Processing Systems 2017, De- cember 4-9, 2017, Long Beach, CA, USA, pages 5998- 6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}