{"type":"ScholarlyArticle","title":"Including Signed Languages in Natural Language Processing (Extended Abstract)","authors":[{"name":"Kayo Yin","slug":"kayo-yin","given":"Kayo","family":"Yin","affiliation":"Carnegie Mellon University"},{"name":"Malihe Alikhani","slug":"malihe-alikhani","given":"Malihe","family":"Alikhani","affiliation":"University of Pittsburgh; Carnegie Mellon University"}],"abstract":"Signed languages are the primary means of communication for many deaf and hard of hearing individuals. Since signed languages exhibit all the fundamental linguistic properties of natural language, we believe that tools and theories of Natural Language Processing (NLP) are crucial towards its modeling. However, existing research in Sign Language Processing (SLP) seldom attempt to explore and leverage the linguistic organization of signed languages. This position paper calls on the NLP community to include signed languages as a research area with high social and scientific impact. We first discuss the linguistic properties of signed languages to consider during their modeling. Then, we review the limitations of current SLP models and identify the open challenges to extend NLP to signed languages. Finally, we urge (1) the adoption of an efficient tokenization method; (2) the development of linguistically-informed models; (3) the collection of real-world signed language data;  (4) the inclusion of local signed language communities as an active and leading voice in research.","doi":"10.24963/ijcai.2022/753","url":"https://ijcai.org/proceedings/2022/753","pdf":"https://ijcai.org/proceedings/2022/0753.pdf","markdown":"https://ijcai.org/proceedings/2022/753.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/753","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Sister Conferences Best Papers","track":"Sister Conferences Best Papers","number":753,"pages":{"first":5369,"last":5373},"keywords":["Artificial Intelligence: General"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987251","length_sec":87,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985394","length_sec":659,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Borg and Camilleri, 2019] Mark Borg and Kenneth P Camilleri. Sign language detection \"in the wild\" with re- current neural networks. In ICASSP 2019-2019 IEEE In- ternational Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1637-1641. IEEE, 2019.","title":"Sign language detection \"in the wild\" with recurrent neural networks","authors":"Camilleri Borg, Mark Borg, Kenneth P Camilleri","year":"2019","venue":"ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bull et al., 2020] Hannah Bull, Michèle Gouiffès, and An- nelies Braffort. Automatic segmentation of sign language into subtitle-units. In European Conference on Computer Vision, pages 186-198. Springer, 2020.","title":"Annelies Braffort. Automatic segmentation of sign language into subtitle-units","authors":"Hannah Bull, Michèle Gouiffès","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Camgöz et al., 2018] Necati Cihan Camgöz, Simon Had- field, Oscar Koller, Hermann Ney, and Richard Bowden. Neural sign language translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pages 7784-7793, 2018.","title":"Neural sign language translation","authors":"Simon Necati Cihan Camgöz, Oscar Hadfield, Hermann Koller, Richard Ney, Bowden","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Camgöz et al., 2020] Necati Cihan Camgöz, Oscar Koller, Simon Hadfield, and Richard Bowden. Multi-channel transformers for multi-articulatory sign language transla- tion. In European Conference on Computer Vision, pages 301-319, 2020.","title":"Multi-channel transformers for multi-articulatory sign language translation","authors":"Oscar Necati Cihan Camgöz, Simon Koller, Richard Hadfield, Bowden","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dreuw et al., 2006] Philippe Dreuw, Thomas Deselaers, Daniel Keysers, and Hermann Ney. Modeling image vari- ability in appearance-based gesture recognition. In ECCV workshop on statistical methods in multi-image and video processing, pages 7-18, 2006.","title":"Modeling image variability in appearance-based gesture recognition","authors":"Philippe Dreuw, Thomas Deselaers, Daniel Keysers, Hermann Ney","year":"2006","venue":"ECCV workshop on statistical methods in multi-image and video processing","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fenlon et al., 2015] Jordan Fenlon, Kearsy Cormier, and Adam Schembri. Building bsl signbank: The lemma dilemma revisited. International Journal of Lexicography, 28(2):169-206, 2015.","title":"Building bsl signbank: The lemma dilemma revisited","authors":"Fenlon","year":"2015","venue":"International Journal of Lexicography","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gebre et al., 2013] Binyam Gebrekidan Gebre, Peter Wit- tenburg, and Tom Heskes. Automatic sign language identi- fication. In 2013 IEEE International Conference on Image Processing, pages 2626-2630. IEEE, 2013.","title":"Binyam Gebrekidan Gebre, Peter Wittenburg, and Tom Heskes. Automatic sign language identification","authors":"Gebre","year":"2013","venue":"2013 IEEE International Conference on Image Processing","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hanke et al., 2020] Thomas Hanke, Marc Schulder, Reiner Konrad, and Elena Jahn. Extending the Public DGS Cor- pus in size and depth. In Proceedings of the LREC2020 9th Workshop on the Representation and Processing of Sign Languages, 2020.","title":"Extending the Public DGS Corpus in size and depth","authors":"Thomas Hanke, Marc Schulder, Reiner Konrad, Elena Jahn","year":"2020","venue":"Proceedings of the LREC2020 9th Workshop on the Representation and Processing of Sign Languages","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Imashev et al., 2020] Alfarabi Imashev, Medet Mukushev, Vadim Kimmelman, and Anara Sandygulova. A dataset for linguistic understanding, visual evaluation, and recog- nition of sign languages: The k-rsl. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 631-640, 2020.","title":"Alfarabi Imashev, Medet Mukushev, Vadim Kimmelman, and Anara Sandygulova. A dataset for linguistic understanding, visual evaluation, and recognition of sign languages: The k-rsl","authors":"Imashev","year":"2020","venue":"Proceedings of the 24th Conference on Computational Natural Language Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kimmelman, 2014] Vadim Kimmelman. Information struc- ture in russian sign language and sign language of the netherlands. Sign Language & Linguistics, 18(1):142- 150, 2014.","title":"Information structure in russian sign language and sign language of the netherlands","authors":"Vadim Kimmelman, Kimmelman","year":"2014","venue":"Sign Language & Linguistics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2020] Dongxu Li, Cristian Rodriguez, Xin Yu, and Hongdong Li. Word-level deep sign language recognition from video: A new large-scale dataset and methods com- parison. In The IEEE Winter Conference on Applications of Computer Vision, pages 1459-1469, 2020.","title":"Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison","authors":"Li","year":"2020","venue":"The IEEE Winter Conference on Applications of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Monteiro et al., 2016] Caio DD Monteiro, Christy Maria Mathew, Ricardo Gutierrez-Osuna, and Frank Shipman. Detecting and identifying sign languages through visual features. In 2016 IEEE International Symposium on Mul- timedia (ISM), pages 287-290. IEEE, 2016.","title":"Detecting and identifying sign languages through visual features","authors":"D D Caio, Christy Monteiro, Maria Mathew, Ricardo Gutierrez-Osuna, Frank Shipman","year":"2016","venue":"IEEE International Symposium on Multimedia (ISM)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Moryossef et al., 2021] Amit Moryossef, Kayo Yin, Gra- ham Neubig, and Yoav Goldberg. Data augmentation for sign language gloss translation. In Proceedings of the 1st International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL), 2021.","title":"Data augmentation for sign language gloss translation","authors":"Amit Moryossef, Kayo Yin, Graham Neubig, Yoav Goldberg","year":"2021","venue":"Proceedings of the 1st International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ormel and Crasborn, 2012] Ellen Ormel and Onno Cras- born. Prosodic correlates of sentences in signed languages: A literature review and suggestions for new types of stud- ies. Sign Language Studies, 12(2):279-315, 2012.","title":"Prosodic correlates of sentences in signed languages: A literature review and suggestions for new types of studies","authors":"Crasborn Ormel","year":"2012","venue":"Sign Language Studies","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Padden, 1988] C. Padden. Interaction of Morphology and Syntax in American Sign Language. Outstanding Disc Lin- guistics Series. Garland, 1988.","title":"Interaction of Morphology and Syntax in American Sign Language","authors":"C Padden","year":"1988","venue":"Outstanding Disc Linguistics Series","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Saunders et al., 2020] Ben Saunders, Necati Cihan Camgöz, and Richard Bowden. Everybody sign now: Translating spoken language to photo realistic sign language video. arXiv preprint arXiv:2011.09846, 2020.","title":"Everybody sign now: Translating spoken language to photo realistic sign language video","authors":"Saunders","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shi et al., 2018] B. Shi, A. Martinez Del Rio, J. Keane, J. Michaux, G. Shakhnarovich D. Brentari, and K. Livescu. American sign language fingerspelling recognition in the wild. SLT, December 2018.","title":"American sign language fingerspelling recognition in the wild","authors":"B Shi, A Martinez Del Rio, J Keane, J Michaux, G Shakhnarovich, D Brentari, K Livescu","year":"2018","venue":"SLT","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sincan and Keles, 2020] Ozge Mercanoglu Sincan and Hacer Yalim Keles. Autsl: A large scale multi-modal turk- ish sign language dataset and baseline methods. IEEE Ac- cess, 8:181340-181355, 2020.","title":"Autsl: A large scale multi-modal turkish sign language dataset and baseline methods","authors":"Keles ; Sincan, Mercanoglu Ozge, Hacer Sincan, Keles Yalim","year":"2020","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sohn et al., 1999] Jongseo Sohn, Nam Soo Kim, and Wony- ong Sung. A statistical model-based voice activity detec- tion. IEEE signal processing letters, 6(1):1-3, 1999.","title":"A statistical model-based voice activity detection","authors":"Sohn","year":"1999","venue":"IEEE signal processing letters","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Stokoe, 1960] Jr. Stokoe, William C. Sign Language Struc- ture: An Outline of the Visual Communication Systems of the American Deaf. The Journal of Deaf Studies and Deaf Education, 10(1):3-37, 01 1960.","title":"Sign Language Structure: An Outline of the Visual Communication Systems of the American Deaf","authors":"Stokoe, C William","year":"1960","venue":"The Journal of Deaf Studies and Deaf Education","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Stoll et al., 2020] Stephanie Stoll, Necati Cihan Camgöz, Simon Hadfield, and Richard Bowden. Text2sign: towards sign language production using neural machine translation and generative adversarial networks. International Journal of Computer Vision, pages 1-18, 2020.","title":"Text2sign: towards sign language production using neural machine translation and generative adversarial networks","authors":"Stephanie Stoll, Cihan Necati, Simon Camgöz, Richard Hadfield, Bowden","year":"2020","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yin and Read, 2020a] Kayo Yin and Jesse Read. Attention is all you sign: sign language translation with transform- ers. In Sign Language Recognition, Translation and Pro- duction (SLRTP) Workshop-Extended Abstracts, volume 4, 2020.","title":"Attention is all you sign: sign language translation with transformers","authors":"Yin, Read, Kayo Yin, Jesse Read","year":"2020","venue":"Sign Language Recognition, Translation and Production (SLRTP) Workshop-Extended Abstracts","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yin and Read, 2020b] Kayo Yin and Jesse Read. Better sign language translation with STMC-transformer. In Pro- ceedings of the 28th International Conference on Compu- tational Linguistics, pages 5975-5989, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics.","title":"Better sign language translation with STMC-transformer","authors":"Yin, Read, Kayo Yin, Jesse Read","year":"2020","venue":"Proceedings of the 28th International Conference on Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yin et al., 2021a] Kayo Yin, Kenneth DeHaan, and Malihe Alikhani. Signed coreference resolution. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 4950-4961, Online and Punta Cana, Dominican Republic, November 2021. Asso- ciation for Computational Linguistics.","title":"Signed coreference resolution","authors":"Yin","year":"2021","venue":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yin et al., 2021b] Kayo Yin, Amit Moryossef, Julie Hochgesang, Yoav Goldberg, and Malihe Alikhani. Including signed languages in natural language process- ing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7347-7360, 2021.","title":"Including signed languages in natural language processing","authors":"Yin","year":"2021","venue":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}