{"type":"ScholarlyArticle","title":"Efficient Sign Language Translation with a Curriculum-based Non-autoregressive Decoder","authors":[{"name":"Pei Yu","slug":"pei-yu","given":"Pei","family":"Yu","affiliation":"School of Informatics , Xiamen University , China; Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University) , Ministry of Culture and Tourism , China"},{"name":"Liang Zhang","slug":"liang-zhang","given":"Liang","family":"Zhang","affiliation":"School of Informatics , Xiamen University , China; Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University) , Ministry of Culture and Tourism , China"},{"name":"Biao Fu","slug":"biao-fu","given":"Biao","family":"Fu","affiliation":"School of Informatics , Xiamen University , China; Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University) , Ministry of Culture and Tourism , China"},{"name":"Yidong Chen","slug":"yidong-chen","given":"Yidong","family":"Chen","affiliation":"School of Informatics , Xiamen University , China; Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University) , Ministry of Culture and Tourism , China"}],"abstract":"Most existing studies on Sign Language Translation (SLT) employ AutoRegressive  Decoding Mechanism (AR-DM) to generate target sentences. However, the main disadvantage of the AR-DM is high inference latency. To address this problem, we introduce Non-AutoRegressive Decoding Mechanism (NAR-DM) into SLT, which generates the whole sentence at once. Meanwhile, to improve its decoding ability, we integrate the advantages of curriculum learning and NAR-DM and propose a Curriculum-based NAR Decoder (CND). Specifically, the lower layers of the CND are expected to predict simple tokens that could be predicted correctly using source-side information solely. Meanwhile, the upper layers could predict complex tokens based on the lower layers' predictions. Therefore, our CND significantly reduces the model's inference latency while maintaining its competitive performance. Moreover, to further boost the performance of our CND, we propose a mutual learning framework, containing two decoders, i.e., an AR decoder and our CND. We jointly train the two decoders and minimize the KL divergence between their outputs, which enables our CND to learn the forward sequential knowledge from the strengthened AR decoder. Experimental results on PHOENIX2014T and CSL-Daily demonstrate that our model consistently outperforms all competitive baselines and achieves 7.92/8.02× speed-up compared to the AR SLT model respectively. Our source code is available at https://github.com/yp20000921/CND.","doi":"10.24963/ijcai.2023/584","url":"https://ijcai.org/proceedings/2023/584","pdf":"https://ijcai.org/proceedings/2023/0584.pdf","markdown":"https://ijcai.org/proceedings/2023/584.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/584","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Natural Language Processing","track":"Main Track","number":584,"pages":{"first":5260,"last":5268},"keywords":["Natural Language Processing: NLP: Machine translation and multilinguality"],"videos":[],"references":[{"position":1,"raw":"[Camgoz et al., 2018] Necati Cihan Camgoz, Simon Had- field, Oscar Koller, Hermann Ney, and Richard Bowden. Neural sign language translation. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 7784-7793, 2018.","title":"Neural sign language translation","authors":"Simon Necati Cihan Camgoz, Oscar Hadfield, Hermann Koller, Richard Ney, Bowden","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Camgoz et al., 2020] Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. Sign language transformers: Joint end-to-end sign language recognition and translation. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pages 10023-10033, 2020.","title":"Sign language transformers: Joint end-to-end sign language recognition and translation","authors":"Oscar Necati Cihan Camgoz, Simon Koller, Richard Hadfield, Bowden","year":"2020","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cao et al., 2022] Yong Cao, Wei Li, Xianzhi Li, Min Chen, Guangyong Chen, Long Hu, Zhengdao Li, and Kai Hwang. Explore more guidance: A task-aware instruction network for sign language translation enhanced with data augmentation. In Findings of the Association for Com- putational Linguistics: NAACL 2022, pages 2679-2690, Seattle, United States, July 2022. Association for Compu- tational Linguistics.","title":"Explore more guidance: A task-aware instruction network for sign language translation enhanced with data augmentation","authors":"Yong Cao, Wei Li, Xianzhi Li, Min Chen, Guangyong Chen, Long Hu, Zhengdao Li, Kai Hwang","year":"2022","venue":"Findings of the Association for Computational Linguistics: NAACL 2022","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2022a] Yutong Chen, Fangyun Wei, Xiao Sun, Zhirong Wu, and Stephen Lin. A simple multi-modality transfer learning baseline for sign language translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5120-5130, 2022.","title":"A simple multi-modality transfer learning baseline for sign language translation","authors":"Chen","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2022b] Yutong Chen, Ronglai Zuo, Fangyun Wei, Yu Wu, Shujie Liu, and Brian Mak. Two-stream net- work for sign language recognition and translation. arXiv preprint arXiv:2211.01367, 2022.","title":"Two-stream network for sign language recognition and translation","authors":"Chen","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ghazvininejad et al., 2019] Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language mod- els. In Proceedings of the 2019 Conference on Empiri- cal Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pages 6112-6121, 2019.","title":"Mask-predict: Parallel decoding of conditional masked language models","authors":"Marjan Ghazvininejad, Omer Levy, Yinhan Liu, Luke Zettlemoyer","year":"2019","venue":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ghazvininejad et al., 2020] Marjan Ghazvininejad, Omer Levy, and Luke Zettlemoyer. Semi-autoregressive train- ing improves mask-predict decoding. arXiv preprint arXiv:2001.08785, 2020.","title":"Semi-autoregressive training improves mask-predict decoding","authors":"Marjan Ghazvininejad, Omer Levy, Luke Zettlemoyer","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Goyal et al., 2017] Priya Goyal, Piotr Dollár, Ross Gir- shick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Ky- rola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Ac- curate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.","title":"Accurate, large minibatch sgd: Training imagenet in 1 hour","authors":"Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, Kaiming He","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gu and Kong, 2021] Jiatao Gu and Xiang Kong. Fully non- autoregressive neural machine translation: Tricks of the trade. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 120-133, 2021.","title":"Jiatao Gu and Xiang Kong. Fully nonautoregressive neural machine translation: Tricks of the trade","authors":"Kong Gu","year":"2021","venue":"Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gu et al., 2018] Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. Non- autoregressive neural machine translation. In International Conference on Learning Representations, 2018.","title":"Nonautoregressive neural machine translation","authors":"Gu","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hao et al., 2021] Aiming Hao, Yuecong Min, and Xilin Chen. Self-mutual distillation learning for continuous sign language recognition. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 11303- 11312, 2021.","title":"Self-mutual distillation learning for continuous sign language recognition","authors":"Hao","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Huang et al., 2021] Xiao Shi Huang, Felipe Perez, and Maksims Volkovs. Improving non-autoregressive transla- tion models without distillation. In International Confer- ence on Learning Representations, 2021.","title":"Improving non-autoregressive translation models without distillation","authors":"Xiao Shi Huang, Felipe Perez, Maksims Volkovs","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Huang et al., 2022a] Chenyang Huang, Hao Zhou, Osmar R Zaïane, Lili Mou, and Lei Li. Non-autoregressive transla- tion with layer-wise prediction and deep supervision. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 36, pages 10776-10784, 2022.","title":"Non-autoregressive translation with layer-wise prediction and deep supervision","authors":"Huang","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Huang et al., 2022b] Fei Huang, Hao Zhou, Yang Liu, Hang Li, and Minlie Huang. Directed acyclic transformer for non-autoregressive machine translation. In Proceedings of the 39th International Conference on Machine Learning, ICML 2022, 2022.","title":"Directed acyclic transformer for non-autoregressive machine translation","authors":"Huang","year":"2022","venue":"Proceedings of the 39th International Conference on Machine Learning, ICML 2022","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kim and Rush, 2016] Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317-1327, 2016.","title":"Sequence-level knowledge distillation","authors":"Yoon Kim, Alexander M Rush","year":"2016","venue":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2022] Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. Elmer: A non- autoregressive pre-trained language model for effi- cient and effective text generation. arXiv preprint arXiv:2210.13304, 2022.","title":"Elmer: A nonautoregressive pre-trained language model for efficient and effective text generation","authors":"Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jian-Yun Nie, Ji-Rong Wen","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lin, 2004] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.","title":"Rouge: A package for automatic evaluation of summaries","authors":"Chin-Yew Lin","year":"2004","venue":"Text summarization branches out","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for au- tomatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Compu- tational Linguistics, pages 311-318, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"Proceedings of the 40th annual meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Qian et al., 2021] Lihua Qian, Hao Zhou, Yu Bao, Mingx- uan Wang, Lin Qiu, Weinan Zhang, Yong Yu, and Lei Li. Glancing transformer for non-autoregressive neural ma- chine translation. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 1993- 2003, 2021.","title":"Glancing transformer for non-autoregressive neural machine translation","authors":"Lihua Qian, Hao Zhou, Yu Bao, Mingxuan Wang, Lin Qiu, Weinan Zhang, Yong Yu, Lei Li","year":"2021","venue":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shao et al., 2019] Chenze Shao, Yang Feng, Jinchao Zhang, Fandong Meng, Xilin Chen, and Jie Zhou. Retrieving sequential information for non-autoregressive neural ma- chine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3013-3024, Florence, Italy, July 2019. Association for Computational Linguistics.","title":"Retrieving sequential information for non-autoregressive neural machine translation","authors":"Shao","year":"2019","venue":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wei et al., 2019] Bingzhen Wei, Mingxuan Wang, Hao Zhou, Junyang Lin, and Xu Sun. Imitation learning for non-autoregressive neural machine translation. In Pro- ceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1304-1312, 2019.","title":"Imitation learning for non-autoregressive neural machine translation","authors":"Wei","year":"2019","venue":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yin et al., 2021] Aoxiong Yin, Zhou Zhao, Jinglin Liu, Weike Jin, Meng Zhang, Xingshan Zeng, and Xiaofei He. Simulslt: End-to-end simultaneous sign language transla- tion. In ACM Multimedia, 2021.","title":"Simulslt: End-to-end simultaneous sign language translation","authors":"Aoxiong Yin, Zhou Zhao, Jinglin Liu, Weike Jin, Meng Zhang, Xingshan Zeng, Xiaofei He","year":"2021","venue":"ACM Multimedia","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2023] Biao Zhang, Mathias Müller, and Rico Sennrich. SLTUNET: A simple unified model for sign lan- guage translation. In The Eleventh International Confer- ence on Learning Representations, 2023.","title":"SLTUNET: A simple unified model for sign language translation","authors":"Biao Zhang, Mathias Müller, Rico Sennrich","year":"2023","venue":"The Eleventh International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zheng et al., 2021] Jiangbin Zheng, Yidong Chen, Chong Wu, Xiaodong Shi, and Suhail Muhammad Kamal. En- hancing neural sign language translation by highlight- ing the facial expression information. Neurocomputing, 464:462-472, 2021.","title":"Enhancing neural sign language translation by highlighting the facial expression information","authors":"Zheng","year":"2021","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zheng et al., 2023] Jiangbin Zheng, Yile Wang, Cheng Tan, Siyuan Li, Ge Wang, Jun Xia, Yidong Chen, and Stan Z Li. Cvt-slr: Contrastive visual-textual transformation for sign language recognition with variational alignment. arXiv preprint arXiv:2303.05725, 2023.","title":"Cvt-slr: Contrastive visual-textual transformation for sign language recognition with variational alignment","authors":"Zheng","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhou et al., 2021a] Hao Zhou, Wengang Zhou, Weizhen Qi, Junfu Pu, and Houqiang Li. Improving sign language translation with monolingual data by sign back-translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1316-1325, 2021.","title":"Improving sign language translation with monolingual data by sign back-translation","authors":"Zhou","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhou et al., 2021b] Hao Zhou, Wengang Zhou, Yun Zhou, and Houqiang Li. Spatial-temporal multi-cue network for sign language recognition and translation. IEEE Transac- tions on Multimedia, 24:768-779, 2021.","title":"Spatial-temporal multi-cue network for sign language recognition and translation","authors":"Zhou","year":"2021","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhou et al., 2022] Chulun Zhou, Fandong Meng, Jie Zhou, Min Zhang, Hongji Wang, and Jinsong Su. Confidence based bidirectional global context aware training frame- work for neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2878-2889, 2022.","title":"Confidence based bidirectional global context aware training framework for neural machine translation","authors":"Zhou","year":"2022","venue":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}