{"type":"ScholarlyArticle","title":"Dense Temporal Convolution Network for Sign Language Translation","authors":[{"name":"Dan Guo","slug":"dan-guo","given":"Dan","family":"Guo","affiliation":"School of Computer Science and Information Engineering , Hefei University of Technology"},{"name":"Shuo Wang","slug":"shuo-wang","given":"Shuo","family":"Wang","affiliation":"School of Computer Science and Information Engineering , Hefei University of Technology"},{"name":"Qi Tian","slug":"qi-tian","given":"Qi","family":"Tian","affiliation":"Huawei Noah's Ark Lab Department of Computer Science , University of Texas at San Antonio"},{"name":"Meng Wang","slug":"meng-wang","given":"Meng","family":"Wang","affiliation":"School of Computer Science and Information Engineering , Hefei University of Technology"}],"abstract":"The sign language translation (SLT) which aims at translating a sign language video into natural language is a weakly supervised task, given that there is no exact mapping relationship between visual actions and textual words in a sentence label. To align the sign language actions and translate them into the respective words automatically, this paper proposes a dense temporal convolution network, termed DenseTCN which captures the actions in hierarchical views. Within this network, a temporal convolution (TC) is designed to learn the short-term correlation among adjacent features and further extended to a dense hierarchical structure. In the kth TC layer, we integrate the outputs of all preceding layers together: (1) The TC in a deeper layer essentially has larger receptive fields, which captures long-term temporal context by the hierarchical content transition. (2) The integration addresses the SLT problem by different views, including embedded short-term and extended longterm sequential learning. Finally, we adopt the CTC loss and a fusion strategy to learn the featurewise classification and generate the translated sentence. The experimental results on two popular sign language benchmarks, i.e. PHOENIX and USTCConSents, demonstrate the effectiveness of our proposed method in terms of various measurements.","doi":"10.24963/ijcai.2019/105","url":"https://ijcai.org/proceedings/2019/105","pdf":"https://ijcai.org/proceedings/2019/0105.pdf","markdown":"https://ijcai.org/proceedings/2019/105.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/105","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Computer Vision","track":"Main track","number":105,"pages":{"first":744,"last":750},"keywords":["Computer Vision: Action Recognition","Computer Vision: Biometrics, Face and Gesture Recognition"],"videos":[],"references":[{"position":1,"raw":"[Camgoz et al., 2017] Necati Cihan Camgoz, Simon Had- field, Oscar Koller, and Richard Bowden. Subunets: End- to-end hand shape and continuous sign language recogni- tion. In ICCV, pages 3075-3084, 2017.","title":"Subunets: Endto-end hand shape and continuous sign language recognition","authors":"Simon Necati Cihan Camgoz, Oscar Hadfield, Richard Koller, Bowden","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cui et al., 2017] Runpeng Cui, Hu Liu, and Changshui Zhang. Recurrent convolutional neural networks for con- tinuous sign language recognition by staged optimization. In CVPR, pages 1610-1618, 2017.","title":"Recurrent convolutional neural networks for continuous sign language recognition by staged optimization","authors":"Cui","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Futane et al., 2012] PR Futane, RV Dharaskar, and VM Thakare. A comparative study for approaches for hand sign language. In NCIPET, pages 36-39, 2012.","title":"PR Futane, RV Dharaskar, and VM Thakare. A comparative study for approaches for hand sign language","authors":"Futane","year":"2012","venue":"NCIPET","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Girshick et al., 2014] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, pages 580-587, 2014.","title":"Rich feature hierarchies for accurate object detection and semantic segmentation","authors":"Ross Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Graves et al., 2006] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In ICML, pages 369- 376, 2006.","title":"Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks","authors":"Alex Graves, Santiago Fernández, Faustino Gomez, Jürgen Schmidhuber","year":"2006","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Guo et al., 2017] Dan Guo, Wengang Zhou, Houqiang Li, and Meng Wang. Online early-late fusion based on adaptive hmm for sign language recognition. TOCCAP, 14(1):1-18, 2017.","title":"Online early-late fusion based on adaptive hmm for sign language recognition","authors":"Dan Guo, Wengang Zhou, Houqiang Li, Meng Wang","year":"2017","venue":"TOCCAP","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Guo et al., 2018] Dan Guo, Wengang Zhou, Houqiang Li, and Meng Wang. Hierarchical lstm for sign language translation. In AAAI, 2018.","title":"Hierarchical lstm for sign language translation","authors":"Dan Guo, Wengang Zhou, Houqiang Li, Meng Wang","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hara et al., 2017] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Learning spatio-temporal features with 3d residual networks for action recognition. In ICCV, vol- ume 2, page 4, 2017.","title":"Learning spatio-temporal features with 3d residual networks for action recognition","authors":"Hara","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang et al., 2015] Jie Huang, Wengang Zhou, Houqiang Li, and Weiping Li. Sign language recognition using 3d convolutional neural networks. In ICME, pages 1-6, 2015.","title":"Sign language recognition using 3d convolutional neural networks","authors":"Jie Huang, Wengang Zhou, Houqiang Li, Weiping Li","year":"2015","venue":"ICME","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2017] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, volume 1, page 3, 2017.","title":"Densely connected convolutional networks","authors":"Gao Huang, Zhuang Liu, Laurens Van Der Maaten, Kilian Q Weinberger","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Huang et al., 2018] Jie Huang, Wengang Zhou, Qilin Zhang, Houqiang Li, and Weiping Li. Video-based sign language recognition without temporal segmentation. In AAAI, 2018.","title":"Video-based sign language recognition without temporal segmentation","authors":"Jie Huang, Wengang Zhou, Qilin Zhang, Houqiang Li, Weiping Li","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Koller et al., 2015] Oscar Koller, Jens Forster, and Hermann Ney. Continuous sign language recognition: Towards large vocabulary statistical recognition systems handling multi- ple signers. CVIU, 141:108-125, 2015.","title":"Continuous sign language recognition: Towards large vocabulary statistical recognition systems handling multiple signers","authors":"Koller","year":"2015","venue":"CVIU","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Koller et al., 2016a] Oscar Koller, Hermann Ney, and Richard Bowden. Deep hand: How to train a cnn on 1 mil- lion hand images when your data is continuous and weakly labelled. In CVPR, pages 3793-3802, 2016.","title":"Deep hand: How to train a cnn on 1 million hand images when your data is continuous and weakly labelled","authors":"Koller","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Koller et al., 2016b] Oscar Koller, O Zargaran, Hermann Ney, and Richard Bowden. Deep sign: Hybrid cnn- hmm for continuous sign language recognition. In BMVC, page 12, 2016.","title":"Deep sign: Hybrid cnnhmm for continuous sign language recognition","authors":"Koller","year":"2016","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Koller et al., 2017] Oscar Koller, Sepehr Zargaran, and Her- mann Ney. Re-sign: Re-aligned end-to-end sequence mod- elling with deep recurrent cnn-hmms. In CVPR, pages 4297-4305, 2017.","title":"Re-sign: Re-aligned end-to-end sequence modelling with deep recurrent cnn-hmms","authors":"Koller","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1097- 1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pan et al., 2016] Yingwei Pan, Tao Mei, Ting Yao, Houqiang Li, and Yong Rui. Jointly modeling embedding and translation to bridge video and language. In CVPR, pages 4594-4602, 2016.","title":"Jointly modeling embedding and translation to bridge video and language","authors":"Pan","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pu et al., 2018] Junfu Pu, Wengang Zhou, and Houqiang Li. Dilated convolutional network with iterative optimization for coutinuous sign language recognition. In IJCAI, pages 885-891, 2018.","title":"Dilated convolutional network with iterative optimization for coutinuous sign language recognition","authors":"Pu","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Quattoni et al., 2007] Ariadna Quattoni, Sybor Wang, Louis-Philippe Morency, Morency Collins, and Trevor Darrell. Hidden conditional random fields. TPAMI, 29(10):1848-1852, 2007.","title":"Hidden conditional random fields","authors":"Ariadna Quattoni, Sybor Wang, Louis-Philippe Morency, Morency Collins, Trevor Darrell","year":"2007","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[René and Hager, 2017] Colin Lea Michael D Flynn René and Vidal Austin Reiter Gregory D Hager. Temporal con- volutional networks for action segmentation and detection. In ICCV, pages 1003-1012, 2017.","title":"Temporal convolutional networks for action segmentation and detection","authors":"Colin Lea, Michael, Flynn René, Vidal Austin, Reiter Gregory, D Hager","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Szegedy et al., 2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich, et al. Going deeper with convolutions. In CVPR, 2015.","title":"Going deeper with convolutions","authors":"Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Venugopalan et al., 2015a] Subhashini Venugopalan, Mar- cus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko. Sequence to sequence- video to text. In ICCV, pages 4534-4542, 2015.","title":"Sequence to sequencevideo to text","authors":"Venugopalan","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Venugopalan et al., 2015b] Subhashini Venugopalan, Hui- juan Xu, Jeff Donahue, Marcus Rohrbach, Raymond Mooney, and Kate Saenko. Translating videos to nat- ural language using deep recurrent neural networks. In NAACL-HLT, 2015.","title":"Translating videos to natural language using deep recurrent neural networks","authors":"Venugopalan","year":"2015","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2018] Shuo Wang, Dan Guo, Wengang Zhou, ZhengJun Zha, and Meng Wang. Connectionist temporal fusion for sign language translation. In MM, pages 1483- 1491, 2018.","title":"Connectionist temporal fusion for sign language translation","authors":"Shuo Wang, Dan Guo, Wengang Zhou, Zhengjun Zha, Meng Wang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang et al., 2016] Zichao Yang, Diyi Yang, Chris Dyer, Xi- aodong He, Alex Smola, and Eduard Hovy. Hierarchical attention networks for document classification. In NAACL HLT, pages 1480-1489, 2016.","title":"Hierarchical attention networks for document classification","authors":"Yang","year":"2016","venue":"NAACL HLT","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yao et al., 2015] Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, and Aaron Courville. Describing videos by exploiting tempo- ral structure. In ICCV, pages 4507-4515, 2015.","title":"Describing videos by exploiting temporal structure","authors":"Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, Aaron Courville","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yu and Koltun, 2016] Fisher Yu and Vladlen Koltun. Multi- scale context aggregation by dilated convolutions. ICLR, 2016.","title":"Multiscale context aggregation by dilated convolutions","authors":"Koltun ; Fisher Yu, Vladlen Yu, Koltun","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2014] Jihai Zhang, Wengang Zhou, and Houqiang Li. A threshold-based hmm-dtw approach for continuous sign language recognition. In ICIMCS, page 237, 2014.","title":"A threshold-based hmm-dtw approach for continuous sign language recognition","authors":"Jihai Zhang, Wengang Zhou, Houqiang Li","year":"2014","venue":"ICIMCS","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2016] Jihai Zhang, Wengang Zhou, Chao Xie, Junfu Pu, and Houqiang Li. Chinese sign language recog- nition with adaptive hmm. In ICME, pages 1-6, 2016.","title":"Chinese sign language recognition with adaptive hmm","authors":"Jihai Zhang, Wengang Zhou, Chao Xie, Junfu Pu, Houqiang Li","year":"2016","venue":"ICME","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}