{"type":"ScholarlyArticle","title":"Dilated Convolutional Network with Iterative Optimization for Continuous Sign Language Recognition","authors":[{"name":"Junfu Pu","slug":"junfu-pu","given":"Junfu","family":"Pu","affiliation":"CAS Key Laboratory of Technology in Geo-spatial Information Processing and Application System , EEIS Department , University of Science and Technology of China"},{"name":"Wengang Zhou","slug":"wengang-zhou","given":"Wengang","family":"Zhou","affiliation":"CAS Key Laboratory of Technology in Geo-spatial Information Processing and Application System , EEIS Department , University of Science and Technology of China"},{"name":"Houqiang Li","slug":"houqiang-li","given":"Houqiang","family":"Li","affiliation":"CAS Key Laboratory of Technology in Geo-spatial Information Processing and Application System , EEIS Department , University of Science and Technology of China"}],"abstract":"This paper presents a novel deep neural architecture with iterative optimization strategy for real-world continuous sign language recognition. Generally, a continuous sign language recognition system consists of visual input encoder for feature extraction and a sequence learning model to learn the correspondence between the input sequence and the output sentence-level labels. We use a 3D residual convolutional network (3D-ResNet) to extract visual features. After that, a stacked dilated convolutional network with Connectionist Temporal Classification (CTC) is applied for learning the mapping between the sequential features and the text sentence. The deep network is hard to train since the CTC loss has limited contribution to early CNN parameters. To alleviate this problem,  we design an iterative optimization strategy to train our architecture. We generate pseudo-labels for video clips from sequence learning model with CTC, and fine-tune the 3D-ResNet with the supervision of pseudo-labels for a better feature representation. We alternately optimize feature extractor and sequence learning model with iterative steps. Experimental results on RWTH-PHOENIX-Weather, a large real-world continuous sign language recognition benchmark, demonstrate the advantages and  effectiveness of our proposed method.","doi":"10.24963/ijcai.2018/123","url":"https://ijcai.org/proceedings/2018/123","pdf":"https://ijcai.org/proceedings/2018/0123.pdf","markdown":"https://ijcai.org/proceedings/2018/123.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/123","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":123,"pages":{"first":885,"last":891},"keywords":["Machine Learning: Deep Learning","Computer Vision: Language and Vision"],"videos":[],"references":[{"position":1,"raw":"[Abadi et al., 2016] Martín Abadi, Paul Barham, Jianmin Chen, et al. Tensorflow: A system for large-scale machine learning. In OSDI, volume 16, pages 265-283, 2016.","title":"Tensorflow: A system for large-scale machine learning","authors":"Martín Abadi, Paul Barham, Jianmin Chen","year":"2016","venue":"OSDI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Camgoz et al., 2017] Necati Cihan Camgoz, Simon Had- field, Oscar Koller, and Richard Bowden. Subunets: End- to-end hand shape and continuous sign language recogni- tion. In ICCV, 2017.","title":"Subunets: Endto-end hand shape and continuous sign language recognition","authors":"Simon Necati Cihan Camgoz, Oscar Hadfield, Richard Koller, Bowden","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the proper- ties of neural machine translation: Encoder-decoder ap- proaches. arXiv preprint arXiv:1409.1259, 2014.","title":"On the properties of neural machine translation: Encoder-decoder approaches","authors":"Cho","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cui et al., 2017] Runpeng Cui, Hu Liu, and Changshui Zhang. Recurrent convolutional neural networks for con- tinuous sign language recognition by staged optimization. In CVPR, pages 7361-7369, 2017.","title":"Recurrent convolutional neural networks for continuous sign language recognition by staged optimization","authors":"Cui","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gehring et al., 2017] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convo- lutional sequence to sequence learning. arXiv preprint arXiv:1705.03122, 2017.","title":"Convolutional sequence to sequence learning","authors":"Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N Dauphin","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Graves et al., 2006] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In ICML, pages 369- 376, 2006.","title":"Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks","authors":"Alex Graves, Santiago Fernández, Faustino Gomez, Jürgen Schmidhuber","year":"2006","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Guo et al., 2018] Dan Guo, Wengang Zhou, Meng Wang, and Houqiang Li. Hierarchical lstm for sign language translation. In AAAI, 2018.","title":"Hierarchical lstm for sign language translation","authors":"Dan Guo, Wengang Zhou, Meng Wang, Houqiang Li","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2015] Jie Huang, Wengang Zhou, Houqiang Li, and Weiping Li. Sign language recognition using 3d convolutional neural networks. In ICME, pages 1-6, 2015.","title":"Sign language recognition using 3d convolutional neural networks","authors":"Jie Huang, Wengang Zhou, Houqiang Li, Weiping Li","year":"2015","venue":"ICME","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Huang et al., 2018] Jie Huang, Wengang Zhou, Qilin Zhang, Houqiang Li, and Weiping Li. Video-based sign language recognition without temporal segmentation. In AAAI, 2018.","title":"Video-based sign language recognition without temporal segmentation","authors":"Jie Huang, Wengang Zhou, Qilin Zhang, Houqiang Li, Weiping Li","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ji et al., 2013] Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 3d convolutional neural networks for human action recognition. TPAMI, 35(1):221-231, 2013.","title":"3d convolutional neural networks for human action recognition","authors":"Ji","year":"2013","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kim et al., 2017] Kyung-Min Kim, Min-Oh Heo, Seong- Ho Choi, and Byoung-Tak Zhang. Deepstory: video story qa by deep embedded memory networks. In IJCAI, 2017.","title":"Deepstory: video story qa by deep embedded memory networks","authors":"Kyung-Min Kim, Min-Oh Heo, Seong-Ho Choi, Byoung-Tak Zhang","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/280"},{"position":14,"raw":"[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Koller et al., 2015] Oscar Koller, Jens Forster, and Hermann Ney. Continuous sign language recognition: Towards large vocabulary statistical recognition systems handling multi- ple signers. CVIU, 141:108-125, 2015.","title":"Continuous sign language recognition: Towards large vocabulary statistical recognition systems handling multiple signers","authors":"Koller","year":"2015","venue":"CVIU","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Koller et al., 2016a] Oscar Koller, Hermann Ney, and Richard Bowden. Deep hand: How to train a cnn on 1 mil- lion hand images when your data is continuous and weakly labelled. In CVPR, pages 3793-3802, 2016.","title":"Deep hand: How to train a cnn on 1 million hand images when your data is continuous and weakly labelled","authors":"Koller","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Koller et al., 2016b] Oscar Koller, O Zargaran, Hermann Ney, and Richard Bowden. Deep sign: Hybrid cnn-hmm for continuous sign language recognition. In BMVC, 2016.","title":"Deep sign: Hybrid cnn-hmm for continuous sign language recognition","authors":"Koller","year":"2016","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Koller et al., 2017] Oscar Koller, Sepehr Zargaran, and Her- mann Ney. Re-sign: Re-aligned end-to-end sequence mod- elling with deep recurrent cnn-hmms. In CVPR, 2017.","title":"Re-sign: Re-aligned end-to-end sequence modelling with deep recurrent cnn-hmms","authors":"Koller","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2017] Xuelong Li, Bin Zhao, and Xiaoqiang Lu. Mam-rnn: Multi-level attention model based rnn for video captioning. In IJCAI, 2017.","title":"Mam-rnn: Multi-level attention model based rnn for video captioning","authors":"Xuelong Li, Bin Zhao, Xiaoqiang Lu","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/307"},{"position":20,"raw":"[Oord et al., 2016a] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, et al. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499, 2016.","title":"Wavenet: A generative model for raw audio","authors":"Oord","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Oord et al., 2016b] Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. Conditional image generation with pixel- cnn decoders. In NIPS, 2016.","title":"Conditional image generation with pixelcnn decoders","authors":"Oord","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Pigou et al., 2015] Lionel Pigou, Aäron van den Oord, Sander Dieleman, Mieke Van Herreweghe, and Joni Dambre. Beyond temporal pooling: Recurrence and tem- poral convolutions for gesture recognition in video. IJCV, pages 1-10, 2015.","title":"Beyond temporal pooling: Recurrence and temporal convolutions for gesture recognition in video","authors":"Lionel Pigou, Aäron Van Den, Sander Oord, Mieke Dieleman, Joni Van Herreweghe, Dambre","year":"2015","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pu et al., 2016] Junfu Pu, Wengang Zhou, and Houqiang Li. Sign language recognition with multi-modal features. In PCM, pages 252-261, 2016.","title":"Sign language recognition with multi-modal features","authors":"Pu","year":"2016","venue":"PCM","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Qiu et al., 2017] Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio-temporal representation with pseudo-3d residual networks. In ICCV, volume 1, page 8, 2017.","title":"Learning spatio-temporal representation with pseudo-3d residual networks","authors":"Qiu","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Song et al., 2017] Jingkuan Song, Zhao Guo, Lianli Gao, Wu Liu, Dongxiang Zhang, and Heng Tao Shen. Hier- archical lstm with adjusted temporal attention for video captioning. In IJCAI, 2017.","title":"Hierarchical lstm with adjusted temporal attention for video captioning","authors":"Song","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/381"},{"position":26,"raw":"[Starner et al., 1998] Thad Starner, Joshua Weaver, and Alex Pentland. Real-time american sign language recognition using desk and wearable computer based video. TPAMI, 20(12):1371-1375, 1998.","title":"Real-time american sign language recognition using desk and wearable computer based video","authors":"Thad Starner, Joshua Weaver, Alex Pentland","year":"1998","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tran et al., 2015] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In ICCV, pages 4489-4497, 2015.","title":"Learning spatiotemporal features with 3d convolutional networks","authors":"Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, Manohar Paluri","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2006] Sy Bor Wang, Ariadna Quattoni, L-P Morency, David Demirdjian, and Trevor Darrell. Hid- den conditional random fields for gesture recognition. In CVPR, volume 2, pages 1521-1527, 2006.","title":"Hidden conditional random fields for gesture recognition","authors":"Wang Sy Bor, Ariadna Quattoni, L-P Morency, David Demirdjian, Trevor Darrell","year":"2006","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2016] Jihai Zhang, Wengang Zhou, Chao Xie, Junfu Pu, and Houqiang Li. Chinese sign language recog- nition with adaptive hmm. In ICME, pages 1-6, 2016.","title":"Chinese sign language recognition with adaptive hmm","authors":"Jihai Zhang, Wengang Zhou, Chao Xie, Junfu Pu, Houqiang Li","year":"2016","venue":"ICME","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhao et al., 2017] Zhou Zhao, Qifan Yang, Deng Cai, Xi- aofei He, and Yueting Zhuang. Video question answering via hierarchical spatio-temporal attention networks. In IJ- CAI, volume 2, 2017.","title":"Video question answering via hierarchical spatio-temporal attention networks","authors":"Zhou Zhao, Qifan Yang, Deng Cai, Xiaofei He, Yueting Zhuang","year":"2017","venue":"IJ-CAI","doi":null,"url":null,"internal_url":"/proceedings/2017/492"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}