{"type":"ScholarlyArticle","title":"Connectionist Temporal Modeling of Video and Language: a Joint Model for Translation and Sign Labeling","authors":[{"name":"Dan Guo","slug":"dan-guo","given":"Dan","family":"Guo","affiliation":"School of Computer Science and Information Engineering , Hefei University of Technology"},{"name":"Shengeng Tang","slug":"shengeng-tang","given":"Shengeng","family":"Tang","affiliation":"School of Computer Science and Information Engineering , Hefei University of Technology"},{"name":"Meng Wang","slug":"meng-wang","given":"Meng","family":"Wang","affiliation":"School of Computer Science and Information Engineering , Hefei University of Technology"}],"abstract":"Online sign interpretation suffers from challenges presented by hybrid semantics learning among sequential variations of visual representations, sign linguistics, and textual grammars. This paper proposes a Connectionist Temporal Modeling (CTM) network for sentence translation and sign labeling. To acquire short-term temporal correlations, a Temporal Convolution Pyramid (TCP) module is performed on 2D CNN features to realize (2D+1D)=pseudo 3D' CNN features. CTM aligns the pseudo 3D' with the original 3D CNN clip features and fuses them. Next, we implement a connectionist decoding scheme for long-term sequential learning. Here, we embed dynamic programming into the decoding scheme, which learns temporal mapping among features, sign labels, and the generated sentence directly. The solution using dynamic programming to sign labeling is considered as pseudo labels. Finally, we utilize the pseudo supervision cues in an end-to-end framework. A joint objective function is designed to measure feature correlation, entropy regularization on sign labeling, and probability maximization on sentence decoding. The experimental results using the RWTH-PHOENIX-Weather and USTC-CSL datasets demonstrate the effectiveness of the proposed approach.","doi":"10.24963/ijcai.2019/106","url":"https://ijcai.org/proceedings/2019/106","pdf":"https://ijcai.org/proceedings/2019/0106.pdf","markdown":"https://ijcai.org/proceedings/2019/106.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/106","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Computer Vision","track":"Main track","number":106,"pages":{"first":751,"last":757},"keywords":["Computer Vision: Biometrics, Face and Gesture Recognition","Computer Vision: Language and Vision"],"videos":[],"references":[{"position":1,"raw":"Methods Off-line Modality VAL(%) TEST(%) Iterations hand traj face des / ins WER des / ins WER HOG-3D [Koller et al., 2015] - √ 25.8 / 4.2 60.9 23.2 / 4.1 58.1 CMLLR [Koller et al., 2015] - √ √ √ 21.8 / 3.9 55.0 20.3 / 4.5 53.0 -Mio-H [Koller et al., 2016a] 3 √ 19.1 / 4.1 51.6 17.5 / 4.5 50.2 -Mio-H+CMLLR [Koller et al., 2016a] 3 √ √ √ 16.3 / 4.6 47.1 15.2 / 4.6 45","title":"Methods Off-line Modality VAL(%) TEST(%) Iterations hand traj face des / ins WER des / ins WER HOG-3D [Koller et al","authors":"Koller","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"1 CNN-Hybrid [Koller et al., 2016b] 3 √ 12.6 / 5.1 38.3 11.1 / 5.7 38","title":"1 CNN-Hybrid","authors":"Koller","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"8 Staged-Opt-init [Cui et al., 2017] - √ 16.3 / 6.7 46.2 15.1 / 7.4 46.9 Staged-Opt [Cui et al., 2017] 3 √ 13.7 / 7.3 39.4 12.2 / 7.5 38.7 SubUNets [Camgoz et al., 2017] - √ 14.6 / 4.0 40.8 14.3 / 4.0 40","title":"8 Staged-Opt-init","authors":"Cui","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"7 Dilated-CNN-init [Pu et al., 2018] - 18.5 / 2.6 60.3 18.1 / 2.8 59.7 Dilated-CNN [Pu et al., 2018] 5 8.3 / 4.8 38.0 7.6 / 4.8 37.3 Our Method - 11.6 / 6.3 38.9 10.9 / 6.4","title":"7 Dilated-CNN-init","authors":"Pu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Camgoz et al., 2017] Necati Cihan Camgoz, Simon Had- field, Oscar Koller, and Richard Bowden. Subunets: end- to-end hand shape and continuous sign language recogni- tion. In ICCV, pages 3075-3084, 2017.","title":"Subunets: endto-end hand shape and continuous sign language recognition","authors":"Simon Necati Cihan Camgoz, Oscar Hadfield, Richard Koller, Bowden","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cihan Camgoz et al., 2018] Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bow- den. Neural sign language translation. In CVPR, pages 7784-7793, 2018.","title":"Neural sign language translation","authors":"Simon Necati Cihan Camgoz, Oscar Hadfield, Hermann Koller, Richard Ney, Bowden","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cui et al., 2017] Runpeng Cui, Hu Liu, and Changshui Zhang. Recurrent convolutional neural networks for con- tinuous sign language recognition by staged optimization. In CVPR, pages 1610-1618, 2017.","title":"Recurrent convolutional neural networks for continuous sign language recognition by staged optimization","authors":"Cui","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Graves et al., 2006] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In ICML, pages 369- 376, 2006.","title":"Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks","authors":"Alex Graves, Santiago Fernández, Faustino Gomez, Jürgen Schmidhuber","year":"2006","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Guo et al., 2017] Dan Guo, Wengang Zhou, Houqiang Li, and Meng Wang. Online early-late fusion based on adap- tive hmm for sign language recognition. TOMM, 14(1):8, 2017.","title":"Online early-late fusion based on adaptive hmm for sign language recognition","authors":"Dan Guo, Wengang Zhou, Houqiang Li, Meng Wang","year":"2017","venue":"TOMM","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Guo et al., 2018] Dan Guo, Wengang Zhou, Houqiang Li, and Meng Wang. Hierarchical lstm for sign language translation. In AAAI, 2018.","title":"Hierarchical lstm for sign language translation","authors":"Dan Guo, Wengang Zhou, Houqiang Li, Meng Wang","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hara et al., 2017] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Learning spatio-temporal features with 3d residual networks for action recognition. In ICCV, pages 3154-3160, 2017.","title":"Learning spatio-temporal features with 3d residual networks for action recognition","authors":"Hara","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Huang et al., 2018] Jie Huang, Wengang Zhou, Qilin Zhang, Houqiang Li, and Weiping Li. Video-based sign language recognition without temporal segmentation. In AAAI, 2018.","title":"Video-based sign language recognition without temporal segmentation","authors":"Jie Huang, Wengang Zhou, Qilin Zhang, Houqiang Li, Weiping Li","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jelodar et al., 2018] Ahmad Babaeian Jelodar, David Paulius, and Yu Sun. Long activity video understanding using functional object-oriented network. TMM, 2018.","title":"Long activity video understanding using functional object-oriented network","authors":"Ahmad Babaeian, Jelodar, David Paulius, Yu Sun","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Joshi et al., 2017] Ajjen Joshi, Soumya Ghosh, Margrit Betke, Stan Sclaroff, and Hanspeter Pfister. Personaliz- ing gesture recognition using hierarchical bayesian neural networks. In CVPR, pages 6513-6522, 2017.","title":"Personalizing gesture recognition using hierarchical bayesian neural networks","authors":"Joshi","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kacem et al., 2018] Anis Kacem, Mohamed Daoudi, Boul- baba Ben Amor, Stefano Berretti, and Juan Carlos Alvarez-Paiva. A novel geometric framework on gram ma- trix trajectories for human behavior understanding. 2018.","title":"A novel geometric framework on gram matrix trajectories for human behavior understanding","authors":"Anis Kacem, Mohamed Daoudi, Boulbaba Ben Amor, Stefano Berretti, Juan Carlos Alvarez-Paiva","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Koller et al., 2015] Oscar Koller, Jens Forster, and Hermann Ney. Continuous sign language recognition: towards large vocabulary statistical recognition systems handling multi- ple signers. CVIU, 141:108-125, 2015.","title":"Continuous sign language recognition: towards large vocabulary statistical recognition systems handling multiple signers","authors":"Koller","year":"2015","venue":"CVIU","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Koller et al., 2016a] Oscar Koller, Hermann Ney, and Richard Bowden. Deep hand: How to train a cnn on 1 mil- lion hand images when your data is continuous and weakly labelled. In CVPR, pages 3793-3802, 2016.","title":"Deep hand: How to train a cnn on 1 million hand images when your data is continuous and weakly labelled","authors":"Koller","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Koller et al., 2016b] Oscar Koller, O Zargaran, Hermann Ney, and Richard Bowden. Deep sign: hybrid cnn-hmm for continuous sign language recognition. In BMVC, 2016.","title":"Deep sign: hybrid cnn-hmm for continuous sign language recognition","authors":"Koller","year":"2016","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Koller et al., 2017] Oscar Koller, Sepehr Zargaran, and Her- mann Ney. Re-sign: Re-aligned end-to-end sequence mod- elling with deep recurrent cnn-hmms. In CVPR, pages 4297-4305, 2017.","title":"Re-sign: Re-aligned end-to-end sequence modelling with deep recurrent cnn-hmms","authors":"Koller","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lefebvre et al., 2015] Grégoire Lefebvre, Samuel Berlemont, Franck Mamalet, and Christophe Garcia. Inertial gesture recognition with blstm-rnn. In ANN, pages 393-410. Springer, 2015.","title":"Inertial gesture recognition with blstm-rnn","authors":"Grégoire Lefebvre, Samuel Berlemont, Franck Mamalet, Christophe Garcia","year":"2015","venue":"ANN","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lin et al., 2014] Yushun Lin, Xiujuan Chai, Yu Zhou, and Xilin Chen. Curve matching from the view of manifold for sign language recognition. In ACCV, pages 233-246, 2014.","title":"Curve matching from the view of manifold for sign language recognition","authors":"Yushun Lin, Xiujuan Chai, Yu Zhou, Xilin Chen","year":"2014","venue":"ACCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Liu et al., 2016] Tao Liu, Wengang Zhou, and Houqiang Li. Sign language recognition with long short-term memory. In ICIP, pages 2871-2875, 2016.","title":"Sign language recognition with long short-term memory","authors":"Tao Liu, Wengang Zhou, Houqiang Li","year":"2016","venue":"ICIP","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Liu et al., 2018] Jun Liu, Amir Shahroudy, Gang Wang, Ling-Yu Duan, and Alex C Kot. Ssnet: scale selection network for online 3d action prediction. In CVPR, pages 8349-8358, 2018.","title":"Ssnet: scale selection network for online 3d action prediction","authors":"Jun Liu, Amir Shahroudy, Gang Wang, Ling-Yu Duan, Alex C Kot","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Neverova et al., 2016] Natalia Neverova, Christian Wolf, Graham Taylor, and Florian Nebout. Moddrop: adap- tive multi-modal gesture recognition. TPAMI, 38(8):1692- 1706, 2016.","title":"Moddrop: adaptive multi-modal gesture recognition","authors":"Natalia Neverova, Christian Wolf, Graham Taylor, Florian Nebout","year":"2016","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Nguyen et al., 2018] Phuc Nguyen, Ting Liu, Gautam Prasad, and Bohyung Han. Weakly supervised action lo- calization by sparse temporal pooling network. In CVPR, pages 6752-6761, 2018.","title":"Weakly supervised action localization by sparse temporal pooling network","authors":"Phuc Nguyen, Ting Liu, Gautam Prasad, Bohyung Han","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Pigou et al., 2018] Lionel Pigou, Aäron Van Den Oord, Sander Dieleman, Mieke Van Herreweghe, and Joni Dambre. Beyond temporal pooling: recurrence and tem- poral convolutions for gesture recognition in video. IJCV, 126(2-4):430-439, 2018.","title":"Beyond temporal pooling: recurrence and temporal convolutions for gesture recognition in video","authors":"Lionel Pigou, Aäron Van Den, Sander Oord, Mieke Dieleman, Joni Van Herreweghe, Dambre","year":"2018","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Pu et al., 2018] Junfu Pu, Wengang Zhou, and Houqiang Li. Dilated convolutional network with iterative optimization for continuous sign language recognition. In IJCAI, pages 885-891, 2018.","title":"Dilated convolutional network with iterative optimization for continuous sign language recognition","authors":"Pu","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/123"},{"position":29,"raw":"[Venugopalan et al., 2015] Subhashini Venugopalan, Mar- cus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko. Sequence to sequence- video to text. In ICCV, pages 4534-4542, 2015.","title":"Sequence to sequencevideo to text","authors":"Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, Kate Saenko","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wu et al., 2016] Di Wu, Lionel Pigou, Pieter-Jan Kinder- mans, Nam Do-Hoang Le, Ling Shao, Joni Dambre, and Jean-Marc Odobez. Deep dynamic neural networks for multimodal gesture segmentation and recognition. TPAMI, 38(8):1583-1597, 2016.","title":"Deep dynamic neural networks for multimodal gesture segmentation and recognition","authors":"Di Wu, Lionel Pigou, Pieter-Jan Kindermans, Nam Do-Hoang Le, Ling Shao, Joni Dambre, Jean-Marc Odobez","year":"2016","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yao et al., 2015] Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, and Aaron Courville. Describing videos by exploiting tempo- ral structure. In ICCV, pages 4507-4515, 2015.","title":"Describing videos by exploiting temporal structure","authors":"Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, Aaron Courville","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}