{"type":"ScholarlyArticle","title":"High Performance Gesture Recognition via Effective and Efficient Temporal Modeling","authors":[{"name":"Yang Yi","slug":"yang-yi","given":"Yang","family":"Yi","affiliation":"Media Lab , Tencent"},{"name":"Feng Ni","slug":"feng-ni","given":"Feng","family":"Ni","affiliation":"Peking University"},{"name":"Yuexin Ma","slug":"yuexin-ma","given":"Yuexin","family":"Ma","affiliation":"University of Hong Kong"},{"name":"Xinge Zhu","slug":"xinge-zhu","given":"Xinge","family":"Zhu","affiliation":"The Chinese University of Hong Kong"},{"name":"Yuankai Qi","slug":"yuankai-qi","given":"Yuankai","family":"Qi","affiliation":"Harbin Institute of Technology , Weihai , China"},{"name":"Riming Qiu","slug":"riming-qiu","given":"Riming","family":"Qiu","affiliation":"Media Lab , Tencent"},{"name":"Shijie Zhao","slug":"shijie-zhao","given":"Shijie","family":"Zhao","affiliation":"Media Lab , Tencent"},{"name":"Feng Li","slug":"feng-li","given":"Feng","family":"Li","affiliation":"Media Lab , Tencent"},{"name":"Yongtao Wang","slug":"yongtao-wang","given":"Yongtao","family":"Wang","affiliation":"Peking University"}],"abstract":"State-of-the-art hand gesture recognition methods have investigated the spatiotemporal features based on 3D convolutional neural networks (3DCNNs) or convolutional long short-term memory (ConvLSTM). However, they often suffer from the inefficiency due to the high computational complexity of  their network structures. In this paper, we focus instead on the 1D convolutional neural networks  and propose a simple and efficient architectural unit, Multi-Kernel Temporal Block (MKTB), that models the multi-scale temporal responses by explicitly applying different temporal kernels. Then, we present a Global Refinement Block (GRB), which is an attention module for shaping the global temporal features based on the cross-channel similarity.  \r\n By incorporating the MKTB and GRB, our architecture can effectively explore the spatiotemporal features within tolerable computational cost. Extensive experiments conducted on public datasets demonstrate that our proposed model achieves the state-of-the-art with higher efficiency. Moreover, the proposed MKTB and GRB are plug-and-play modules and the experiments on other tasks, like video understanding and video-based person re-identification, also display their good performance in efficiency and capability of generalization.","doi":"10.24963/ijcai.2019/141","url":"https://ijcai.org/proceedings/2019/141","pdf":"https://ijcai.org/proceedings/2019/0141.pdf","markdown":"https://ijcai.org/proceedings/2019/141.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/141","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Computer Vision","track":"Main track","number":141,"pages":{"first":1003,"last":1009},"keywords":["Computer Vision: Action Recognition","Computer Vision: Biometrics, Face and Gesture Recognition"],"videos":[],"references":[{"position":1,"raw":"[Carreira and Zisserman, 2017] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, pages 6299-6308, 2017.","title":"Quo vadis, action recognition? a new model and the kinetics dataset","authors":"Joao Carreira, Andrew Zisserman","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chollet, 2017] Franc ¸ois Chollet. Xception: Deep learning with depthwise separable convolutions. In CVPR, pages 1251-1258, 2017.","title":"Franc ¸ois Chollet. Xception: Deep learning with depthwise separable convolutions","authors":"Chollet","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248-255, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Goyal et al., 2017] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The\" something something\" video database for learning and evaluating vi- sual common sense. In ICCV, volume 2, page 8, 2017.","title":"The\" something something\" video database for learning and evaluating visual common sense","authors":"Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kopuklu et al., 2018] Okan Kopuklu, Neslihan Kose, and Gerhard Rigoll. Motion fused frames: Data level fusion strategy for hand gesture recognition. In CVPR Work- shops, pages 2103-2111, 2018.","title":"Motion fused frames: Data level fusion strategy for hand gesture recognition","authors":"Okan Kopuklu, Neslihan Kose, Gerhard Rigoll","year":"2018","venue":"CVPR Workshops","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Li et al., 2016] Yunan Li, Qiguang Miao, Kuan Tian, Yingying Fan, Xin Xu, Rui Li, and Jianfeng Song. Large- scale gesture recognition with a fusion of rgb-d data based on the c3d model. In ICPR, pages 25-30. IEEE, 2016.","title":"Largescale gesture recognition with a fusion of rgb-d data based on the c3d model","authors":"Yunan Li, Qiguang Miao, Kuan Tian, Yingying Fan, Xin Xu, Rui Li, Jianfeng Song","year":"2016","venue":"ICPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lin et al., 2018] Ji Lin, Chuang Gan, and Song Han. Tem- poral shift module for efficient video understanding. arXiv preprint arXiv:1811.08383, 2018.","title":"Temporal shift module for efficient video understanding","authors":"Ji Lin, Chuang Gan, Song Han","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ma et al., 2018] Yuexin Ma, Xinge Zhu, Sibo Zhang, Ruigang Yang, Wenping Wang, and Dinesh Manocha. Trafficpredict: Trajectory prediction for heterogeneous traffic-agents. arXiv preprint arXiv:1811.02146, 2018.","title":"Trafficpredict: Trajectory prediction for heterogeneous traffic-agents","authors":"Ma","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Miao et al., 2017] Qiguang Miao, Yunan Li, Wanli Ouyang, Zhenxin Ma, Xin Xu, Weikang Shi, and Xiaochun Cao. Multimodal gesture recognition based on the resc3d net- work. In ICCV, pages 3047-3055, 2017.","title":"Multimodal gesture recognition based on the resc3d network","authors":"Miao","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Narayana et al., 2018] Pradyumna Narayana, Ross Bev- eridge, and Bruce A Draper. Gesture recognition: Focus on the hands. In CVPR, pages 5235-5244, 2018.","title":"Gesture recognition: Focus on the hands","authors":"Pradyumna Narayana, Ross Beveridge, Bruce A Draper","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Qiu et al., 2017] Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio-temporal representation with pseudo-3d residual networks. In ICCV, pages 5533-5541, 2017.","title":"Learning spatio-temporal representation with pseudo-3d residual networks","authors":"Qiu","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Tran et al., 2018] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In CVPR, pages 6450-6459, 2018.","title":"A closer look at spatiotemporal convolutions for action recognition","authors":"Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann Lecun, Manohar Paluri","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[TwentyBn, 2017] TwentyBn. The 20bn-jester dataset v1. https://20bn.com/datasets/jester, 2017.","title":"TwentyBn. The 20bn-jester dataset v1","authors":"Twentybn","year":"2017","venue":null,"doi":null,"url":"https://20bn.com/datasets/jester,2017","internal_url":null},{"position":15,"raw":"[Wan et al., 2016] Jun Wan, Yibing Zhao, Shuai Zhou, Is- abelle Guyon, Sergio Escalera, and Stan Z Li. Chalearn looking at people rgb-d isolated and continuous datasets for gesture recognition. In CVPR Workshops, pages 56- 64, 2016.","title":"Chalearn looking at people rgb-d isolated and continuous datasets for gesture recognition","authors":"Jun Wan, Yibing Zhao, Shuai Zhou, Isabelle Guyon, Sergio Escalera, Stan Z Li","year":"2016","venue":"CVPR Workshops","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Wang et al., 2016] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, pages 20- 36. Springer, 2016.","title":"Temporal segment networks: Towards good practices for deep action recognition","authors":"Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, Luc Van Gool","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wang et al., 2018] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, pages 7794-7803, 2018.","title":"Non-local neural networks","authors":"Xiaolong Wang, Ross Girshick, Abhinav Gupta, Kaiming He","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Zhang et al., 2017] Liang Zhang, Guangming Zhu, Peiyi Shen, Juan Song, Syed Afaq Shah, and Mohammed Ben- namoun. Learning spatiotemporal features using 3dcnn and convolutional lstm for gesture recognition. In ICCV, pages 3120-3128, 2017.","title":"Learning spatiotemporal features using 3dcnn and convolutional lstm for gesture recognition","authors":"Liang Zhang, Guangming Zhu, Peiyi Shen, Juan Song, Afaq Syed, Mohammed Shah, Bennamoun","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Zhang et al., 2018] Liang Zhang, Guangming Zhu, Lin Mei, Peiyi Shen, Syed Afaq Ali Shah, and Mohammed Bennamoun. Attention in convolutional lstm for gesture recognition. In NeurIPS, pages 1957-1966, 2018.","title":"Attention in convolutional lstm for gesture recognition","authors":"Liang Zhang, Guangming Zhu, Lin Mei, Peiyi Shen, Syed Afaq, Ali Shah, Mohammed Bennamoun","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zheng et al., 2016] Liang Zheng, Zhi Bie, Yifan Sun, Jing- dong Wang, Chi Su, Shengjin Wang, and Qi Tian. Mars: A video benchmark for large-scale person re-identification. In ECCV, pages 868-884. Springer, 2016.","title":"Mars: A video benchmark for large-scale person re-identification","authors":"Liang Zheng, Zhi Bie, Yifan Sun, Jingdong Wang, Chi Su, Shengjin Wang, Qi Tian","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhou et al., 2018] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In ECCV, pages 803-818, 2018.","title":"Temporal relational reasoning in videos","authors":"Zhou","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhu et al., 2016] Guangming Zhu, Liang Zhang, Lin Mei, Jie Shao, Juan Song, and Peiyi Shen. Large-scale isolated gesture recognition using pyramidal 3d convolutional net- works. In ICPR, pages 19-24. IEEE, 2016.","title":"Large-scale isolated gesture recognition using pyramidal 3d convolutional networks","authors":"Guangming Zhu, Liang Zhang, Lin Mei, Jie Shao, Juan Song, Peiyi Shen","year":"2016","venue":"ICPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhu et al., 2017] Xinge Zhu, Liang Li, Weigang Zhang, Tianrong Rao, Min Xu, Qingming Huang, and Dong Xu. Dependency exploitation: A unified cnn-rnn approach for visual emotion recognition. In IJCAI, pages 3595-3601, 2017.","title":"Dependency exploitation: A unified cnn-rnn approach for visual emotion recognition","authors":"Xinge Zhu, Liang Li, Weigang Zhang, Tianrong Rao, Min Xu, Qingming Huang, Dong Xu","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/503"},{"position":24,"raw":"[Zolfaghari et al., 2018] Mohammadreza Zolfaghari, Ka- maljeet Singh, and Thomas Brox. Eco: Efficient convolu- tional network for online video understanding. In ECCV, pages 695-712, 2018.","title":"Eco: Efficient convolutional network for online video understanding","authors":"Mohammadreza Zolfaghari, Kamaljeet Singh, Thomas Brox","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}