{"type":"ScholarlyArticle","title":"GestureDet: Real-time Student Gesture Analysis with Multi-dimensional Attention-based Detector","authors":[{"name":"Rui Zheng","slug":"rui-zheng","given":"Rui","family":"Zheng","affiliation":"Department of Computer Science and Engineering , Shanghai Jiao Tong University , China"},{"name":"Fei Jiang","slug":"fei-jiang","given":"Fei","family":"Jiang","affiliation":"Department of Computer Science and Engineering , Shanghai Jiao Tong University , China"},{"name":"Ruimin Shen","slug":"ruimin-shen","given":"Ruimin","family":"Shen","affiliation":"Department of Computer Science and Engineering , Shanghai Jiao Tong University , China"}],"abstract":"Students’ gestures, hand-raising, stand-up, and sleeping, indicates the engagement of students in classrooms and partially reflects teaching quality. Therefore, fast and automatically recognizing these gestures are of great importance. Due to limited computational resources in primary and secondary schools, we propose a real-time student behavior detector based on light-weight MobileNetV2-SSD to reduce the dependency of GPUs. Firstly, we build a large-scale corpus from real schools to capture various behavior gestures. Based on such a corpus, we transfer the gesture recognition task into object detections. Secondly, we design a multi-dimensional attention-based detector, named GestureDet, for real-time and accurate gesture analysis. The multi-dimensional attention mechanisms simultaneously consider all the dimensions of the training set, aiming to pay more attention to discriminative features and samples that are important for the final performance. Specifically, the spatial attention is constructed with stacked dilated convolution layers to generate a soft and learnable mask for re-weighting foreground and background features; the channel attention introduces the context modeling and squeeze-and-excitation module to focus on discriminative features; the batch attention discriminates important samples with a new designed reweight strategy. Experimental results demonstrate the effectiveness and versatility of GestureDet, which achieves 75.2% mAP on real student behavior dataset, and 74.5% on public PASCAL VOC dataset at 20fps on embedding device Nvidia Jetson TX2. Code will be made publicly available.","doi":"10.24963/ijcai.2020/95","url":"https://ijcai.org/proceedings/2020/95","pdf":"https://ijcai.org/proceedings/2020/0095.pdf","markdown":"https://ijcai.org/proceedings/2020/95.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/95","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":95,"pages":{"first":680,"last":686},"keywords":["Computer Vision: Biometrics, Face and Gesture Recognition","Multidisciplinary Topics and Applications: Real-Time Systems","Humans and AI: Computer-Aided Education"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24332/file_24332.m3u8","length_sec":273,"provider":"hls"}],"references":[{"position":1,"raw":"[Cao et al., 2019] Yuhang Cao, Kai Chen, Chen Change Loy, and Dahua Lin. Prime sample attention in object detection. arXiv preprint arXiv:1904.04821, 2019.","title":"Prime sample attention in object detection","authors":"Yuhang Cao, Kai Chen, Chen Change Loy, Dahua Lin","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2017] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587, 2017.","title":"Rethinking atrous convolution for semantic image segmentation","authors":"Liang-Chieh Chen, George Papandreou, Florian Schroff, Hartwig Adam","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2019] Yuntao Chen, Chenxia Han, Naiyan Wang, and Zhaoxiang Zhang. Revisiting feature align- ment for one-stage object detection. arXiv preprint arXiv:1908.01570, 2019.","title":"Revisiting feature alignment for one-stage object detection","authors":"Yuntao Chen, Chenxia Han, Naiyan Wang, Zhaoxiang Zhang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Everingham et al., 2010] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zis- serman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303-338, 2010.","title":"The pascal visual object classes (voc) challenge","authors":"Mark Everingham, Luc Van Gool, K I Christopher, John Williams, Andrew Winn, Zisserman","year":"2010","venue":"International journal of computer vision","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Girshick et al., 2014] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580-587, 2014.","title":"Rich feature hierarchies for accurate object detection and semantic segmentation","authors":"Ross Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik","year":"2014","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Howard et al., 2017] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mo- bilenets: Efficient convolutional neural networks for mo- bile vision applications. arXiv preprint arXiv:1704.04861, 2017.","title":"Mobilenets: Efficient convolutional neural networks for mobile vision applications","authors":"Menglong Andrew G Howard, Bo Zhu, Dmitry Chen, Weijun Kalenichenko, Tobias Wang, Marco Weyand, Hartwig Andreetto, Adam","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2018] Jie Hu, Li Shen, and Gang Sun. Squeeze- and-excitation networks. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pages 7132-7141, 2018.","title":"Squeezeand-excitation networks","authors":"Jie Hu, Li Shen, Gang Sun","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Li et al., 2018] Yuxi Li, Jiuwei Li, Weiyao Lin, and Jianguo Li. Tiny-dsod: Lightweight object detection for resource- restricted usages. arXiv preprint arXiv:1807.11013, 2018.","title":"Tiny-dsod: Lightweight object detection for resourcerestricted usages","authors":"Yuxi Li, Jiuwei Li, Weiyao Lin, Jianguo Li","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In European conference on com- puter vision, pages 740-755. Springer, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lin et al., 2017] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense ob- ject detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.","title":"Focal loss for dense object detection","authors":"Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollár","year":"2017","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2016] Wei Liu, Dragomir Anguelov, Dumitru Er- han, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference on computer vision, pages 21-37. Springer, 2016.","title":"Ssd: Single shot multibox detector","authors":"Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, Alexander C Berg","year":"2016","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Qin et al., 2019] Zheng Qin, Zeming Li, Zhaoning Zhang, Yiping Bao, Gang Yu, Yuxing Peng, and Jian Sun. Thun- dernet: Towards real-time generic object detection. arXiv preprint arXiv:1903.11752, 2019.","title":"Thundernet: Towards real-time generic object detection","authors":"Zheng Qin, Zeming Li, Zhaoning Zhang, Yiping Bao, Gang Yu, Yuxing Peng, Jian Sun","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Redmon and Farhadi, 2017] Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7263-7271, 2017.","title":"Yolo9000: better, faster, stronger","authors":"Farhadi Redmon, Joseph Redmon, Ali Farhadi","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Redmon et al., 2016] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 779-788, 2016.","title":"You only look once: Unified, real-time object detection","authors":"Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster r-cnn: Towards real-time ob- ject detection with region proposal networks. In Advances in neural information processing systems, pages 91-99, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sandler et al., 2018] Mark Sandler, Andrew Howard, Men- glong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4510-4520, 2018.","title":"Mobilenetv2: Inverted residuals and linear bottlenecks","authors":"Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, Liang-Chieh Chen","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shrivastava et al., 2016] Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region-based object detectors with online hard example mining. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 761-769, 2016.","title":"Training region-based object detectors with online hard example mining","authors":"Abhinav Shrivastava, Abhinav Gupta, Ross Girshick","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing sys- tems, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2017] Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang, and Xiaoou Tang. Residual attention network for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3156- 3164, 2017.","title":"Residual attention network for image classification","authors":"Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang, Xiaoou Tang","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2018] Robert J Wang, Xiang Li, and Charles X Ling. Pelee: A real-time object detection system on mo- bile devices. In Advances in Neural Information Process- ing Systems, pages 1963-1972, 2018.","title":"Pelee: A real-time object detection system on mobile devices","authors":"J Robert, Xiang Wang, Charles X Li, Ling","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}