{"type":"ScholarlyArticle","title":"MMNet: Muscle Motion-Guided Network for Micro-Expression Recognition","authors":[{"name":"Hanting Li","slug":"hanting-li","given":"Hanting","family":"Li","affiliation":"University of Science and Technology of China"},{"name":"Mingzhe Sui","slug":"mingzhe-sui","given":"Mingzhe","family":"Sui","affiliation":"University of Science and Technology of China"},{"name":"Zhaoqing Zhu","slug":"zhaoqing-zhu","given":"Zhaoqing","family":"Zhu","affiliation":"University of Science and Technology of China"},{"name":"Feng Zhao","slug":"feng-zhao","given":"Feng","family":"Zhao","affiliation":"University of Science and Technology of China"}],"abstract":"Facial micro-expressions (MEs) are involuntary facial motions revealing people’s real feelings and play an important role in the early intervention of mental illness, the national security, and many human-computer interaction systems. However, existing micro-expression datasets are limited and usually pose some challenges for training good classifiers. To model the subtle facial muscle motions, we propose a robust micro-expression recognition (MER) framework, namely muscle motion-guided network (MMNet). Specifically,  a continuous attention (CA) block is introduced to focus on modeling local subtle muscle motion patterns with little identity information, which is different from most previous methods that directly extract features from complete video frames with much identity information. Besides, we design a position calibration (PC) module based on the vision transformer. By adding the position embeddings of the face generated by the PC module at the end of the two branches, the PC module can help to add position information to facial muscle motion-pattern features for the MER. Extensive experiments on three public micro-expression datasets demonstrate that our approach outperforms state-of-the-art methods by a large margin. Code is available at https://github.com/muse1998/MMNet.","doi":"10.24963/ijcai.2022/150","url":"https://ijcai.org/proceedings/2022/150","pdf":"https://ijcai.org/proceedings/2022/0150.pdf","markdown":"https://ijcai.org/proceedings/2022/150.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/150","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":150,"pages":{"first":1074,"last":1080},"keywords":["Computer Vision: Biometrics, Face, Gesture and Pose Recognition","Computer Vision: Video analysis and understanding","Computer Vision: Recognition (object detection, categorization)"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986496","length_sec":75,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984639","length_sec":322,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Ben et al., 2021] Xianye Ben, Yi Ren, Junping Zhang, Su- Jing Wang, Kidiyo Kpalma, et al. Video-based facial micro-expression analysis: A survey of datasets, features and algorithms. IEEE TPAMI, 2021.","title":"Video-based facial micro-expression analysis: A survey of datasets, features and algorithms","authors":"Ben","year":"2021","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Davison et al., 2016] Adrian K Davison, Cliff Lansley, Nicholas Costen, Kevin Tan, and Moi Hoon Yap. SAMM: A spontaneous micro-facial movement dataset. IEEE TAC, 9(1):116-129, 2016.","title":"SAMM: A spontaneous micro-facial movement dataset","authors":"Adrian K Davison, Cliff Lansley, Nicholas Costen, Kevin Tan, Moi Hoon, Yap","year":"2016","venue":"IEEE TAC","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ekman and Friesen, 1978] Paul Ekman and Wallace V Friesen. Facial action coding system. Environmental Psy- chology & Nonverbal Behavior, 1978.","title":"Facial action coding system","authors":"Friesen Ekman, Paul Ekman, Wallace V Friesen","year":"1978","venue":"Environmental Psychology & Nonverbal Behavior","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ekman, 2009] Paul Ekman. Telling lies: Clues to deceit in the marketplace, politics, and marriage (revised edition).","title":"Telling lies: Clues to deceit in the marketplace, politics, and marriage","authors":"Paul Ekman, Ekman","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gan et al., 2019] Yee Siang Gan, Sze-Teng Liong, Wei- Chuen Yau, Yen-Chang Huang, and Lit-Ken Tan. OFF- ApexNet on micro-expression recognition system. Signal Processing: Image Communication, 74:129-139, 2019.","title":"OFF-ApexNet on micro-expression recognition system","authors":"Yee Siang Gan, Sze-Teng Liong, Wei-Chuen Yau, Yen-Chang Huang, Lit-Ken Tan","year":"2019","venue":"Signal Processing: Image Communication","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Happy and Routray, 2017] SL Happy and Aurobinda Routray. Fuzzy histogram of optical flow orientations for micro-expression recognition. IEEE TAC, 10(3):394-406, 2017.","title":"SL Happy and Aurobinda Routray. Fuzzy histogram of optical flow orientations for micro-expression recognition","authors":"Routray Happy","year":"2017","venue":"IEEE TAC","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Khor et al., 2019] Huai-Qian Khor, John See, Sze-Teng Li- ong, Raphael CW Phan, and Weiyao Lin. Dual-stream shallow networks for facial micro-expression recognition. In ICIP, pages 36-40. IEEE, 2019.","title":"Dual-stream shallow networks for facial micro-expression recognition","authors":"Huai-Qian, John Khor, Sze-Teng See, Liong, Weiyao Raphael Cw Phan, Lin","year":"2019","venue":"ICIP","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kumar and Bhanu, 2021] Ankith Jain Rakesh Kumar and Bir Bhanu. Micro-expression classification based on land- mark relations with graph attention convolutional network. In CVPRW, pages 1511-1520, 2021.","title":"Micro-expression classification based on landmark relations with graph attention convolutional network","authors":"Bhanu ; Kumar, Bir Bhanu","year":"2021","venue":"CVPRW","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Le Ngo et al., 2016] Anh Cat Le Ngo, John See, and Raphael C-W Phan. Sparsity in dynamics of spontaneous subtle emotions: Analysis and application. IEEE TAC, 8(3):396-411, 2016.","title":"Sparsity in dynamics of spontaneous subtle emotions: Analysis and application","authors":"Anh Cat, Le Ngo, John See, C-W Raphael, Phan","year":"2016","venue":"IEEE TAC","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lei et al., 2020] Ling Lei, Jianfeng Li, Tong Chen, and Shi- gang Li. A novel Graph-TCN with a graph structured representation for micro-expression recognition. In MM, pages 2237-2245, 2020.","title":"A novel Graph-TCN with a graph structured representation for micro-expression recognition","authors":"Ling Lei, Jianfeng Li, Tong Chen, Shigang Li","year":"2020","venue":"MM","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lei et al., 2021] Ling Lei, Tong Chen, Shigang Li, and Jian- feng Li. Micro-expression recognition based on facial graph representation learning and facial action unit fusion. In CVPRW, pages 1571-1580, 2021.","title":"Micro-expression recognition based on facial graph representation learning and facial action unit fusion","authors":"Ling Lei, Tong Chen, Shigang Li, Jianfeng Li","year":"2021","venue":"CVPRW","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li and Deng, 2020] Shan Li and Weihong Deng. Deep fa- cial expression recognition: A survey. IEEE TAC, 2020.","title":"Deep facial expression recognition: A survey","authors":"Shan Li, Weihong Deng","year":"2020","venue":"IEEE TAC","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2017] Xiaobai Li, Xiaopeng Hong, Antti Moila- nen, Xiaohua Huang, Tomas Pfister, Guoying Zhao, and Matti Pietikäinen. Towards reading hidden emotions: A comparative study of spontaneous micro-expression spot- ting and recognition methods. IEEE TAC, 9(4):563-577, 2017.","title":"Towards reading hidden emotions: A comparative study of spontaneous micro-expression spotting and recognition methods","authors":"Xiaobai Li, Xiaopeng Hong, Antti Moilanen, Xiaohua Huang, Tomas Pfister, Guoying Zhao, Matti Pietikäinen","year":"2017","venue":"IEEE TAC","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2020] Yante Li, Xiaohua Huang, and Guoying Zhao. Joint local and global information learning with sin- gle apex frame detection for micro-expression recognition. IEEE TIP, 30:249-263, 2020.","title":"Joint local and global information learning with single apex frame detection for micro-expression recognition","authors":"Li","year":"2020","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liong et al., 2019] Sze-Teng Liong, Yee Siang Gan, John See, Huai-Qian Khor, and Yen-Chang Huang. Shal- low triple stream three-dimensional CNN (STSTNet) for micro-expression recognition. In FG, pages 1-5. IEEE, 2019.","title":"Shallow triple stream three-dimensional CNN (STSTNet) for micro-expression recognition","authors":"Sze-Teng Liong, Yee Siang Gan, John See, Huai-Qian, Yen-Chang Khor, Huang","year":"2019","venue":"FG","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2020] Jiateng Liu, Wenming Zheng, and Yuan Zong. SMA-STN: Segmented movement-attending spatiotemporal network formicro-expression recognition. arXiv preprint arXiv:2010.09342, 2020.","title":"SMA-STN: Segmented movement-attending spatiotemporal network formicro-expression recognition","authors":"Liu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mollahosseini et al., 2017] Ali Mollahosseini, Behzad Hasani, and Mohammad H Mahoor. Affectnet: A database for facial expression, valence, and arousal computing in the wild. IEEE TAC, 10(1):18-31, 2017.","title":"Affectnet: A database for facial expression, valence, and arousal computing in the wild","authors":"Ali Mollahosseini, Behzad Hasani, Mohammad H Mahoor","year":"2017","venue":"IEEE TAC","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nie et al., 2021] Xuan Nie, Madhumita A Takalkar, Mengyang Duan, Haimin Zhang, and Min Xu. GEME: Dual-stream multi-task gender-based micro-expression recognition. Neurocomputing, 427:13-28, 2021.","title":"GEME: Dual-stream multi-task gender-based micro-expression recognition","authors":"Xuan Nie, A Madhumita, Mengyang Takalkar, Haimin Duan, Min Zhang, Xu","year":"2021","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Song et al., 2019] Baolin Song, Ke Li, Yuan Zong, Jie Zhu, Wenming Zheng, et al. Recognizing spontaneous micro- expression using a three-stream convolutional neural net- work. IEEE Access, 7:184537-184551, 2019.","title":"Recognizing spontaneous microexpression using a three-stream convolutional neural network","authors":"Baolin Song, Ke Li, Yuan Zong, Jie Zhu, Wenming Zheng","year":"2019","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sun et al., 2020] Bo Sun, Siming Cao, Dongliang Li, Jun He, and Lejun Yu. Dynamic micro-expression recognition using knowledge distillation. IEEE TAC, 2020.","title":"Dynamic micro-expression recognition using knowledge distillation","authors":"Bo Sun, Siming Cao, Dongliang Li, Jun He, Lejun Yu","year":"2020","venue":"IEEE TAC","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Woo et al., 2018] Sanghyun Woo, Jongchan Park, Joon- Young Lee, and In So Kweon. CBAM: Convolutional block attention module. In ECCV, pages 3-19, 2018.","title":"CBAM: Convolutional block attention module","authors":"Sanghyun Woo, Jongchan Park, Joon-Young Lee, In So Kweon","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xia and Wang, 2021] Bin Xia and Shangfei Wang. Micro- expression recognition enhanced by macro-expression from spatial-temporal domain. IJCAI, 2021.","title":"Microexpression recognition enhanced by macro-expression from spatial-temporal domain","authors":"Wang ; Xia, Bin Xia, Shangfei Wang","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xia et al., 2020] Bin Xia, Weikang Wang, Shangfei Wang, and Enhong Chen. Learning from macro-expression: A micro-expression recognition framework. In MM, page 2936-2944. Association for Computing Machinery, 2020.","title":"Learning from macro-expression: A micro-expression recognition framework","authors":"Bin Xia, Weikang Wang, Shangfei Wang, Enhong Chen","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xie et al., 2020] Hong-Xia Xie, Ling Lo, Hong-Han Shuai, and Wen-Huang Cheng. AU-assisted graph attention con- volutional network for micro-expression recognition. In MM, pages 2871-2880, 2020.","title":"AU-assisted graph attention convolutional network for micro-expression recognition","authors":"Xie","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yan et al., 2014] Wen-Jing Yan, Xiaobai Li, Su-Jing Wang, Guoying Zhao, Yong-Jin Liu, Yu-Hsin Chen, and Xi- aolan Fu. CASME II: An improved spontaneous micro- expression database and the baseline evaluation. PloS one, 9(1):e86041, 2014.","title":"CASME II: An improved spontaneous microexpression database and the baseline evaluation","authors":"Wen-Jing Yan, Xiaobai Li, Su-Jing Wang, Guoying Zhao, Yong-Jin Liu, Yu-Hsin Chen, Xiaolan Fu","year":"2014","venue":"PloS one","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhao et al., 2021] Sirui Zhao, Hanqing Tao, Yangsong Zhang, Tong Xu, Kun Zhang, Zhongkai Hao, and Enhong Chen. A two-stage 3D CNN based learning method for spontaneous micro-expression recognition. Neurocomput- ing, 448:276-289, 2021.","title":"A two-stage 3D CNN based learning method for spontaneous micro-expression recognition","authors":"Sirui Zhao, Hanqing Tao, Yangsong Zhang, Tong Xu, Kun Zhang, Zhongkai Hao, Enhong Chen","year":"2021","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}