{"type":"ScholarlyArticle","title":"Weakly Supervised Local-Global Relation Network for Facial Expression Recognition","authors":[{"name":"Haifeng Zhang","slug":"haifeng-zhang","given":"Haifeng","family":"Zhang","affiliation":"Department of Automation , University of Science and Technology of China"},{"name":"Wen Su","slug":"wen-su","given":"Wen","family":"Su","affiliation":"Faculty of Mechanical Engineering and Automation , Zhejiang Sci-Tech University"},{"name":"Jun Yu","slug":"jun-yu","given":"Jun","family":"Yu","affiliation":"Department of Automation , University of Science and Technology of China"},{"name":"Zengfu Wang","slug":"zengfu-wang","given":"Zengfu","family":"Wang","affiliation":"Department of Automation , University of Science and Technology of China; Institute of Intelligent Machines , Chinese Academy of Sciences"}],"abstract":"To extract crucial local features and enhance the complementary relation between local and global features, this paper proposes a Weakly Supervised Local-Global Relation Network (WS-LGRN), which uses the attention mechanism to deal with part location and feature fusion problems. Firstly, the Attention Map Generator quickly finds the local regions-of-interest under the supervision of image-level labels. Secondly, bilinear attention pooling is employed to generate and refine local features. Thirdly, Relational Reasoning Unit is designed to model the relation among all features before making classification. The weighted fusion mechanism in the Relational Reasoning Unit makes the model benefit from the complementary advantages between different features. In addition, contrastive losses are introduced for local and global features to increase the inter-class dispersion and intra-class compactness at different granularities. Experiments on lab-controlled and real-world facial expression dataset show that WS-LGRN achieves state-of-the-art performance, which demonstrates its superiority in FER.","doi":"10.24963/ijcai.2020/145","url":"https://ijcai.org/proceedings/2020/145","pdf":"https://ijcai.org/proceedings/2020/0145.pdf","markdown":"https://ijcai.org/proceedings/2020/145.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/145","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":145,"pages":{"first":1040,"last":1046},"keywords":["Machine Learning: Deep Learning: Convolutional networks","Machine Learning: Classification","Computer Vision: Biometrics, Face and Gesture Recognition"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25380/file_25380.m3u8","length_sec":225,"provider":"hls"}],"references":[{"position":1,"raw":"[Ali and Hughes, 2019] Kamran Ali and Charles E Hughes. Facial expression recognition using disentangled adversar- ial learning. arXiv preprint arXiv:1909.13135, 2019.","title":"Facial expression recognition using disentangled adversarial learning","authors":"Kamran Ali, Charles E Hughes","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cai et al., 2018a] Jie Cai, Zibo Meng, Ahmed Shehab Khan, Zhiyuan Li, James O'Reilly, and Yan Tong. Probabilistic attribute tree in convolutional neural net- works for facial expression recognition. arXiv preprint arXiv:1812.07067, 2018.","title":"Probabilistic attribute tree in convolutional neural networks for facial expression recognition","authors":"Cai","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cai et al., 2018b] Jie Cai, Zibo Meng, Ahmed Shehab Khan, Zhiyuan Li, James O'Reilly, and Yan Tong. Island loss for learning discriminative features in facial expres- sion recognition. In 2018 13th IEEE International Con- ference on Automatic Face & Gesture Recognition (FG 2018), pages 302-309. IEEE, 2018.","title":"Island loss for learning discriminative features in facial expression recognition","authors":"Cai","year":"2018","venue":"13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2019] Yuedong Chen, Jianfeng Wang, Shikai Chen, Zhongchao Shi, and Jianfei Cai. Facial motion prior networks for facial expression recognition. arXiv preprint arXiv:1902.08788, 2019.","title":"Facial motion prior networks for facial expression recognition","authors":"Chen","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cohn and Zlochower, 1995] JF Cohn and A Zlochower. A computerized analysis of facial expression: Feasibility of automated discrimination. American Psychological Soci- ety, 2:6, 1995.","title":"A computerized analysis of facial expression: Feasibility of automated discrimination","authors":"Zlochower ; Jf Cohn, A Cohn, Zlochower","year":"1995","venue":"American Psychological Society","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ding et al., 2017] Hui Ding, Shaohua Kevin Zhou, and Rama Chellappa. Facenet2expnet: Regularizing a deep face recognition net for expression recognition. In 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), pages 118-126. IEEE, 2017.","title":"Facenet2expnet: Regularizing a deep face recognition net for expression recognition","authors":"Hui Ding, Kevin Shaohua, Rama Zhou, Chellappa","year":"2017","venue":"12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Goodfellow et al., 2015] Ian J Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong-Hyun Lee, et al. Challenges in representation learn- ing: A report on three machine learning contests. Neural Networks, 64:59-63, 2015.","title":"Challenges in representation learning: A report on three machine learning contests","authors":"Ian J Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong-Hyun Lee","year":"2015","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Happy and Routray, 2014] SL Happy and Aurobinda Routray. Automatic facial expression recognition using features of salient facial patches. IEEE transactions on Affective Computing, 6(1):1-12, 2014.","title":"SL Happy and Aurobinda Routray. Automatic facial expression recognition using features of salient facial patches","authors":"Routray Happy","year":"2014","venue":"IEEE transactions on Affective Computing","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Itti and Koch, 2001] Laurent Itti and Christof Koch. Com- putational modelling of visual attention. Nature reviews neuroscience, 2(3):194, 2001.","title":"Computational modelling of visual attention","authors":"Koch Itti, Laurent Itti, Christof Koch","year":"2001","venue":"Nature reviews neuroscience","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li and Deng, 2018] Shan Li and Weihong Deng. Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition. IEEE Trans- actions on Image Processing, 28(1):356-370, 2018.","title":"Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition","authors":"Shan Li, Weihong Deng","year":"2018","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2019] Zhiyuan Li, Shizhong Han, Ahmed Shehab Khan, Jie Cai, Zibo Meng, James O'Reilly, and Yan Tong. Pooling map adaptation in convolutional neural network for facial expression recognition. In 2019 IEEE Interna- tional Conference on Multimedia and Expo (ICME), pages 1108-1113. IEEE, 2019.","title":"Pooling map adaptation in convolutional neural network for facial expression recognition","authors":"Zhiyuan Li, Shizhong Han, Ahmed Shehab Khan, Jie Cai, Zibo Meng, O' James, Yan Reilly, Tong","year":"2019","venue":"IEEE International Conference on Multimedia and Expo (ICME)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lin et al., 2018] Feng Lin, Richang Hong, Wengang Zhou, and Houqiang Li. Facial expression recognition with data augmentation and compact feature learning. In 2018 25th IEEE International Conference on Image Processing (ICIP), pages 1957-1961. IEEE, 2018.","title":"Facial expression recognition with data augmentation and compact feature learning","authors":"Feng Lin, Richang Hong, Wengang Zhou, Houqiang Li","year":"2018","venue":"25th IEEE International Conference on Image Processing (ICIP)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2014] Ping Liu, Shizhong Han, Zibo Meng, and Yan Tong. Facial expression recognition via a boosted deep belief network. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 1805-1812, 2014.","title":"Facial expression recognition via a boosted deep belief network","authors":"Ping Liu, Shizhong Han, Zibo Meng, Yan Tong","year":"2014","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2015] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730-3738, 2015.","title":"Deep learning face attributes in the wild","authors":"Ziwei Liu, Ping Luo, Xiaogang Wang, Xiaoou Tang","year":"2015","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2017] Xiaofeng Liu, BVK Vijaya Kumar, Jane You, and Ping Jia. Adaptive deep metric learning for identity-aware facial expression recognition. In Proceed- ings of the IEEE Conference on Computer Vision and Pat- tern Recognition Workshops, pages 20-29, 2017.","title":"Adaptive deep metric learning for identity-aware facial expression recognition","authors":"Xiaofeng Liu, Jane Bvk Vijaya Kumar, Ping You, Jia","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lucey et al., 2010] Patrick Lucey, Jeffrey F Cohn, Takeo Kanade, Jason Saragih, Zara Ambadar, and Iain Matthews. The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, pages 94-101. IEEE, 2010.","title":"The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression","authors":"Patrick Lucey, Jeffrey F Cohn, Takeo Kanade, Jason Saragih, Zara Ambadar, Iain Matthews","year":"2010","venue":"IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Meng et al., 2017] Zibo Meng, Ping Liu, Jie Cai, Shizhong Han, and Yan Tong. Identity-aware convolutional neural network for facial expression recognition. In 2017 12th IEEE International Conference on Automatic Face & Ges- ture Recognition (FG 2017), pages 558-565. IEEE, 2017.","title":"Identity-aware convolutional neural network for facial expression recognition","authors":"Zibo Meng, Ping Liu, Jie Cai, Shizhong Han, Yan Tong","year":"2017","venue":"12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Xie and Hu, 2018] Siyue Xie and Haifeng Hu. Facial expression recognition using hierarchical features with deep comprehensive multipatches aggregation convolu- tional neural networks. IEEE Transactions on Multimedia, 21(1):211-220, 2018.","title":"Facial expression recognition using hierarchical features with deep comprehensive multipatches aggregation convolutional neural networks","authors":"Hu ; ] Siyue Xie, Haifeng Xie, Hu","year":"2018","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Xiong and La Torre, 2013] Xuehan Xiong and Fernando De La Torre. Supervised descent method and its applications to face alignment. 2013:532-539, 2013.","title":"Xuehan Xiong and Fernando De La Torre. Supervised descent method and its applications to face alignment","authors":"La Xiong, Torre","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yang et al., 2018] Huiyuan Yang, Umur Ciftci, and Lijun Yin. Facial expression recognition by de-expression residue learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2168- 2177, 2018.","title":"Facial expression recognition by de-expression residue learning","authors":"Yang","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhao et al., 2018] Shuwen Zhao, Haibin Cai, Honghai Liu, Jianhua Zhang, and Shengyong Chen. Feature selection mechanism in cnns for facial expression recognition. In BMVC, page 317, 2018.","title":"Feature selection mechanism in cnns for facial expression recognition","authors":"Shuwen Zhao, Haibin Cai, Honghai Liu, Jianhua Zhang, Shengyong Chen","year":"2018","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhou et al., 2016] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceed- ings of the IEEE conference on computer vision and pat- tern recognition, pages 2921-2929, 2016.","title":"Learning deep features for discriminative localization","authors":"Zhou","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}