{"type":"ScholarlyArticle","title":"LION: Label Disambiguation for Semi-supervised Facial Expression Recognition with Progressive Negative Learning","authors":[{"name":"Zhongjing Du","slug":"zhongjing-du","given":"Zhongjing","family":"Du","affiliation":""},{"name":"Xu Jiang","slug":"xu-jiang","given":"Xu","family":"Jiang","affiliation":""},{"name":"Peng Wang","slug":"peng-wang","given":"Peng","family":"Wang","affiliation":""},{"name":"Qizheng Zhou","slug":"qizheng-zhou","given":"Qizheng","family":"Zhou","affiliation":""},{"name":"Xi Wu","slug":"xi-wu","given":"Xi","family":"Wu","affiliation":""},{"name":"Jiliu Zhou","slug":"jiliu-zhou","given":"Jiliu","family":"Zhou","affiliation":""},{"name":"Yan Wang","slug":"yan-wang","given":"Yan","family":"Wang","affiliation":""}],"abstract":"Semi-supervised deep facial expression recognition (SS-DFER) has recently attracted rising research interest due to its more practical setting of abundant unlabeled data. However, there are two main problems unconsidered in current SS-DFER methods: 1) label ambiguity, i.e., given labels mismatch with facial expressions; 2) inefficient utilization of unlabeled data with low-confidence. In this paper, we propose a novel SS-DFER method, including a Label DIsambiguation module and a PrOgressive Negative Learning module, namely LION, to simultaneously address both problems. Specifically, the label disambiguation module operates on labeled data, including data with accurate labels (clear data) and ambiguous labels (ambiguous data). It first uses clear data to calculate prototypes for all the expression classes, and then re-assign a candidate label set to all the ambiguous data. Based on the prototypes and the candidate label set, the ambiguous data can be relabeled more accurately. As for unlabeled data with low-confidence, the progressive negative learning module is developed to iteratively mine more complete complementary labels, which can guide the model to reduce the association between data and corresponding complementary labels. Experiments on three challenging datasets show that our method significantly outperforms the current state-of-the-art approaches in SS-DFER and surpasses fully-supervised baselines. Code will be available at https://github.com/NUM-7/LION.","doi":"10.24963/ijcai.2023/78","url":"https://ijcai.org/proceedings/2023/78","pdf":"https://ijcai.org/proceedings/2023/0078.pdf","markdown":"https://ijcai.org/proceedings/2023/78.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/78","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":78,"pages":{"first":699,"last":707},"keywords":["Computer Vision: CV: Applications","Computer Vision: CV: Transfer, low-shot, semi- and un- supervised learning"],"videos":[],"references":[{"position":1,"raw":"[Berthelot et al., 2019] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, Colin A. Raffel. Mixmatch: A holistic approach to semi-supervised learning. In NeurIPS, 2019.","title":"Mixmatch: A holistic approach to semi-supervised learning","authors":"David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, Colin A Raffel","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2020] Shikai Chen, Jianfeng Wang, Y uedong Chen, ZhongchaoShi, Xin Geng, and Y ong Rui. Label distribution learningon auxiliary label space graphs for fa- cial expression recognition. In CVPR, 2020.","title":"Label distribution learningon auxiliary label space graphs for facial expression recognition","authors":"Shikai Chen, Jianfeng Wang, Y Uedong Chen, Xin Zhongchaoshi, Y Geng, Rui Ong","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li and Li Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR, 2009.","title":"ImageNet: A Large-Scale Hierarchical Image Database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Denton et al., 2016] Emily Denton, Sam Gross, Rob Fergus. Semi-Supervised Learning with Context-Conditional Generative Adversarial Networks. arXiv:1611.06430, 2016.","title":"Semi-Supervised Learning with Context-Conditional Generative Adversarial Networks","authors":"Emily Denton, Sam Gross, Rob Fergus","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dhall et al., 2011] Abhinav Dhall, Roland Goecke, Simon Lucey, Tom Gedeon. Static facial expression analysis in tough conditions: Data, evaluation protocol and bench- mark. In ICCV, 2011.","title":"Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark","authors":"Abhinav Dhall, Roland Goecke, Simon Lucey, Tom Gedeon","year":"2011","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Donahue et al., 2016] Jeff Donahue, Philipp Krähen- bühl, Trevor Darrell. Adversarial feature learning. arXiv:1605.09782, 2016.","title":"Adversarial feature learning","authors":"Jeff Donahue, Philipp Krähenbühl, Trevor Darrell","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Duan et al., 2022] Yue Duan, Zhen Zhao, Lei Qi, Lei Wang, Luping Zhou, Yinghuan Shi, Yang Gao. MutexMatch: Semi-supervised Learning with Mutex- based Consistency Regularization. arXiv:2203.14316, 2022.","title":"MutexMatch: Semi-supervised Learning with Mutexbased Consistency Regularization","authors":"Zhen Yue Duan, Lei Zhao, Lei Qi, Luping Wang, Yinghuan Zhou, Yang Shi, Gao","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Florea et al., 2020] Corneliu Florea, Mihai Badea, Laura Florea, AndreiRacoviteanu, Constantin Vertan. Margin- mix: Semi-supervised learning for face expression recog- nition. In ECCV, 2020.","title":"Marginmix: Semi-supervised learning for face expression recognition","authors":"Corneliu Florea, Mihai Badea, Laura Florea, Constantin Andreiracoviteanu, Vertan","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Guo et al., 2016] Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, Jianfeng Gao. Ms-celeb-1m: A dataset and benchmark for large-scale face recognition. In ECCV, 2016.","title":"Ms-celeb-1m: A dataset and benchmark for large-scale face recognition","authors":"Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, Jianfeng Gao","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kim et al., 2019] Youngdong Kim, Junho Yim, Juseung Yun, Junmo Kim. NLNL: Negative learning for noisy labels. In ICCV, 2019. [Krizhevsky et al., 2009] Krizhevsky, Alex, and Geoffrey Hinton. Learning multiple layers of features from tiny im- ages. In Technical Report TR-2009, University of To- ronto, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Kim","year":"2009","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2017] Shan Li and Weihong Deng. Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition. In CVPR, 2017.","title":"Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition","authors":"Shan Li, Weihong Deng","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2019] Shan Li, Weihong Deng. Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition. In IEEE Transactions on Image Processing, 2019","title":"Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition","authors":"Shan Li, Weihong Deng","year":"2019","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2022] Hangyu Li, Nannan Wang, Xi Yang, Xiaoyu Wang, Xinbo Gao. Towards Semi-Supervised Deep Fa- cial Expression Recognition with An Adaptive Confi- dence Margin. In CVPR, 2022.","title":"Towards Semi-Supervised Deep Facial Expression Recognition with An Adaptive Confidence Margin","authors":"Hangyu Li, Nannan Wang, Xi Yang, Xiaoyu Wang, Xinbo Gao","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lucey et al., 2010] Patrick Lucey, Jeffrey F. Cohn, Takeo Kanade, Jason Saragih, Zara Ambadar, and Iain Matthews. The extended cohn-kanade dataset (ck+): A complete da- taset for action unit and emotion-specified expression. In CVPR, 2010.","title":"The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression","authors":"Patrick Lucey, Jeffrey F Cohn, Takeo Kanade, Jason Saragih, Zara Ambadar, Iain Matthews","year":"2010","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Marino et al., 2016] Kenneth Marino, Ruslan Salakhutdi- nov, Abhinav Gupta. The more you know: Using knowledge graphs for image classification. arXiv:1612.04844, 2016.","title":"The more you know: Using knowledge graphs for image classification","authors":"Kenneth Marino, Ruslan Salakhutdinov, Abhinav Gupta","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mollahosseini et al., 2017] Ali Mollahosseini, Behzad Hasani, Mohammad H Mahoor. Affectnet: A database for facial expression, valence, arousal computing in the wild. In IEEE Transactions on Affective Computing, 2019.","title":"Affectnet: A database for facial expression, valence, arousal computing in the wild","authors":"Ali Mollahosseini, Behzad Hasani, H Mohammad, Mahoor","year":"2019","venue":"IEEE Transactions on Affective Computing","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pham et al., 2021] Hieu Pham, Zihang Dai, Qizhe Xie, Quoc V. Le. Meta Pseudo Labels. In CVPR, 2021.","title":"Meta Pseudo Labels","authors":"Hieu Pham, Zihang Dai, Qizhe Xie, V Quoc, Le","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rizve et al., 2021] Mamshad Nayeem Rizve, Kevin Du- arte, Yogesh S Rawat, Mubarak Shah. In defense of pseudo-labeling: An uncertainty-aware pseudo-label se- lection framework for semi-supervised learning. In ICLR, 2021.","title":"In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning","authors":"Rizve","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sajjadi et al., 2016] Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen.Regularization with stochastic transfor- mations and perturbations for deep semi-supervised learn- ing. In NeurIPS, 2016.","title":"Regularization with stochastic transformations and perturbations for deep semi-supervised learning","authors":"Mehdi Sajjadi, Mehran Javanmardi, Tolga Tasdizen","year":"2016","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[She et al., 2021] Jiahui She, Yibo Hu, Hailin Shi, Jun Wang, Qiu Shen, Tao Mei. Dive into ambiguity: Latent distribu- tion mining and pairwise uncertainty estimation for facial expression recognition. In CVPR, 2021.","title":"Dive into ambiguity: Latent distribution mining and pairwise uncertainty estimation for facial expression recognition","authors":"Jiahui She, Yibo Hu, Hailin Shi, Jun Wang, Qiu Shen, Tao Mei","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sohn et al., 2020] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In NeurIPS, 2020.","title":"Fixmatch: Simplifying semi-supervised learning with consistency and confidence","authors":"Sohn","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2020] Haibo Wang, Chuan Zhou, Xin Chen, Jia Wu, Shirui Pan, Jilong Wang. Graph stochastic neural networks for semi-supervised learning. In NeurIPS, 2020.","title":"Graph stochastic neural networks for semi-supervised learning","authors":"Haibo Wang, Chuan Zhou, Xin Chen, Jia Wu, Shirui Pan, Jilong Wang","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2020] Kai Wang, Xiaojiang Peng, Jianfei Yang, Debin Meng, Yu Qiao. Region attention networks for pose and occlusion robust facial expression recognition. In IEEE Transactions on Image Processing, 2020.","title":"Region attention networks for pose and occlusion robust facial expression recognition","authors":"Kai Wang, Xiaojiang Peng, Jianfei Yang, Debin Meng, Yu Qiao","year":"2020","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2020] Kai Wang, Xiaojiang Peng, Jianfei Yang, Shijian Lu, and Yu Qiao. Suppressing uncertainties for large-scale facial expression recognition. In CVPR, 2020.","title":"Suppressing uncertainties for large-scale facial expression recognition","authors":"Kai Wang, Xiaojiang Peng, Jianfei Yang, Shijian Lu, Yu Qiao","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2022] Haobo Wang, Ruixuan Xiao, Yixuan Li, Lei Feng, Gang Niu, Gang Chen, Junbo Zhao. PICO: Contrastive Label Disambiguation for Partial Label Learning. In ICLR, 2022.","title":"PICO: Contrastive Label Disambiguation for Partial Label Learning","authors":"Wang","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xie et al., 2020] Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmen- tation for consistency training. In NeurIPS, 2020.","title":"Unsupervised data augmentation for consistency training","authors":"Xie","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xu et al., 2021] Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Y u-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi- supervised learning with dynamic thresholding. In ICML, 2021.","title":"Dash: Semisupervised learning with dynamic thresholding","authors":"Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Y U-Feng Li, Baigui Sun, Hao Li, Rong Jin","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xue et al., 2021] Xue, Fanglei, Qiangchang Wang, and Guodong Guo. Transfer: Learning relation-aware facial expression representations with transformers. In ICCV, 2021.","title":"Transfer: Learning relation-aware facial expression representations with transformers","authors":"Fanglei Xue, Qiangchang Wang, Guodong Guo","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yang et al., 2023] Yujie Yang, Lin Hu, Chen Zu, Qizheng Zhou, Xi Wu, Jiliu Zhou, Yan Wang: Facial Expression Recognition with Contrastive Learning and Uncertainty- Guided Relabeling. In International Journal of Neural Systems, 2023.","title":"Facial Expression Recognition with Contrastive Learning and Uncertainty-Guided Relabeling","authors":"Yujie Yang, Lin Hu, Chen Zu, Qizheng Zhou, Xi Wu, Jiliu Zhou, Yan Wang","year":"2023","venue":"International Journal of Neural Systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zeng et al., 2018] Jiabei Zeng, Shiguang Shan, and Xilin Chen. Facial expression recognition with inconsistently annotated datasets. In ECCV, 2018.","title":"Facial expression recognition with inconsistently annotated datasets","authors":"Zeng","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2016] Kaipeng Zhang, Zhanpeng Zhang, Zhif- eng Li, Y u Qiao. Joint face detection and alignment using multitask cascaded convolutional networks. In IEEE Sig- nal Processing Letters, 2016.","title":"Joint face detection and alignment using multitask cascaded convolutional networks","authors":"Zhang","year":"2016","venue":"IEEE Signal Processing Letters","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2021] Bowen Zhang, Yidong Wang, Wenxin Hou, HAO WU, Jindong Wang, Manabu Okumura, Taka- hiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. In NeurIPS, 2021.","title":"Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling","authors":"Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, Takahiro Shinozaki","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhao et al., 2011] Guoying Zhao, Xiaohua Huang, Matti Taini, Stan Z Li, and Matti PietikäInen. Facial expression recognition from near-infrared videos. Image and Vision Computing, 2011.","title":"Facial expression recognition from near-infrared videos","authors":"Guoying Zhao, Xiaohua Huang, Matti Taini, Stan Z Li, Matti Pietikäinen","year":"2011","venue":"Image and Vision Computing","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}