{"type":"ScholarlyArticle","title":"Analyzing and Combating Attribute Bias for Face Restoration","authors":[{"name":"Zelin Li","slug":"zelin-li","given":"Zelin","family":"Li","affiliation":"Research Institute of Trustworthy Autonomous Systems , Southern University of Science and Technology; Department of Computer Science and Engineering , Southern University of Science and Technology"},{"name":"Dan Zeng","slug":"dan-zeng","given":"Dan","family":"Zeng","affiliation":"Research Institute of Trustworthy Autonomous Systems , Southern University of Science and Technology; Department of Computer Science and Engineering , Southern University of Science and Technology"},{"name":"Xiao Yan","slug":"xiao-yan","given":"Xiao","family":"Yan","affiliation":"Department of Computer Science and Engineering , Southern University of Science and Technology"},{"name":"Qiaomu Shen","slug":"qiaomu-shen","given":"Qiaomu","family":"Shen","affiliation":"Research Institute of Trustworthy Autonomous Systems , Southern University of Science and Technology; Department of Computer Science and Engineering , Southern University of Science and Technology"},{"name":"Bo Tang","slug":"bo-tang","given":"Bo","family":"Tang","affiliation":"Research Institute of Trustworthy Autonomous Systems , Southern University of Science and Technology; Department of Computer Science and Engineering , Southern University of Science and Technology"}],"abstract":"Face restoration (FR) recovers high resolution (HR) faces from low resolution (LR) faces and is challenging due to its ill-posed nature. With years of development, existing methods can produce quality HR faces with realistic details. However, we observe that key facial attributes (e.g., age and gender) of the restored faces could be dramatically different from the LR faces and call this phenomenon attribute bias, which is fatal when using FR for applications such as surveillance and security. Thus, we argue that FR should consider not only image quality as in existing works but also attribute bias. To this end, we thoroughly analyze attribute bias with extensive experiments and find that two major causes are the lack of attribute information in LR faces and bias in the training data. Moreover, we propose the DebiasFR framework to produce HR faces with high image quality and accurate facial attributes. The key design is to explicitly model the facial attributes, which also allows to adjust facial attributes for the output HR faces. Experiment results show that DebiasFR has comparable image quality but significantly smaller attribute bias when compared with state-of-the-art FR methods.","doi":"10.24963/ijcai.2023/128","url":"https://ijcai.org/proceedings/2023/128","pdf":"https://ijcai.org/proceedings/2023/0128.pdf","markdown":"https://ijcai.org/proceedings/2023/128.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/128","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":128,"pages":{"first":1151,"last":1159},"keywords":["Computer Vision: CV: Applications","Computer Vision: CV: Bias, fairness and privacy","Computer Vision: CV: Neural generative models, auto encoders, GANs"],"videos":[],"references":[{"position":1,"raw":"[Chen et al., 2018] Yu Chen, Ying Tai, Xiaoming Liu, Chun- hua Shen, and Jian Yang. Fsrnet: End-to-end learning face super-resolution with facial priors. In CVPR, pages 2492- 2501, 2018.","title":"Fsrnet: End-to-end learning face super-resolution with facial priors","authors":"Yu Chen, Ying Tai, Xiaoming Liu, Chunhua Shen, Jian Yang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2021] Chaofeng Chen, Xiaoming Li, Lingbo Yang, Xianhui Lin, Lei Zhang, and Kwan-Yee K. Wong. Progressive semantic-aware style transformation for blind face restoration. In CVPR, pages 11896-11905. Computer Vision Foundation / IEEE, 2021.","title":"Progressive semantic-aware style transformation for blind face restoration","authors":"Chaofeng Chen, Xiaoming Li, Lingbo Yang, Xianhui Lin, Lei Zhang, K Kwan-Yee, Wong","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Courbariaux et al., 2015] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propaga- tions. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, NeurIPS, 2015.","title":"Binaryconnect: Training deep neural networks with binary weights during propagations","authors":"Matthieu Courbariaux, Yoshua Bengio, Jean-Pierre David","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dong et al., 2015] Chao Dong, Yubin Deng, Chen Change Loy, and Xiaoou Tang. Compression artifacts reduction by a deep convolutional network. In ICCV, pages 576-584, 2015.","title":"Compression artifacts reduction by a deep convolutional network","authors":"Chao Dong, Yubin Deng, Chen Change Loy, Xiaoou Tang","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gondal et al., 2018] Muhammad Waleed Gondal, Bernhard Schölkopf, and Michael Hirsch. The unreasonable ef- fectiveness of texture transfer for single image super- resolution. In ECCV, pages 80-97. Springer, 2018.","title":"The unreasonable effectiveness of texture transfer for single image superresolution","authors":"Gondal","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Grm et al., 2019] Klemen Grm, Walter J Scheirer, and Vit- omir Štruc. Face hallucination using cascaded super- resolution and identity priors. IEEE Transactions on Im- age Processing, 29:2150-2165, 2019.","title":"Face hallucination using cascaded superresolution and identity priors","authors":"Klemen Grm, Walter J Scheirer, Vitomir Štruc","year":"2019","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gu et al., 2022] Yuchao Gu, Xintao Wang, Liangbin Xie, Chao Dong, Gen Li, Ying Shan, and Ming-Ming Cheng. VQFR: blind face restoration with vector-quantized dictio- nary and parallel decoder. In ECCV, volume 13678, pages 126-143, 2022.","title":"VQFR: blind face restoration with vector-quantized dictionary and parallel decoder","authors":"Yuchao Gu, Xintao Wang, Liangbin Xie, Chao Dong, Gen Li, Ying Shan, Ming-Ming Cheng","year":"2022","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hassan et al., 2021] Bilal Hassan, Ebroul Izquierdo, and Tomas Piatrik. Soft biometrics: a survey. Multimedia Tools and Applications, pages 1-44, 2021.","title":"Soft biometrics: a survey","authors":"Hassan","year":"2021","venue":"Multimedia Tools and Applications","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Heusel et al., 2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, NeurlPS, pages 6626-6637, 2017.","title":"Gans trained by a two time-scale update rule converge to a local nash equilibrium","authors":"Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Sepp Hochreiter","year":"2017","venue":"NeurlPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang and Liu, 2016] Dongdong Huang and Heng Liu. Face hallucination using convolutional neural network with iterative back projection. In Zhisheng You, Jie Zhou, Yunhong Wang, Zhenan Sun, Shiguang Shan, Wei-Shi Zheng, Jianjiang Feng, and Qijun Zhao, editors, CCBR, 2016. Huang et al., 2015]","title":"Face hallucination using convolutional neural network with iterative back projection","authors":"Dongdong Huang, Heng Liu, ; Huang","year":"2015","venue":"CCBR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"Zhiwu Huang, Shiguang Shan, Ruiping Wang, Haihong Zhang, Shihong Lao, Alifu Kuerban, and Xilin Chen. A benchmark and comparative study of video- based face recognition on COX face database. IEEE Trans. Image Process., 24(12):5967-5981, 2015.","title":"A benchmark and comparative study of videobased face recognition on COX face database","authors":"Zhiwu Huang, Shiguang Shan, Ruiping Wang, Haihong Zhang, Shihong Lao, Alifu Kuerban, Xilin Chen","year":"2015","venue":"IEEE Trans. Image Process","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Härkönen et al., 2020] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discov- ering interpretable gan controls. In NeurIPS, 2020.","title":"Ganspace: Discovering interpretable gan controls","authors":"Härkönen","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Karras et al., 2018] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. In ICLR. OpenRe- view.net, 2018.","title":"Progressive growing of gans for improved quality, stability, and variation","authors":"Tero Karras, Timo Aila, Samuli Laine, Jaakko Lehtinen","year":"2018","venue":"ICLR. OpenReview.net","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Karras et al., 2019] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, pages 4401-4410, 2019.","title":"A style-based generator architecture for generative adversarial networks","authors":"Tero Karras, Samuli Laine, Timo Aila","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kupyn et al., 2018] Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiří Matas. Deblurgan: Blind motion deblurring using conditional adversarial networks. In CVPR, pages 8183-8192, 2018.","title":"Deblurgan: Blind motion deblurring using conditional adversarial networks","authors":"Volodymyr Orest Kupyn, Mykola Budzan, Dmytro Mykhailych, Jiří Mishkin, Matas","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lee et al., 2018] Cheng-Han Lee, Kaipeng Zhang, Hu- Cheng Lee, Chia-Wen Cheng, and Winston Hsu. Attribute augmented convolutional neural network for face halluci- nation. In CVPRW, pages 721-729, 2018.","title":"Attribute augmented convolutional neural network for face hallucination","authors":"Kaipeng Cheng-Han Lee, Hu-Cheng Zhang, Chia-Wen Lee, Winston Cheng, Hsu","year":"2018","venue":"CVPRW","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2018] Xiaoming Li, Ming Liu, Yuting Ye, Wang- meng Zuo, Liang Lin, and Ruigang Yang. Learning warped guidance for blind face restoration. In ECCV, 2018.","title":"Learning warped guidance for blind face restoration","authors":"Xiaoming Li, Ming Liu, Yuting Ye, Wangmeng Zuo, Liang Lin, Ruigang Yang","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2019] Mengyan Li, Yuechuan Sun, Zhaoyu Zhang, Haonian Xie, and Jun Yu. Deep learning face hallucination via attributes transfer and enhancement. In ICME, pages 604-609. IEEE, 2019.","title":"Deep learning face hallucination via attributes transfer and enhancement","authors":"Li","year":"2019","venue":"ICME","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2020] Xiaoming Li, Wenyu Li, Dongwei Ren, Hongzhi Zhang, Meng Wang, and Wangmeng Zuo. En- hanced blind face restoration with multi-exemplar images and adaptive spatial feature fusion. In CVPR, 2020.","title":"Enhanced blind face restoration with multi-exemplar images and adaptive spatial feature fusion","authors":"Xiaoming Li, Wenyu Li, Dongwei Ren, Hongzhi Zhang, Meng Wang, Wangmeng Zuo","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lu et al., 2018] Yongyi Lu, Yu-Wing Tai, and Chi-Keung Tang. Attribute-guided face generation using conditional cyclegan. In ECCV, pages 282-297, 2018.","title":"Attribute-guided face generation using conditional cyclegan","authors":"Yongyi Lu, Yu-Wing Tai, Chi-Keung Tang","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Menon et al., 2020] Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. PULSE: self- supervised photo upsampling via latent space exploration of generative models. In CVPR, pages 2434-2442, 2020.","title":"PULSE: selfsupervised photo upsampling via latent space exploration of generative models","authors":"Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, Cynthia Rudin","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Mittal et al., 2012] Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. Making a \"completely blind\" image quality analyzer. IEEE Signal processing letters, 20(3):209-212, 2012.","title":"Making a \"completely blind\" image quality analyzer","authors":"Anish Mittal, Rajiv Soundararajan, Alan C Bovik","year":"2012","venue":"IEEE Signal processing letters","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Or-El et al., 2020] Roy Or-El, Soumyadip Sengupta, Ohad Fried, Eli Shechtman, and Ira Kemelmacher-Shlizerman. Lifespan age transformation synthesis. In ECCV, pages 739-755. Springer, 2020.","title":"Lifespan age transformation synthesis","authors":"Or-El","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from nat- ural language supervision. In ICML, pages 8748-8763, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Rothe et al., 2018] Rasmus Rothe, Radu Timofte, and Luc Van Gool. Deep expectation of real and apparent age from a single image without facial landmarks. Inter- national Journal of Computer Vision, 126(2-4):144-157, 2018.","title":"Deep expectation of real and apparent age from a single image without facial landmarks","authors":"Radu Rasmus Rothe, Luc Timofte, Van Gool","year":"2018","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Shen and Zhou, 2021] Yujun Shen and Bolei Zhou. Closed- form factorization of latent semantics in gans. In CVPR, pages 1532-1540, 2021.","title":"Closedform factorization of latent semantics in gans","authors":"Zhou ; ] Yujun Shen, Bolei Shen, Zhou","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Shen et al., 2020] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for se- mantic face editing. In CVPR, pages 9243-9252, 2020.","title":"Interpreting the latent space of gans for semantic face editing","authors":"Yujun Shen, Jinjin Gu, Xiaoou Tang, Bolei Zhou","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun, editors, ICLR, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Song et al., 2017] Yibing Song, Jiawei Zhang, Shengfeng He, Linchao Bao, and Qingxiong Yang. Learning to hallu- cinate face images via component generation and enhance- ment. In Carles Sierra, editor, IJCAI, pages 4537-4543.","title":"Learning to hallucinate face images via component generation and enhancement","authors":"Yibing Song, Jiawei Zhang, Shengfeng He, Linchao Bao, Qingxiong Yang","year":null,"venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2017/633"},{"position":30,"raw":"[van den Oord et al., 2017] Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete repre- sentation learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, NeurlPS, pages 6306-6315, 2017.","title":"Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning","authors":"Oord Van Den","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang et al., 2021] Xintao Wang, Yu Li, Honglun Zhang, and Ying Shan. Towards real-world blind face restoration with generative facial prior. In CVPR, pages 9168-9178, 2021.","title":"Towards real-world blind face restoration with generative facial prior","authors":"Wang","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yang et al., 2021] Tao Yang, Peiran Ren, Xuansong Xie, and Lei Zhang. Gan prior embedded network for blind face restoration in the wild. In CVPR, pages 672-681, 2021.","title":"Gan prior embedded network for blind face restoration in the wild","authors":"Tao Yang, Peiran Ren, Xuansong Xie, Lei Zhang","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yu et al., 2018] Xin Yu, Basura Fernando, Richard Hartley, and Fatih Porikli. Super-resolving very low-resolution face images with supplementary attributes. In CVPR, pages 908-917, 2018.","title":"Super-resolving very low-resolution face images with supplementary attributes","authors":"Xin Yu, Basura Fernando, Richard Hartley, Fatih Porikli","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhang et al., 2017] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian de- noiser: Residual learning of deep cnn for image denoising. IEEE transactions on image processing, 26(7):3142-3155, 2017.","title":"Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising","authors":"Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, Lei Zhang","year":"2017","venue":"IEEE transactions on image processing","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhao et al., 2022] Yang Zhao, Yu-Chuan Su, Chun-Te Chu, Yandong Li, Marius Renn, Yukun Zhu, Changyou Chen, and Xuhui Jia. Rethinking deep face restoration. In CVPR, pages 7652-7661, 2022.","title":"Rethinking deep face restoration","authors":"Yang Zhao, Yu-Chuan Su, Chun-Te Chu, Yandong Li, Marius Renn, Yukun Zhu, Changyou Chen, Xuhui Jia","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhou et al., 2015] Erjin Zhou, Haoqiang Fan, Zhimin Cao, Yuning Jiang, and Qi Yin. Learning face hallucination in the wild. In Blai Bonet and Sven Koenig, editors, Proceed- ings of the Twenty-Ninth AAAI Conference on Artificial In- telligence, January 25-30, 2015, Austin, Texas, USA, pages 3871-3877. AAAI, 2015.","title":"Learning face hallucination in the wild","authors":"Zhou","year":"2015","venue":"Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhu et al., 2022] Feida Zhu, Junwei Zhu, Wenqing Chu, Xinyi Zhang, Xiaozhong Ji, Chengjie Wang, and Ying Tai. Blind face restoration via integrating face shape and gen- erative priors. In CVPR, pages 7662-7671, 2022.","title":"Blind face restoration via integrating face shape and generative priors","authors":"Feida Zhu, Junwei Zhu, Wenqing Chu, Xinyi Zhang, Xiaozhong Ji, Chengjie Wang, Ying Tai","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}