{"type":"ScholarlyArticle","title":"Reference Guided Face Component Editing","authors":[{"name":"Qiyao Deng","slug":"qiyao-deng","given":"Qiyao","family":"Deng","affiliation":"Center for Research on Intelligent Perception and Computing , NLPR , CASIA , Beijing , China; School of Artificial Intelligence , University of Chinese Academy of Sciences , Beijing , China"},{"name":"Jie Cao","slug":"jie-cao","given":"Jie","family":"Cao","affiliation":"Center for Research on Intelligent Perception and Computing , NLPR , CASIA , Beijing , China; School of Artificial Intelligence , University of Chinese Academy of Sciences , Beijing , China"},{"name":"Yunfan Liu","slug":"yunfan-liu","given":"Yunfan","family":"Liu","affiliation":"Center for Research on Intelligent Perception and Computing , NLPR , CASIA , Beijing , China; School of Artificial Intelligence , University of Chinese Academy of Sciences , Beijing , China"},{"name":"Zhenhua Chai","slug":"zhenhua-chai","given":"Zhenhua","family":"Chai","affiliation":"Vision Intelligence Center , AI Platform , Meituandianping Group"},{"name":"Qi Li","slug":"qi-li","given":"Qi","family":"Li","affiliation":"Center for Research on Intelligent Perception and Computing , NLPR , CASIA , Beijing , China; Artificial Intelligence Research , CAS , Qingdao , China; School of Artificial Intelligence , University of Chinese Academy of Sciences , Beijing , China"},{"name":"Zhenan Sun","slug":"zhenan-sun","given":"Zhenan","family":"Sun","affiliation":"Center for Research on Intelligent Perception and Computing , NLPR , CASIA , Beijing , China; Center for Excellence in Brain Science and Intelligence Technology , CAS , Beijing , China; School of Artificial Intelligence , University of Chinese Academy of Sciences , Beijing , China"}],"abstract":"Face portrait editing has achieved great progress in recent years. However, previous methods either 1) operate on pre-defined face attributes, lacking the flexibility of controlling shapes of high-level semantic facial components (e.g., eyes, nose, mouth), or 2) take manually edited mask or sketch as an intermediate representation for observable changes, but such additional input usually requires extra efforts to obtain. To break the limitations (e.g. shape, mask or sketch) of the existing methods, we propose a novel framework termed r FACE (Reference Guided FAce Component Editing) for diverse and controllable face component editing with geometric changes. Specifically, r-FACE takes an image inpainting model as the backbone, utilizing reference images as conditions for controlling the shape of face components. In order to encourage the framework to concentrate on the target face components, an example-guided attention module is designed to fuse attention features and the target face component features extracted from the reference image. Through extensive experimental validation and comparisons, we verify the effectiveness of the proposed framework.","doi":"10.24963/ijcai.2020/70","url":"https://ijcai.org/proceedings/2020/70","pdf":"https://ijcai.org/proceedings/2020/0070.pdf","markdown":"https://ijcai.org/proceedings/2020/70.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/70","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":70,"pages":{"first":502,"last":508},"keywords":["Computer Vision: 2D and 3D Computer Vision","Computer Vision: Biometrics, Face and Gesture Recognition"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24473/file_24473.m3u8","length_sec":258,"provider":"hls"}],"references":[{"position":1,"raw":"[Bertalmio et al., 2000] Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. Image inpainting. In SIGGRAPH, pages 417-424, 2000.","title":"Image inpainting","authors":"Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, Coloma Ballester","year":"2000","venue":"SIGGRAPH","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bertalmio et al., 2003] Marcelo Bertalmio, Luminita Vese, Guillermo Sapiro, and Stanley Osher. Simultaneous struc- ture and texture image inpainting. IEEE TIP, 12(8):882- 889, 2003.","title":"Simultaneous structure and texture image inpainting","authors":"Marcelo Bertalmio, Luminita Vese, Guillermo Sapiro, Stanley Osher","year":"2003","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cao et al., 2019] Jie Cao, Yibo Hu, Bing Yu, Ran He, and Zhenan Sun. 3D aided duet GANs for multi-view face image synthesis. IEEE TIFS, 14(8):2028-2042, 2019.","title":"3D aided duet GANs for multi-view face image synthesis","authors":"Jie Cao, Yibo Hu, Bing Yu, Ran He, Zhenan Sun","year":"2019","venue":"IEEE TIFS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Choi et al., 2018] Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. Star- GAN: Unified generative adversarial networks for multi- domain image-to-image translation. In CVPR, pages 8789-8797, 2018.","title":"Star-GAN: Unified generative adversarial networks for multidomain image-to-image translation","authors":"Choi","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Criminisi et al., 2003] Antonio Criminisi, Patrick Perez, and Kentaro Toyama. Object removal by exemplar-based inpainting. In CVPR, volume 2, pages II-II, 2003.","title":"Object removal by exemplar-based inpainting","authors":"Criminisi","year":"2003","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dolhansky and Canton Ferrer, 2018] Brian Dolhansky and Cristian Canton Ferrer. Eye in-painting with exemplar generative adversarial networks. In CVPR, pages 7902- 7911, 2018.","title":"Eye in-painting with exemplar generative adversarial networks","authors":"Brian Dolhansky, Cristian Canton Ferrer","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In NeurIPS, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gu et al., 2019] Shuyang Gu, Jianmin Bao, Hao Yang, Dong Chen, Fang Wen, and Lu Yuan. Mask-guided por- trait editing with conditional gans. In CVPR, pages 3436- 3445, 2019.","title":"Mask-guided portrait editing with conditional gans","authors":"Shuyang Gu, Jianmin Bao, Hao Yang, Dong Chen, Fang Wen, Lu Yuan","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2019] Zhenliang He, Wangmeng Zuo, Meina Kan, Shiguang Shan, and Xilin Chen. AttGAN: Facial attribute editing by only changing what you want. IEEE TIP, 2019.","title":"AttGAN: Facial attribute editing by only changing what you want","authors":"He","year":"2019","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Heusel et al., 2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. GANs trained by a two time-scale update rule converge to a local Nash Equilibrium. In NeurIPS, pages 6626-6637, 2017.","title":"GANs trained by a two time-scale update rule converge to a local Nash Equilibrium","authors":"Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Sepp Hochreiter","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Iizuka et al., 2017] Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Globally and locally consistent image completion. ACM ToG, 36(4):107, 2017.","title":"Globally and locally consistent image completion","authors":"Satoshi Iizuka, Edgar Simo-Serra, Hiroshi Ishikawa","year":"2017","venue":"ACM ToG","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jo and Park, 2019] Youngjoo Jo and Jongyoul Park. SC- FEGAN: Face editing generative adversarial network with user's sketch and color. In ICCV, October 2019.","title":"Youngjoo Jo and Jongyoul Park. SC-FEGAN: Face editing generative adversarial network with user's sketch and color","authors":"Jo, Park","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Johnson et al., 2016] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, pages 694-711, 2016.","title":"Perceptual losses for real-time style transfer and super-resolution","authors":"Justin Johnson, Alexandre Alahi, Li Fei-Fei","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Langner et al., 2010] Oliver Langner, Ron Dotsch, Gijs- bert Bijlstra, Daniel HJ Wigboldus, Skyler T Hawk, and AD Van Knippenberg. Presentation and validation of the Radboud Faces Database. Cognition and Emotion, 24(8):1377-1388, 2010.","title":"Presentation and validation of the Radboud Faces Database","authors":"Oliver Langner, Ron Dotsch, Gijsbert Bijlstra, H J Daniel, Skyler T Wigboldus, Hawk, Van Knippenberg","year":"2010","venue":"Cognition and Emotion","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lee et al., 2019] Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. MaskGAN: Towards diverse and in- teractive facial image manipulation. arXiv preprint arXiv:1907.11922, 2019.","title":"MaskGAN: Towards diverse and interactive facial image manipulation","authors":"Ziwei Cheng-Han Lee, Lingyun Liu, Ping Wu, Luo","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2015] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, pages 3730-3738, 2015.","title":"Deep learning face attributes in the wild","authors":"Ziwei Liu, Ping Luo, Xiaogang Wang, Xiaoou Tang","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2019] Yunfan Liu, Qi Li, and Zhenan Sun. Attribute-aware face aging with wavelet-based genera- tive adversarial networks. In CVPR, pages 11877-11886, 2019.","title":"Attribute-aware face aging with wavelet-based generative adversarial networks","authors":"Liu","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mao et al., 2017] Xudong Mao, Qing Li, Haoran Xie, Ray- mond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In ICCV, pages 2794-2802, 2017.","title":"Least squares generative adversarial networks","authors":"Xudong Mao, Qing Li, Haoran Xie, Raymond Yk Lau, Zhen Wang, Stephen Paul Smolley","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mechrez et al., 2018] Roey Mechrez, Itamar Talmi, and Lihi Zelnik-Manor. The contextual loss for image transfor- mation with non-aligned data. In ECCV, pages 768-783, 2018.","title":"The contextual loss for image transformation with non-aligned data","authors":"Roey Mechrez, Itamar Talmi, Lihi Zelnik-Manor","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Song et al., 2019] Linsen Song, Jie Cao, Linxiao Song, Yibo Hu, and Ran He. Geometry-aware face completion and editing. In AAAI, pages 2506-2513, 2019.","title":"Geometry-aware face completion and editing","authors":"Linsen Song, Jie Cao, Linxiao Song, Yibo Hu, Ran He","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2003] Zhou Wang, Eero P Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assessment. In ACSSC, volume 2, pages 1398- 1402, 2003.","title":"Multiscale structural similarity for image quality assessment","authors":"Zhou Wang, Eero P Simoncelli, Alan C Bovik","year":"2003","venue":"ACSSC","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu et al., 2019] Po-Wei Wu, Yu-Jing Lin, Che-Han Chang, Edward Y Chang, and Shih-Wei Liao. RelGAN: Multi- domain image-to-image translation via relative attributes. In ICCV, pages 5914-5922, 2019.","title":"RelGAN: Multidomain image-to-image translation via relative attributes","authors":"Po-Wei Wu, Yu-Jing Lin, Che-Han Chang, Edward Y Chang, Shih-Wei Liao","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xiao et al., 2018] Taihong Xiao, Jiapeng Hong, and Jinwen Ma. Elegant: Exchanging latent encodings with gan for transferring multiple face attributes. In ECCV, pages 168- 184, 2018.","title":"Elegant: Exchanging latent encodings with gan for transferring multiple face attributes","authors":"Xiao","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yang et al., 2018] Hongyu Yang, Di Huang, Yunhong Wang, and Anil K Jain. Learning face age progression: A pyramid architecture of gans. In CVPR, pages 31-39, 2018.","title":"Learning face age progression: A pyramid architecture of gans","authors":"Hongyu Yang, Di Huang, Yunhong Wang, Anil K Jain","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zheng et al., 2019] Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. Pluralistic image completion. In CVPR, pages 1438-1447, 2019.","title":"Pluralistic image completion","authors":"Zheng","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}