{"type":"ScholarlyArticle","title":"HifiHead: One-Shot High Fidelity Neural Head Synthesis with 3D Control","authors":[{"name":"Feida Zhu","slug":"feida-zhu","given":"Feida","family":"Zhu","affiliation":"Youtu Lab , Tencent"},{"name":"Junwei Zhu","slug":"junwei-zhu","given":"Junwei","family":"Zhu","affiliation":"Youtu Lab , Tencent"},{"name":"Wenqing Chu","slug":"wenqing-chu","given":"Wenqing","family":"Chu","affiliation":"Youtu Lab , Tencent"},{"name":"Ying Tai","slug":"ying-tai","given":"Ying","family":"Tai","affiliation":"Youtu Lab , Tencent"},{"name":"Zhifeng Xie","slug":"zhifeng-xie","given":"Zhifeng","family":"Xie","affiliation":"Shanghai Film Academy , Shanghai University"},{"name":"Xiaoming Huang","slug":"xiaoming-huang","given":"Xiaoming","family":"Huang","affiliation":"Youtu Lab , Tencent"},{"name":"Chengjie Wang","slug":"chengjie-wang","given":"Chengjie","family":"Wang","affiliation":"Youtu Lab , Tencent"}],"abstract":"We propose HifiHead, a high fidelity neural talking head synthesis method, which can well preserve the source image's appearance and control the motion (e.g., pose, expression, gaze) flexibly with 3D morphable face models (3DMMs) parameters derived from a driving image or indicated by users. Existing head synthesis works mainly focus on low-resolution inputs. Instead, we exploit the powerful generative prior embedded in StyleGAN to achieve high-quality head synthesis and editing. Specifically, we first extract the source image's appearance and driving image's motion to construct 3D face descriptors, which are employed as latent style codes for the generator. Meanwhile, hierarchical representations are extracted from the source and rendered 3D images respectively to provide faithful appearance and shape guidance. Considering the appearance representations need high-resolution flow fields for spatial transform, we propose a coarse-to-fine style-based generator consisting of a series of feature alignment and refinement (FAR) blocks. Each FAR block updates the dense flow fields and refines RGB outputs simultaneously for efficiency. Extensive experiments show that our method blends source appearance and target motion more accurately along with more photo-realistic results than previous state-of-the-art approaches.","doi":"10.24963/ijcai.2022/244","url":"https://ijcai.org/proceedings/2022/244","pdf":"https://ijcai.org/proceedings/2022/0244.pdf","markdown":"https://ijcai.org/proceedings/2022/244.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/244","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":244,"pages":{"first":1750,"last":1756},"keywords":["Computer Vision: Applications","Computer Vision: 3D Computer Vision","Computer Vision: Neural generative models, auto encoders, GANs"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986678","length_sec":257,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984821","length_sec":294,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Chu et al., 2020] Wenqing Chu, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, and Rongrong Ji. Sscgan: Fa- cial attribute editing via style skip connections. In ECCV, 2020.","title":"Sscgan: Facial attribute editing via style skip connections","authors":"Wenqing Chu, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, Rongrong Ji","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Deng et al., 2019a] Jiankang Deng, J. Guo, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. CVPR, 2019.","title":"Arcface: Additive angular margin loss for deep face recognition","authors":"Deng","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deng et al., 2019b] Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. Accurate 3d face re- construction with weakly-supervised learning: From sin- gle image to image set. In CVPRW, 2019.","title":"Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set","authors":"Deng","year":"2019","venue":"CVPRW","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Doukas et al., 2021] Michail Christos Doukas, Stefanos Zafeiriou, and Viktoriia Sharmanska. Headgan: One-shot neural head synthesis and editing. In ICCV, 2021.","title":"Headgan: One-shot neural head synthesis and editing","authors":"Christos Michail, Stefanos Doukas, Viktoriia Zafeiriou, Sharmanska","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ha et al., 2020] Sungjoo Ha, Martin Kersner, Beomsu Kim, Seokjun Seo, and Dongyoung Kim. Marionette: Few-shot face reenactment preserving identity of unseen targets. In AAAI, 2020.","title":"Marionette: Few-shot face reenactment preserving identity of unseen targets","authors":"Ha","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Heusel et al., 2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS, 2017.","title":"Gans trained by a two time-scale update rule converge to a local nash equilibrium","authors":"Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Sepp Hochreiter","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Huang et al., 2020] Y. Huang, Yuhan Wang, Ying Tai, Xi- aoming Liu, Pengcheng Shen, Shaoxin Li, Jilin Li, and Feiyue Huang. Curricularface: Adaptive curriculum learn- ing loss for deep face recognition. In CVPR, 2020.","title":"Curricularface: Adaptive curriculum learning loss for deep face recognition","authors":"Y Huang, Yuhan Wang, Ying Tai, Xiaoming Liu, Pengcheng Shen, Shaoxin Li, Jilin Li, Feiyue Huang","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Johnson et al., 2016] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016.","title":"Perceptual losses for real-time style transfer and super-resolution","authors":"Justin Johnson, Alexandre Alahi, Li Fei-Fei","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Karras et al., 2017] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. In ICLR, 2017.","title":"Progressive growing of gans for improved quality, stability, and variation","authors":"Tero Karras, Timo Aila, Samuli Laine, Jaakko Lehtinen","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Karras et al., 2019] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.","title":"A style-based generator architecture for generative adversarial networks","authors":"Tero Karras, Samuli Laine, Timo Aila","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Karras et al., 2020] Tero Karras, Samuli Laine, Miika Ait- tala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. An- alyzing and improving the image quality of stylegan. In CVPR, 2020.","title":"Analyzing and improving the image quality of stylegan","authors":"Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, Timo Aila","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Karras et al., 2021] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. In NeurIPS, 2021.","title":"Alias-free generative adversarial networks","authors":"Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, Timo Aila","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Nagrani et al., 2017] Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. Voxceleb: A large-scale speaker iden- tification dataset. In INTERSPEECH, 2017.","title":"Voxceleb: A large-scale speaker identification dataset","authors":"Arsha Nagrani, Joon, Son Chung, Andrew Zisserman","year":"2017","venue":"INTERSPEECH","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pumarola et al., 2021] Albert Pumarola, Enric Corona, Ger- ard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. CVPR, 2021.","title":"Albert Pumarola, Enric Corona","authors":"Pumarola","year":"2021","venue":"Neural radiance fields for dynamic scenes. CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ren et al., 2021] Yurui Ren, Ge Li, Yuanqi Chen, Thomas H Li, and Shan Liu. Pirenderer: Control- lable portrait image generation via semantic neural rendering. In ICCV, 2021.","title":"Pirenderer: Controllable portrait image generation via semantic neural rendering","authors":"Ren","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Siarohin et al., 2019a] Aliaksandr Siarohin, Stéphane Lath- uilière, S. Tulyakov, Elisa Ricci, and N. Sebe. First order motion model for image animation. In NeurIPS, 2019.","title":"First order motion model for image animation","authors":"Siarohin","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Siarohin et al., 2019b] Aliaksandr Siarohin, Stéphane Lath- uilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. An- imating arbitrary objects via deep motion transfer. In CVPR, 2019.","title":"Animating arbitrary objects via deep motion transfer","authors":"Siarohin","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Siarohin et al., 2021] Aliaksandr Siarohin, Oliver J Wood- ford, Jian Ren, Menglei Chai, and Sergey Tulyakov. Mo- tion representations for articulated animation. In CVPR, 2021.","title":"Motion representations for articulated animation","authors":"Aliaksandr Siarohin, J Oliver, Jian Woodford, Menglei Ren, Sergey Chai, Tulyakov","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tewari et al., 2020a] Ayush Tewari, Mohamed Elgharib, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, and Christian Theobalt. Pie: Portrait image embedding for semantic control. TOG, 2020.","title":"Pie: Portrait image embedding for semantic control","authors":"Tewari","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tewari et al., 2020b] Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhofer, and Christian Theobalt. Stylerig: Rigging stylegan for 3d control over portrait im- ages. In CVPR, 2020.","title":"Stylerig: Rigging stylegan for 3d control over portrait images","authors":"Tewari","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2021a] Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. One-shot free-view neural talking-head syn- thesis for video conferencing. CVPR, 2021.","title":"One-shot free-view neural talking-head synthesis for video conferencing","authors":"Wang","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2021b] Yuhan Wang, Xu Chen, Junwei Zhu, Wenqing Chu, Ying Tai, Chengjie Wang, Jilin Li, Yongjian Wu, Feiyue Huang, and Rongrong Ji. Hififace: 3d shape and semantic prior guided high fidelity face swapping. In IJCAI, 2021.","title":"Hififace: 3d shape and semantic prior guided high fidelity face swapping","authors":"Wang","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/157"},{"position":23,"raw":"[Wiles et al., 2018] Olivia Wiles, A Koepke, and Andrew Zisserman. X2face: A network for controlling face gen- eration using images, audio, and pose codes. In ECCV, 2018.","title":"X2face: A network for controlling face generation using images, audio, and pose codes","authors":"Olivia Wiles, Andrew Koepke, Zisserman","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu et al., 2018] Wayne Wu, Yunxuan Zhang, Cheng Li, Chen Qian, and Chen Change Loy. Reenactgan: Learn- ing to reenact faces via boundary transfer. In ECCV, 2018.","title":"Reenactgan: Learning to reenact faces via boundary transfer","authors":"Wayne Wu, Yunxuan Zhang, Cheng Li, Chen Qian, Chen Change Loy","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yang et al., 2021] Tao Yang, Peiran Ren, Xuansong Xie, and Lei Zhang. Gan prior embedded network for blind face restoration in the wild. In CVPR, 2021.","title":"Gan prior embedded network for blind face restoration in the wild","authors":"Tao Yang, Peiran Ren, Xuansong Xie, Lei Zhang","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yao et al., 2021] Guangming Yao, Yi Yuan, Tianjia Shao, Shuang Li, Shanqi Liu, Yong Liu, Mengmeng Wang, and Kun Zhou. One-shot face reenactment using appearance adaptive normalization. In AAAI, 2021.","title":"One-shot face reenactment using appearance adaptive normalization","authors":"Guangming Yao, Yi Yuan, Tianjia Shao, Shuang Li, Shanqi Liu, Yong Liu, Mengmeng Wang, Kun Zhou","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zakharov et al., 2019] Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky. Few-shot adversarial learning of realistic neural talking head models. In ICCV, 2019.","title":"Few-shot adversarial learning of realistic neural talking head models","authors":"Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, Victor Lempitsky","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zakharov et al., 2020] Egor Zakharov, Aleksei Ivakhnenko, Aliaksandra Shysheya, and Victor Lempitsky. Fast bi- layer neural synthesis of one-shot realistic head avatars. In ECCV, 2020.","title":"Fast bilayer neural synthesis of one-shot realistic head avatars","authors":"Egor Zakharov, Aleksei Ivakhnenko, Aliaksandra Shysheya, Victor Lempitsky","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.","title":"The unreasonable effectiveness of deep features as a perceptual metric","authors":"Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, Oliver Wang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}