{"type":"ScholarlyArticle","title":"Emotion-Controllable Generalized Talking Face Generation","authors":[{"name":"Sanjana Sinha","slug":"sanjana-sinha","given":"Sanjana","family":"Sinha","affiliation":"TCS Research , India"},{"name":"Sandika Biswas","slug":"sandika-biswas","given":"Sandika","family":"Biswas","affiliation":"TCS Research , India"},{"name":"Ravindra Yadav","slug":"ravindra-yadav","given":"Ravindra","family":"Yadav","affiliation":"IIT Kanpur , India"},{"name":"Brojeshwar Bhowmick","slug":"brojeshwar-bhowmick","given":"Brojeshwar","family":"Bhowmick","affiliation":"TCS Research , India"}],"abstract":"Despite the significant progress in recent years, very few of the AI-based talking face generation methods attempt to render natural emotions. Moreover, the scope of the methods is majorly limited to the characteristics of the training dataset, hence they fail to generalize to arbitrary unseen faces. In this paper, we propose a one-shot facial geometry-aware emotional talking face generation method that can generalize to arbitrary faces. We propose a graph convolutional neural network that uses speech content feature, along with an independent emotion input to generate emotion and speech-induced motion on facial geometry-aware landmark representation.  This representation is further used in our optical flow-guided texture generation network for producing the texture. We propose a two-branch texture generation network, with motion and texture branches designed to consider the motion and texture content independently. Compared to the previous emotion talking face methods, our method can adapt to arbitrary faces captured in-the-wild by fine-tuning with only a single image of the target identity in neutral emotion.","doi":"10.24963/ijcai.2022/184","url":"https://ijcai.org/proceedings/2022/184","pdf":"https://ijcai.org/proceedings/2022/0184.pdf","markdown":"https://ijcai.org/proceedings/2022/184.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/184","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":184,"pages":{"first":1320,"last":1327},"keywords":["Humans and AI: Human-Computer Interaction","Computer Vision: Applications","Computer Vision: Machine Learning for Vision","Computer Vision: Neural generative models, auto encoders, GANs","Machine Learning: Applications"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987114","length_sec":76,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985257","length_sec":615,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Cai et al., 2019] Yujun Cai, Liuhao Ge, Jun Liu, Jianfei Cai, Tat-Jen Cham, Junsong Yuan, and Nadia Magnenat Thal- mann. Exploiting spatial-temporal relationships for 3d pose estimation via graph convolutional networks. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 2272-2281, 2019.","title":"Exploiting spatial-temporal relationships for 3d pose estimation via graph convolutional networks","authors":"Yujun Cai, Liuhao Ge, Jun Liu, Jianfei Cai, Tat-Jen Cham, Junsong Yuan, Nadia Magnenat Thalmann","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cao et al., 2014] Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Figure 7: User Study results. The bar plots represent the average score (range 0-5, high score indicates better performance).","title":"Ragini Figure 7: User Study results. The bar plots represent the average score (range 0-5, high score indicates better performance)","authors":"Cao","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"Verma. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions on affective computing, 5:377-390, 2014.","title":"Crema-d: Crowd-sourced emotional multimodal actors dataset","authors":"Verma","year":"2014","venue":"IEEE transactions on affective computing","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2019] Lele Chen, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. Hierarchical cross-modal talk- ing face generation with dynamic pixel-wise loss. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7832-7841, 2019.","title":"Hierarchical cross-modal talking face generation with dynamic pixel-wise loss","authors":"Lele Chen, Ross K Maddox, Zhiyao Duan, Chenliang Xu","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2020] Lele Chen, Guofeng Cui, Celong Liu, Zhong Li, Ziyi Kou, Yi Xu, and Chenliang Xu. Talking- head generation with rhythmic head motion. In European Conference on Computer Vision, pages 35-51. Springer, 2020.","title":"Talkinghead generation with rhythmic head motion","authors":"Lele Chen, Guofeng Cui, Celong Liu, Zhong Li, Ziyi Kou, Yi Xu, Chenliang Xu","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chung and Zisserman, 2016] J. S. Chung and A. Zisserman. Out of time: automated lip sync in the wild. In Workshop on Multi-view Lip-reading, ACCV, 2016.","title":"Out of time: automated lip sync in the wild","authors":"J S Chung, A Zisserman","year":"2016","venue":"Workshop on Multi-view Lip-reading","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chung et al., 2017] Joon Son Chung, Amir Jamaludin, and Andrew Zisserman. You said that? arXiv preprint arXiv:1705.02966, 2017.","title":"You said that","authors":"Son Joon, Amir Chung, Andrew Jamaludin, Zisserman","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chung et al., 2018] Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. Voxceleb2: Deep speaker recognition. arXiv preprint arXiv:1806.05622, 2018.","title":"Voxceleb2: Deep speaker recognition","authors":"Son Joon, Arsha Chung, Andrew Nagrani, Zisserman","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Das et al., 2020] Dipanjan Das, Sandika Biswas, Sanjana Sinha, and Brojeshwar Bhowmick. Speech-driven facial animation using cascaded gans for learning of motion and texture. In European Conference on Computer Vision, 2020.","title":"Speech-driven facial animation using cascaded gans for learning of motion and texture","authors":"Dipanjan Das, Sandika Biswas, Sanjana Sinha, Brojeshwar Bhowmick","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Delaunay et al., 1934] Boris Delaunay, S Vide, A Lamémoire, and V De Georges. Bulletin de l'académie des sciences de l'urss. Classe des sciences mathématiques et naturelles, 6:793-800, 1934.","title":"Bulletin de l'académie des sciences de l'urss","authors":"Boris Delaunay, Vide, V Lamémoire, Georges De","year":"1934","venue":"Classe des sciences mathématiques et naturelles","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Deng et al., 2019] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4690-4699, 2019.","title":"Arcface: Additive angular margin loss for deep face recognition","authors":"Deng","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Eskimez et al., 2020] Sefik Emre Eskimez, You Zhang, and Zhiyao Duan. Speech driven talking face generation from a single image and an emotion condition. arXiv preprint arXiv:2008.03592, 2020.","title":"Speech driven talking face generation from a single image and an emotion condition","authors":"Eskimez","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Guo et al., 2020] Jianzhu Guo, Xiangyu Zhu, Yang Yang, Fan Yang, Zhen Lei, and Stan Z Li. Towards fast, accurate and stable 3d dense face alignment. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.","title":"Towards fast, accurate and stable 3d dense face alignment","authors":"Jianzhu Guo, Xiangyu Zhu, Yang Yang, Fan Yang, Zhen Lei, Stan Z Li","year":"2020","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hannun et al., 2014] Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.5567, 2014.","title":"Deep speech: Scaling up end-to-end speech recognition","authors":"Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Heusel et al., 2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in neural information processing systems, pages 6626-6637, 2017.","title":"Gans trained by a two time-scale update rule converge to a local nash equilibrium","authors":"Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Sepp Hochreiter","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ji et al., 2021] Xinya Ji, Hang Zhou, Kaisiyuan Wang, Wayne Wu, Chen Change Loy, Xun Cao, and Feng Xu. Audio-driven emotional video portraits. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 14080-14089, 2021.","title":"Audio-driven emotional video portraits","authors":"Xinya Ji, Hang Zhou, Kaisiyuan Wang, Wayne Wu, Chen Change Loy, Xun Cao, Feng Xu","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kipf and Welling, 2016] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.","title":"Semi-supervised classification with graph convolutional networks","authors":"N Thomas, Max Kipf, Welling","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Livingstone and Russo, 2018] Steven R Livingstone and Frank A Russo. The ryerson audio-visual database of emo- tional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american en- glish. PloS one, 13(5):e0196391, 2018.","title":"The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english","authors":"R Steven, Frank A Livingstone, Russo","year":"2018","venue":"PloS one","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mao et al., 2017] Xudong Mao, Qing Li, Haoran Xie, Ray- mond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceed- ings of the IEEE international conference on computer vi- sion, pages 2794-2802, 2017.","title":"Least squares generative adversarial networks","authors":"Xudong Mao, Qing Li, Haoran Xie, Raymond Yk Lau, Zhen Wang, Stephen Paul Smolley","year":"2017","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Narvekar and Karam, 2009] Niranjan D Narvekar and Lina J Karam. A no-reference perceptual image sharpness metric based on a cumulative probability of blur detection. In 2009 International Workshop on Quality of Multimedia Experience, pages 87-91. IEEE, 2009.","title":"A no-reference perceptual image sharpness metric based on a cumulative probability of blur detection","authors":"D Niranjan, Lina J Narvekar, Karam","year":"2009","venue":"International Workshop on Quality of Multimedia Experience","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Prajwal et al., 2020] KR Prajwal, Rudrabha Mukhopad- hyay, Vinay P Namboodiri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM International Con- ference on Multimedia, pages 484-492, 2020.","title":"A lip sync expert is all you need for speech to lip generation in the wild","authors":"Prajwal","year":"2020","venue":"Proceedings of the 28th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sinha et al., 2020] Sanjana Sinha, Sandika Biswas, and Brojeshwar Bhowmick. Identity-preserving realistic talk- ing face generation. In 2020 International Joint Con- ference on Neural Networks (IJCNN), pages 1-10. IEEE, 2020.","title":"Identity-preserving realistic talking face generation","authors":"Sanjana Sinha, Sandika Biswas, Brojeshwar Bhowmick","year":"2020","venue":"International Joint Conference on Neural Networks (IJCNN)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Song et al., 2020] Linsen Song, Wayne Wu, Chen Qian, Ran He, and Chen Change Loy. Everybody's talkin': Let me talk as you want. arXiv preprint arXiv:2001.05201, 2020.","title":"Everybody's talkin': Let me talk as you want","authors":"Linsen Song, Wayne Wu, Chen Qian, Ran He, Chen Change Loy","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Suwajanakorn et al., 2017] Supasorn Suwajanakorn, Steven M Seitz, and Ira Kemelmacher-Shlizerman. Syn- thesizing obama: learning lip sync from audio. ACM Transactions on Graphics (TOG), 36(4):1-13, 2017.","title":"Synthesizing obama: learning lip sync from audio","authors":"Suwajanakorn","year":"2017","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Thies et al., 2019] Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, and Matthias Nießner. Neural voice puppetry: Audio-driven facial reenactment. arXiv preprint arXiv:1912.05566, 2019.","title":"Neural voice puppetry: Audio-driven facial reenactment","authors":"Thies","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Vougioukas et al., 2019] Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. Realistic speech-driven facial animation with gans. International Journal of Computer Vision, pages 1-16, 2019.","title":"Realistic speech-driven facial animation with gans","authors":"Konstantinos Vougioukas, Stavros Petridis, Maja Pantic","year":"2019","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transac- tions on image processing, 13(4):600-612, 2004.","title":"Image quality assessment: from error visibility to structural similarity","authors":"Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli","year":"2004","venue":"IEEE transactions on image processing","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2020] Kaisiyuan Wang, Qianyi Wu, Linsen Song, Zhuoqian Yang, Wayne Wu, Chen Qian, Ran He, Yu Qiao, and Chen Change Loy. Mead: A large-scale audio-visual dataset for emotional talking-face generation. In European Conference on Computer Vision, pages 700- 717. Springer, 2020.","title":"Mead: A large-scale audio-visual dataset for emotional talking-face generation","authors":"Kaisiyuan Wang, Qianyi Wu, Linsen Song, Zhuoqian Yang, Wayne Wu, Chen Qian, Ran He, Yu Qiao, Chen Change Loy","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2021] Suzhen Wang, Lincheng Li, Yu Ding, Changjie Fan, and Xin Yu. Audio2head: Audio-driven one-shot talking-head generation with natural head mo- tion. IJCAI, 2021.","title":"Audio2head: Audio-driven one-shot talking-head generation with natural head motion","authors":"Suzhen Wang, Lincheng Li, Yu Ding, Changjie Fan, Xin Yu","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/152"},{"position":30,"raw":"[Wen et al., 2020] Xin Wen, Miao Wang, Christian Richardt, Ze-Yin Chen, and Shi-Min Hu. Photorealistic audio-driven video portraits. IEEE Transactions on Visualization and Computer Graphics, 26(12):3457-3466, 2020.","title":"Photorealistic audio-driven video portraits","authors":"Xin Wen, Miao Wang, Christian Richardt, Ze-Yin Chen, Shi-Min Hu","year":"2020","venue":"IEEE Transactions on Visualization and Computer Graphics","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yan et al., 2018] Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton-based action recognition. In Thirty-second AAAI conference on artificial intelligence, 2018.","title":"Spatial temporal graph convolutional networks for skeleton-based action recognition","authors":"Sijie Yan, Yuanjun Xiong, Dahua Lin","year":"2018","venue":"Thirty-second AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yu et al., 2018] Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. Bisenet: Bi- lateral segmentation network for real-time semantic seg- mentation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 325-341, 2018.","title":"Bisenet: Bilateral segmentation network for real-time semantic segmentation","authors":"Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, Nong Sang","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2021] Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. Flow-guided one-shot talking face gen- eration with a high-resolution audio-visual dataset. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3661-3670, 2021.","title":"Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset","authors":"Zhang","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhou et al., 2019] Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, and Xiaogang Wang. Talking face generation by adver- sarially disentangled audio-visual representation. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9299-9306, 2019.","title":"Talking face generation by adversarially disentangled audio-visual representation","authors":"Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, Xiaogang Wang","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhou et al., 2020] Yang Zhou, Xintong Han, Eli Shecht- man, Jose Echevarria, Evangelos Kalogerakis, and Dingzeyu Li. Makelttalk: speaker-aware talking-head an- imation. ACM Transactions on Graphics (TOG), 39(6):1- 15, 2020.","title":"Makelttalk: speaker-aware talking-head animation","authors":"Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevarria, Evangelos Kalogerakis, Dingzeyu Li","year":"2020","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhou et al., 2021] Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Ziwei Liu. Pose- controllable talking face generation by implicitly modu- larized audio-visual representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4176-4186, 2021.","title":"Posecontrollable talking face generation by implicitly modularized audio-visual representation","authors":"Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, Ziwei Liu","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}