{"type":"ScholarlyArticle","title":"Talking Face Generation by Conditional Recurrent Adversarial Network","authors":[{"name":"Yang Song","slug":"yang-song","given":"Yang","family":"Song","affiliation":"University of Tennessee , Knoxville"},{"name":"Jingwen Zhu","slug":"jingwen-zhu","given":"Jingwen","family":"Zhu","affiliation":"Samsung Research America"},{"name":"Dawei Li","slug":"dawei-li","given":"Dawei","family":"Li","affiliation":"Samsung Research America"},{"name":"Andy Wang","slug":"andy-wang","given":"Andy","family":"Wang","affiliation":"IBM"},{"name":"Hairong Qi","slug":"hairong-qi","given":"Hairong","family":"Qi","affiliation":"University of Tennessee , Knoxville"}],"abstract":"Given an arbitrary face image and an arbitrary speech clip, the proposed work attempts to generate the talking face video with accurate lip synchronization. Existing works either do not consider temporal dependency across video frames thus yielding abrupt facial and lip movement or are limited to the generation of talking face video for a specific person thus lacking generalization capacity. We propose a novel conditional recurrent generation network that incorporates both image and audio features in the recurrent unit for  temporal dependency. To achieve both image- and video-realism, a pair of spatial-temporal discriminators are included in the network for better image/video quality. Since accurate lip synchronization is essential to the success of talking face video generation, we also construct a lip-reading discriminator to boost the accuracy of lip synchronization. We also extend the network to model the natural pose and expression of talking face on the Obama Dataset. Extensive experimental results demonstrate the superiority of our framework over the state-of-the-arts in terms of visual quality, lip sync accuracy, and smooth transition pertaining to both lip and facial movement.","doi":"10.24963/ijcai.2019/129","url":"https://ijcai.org/proceedings/2019/129","pdf":"https://ijcai.org/proceedings/2019/0129.pdf","markdown":"https://ijcai.org/proceedings/2019/129.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/129","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Computer Vision","track":"Main track","number":129,"pages":{"first":919,"last":925},"keywords":["Computer Vision: Computer Vision","Computer Vision: Language and Vision"],"videos":[],"references":[{"position":1,"raw":"[Chen et al., 2018] Lele Chen, Zhiheng Li, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. Lip movements genera- tion at a glance. arXiv preprint arXiv:1803.10404, 2018.","title":"Lip movements generation at a glance","authors":"Lele Chen, Zhiheng Li, Ross K Maddox, Zhiyao Duan, Chenliang Xu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chung and Zisserman, 2016] Joon Son Chung and Andrew Zisserman. Lip reading in the wild. In Asian Conference on Computer Vision, pages 87-103. Springer, 2016.","title":"Lip reading in the wild","authors":"Chung, Zisserman ; Joon, Son Chung, Andrew Zisserman","year":"2016","venue":"Asian Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chung et al., 2017] Joon Son Chung, Amir Jamaludin, and Andrew Zisserman. You said that? British Machine Vision Conference (BMVC), 2017.","title":"You said that? British Machine Vision Conference (BMVC)","authors":"Son Joon, Amir Chung, Andrew Jamaludin, Zisserman","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Harte and Gillen, 2015] Naomi Harte and Eoin Gillen. Tcd- timit: An audio-visual corpus of continuous speech. IEEE Transactions on Multimedia, 17(5):603-615, 2015.","title":"Tcdtimit: An audio-visual corpus of continuous speech","authors":"Gillen Harte, Naomi Harte, Eoin Gillen","year":"2015","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Karras et al., 2017] Tero Karras, Timo Aila, Samuli Laine, Antti Herva, and Jaakko Lehtinen. Audio-driven facial an- imation by joint end-to-end learning of pose and emotion. ACM Transactions on Graphics (TOG), 36(4):94, 2017.","title":"Audio-driven facial animation by joint end-to-end learning of pose and emotion","authors":"Tero Karras, Timo Aila, Samuli Laine, Antti Herva, Jaakko Lehtinen","year":"2017","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Liang et al., 2017] Xiaodan Liang, Lisa Lee, Wei Dai, and Eric P Xing. Dual motion gan for future-flow embedded video prediction. arXiv preprint, 2017.","title":"Dual motion gan for future-flow embedded video prediction","authors":"Xiaodan Liang, Lisa Lee, Wei Dai, Eric P Xing","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Mathieu et al., 2015] Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction be- yond mean square error. arXiv preprint arXiv:1511.05440, 2015.","title":"Deep multi-scale video prediction beyond mean square error","authors":"Michael Mathieu, Camille Couprie, Yann Lecun","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Nagrani et al., 2017] Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. Voxceleb: a large-scale speaker iden- tification dataset. arXiv preprint arXiv:1706.08612, 2017.","title":"Voxceleb: a large-scale speaker identification dataset","authors":"Arsha Nagrani, Joon, Son Chung, Andrew Zisserman","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Oh et al., 2015] Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in atari games. In Advances in Neural Information Processing Systems, pages 2863-2871, 2015.","title":"Action-conditional video prediction using deep networks in atari games","authors":"Oh","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Srivastava et al., 2015] Nitish Srivastava, Elman Mansi- mov, and Ruslan Salakhudinov. Unsupervised learning of video representations using lstms. In International confer- ence on machine learning, pages 843-852, 2015.","title":"Unsupervised learning of video representations using lstms","authors":"Nitish Srivastava, Elman Mansimov, Ruslan Salakhudinov","year":"2015","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Suwajanakorn et al., 2017] Supasorn Suwajanakorn, Steven M Seitz, and Ira Kemelmacher-Shlizerman. Syn- thesizing obama: learning lip sync from audio. ACM Transactions on Graphics (TOG), 36(4):95, 2017.","title":"Synthesizing obama: learning lip sync from audio","authors":"Suwajanakorn","year":"2017","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Taylor et al., 2017] Sarah Taylor, Taehwan Kim, Yisong Yue, Moshe Mahler, James Krahe, Anastasio Garcia Ro- driguez, Jessica Hodgins, and Iain Matthews. A deep learning approach for generalized speech animation. ACM Transactions on Graphics (TOG), 36(4):93, 2017.","title":"A deep learning approach for generalized speech animation","authors":"Sarah Taylor, Taehwan Kim, Yisong Yue, Moshe Mahler, James Krahe, Anastasio Garcia Rodriguez, Jessica Hodgins, Iain Matthews","year":"2017","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Tulyakov et al., 2017] Sergey Tulyakov, Ming-Yu Liu, Xi- aodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. arXiv preprint arXiv:1707.04993, 2017.","title":"Mocogan: Decomposing motion and content for video generation","authors":"Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, Jan Kautz","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Villegas et al., 2017] Ruben Villegas, Jimei Yang, Se- unghoon Hong, Xunyu Lin, and Honglak Lee. Decom- posing motion and content for natural video sequence pre- diction. arXiv preprint arXiv:1706.08033, 2017.","title":"Decomposing motion and content for natural video sequence prediction","authors":"Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, Honglak Lee","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Vondrick et al., 2016] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dy- namics. In Advances In Neural Information Processing Systems, pages 613-621, 2016.","title":"Generating videos with scene dynamics","authors":"Carl Vondrick, Hamed Pirsiavash, Antonio Torralba","year":"2016","venue":"Advances In Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2018] Ting-Chun Wang, Ming-Yu Liu, Jun- Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-video synthesis. arXiv preprint arXiv:1808.06601, 2018.","title":"Video-to-video synthesis","authors":"Ting-Chun, Ming-Yu Wang, Jun-Yan Liu, Guilin Zhu, Andrew Liu, Jan Tao, Bryan Kautz, Catanzaro","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wei, 2014] Shuang Wei. Computer vision aided lip move- ment correction to improve english pronunciation. 2014.","title":"Computer vision aided lip movement correction to improve english pronunciation","authors":"Shuang Wei","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zhang et al., 2017] Zhifei Zhang, Yang Song, and Hairong Qi. Age progression/regression by conditional adversarial autoencoder. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.","title":"Age progression/regression by conditional adversarial autoencoder","authors":"Zhang","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhang et al., 2019] Zhifei Zhang, Zhaowen Wang, Zhe Lin, and Hairong Qi. Image super-resolution by neural texture transfer. arXiv preprint arXiv:1903.00834, 2019.","title":"Image super-resolution by neural texture transfer","authors":"Zhang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhou et al., 2019] Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, and Xiaogang Wang. Talking face generation by adversar- ially disentangled audio-visual representation. In AAAI, 2019.","title":"Talking face generation by adversarially disentangled audio-visual representation","authors":"Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, Xiaogang Wang","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}