{"type":"ScholarlyArticle","title":"Arbitrary Talking Face Generation via Attentional Audio-Visual Coherence Learning","authors":[{"name":"Hao Zhu","slug":"hao-zhu","given":"Hao","family":"Zhu","affiliation":"School of Computer Science and Technology , Anhui University , Hefei , China; NLPR&CEBSIT&CRIPAC , Institute of Automation , CAS , Beijing , China"},{"name":"Huaibo Huang","slug":"huaibo-huang","given":"Huaibo","family":"Huang","affiliation":"NLPR&CEBSIT&CRIPAC , Institute of Automation , CAS , Beijing , China; School of Artificial Intelligence , University of Chinese Academy of Sciences , Beijing , China"},{"name":"Yi Li","slug":"yi-li","given":"Yi","family":"Li","affiliation":"NLPR&CEBSIT&CRIPAC , Institute of Automation , CAS , Beijing , China; School of Artificial Intelligence , University of Chinese Academy of Sciences , Beijing , China"},{"name":"Aihua Zheng","slug":"aihua-zheng","given":"Aihua","family":"Zheng","affiliation":"School of Computer Science and Technology , Anhui University , Hefei , China"},{"name":"Ran He","slug":"ran-he","given":"Ran","family":"He","affiliation":"NLPR&CEBSIT&CRIPAC , Institute of Automation , CAS , Beijing , China; School of Artificial Intelligence , University of Chinese Academy of Sciences , Beijing , China"}],"abstract":"Talking face generation aims to synthesize a face video with precise lip synchronization as well as a smooth transition of facial motion over the entire video via the given speech clip and facial image. Most existing methods mainly focus on either disentangling the information in a single image or learning temporal information between frames. However, cross-modality coherence between audio and video information has not been well addressed during synthesis. In this paper, we propose a novel arbitrary talking face generation framework by discovering the audio-visual coherence via the proposed Asymmetric Mutual Information Estimator (AMIE). In addition, we propose a Dynamic Attention (DA) block by selectively focusing the lip area of the input image during the training stage, to further enhance lip synchronization. Experimental results on benchmark LRW dataset and GRID dataset transcend the state-of-the-art methods on prevalent metrics with robust high-resolution synthesizing on gender and pose variations.","doi":"10.24963/ijcai.2020/327","url":"https://ijcai.org/proceedings/2020/327","pdf":"https://ijcai.org/proceedings/2020/0327.pdf","markdown":"https://ijcai.org/proceedings/2020/327.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/327","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":327,"pages":{"first":2362,"last":2368},"keywords":["Humans and AI: Human-Computer Interaction","Machine Learning: Deep Generative Models"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24213/file_24213.m3u8","length_sec":295,"provider":"hls"}],"references":[{"position":1,"raw":"[Belghazi et al., 2018] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, R. Devon Hjelm, and Aaron C. Courville. Mutual infor- mation neural estimation. In International Conference on Machine Learning (ICML), 2018.","title":"Mutual information neural estimation","authors":"Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, R Devon Hjelm, Aaron C Courville","year":"2018","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2016] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Info- gan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pages 2172-2180, 2016.","title":"Infogan: Interpretable representation learning by information maximizing generative adversarial nets","authors":"Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, Pieter Abbeel","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2018] Lele Chen, Zhiheng Li, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. Lip movements gener- ation at a glance. In European Conference on Computer Vision (ECCV), 2018.","title":"Lip movements generation at a glance","authors":"Lele Chen, Zhiheng Li, Ross K Maddox, Zhiyao Duan, Chenliang Xu","year":"2018","venue":"European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2019] Lele Chen, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. Hierarchical cross-modal talk- ing face generation with dynamic pixel-wise loss. In IEEE Conference on Computer Vision and Pattern Recognition, pages 7832-7841, 2019.","title":"Hierarchical cross-modal talking face generation with dynamic pixel-wise loss","authors":"Lele Chen, Ross K Maddox, Zhiyao Duan, Chenliang Xu","year":"2019","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chung and Zisserman, 2016] Joon Son Chung and Andrew Zisserman. Lip reading in the wild. In Asian Conference on Computer Vision (ACCV), 2016.","title":"Lip reading in the wild","authors":"Chung, Zisserman ; Joon, Son Chung, Andrew Zisserman","year":"2016","venue":"Asian Conference on Computer Vision (ACCV)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chung et al., 2017] Joon Son Chung, Amir Jamaludin, and Andrew Zisserman. You said that? In British Machine Vision Conference (BMVC), 2017.","title":"You said that?","authors":"Son Joon, Amir Chung, Andrew Jamaludin, Zisserman","year":"2017","venue":"British Machine Vision Conference (BMVC)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cooke et al., 2006] Martin Cooke, Jon Barker, Stuart Cun- ningham, and Xu Shao. An audio-visual corpus for speech perception and automatic speech recognition. Journal of the Acoustical Society of America, 120(5):2421-2424, 2006.","title":"An audio-visual corpus for speech perception and automatic speech recognition","authors":"Martin Cooke, Jon Barker, Stuart Cunningham, Xu Shao","year":"2006","venue":"Journal of the Acoustical Society of America","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Donsker and Varadhan, 1983] Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time. iv. Communications on Pure and Applied Mathematics, 36(2):183-212, 1983.","title":"Asymptotic evaluation of certain markov process expectations for large time","authors":"D Monroe, Donsker, Sr Srinivasa Varadhan","year":"1983","venue":"Communications on Pure and Applied Mathematics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gupta and Aggarwal, 2010] Neelima Gupta and Seema Ag- garwal. Mib: Using mutual information for biclustering gene expression data. Pattern Recognition, 43(8):2692- 2697, 2010.","title":"Neelima Gupta and Seema Aggarwal. Mib: Using mutual information for biclustering gene expression data","authors":"Aggarwal Gupta","year":"2010","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2009] Ran He, Bao-Gang Hu, and Xiao-Tong Yuan. Robust discriminant analysis based on nonparamet- ric maximum entropy. In Asian Conference on Machine Learning, pages 120-134. Springer, 2009.","title":"Robust discriminant analysis based on nonparametric maximum entropy","authors":"Ran He, Bao-Gang Hu, Xiao-Tong Yuan","year":"2009","venue":"Asian Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2014] Ran He, Baogang Hu, Xiaotong Yuan, and Liang Wang. Robust recognition via information theoretic learning. Springer, 2014.","title":"Robust recognition via information theoretic learning","authors":"Ran He, Baogang Hu, Xiaotong Yuan, Liang Wang","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hjelm et al., 2018] R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670, 2018.","title":"Learning deep representations by mutual information estimation and maximization","authors":"Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Adam Trischler, Yoshua Bengio","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hulle, 2005] Marc M Van Hulle. Edgeworth approximation of multivariate differential entropy. Neural Computation, 17(9):1903-1910, 2005.","title":"Edgeworth approximation of multivariate differential entropy","authors":"M Marc, Van Hulle","year":"2005","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Johnson et al., 2016] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Com- puter Vision (ECCV), 2016.","title":"Perceptual losses for real-time style transfer and super-resolution","authors":"Justin Johnson, Alexandre Alahi, Li Fei-Fei","year":"2016","venue":"European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jun et al., 2011] Bongjin Jun, Taewan Kim, and Daijin Kim. A compact local binary pattern using maximization of mu- tual information for face analysis. Pattern Recognition, 44(3):532-543, 2011.","title":"Bongjin Jun, Taewan Kim, and Daijin Kim. A compact local binary pattern using maximization of mutual information for face analysis","authors":"Jun","year":"2011","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kraskov et al., 2004] Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mu- tual information. Physical review E, 69(6):066138, 2004.","title":"Estimating mutual information","authors":"Alexander Kraskov, Harald Stögbauer, Peter Grassberger","year":"2004","venue":"Physical review E","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kumar et al., 2017] Rithesh Kumar, Jose Sotelo, Kundan Kumar, Alexandre de Brébisson, and Yoshua Bengio. Obamanet: Photo-realistic lip-sync from text. arXiv preprint arXiv:1801.01442, 2017.","title":"Obamanet: Photo-realistic lip-sync from text","authors":"Kumar","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2009] Huawen Liu, Jigui Sun, Lei Liu, and Hui- jie Zhang. Feature selection with dynamic mutual infor- mation. Pattern Recognition, 42(7):1330-1339, 2009.","title":"Feature selection with dynamic mutual information","authors":"Liu","year":"2009","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ruderman et al., 2012] Avraham Ruderman, Mark Reid, Darío García-García, and James Petterson. Tighter vari- ational representations of f-divergences via restriction to probability measures. arXiv preprint arXiv:1206.4664, 2012.","title":"Tighter variational representations of f-divergences via restriction to probability measures","authors":"Avraham Ruderman, Mark Reid, Darío García-García, James Petterson","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sotoca and Pla, 2010] José Martínez Sotoca and Filiberto Pla. Supervised feature selection by clustering using conditional mutual information-based distances. Pattern Recognition, 43(6):2068-2081, 2010.","title":"José Martínez Sotoca and Filiberto Pla. Supervised feature selection by clustering using conditional mutual information-based distances","authors":"Pla Sotoca","year":"2010","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Suwajanakorn et al., 2017] Supasorn Suwajanakorn, Steven M Seitz, and Ira Kemelmacher-Shlizerman. Syn- thesizing obama: learning lip sync from audio. ACM Transactions on Graphics (TOG), 36(4):95:1-95:13, 2017.","title":"Synthesizing obama: learning lip sync from audio","authors":"Suwajanakorn","year":"2017","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Vincent et al., 2010] Pascal Vincent, Hugo Larochelle, Is- abelle Lajoie, Yoshua Bengio, and Pierre-Antoine Man- zagol. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of Machine Learning Research (JMLR), 11(12):3371-3408, 2010.","title":"Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion","authors":"Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol","year":"2010","venue":"Journal of Machine Learning Research (JMLR)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vougioukas et al., 2018] Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. End-to-end speech-driven facial animation with temporal gans. In British Machine Vision Conference (BMVC), 2018.","title":"End-to-end speech-driven facial animation with temporal gans","authors":"Konstantinos Vougioukas, Stavros Petridis, Maja Pantic","year":"2018","venue":"British Machine Vision Conference (BMVC)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Trans- actions on Image Processing (TIP), 13(4):600-612, 2004.","title":"Image quality assessment: from error visibility to structural similarity","authors":"Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli","year":"2004","venue":"IEEE Transactions on Image Processing (TIP)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhou et al., 2019] Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, and Xiaogang Wang. Talking face generation by adversar- ially disentangled audio-visual representation. In Interna- tional AAAI Conference on Artificial Intelligence (AAAI), 2019.","title":"Talking face generation by adversarially disentangled audio-visual representation","authors":"Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, Xiaogang Wang","year":"2019","venue":"International AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}