Speech2Talking-Face: Inferring and Driving a Face with Synchronized Audio-Visual Representation

1 Introduction

It has been proven in neuroscience and psychology [Kamachi et al., 2003; Mavica and Barenholtz, 2013] that detailed facial appearances can somehow be inferred from human voices. The same genetic, physical or environmental factors may biologically affect both voice and face [Wen et al., 2019a]. Based on this knowledge, recent studies in artificial intelligence have sought to associate facial appearances with the human speech in a data-driven manner. They propose to retrieve faces with speech [Nagrani et al., 2018b; Kim et al., 2018a; Nagrani et al., 2018a] and even directly generate face images from speech [Oh et al., 2019; Wen et al., 2019b; Choi et al., 2020], to fulfill human's visual imagination when hearing a voice.

∗Equal contribution.

Previous studies focus more on depicting a still face image. They normally match audio features with visual ones [Oh et al., 2019; Wen et al., 2019b; Choi et al., 2020], and then leverage a pretrained face normalization or generation network for reconstruction. However, facial appearance is not the only thing that we can imagine when listening to a clip of speech. We shall also picture the mouth movement which is strongly correlated with the content of the speech. Meanwhile, researchers have succeeded in driving a known facial image to speak conditioning on audios [Zhou et al., 2019; Chen et al., 2019], but the reference identity has to be given. The ability to directly generate the appearance and movements of a face could be useful in multiple scenarios.

In this paper, we investigate deeper into the correlations between speech and faces and try to visualize as much information as possible from a clip of speech in a unified model. This is a non-trivial task. On one hand, there is no fixed matching between voice and faces. It is not possible to reconstruct the exact face given a clip of speech. On the other hand, identity and lip motion information are entangled within both faces and audios. It is difficult to build supervision for both information individually at the same time.

To this end, we propose Speech2Talking-Face (S2TF), a generative framework that is not only capable of inferring appearances but also driving the faces to talk with speech. The key insight is to synchronize speech with visual representations in two branches, then utilize a style-based generator for multi-modal information balancing. Specifically, two latent spaces, namely the identity space and the identity-irrelevant space, are defined through the reconstruction training of visual features as [Burkov et al., 2020; Zhou et al., 2021]. Proven to be effective for learning coherent audio-visual representations [Nagrani et al., 2018a; Chung and Zisserman, 2016], the contrastive loss is leveraged for our modality synchronization in the two spaces. Detailedly, due to the fact that identity-irrelevant space contains speech-irrelevant information such as head poses, we map speech features to a content subspace in the identity-irrelevant space by synchronizing speech with the differences of visual feature. In this way, the initial pose of our generated face can be determined by any facial image. Furthermore, inspired by discoveries in face recognition [Deng et al., 2019], class centroids of identities are leveraged for more compact identity representation learning.

Finally, the features from the two spaces are integrated in the style-based generator. Visual-to-audio curriculum learning is adopted to support the generator in balancing information from two latent spaces and two modalities. After training, our model is capable of leveraging one clip of speech to generate a dynamic face whose appearance is controlled by the identity space, with accurate lip motion controlled by the content subspace. Compared with previous methods, our generated results are of higher quality and more attractive given our particular ability to reflect speech content information. Moreover, experiments show that our learned speech-identity association is comparable to previous studies.

Our contributions are summarized as follows: 1) We propose to extract as much speech-facial associated information as possible with two branches of audio-visual synchronization. 2) We strengthen the synchronized identity space with the concept of class centroids, and learning a content subspace within the identity-irrelevant space. 3) We successfully achieve speech-inferred facial appearance reconstruction and speech-driven talking face generation within a unified model called Speech2Talking-Face. To the best of our knowledge, this problem has not been addressed before.

2 RelatedWork

Face Reconstruction from Voice. Reconstructing face from human voice has recently gained great attention in the field of artificial intelligence, given its potential applications in entertainment and security. Speech2Face [Oh et al., 2019] leverages a pretrained face encoder, a fixed face normalization decoder, and attempts to map speech embeddings to the face encoded ones. Their results suffer from severe artifacts due to the weak capability of the decoder. Wav2Pix [Duarte et al., 2019] employs a speech-conditioned GAN architecture, but could not generalize to speakers outside of its training set. Voice2Face [Wen et al., 2019b] utilizes an identity classifier and an image discriminator for better identity preserving while generating faces from voices. However, this work only generates fixed frontal-view faces. To disentangle the factors that are unrelated to facial characteristics, [Choi et al., 2020] add an extra random latent code to implicitly cover them. Particularly, [Meishvili et al., 2020] advocate that audio carries information that can help the recovery of reasonable highresolution faces by mixing audio with low-resolution faces. Nevertheless, previous studies only consider the appearance information within speech and attempt to generate static faces from voice.

Talking Face Generation. Driving a face to talk with either a clip of audio [Chen et al., 2019; Zhu et al., 2020; Zhou et al., 2019; Prajwal et al., 2020] or video [Zakharov et al., 2019; Burkov et al., 2020; Kim et al., 2018b] has long been the topic of interest for both the fields of computer graphics and artificial intelligence. Earlier methods focus more on modeling a specific person [Suwajanakorn et al., 2017; Kim et al., 2018b]. Recent studies propose to achieve arbitrary subject or one-shot talking face generation driven by audios or videos. A number of studies borrow intermediate representations such as landmarks [Zakharov et al., 2019; Chen et al., 2019] into this task. But this is not convenient in our settings, as voice-reconstructed faces have no fixed landmarks. Certain landmark-free methods [Zhou et al., 2019; Prajwal et al., 2020] rely on skip-connections thus require an image as network input. Specifically, [Burkov et al., 2020] disentangles facial identities and poses in a style-based framework. However, they require a meta-learning stage on ground truth videos. Our work is inspired by theirs and extended to the scope of speech.

Face-Voice Association Learning. The studies on face-voice association learning [Nagrani et al., 2018b; Nagrani et al., 2018a; Kim et al., 2018a; Horiguchi et al., 2018] aim to reveal the relationship between facial appearance and voice by cross-modal matching. [Nagrani et al., 2018b] introduce both binary and multi-way cross-modal face and voice matching tasks through classification. In their following work [Nagrani et al., 2018a], they propose to learn common cross-modal embeddings for person identity under self-supervised framework through contrastive loss. Metric learning [Kim et al., 2018a] and multi-task learning [Wen et al., 2019a] are adopted to learn a shared representation for different modalities. In our work, we also leverage a similar contrastive learning idea for our identity space embedding. We show that our synchronized speech-identity representation is also beneficial for the task of face-voice association learning.

3 Our Approach

We present Speech2Talking-Face (S2TF), a unified framework that achieves face inferring and driving simultaneously from audio. The pipeline of our method is illustrated in Fig. 1. Below we first introduce the overview of our framework (Sec. 3.1), then we explain the synchronization between audio and visual representations in two latent spaces (Sec. 3.2). Finally, we show the procedures of our visually assisted curriculum learning for visual generation (Sec. 3.3).

3.1 Framework Overview

Normally the problem of face reconstruction from voice is formulated in a way of conditional image generation. Given a clip of speech A (transformed to spectrogram S), the ideal result would be an image I ′ with the same identity (appearance) as the source person. In our setting, rather than generating an image, we propose to generate a clip of video V ′ = {I ′1 . . . , I ′K } which should be close to the video V = {I1, . . . , IK} which produces the speech A. The generated frames should not only have the visual appearance of the source identity but also have accurate lip motion synced with audio.

However, it is difficult to build supervision for both appearance and lip motion at the same time. To this end, we adopt a similar framework as [Burkov et al., 2020] with a style-based generator [Karras et al., 2020]. The idea is to explicitly encode frame Ik to an identity space and a frame I + i with data augmentation to an identity-irrelevant space. The style-based generator is then leveraged to reconstruct the original frame Ii. The data augmentation aims at preserving the pose and expression information while changing the texture and shape which are highly correlated with identity. The style-based generator enables separate control of both the information required through the training of frame reconstruction.

Based on this framework, our goal is to encode the speech information to both the identity and identity-irrelevant latent spaces, where they can be mapped to the image domain by the generator.

3.2 Audio-Visual Representation Synchronization

The solution is to synchronize the audio-visual representation within two spaces from the same source with contrastive learning [Chung and Zisserman, 2016; Zhou et al., 2019].

Identity Space Learning. The learning of the identity space is of more importance given that the question of recovering appearances from voices is still not well-solved. We name the visual-to-identity encoder Evid , which encodes feature f v id from the image Ik. Then we leverage a speech encoder Eaid , for encoding the audio identity feature f a id from speech S.

We first build positive and negative pairs from different talking face videos and their corresponding speeches. As the speech embedding f a id should be close to the image embedding f v id , image embeddings from other videos f vid (i) can be served as negative samples that our speech embedding should repel. We use the term cos θ(f1,f2) = f T 1 ∗f2 |f1|·|f2| to define the cosine distance between two features f1 and f2. Then the contrastive loss for the identity information synchronization can be written as:

Where M is the number of negative samples in one batch.

Additionally, as our model is trained on the VoxCeleb2 [Chung et al., 2018] dataset, identity labels are used for more compact identity representation learning. Normally a fully connected (fc) layer with weight W would be leveraged. Then a softmax cross-entropy loss is served for classification.

It has been verified that after the convergence of a classification task, the weight Wj for each class j actually lies in the centroid of all feature embeddings from identity j. As a result, we share the W for both audio and visual so that given a same identity, the features from both modalities will be encoded around a same centroid. Further more, we would expect the encoded features to be more distinctive, thus the angular-based loss function ArcFace [Deng et al., 2019] is leveraged for any fid from both audio and visual modality. The idea is to bring closer the features within one class and their centroid and repel other centroids. The loss function can be written as:

where s is a scale factor and m is a margin. The loss form is quite similar to the contrastive loss in Eq. 1.

Speech Content Subspace Learning. The identityirrelevant space is encoded by encoder Evir in the visual domain through data augmentation, where an image I + i would be encoded to feature f vi ir . Thus the audio spectrogram S is divided to S = {s1, . . . , sK} to match the frame numbers with a sliding window. Normally, we would expect the speech features f ai ir encoded from speech encoder Eas to be the same as the visual features f vi ir . However, the identity-irrelevant space contains information such as head pose, which cannot be inferred from speech. Instead of synchronizing f a s with f v ir , we focus more on learning the dynamic changes between frames, which includes mainly speech content information.

Specifically, we select the input frame to the identity spaces Ik as the first frame I1, and send it also to Evir to obtain the feature f v1 ir . Then we compute the changes of the rest of the frames as ∆fvir = {f v2 ir -f v1 ir , . . . , f vK ir -f v1 ir }. The encoded speech features fas = {f a2 s , . . . , f aK s } are thus synchronized with ∆fvir through contrastive loss:

The formulation of the negative pair is the same as [Chung and Zisserman, 2016].

After the learning procedure, we obtain a speech feature f a s which lies also in the identity-irrelevant space. However, as only content information can be preserved in the speech feature, we identify that these speech features actually formulate a speech content subspace within the identity-irrelevant space.

3.3 Training Paradigm

We adopt a curriculum training paradigm to enable convergence and information balancing within this framework. The idea is to leverage network training in the visual domain as a probe to guide the learning of audio information.

Visual Domain Training. Having obtained the identity feature f v id and identity-irrelevant feature f v ir from the visual embedding, the style-based generator [Karras et al., 2020] G targets to decode the concatenated feature f v = cat(f v id , f v ir ) to a face image I v′ . Detailedly, the feature is sent throughMLPs and served to modulate the weights of the generator's convolution kernels. Then the weights are normalized through the demodulation operation. Such design has been proven to be effective in general image generation tasks. The reconstructed image can be written as I v′ = G(f v ). At this stage, the identity encoder is supervised by the angular loss written in Eq. 2.

The training of the generator relies on the image reconstruction loss Lrec and adversarial loss Ladv. For image reconstruction, we employ L1 loss in RGB space, and the perceptual loss which constrains high-level features. For any generated image I ′ :

Where φl denotes the lth layer of a pretrained VGG19 [Simonyan and Zisserman, 2015] network. For adversarial loss, we apply a multi-scale discriminator [Wang et al., 2018] D with ND layers. The adversarial loss can be written as

Audio to Visual Synchronization Training. At the second stage, we attempt to synchronize audio features to visual features. Thus the visual identity encoder Evid and visual identity-irrelevant encoder Evir are fixed.

The audio identity encoder Eaid is trained on the shared W with Evid . It is also supervised by the angular loss (Eq. 2), and the contrastive loss (Eq. 1). While the audio speech content encoder Eas is trained under the supervision of Eq. 3. During this stage, we also adopt the curriculum training strategy that first selects videos of different identities as negative samples, then moving to the videos of the sample identity but speaking different contents.

Final Finetuning. At last we gradually finetune all models. Particularly, a feature of f va = cat(f v id , f a s + f v1 ir ) is firstly formed and finetuned with the generator G to reconstruct I va′ = G(f va ) , where f v1 ir provides the initial pose information. This is for the generator to adapt to the speech content information while keeping the identity unchanged. Specifically, if both identity and content are sourced from speech, the reconstruction loss (Eq. 5) would emphasize only on the appearance of the face but neglect the mouth shapes.

Finally the audio features are combined together as f a = cat(f a id , f a s +f v1 ir ) to generate I a′ = G(f a ), and fintuned with all models. This procedure is also crucial given that the strong expressive ability of style-based generator would implicitly balance the information between the two features. The final cost function can be written as:

where the λs weight different loss terms.

Quality

-

Please be noted that during testing, the feature f a for reconstruction can be derived in two ways. The first is to encode an arbitrary f ˆ v ir from any face image ˆ I to form f a = cat(f a id , f ˆ v ir + f a s ). In this way, our generated results will preserve the initial pose of the image ˆ I. On the other hand, we could simply omit this pose guidance feature and direct leverage f a = cat(f a id , f a s ). Frontalized faces can thus be generated under such scenario.

4 Experiments

4.1 Experimental Settings

Dataset. We use the popular in-the-wild dataset VoxCeleb2 [Chung et al., 2018] in our experiments. It contains a total of 6,112 celebrities. 5994 speakers are selected for training and 118 speakers are selected for testing. Images are extremely diverse in this dataset, containing different light conditions, large head poses, and varied video quality.

Implementation Details. We adopt the encoder from [Zhou et al., 2019] as Evir . The structure of visual identity encoder Ev id is ResNeXt50 [Xie et al., 2017] and the audio encoders are ResNetSE34. Our generator G consists of 6 blocks of style modulated convolutions. The length of identity features f a id and f v id are set as 2048 while identity-irrelevant features f v ir and f a s are set as 512.

We conduct our experiments using PyTorch deep learning framework with eight 16 GB Tesla V100 GPUs. Images are cropped to 224×224. The audio inputs are mel-spectrograms processed with FFT window size 1280, hop length 160 with 80 Mel filter-banks. A clip of human voice lasting 3.2 seconds is used for our speech-to-identity mapping. During testing, We retrieve an arbitrary image from other identities as the pose source for our generated identity. All λs in loss functions are empirically set to 1.

4.2 Voice-to-Face Mapping Evaluation

As no previous study tackles the problem of generating the appearance and lip motion from speech in the same model, we focus more on the evaluations of voice-to-face mapping, which has not been thoroughly studied before.

We compare our model with state-of-the-art method voice2face [Wen et al., 2019b]. The generated image resolution in their official release is 64 × 64. For a fair comparison, we also re-implement their method to produce 128×128 facial images on the same dataset as ours. We also reimplement Speech2Face [Oh et al., 2019] as described in their paper. The test set of VoxCeleb2 is leveraged in our evaluation, where all identities are not seen during training. Qualitative Results on Voice-to-Face Generation. As illustrated in Fig. 2, images generated by Voice2Face and V2F official contain unstable backgrounds and unnatural artifacts. While results from Speech2Face are blurry with indistinct facial appearances, the quality of our generated faces is comparable to real reference facial images. Moreover, we can observe that our faces synthesized from different voices are ageand gender-matched with the speaker. This proves that our model can capture more underlying relationships between vocal and physical properties. Furthermore, with the help of the pose reference feature, our generated faces have more diverse poses, which expands our application in real-world scenarios. Please refer to https://hangz-nju-cuhk.github.io/projects/ S2TF for our video code and models.

Quantitative Results on Voice-to-Face Generation. Similar to previous work [Oh et al., 2019], metrics on similarity, retrieval performance and image quality are adopted for evaluation. The similarity is measured by the cosine and L1 distances between the synthesized face and its corresponding ground truth embedded by a pre-trained FaceNet [Schroff et al., 2015]. Retrieval performance is validated by querying the dataset with a produced image, targeting to retrieve the ground truth speaker image. We calculate R@K, indicating the ratio of successful retrievals in the top-K similar faces. Image quality is validated by VGGFace score (VFS), which is proposed to replace the Inception Score in face settings. We also conduct the ablation study on whether our centroid learning procedure is effective, thus our model without it (S2TF w/o CL) is also evaluated on these metrics.

The results are listed in Table 1. Our model achieves higher similarity scores, and higher retrieval ratios than other methods, suggesting that our model generates more similar facial images with ground truth. Some examples of the top-5 retrieval results are visualized in Fig. 3. It can be seen that our model could retrieve images with similar facial attributes as real images, verifying that our identity encoder successfully captures the audio-visual association. Additionally, our model achieves the highest VFS among all methods, which we attribute to the expressiveness of style-modulation.

Voice-Face Association Matching. We conduct an experiment on voice-face matching with PIN [Nagrani et al., 2018a] which is specifically designed for this task. We use standard metrics, area under the ROC curve (AUC), in speaker verification following their work. The model is evaluated by sampling negative test pairs while holding constant each of the following demographic criteria: gender (G), nationality (N), and age (A).

The results are listed in Table 2. Though our method is not designed for cross-modal matching, our results are comparable to theirs. Particularly, we outperform their results on gender and the average of the three aspects.

4.3 Additional Evaluation

Experiments on Talking Face Generation. As we can directly generate dynamic talking faces from audio, we also show the lip sync results. Please be noted that our goal can also be achieved by combining voice-to-face reconstruction and audio-driven talking face generation. However, such a pipeline is too complex and non-flexible. Here we send our generated one frame to the talking face model of ATVG [Chen et al., 2019] and Wav2Lip [Prajwal et al., 2020]. It can be seen from Fig. 4 that while ATVG has lower qualities, the lip motion of Wav2Lip is not as accurate as ours. In Table 3 shows the LMD landmark distance, and the confidence score from SyncNet [Chung and Zisserman, 2016].

It can be seen that we also achieve comparable results as the methods specifically designed for this area. Particularly, if we change f a id to any f v id of an existing frame, we can directly generate talking face videos with this framework.

More Ablation Studies. Apart from the evaluation of S2TF without centroid learning above, we have also experimented on other types of contrastive losses in Eq. 1 such as the traditional L2 form or cosine form. Their influences on the final results are subtle. However, using the softmax form (Info NCE) stables the training process. The model cannot render reasonable results if there is no contrastive loss.

As for the curriculum paradigm, the visual domain pretraining is essential for the network to balance the identity and identity-irrelevant information. Intuitively, reconstructing a face from visual features is definitely easier than from audio features.

User Study. We conduct a user study to further evaluate all methods. We randomly select 40 generated images from different identities in the test split of VoxCeleb2 [Chung et al., 2018]. A total of 18 participants are asked to rate the image quality and similarity between the synthesized image and its reference with the widely used mean Opinion Scores (MOS) rating protocol (from 1 to 5). The results are summarized in Table 4. We can see that our model achieves the highest MOS on both metrics. Particularly, our approach outperforms Speech2Face [Oh et al., 2019] and Voice2Face [Wen et al., 2019b] on image quality by a large margin.

5 Conclusion

In this paper, we propose Speech2Talking-Face (S2TF) framework, which aims at inferring and driving a face from speech in a unified framework. We emphasize the properties of our method: 1) Through our explicit design of two synchronized spaces, we successfully generate high-quality facial appearances with accurate lip sync from speech only. 2) Our synchronization in the identity space learns a more compact identity association between speech and human faces. 3) As a unified framework, our model has the capacity to go beyond its designed purpose. For example, the task of audiodriven talking face generation can be directly handled.

Acknowledgements

This research was conducted in collaboration with SenseTime. It is supported in part by the General Research Fund through the Research Grants Council of Hong Kong under Grants (Nos. 14202217,14203118, 14208619), in part by Research Impact Fund Grant No.R5001-18, and in part by NTU NAP and A*STAR through the Industry Alignment Fund - Industry Collaboration Projects Grant.