A Sketch-Transformer Network for Face Photo-Sketch Synthesis

1 Introduction

Generating a face sketch (photo) from a face photo (sketch), often referred as face photo-sketch synthesis, is an important task in computer vision. It has many applications in digital entertainment, animation production and law enforcement [Wang et al., 2014; Li et al., 2016]. The core challenge of face photo-sketch synthesis lies in synthesizing visually realistic and semantically plausible images and surpassing the considerable discrepancies (shape, texture and color) barrier.

Early studies [Liu et al., 2005; Liang Chang et al., 2010; Zhu et al., 2017b] attempt to solve the problem in an exemplar-based manner, i.e. matching and combining sample images (image patches) in a reference set of photo-sketch pairs to synthesize the target image. These approaches work well under constrained conditions such as less illumination variations, pose changes, and deformations, but will fail when come across more complicate conditions. Moreover, two main flaws often limit their performance: 1) blurry or over smooth, i.e not realistic; 2) time-consuming. Rapid progress in deep convolutional neural networks (CNN), especially in generative adversarial networks (GAN) [Goodfellow et al., 2014], has inspired recent studies [Wang et al., 2018; Yu et al., 2020; Chen et al., 2018] to formulate face photo sketch synthesis as a image-to-image translation [Isola et al., 2017; Zhu et al., 2017a] problem. With the assistance of the adversarial loss, these approaches have capacity to generate images with realistic textures.

∗Corresponding Author: Nannan Wang

Although promising results have been obtained, the intrinsic shortage of convolutional architectures that lacks of the ability of capturing long-range spatial dependency has limited the performance of existing approaches, which may results in the loss of global structure information and thus generating images with compromised visual quality. As shown in Figure 1, the results of a state-of-the-art (SOTA) method [Yu et al., 2020] have undesirable artifacts and distorted structures. Recently, transformer models [Vaswani et al., 2017] which mainly based on self-attention mechanism have demonstrated exemplary performance on natural language processing (NLP) tasks and intrigued the vision community to investigate their application to computer vision problems [Dosovitskiy et al., 2020]. Inspired by the power of transformer in NLP and many computer vision tasks, we investigate its application in face photo-sketch synthesis task in this work. However, there are three factors that limit the application of existing transformer models in this task: 1) The training samples are limited so that the model should not be too large; 2) The resolution of the image is relatively large so that the self-attention module consumes lots of computing resources; 3) The self-attention module is unable to capture positional information of the tokens in an image.

To address these problems, we propose a SketchTransformer which can properly introduce the self-attention mechanism into the face photo-sketch synthesis task. Specifically, three closely-related modules are proposed. First, we propose a multi-scale feature and position encoder (MFPEncoder) which integrates convolutional architectures and a face parsing model to extract multi-scale feature embeddings and positional encodings in each local area. Second, we stack several residual self-attention layers in the bottleneck to capture the long-range spatial dependency between the tokens (local embeddings). Finally, we propose a multi-scale spatially-adaptive de-normalization decoder (MSPADE-Decoder) which takes as input the output of the self-attention module, multi-scale feature embeddings and positional encodings generated by the multi-scale feature and position encoder to reconstruct the target image. The overall design enables our Sketch-Transformer to capture long-range spatial dependency while generate reasonable detail texture and therefore achieve a better visual result compared with state-of-the-art approaches (as shown in Figure 1).

The contributions of this work are summarized as follows:

• We propose to learn the key elements of the transformer architecture and adapt them to face photo-sketch synthesis task.
• We propose a Sketch-Transformer with three closelyrelated modules to properly introduce the self-attention mechanism. The proposed model can capture longrange spatial dependency while generate reasonable de-

tail texture.

• Quantitative and qualitative experiments demonstrate that the proposed model achieves superior performance compared with other state-of-the-art methods on public benchmarks and face images in real scenarios.

2 RelatedWork

In this section, we review previous studies of face photosketch synthesis and transformer which are the most relevant to our work.

2.1 Face Photo-Sketch Synthesis

Existing works for face photo-sketch synthesis can be mainly divided into two categories. Exemplar-based methods reconstruct target image by mining correspondences between input image (image patch) and images (image patches) in a reference set of photo-sketch pairs. Deep learning-based methods attempt to predict the target image pixels from the source image pixels through an end-to-end convolutional neural networks.

Exemplar-based methods can be further grouped into three types: subspace learning-based approaches [Liu et al., 2005], sparse representation-based approaches [Liang Chang et al., 2010], and Bayesian inference-based approaches [Zhu et al., 2017b]. A detailed overview of existing exemplar-based methods can be found in [Wang et al., 2014].

Recently, CNN-based and GAN-based approaches have emerged as a promising paradigm for face photo-sketch synthesis. Initial effort [Zhang et al., 2015] trains an end-to-end fully convolutional neural networks (FCN) for directly modeling the nonlinear mapping between face photos and face sketches. Limited by shallow layers and pixel-level loss, however, it fails to capture texture details and fails to preserve reasonable structures. Isola et al. [2017] use conditional GAN (cGAN) as a unified solution (pix2pix) for several image-to-image translation tasks such as edges to photos, labels to street scenes, day to night, etc. Zhu et al. [2017a] propose a CycleGAN model for unpaired image-to-image translation by introducing a cycle consistency loss. These two models can be directly applied to face photo-sketch synthesis task. Several works follow ideas from image-to-image translation and focus on improving face photo-sketch synthesis performance by adding prior information. Wang et al. [2018] propose a multi-scale discriminator to provide adversarial supervision on different image resolution. SCAGAN [Yu et al., 2020] introduces facial composition information as additional input to help the generation of sketch portraits and proposes a compositional loss based on facial composition information. To tackle the problem of insufficient paired training data, Chen et al. [2018] propose a semi-supervised learning method to augment paired training samples by synthesizing pseudo sketch features of additional training photos and learn the mapping function between them. Although great progress has been made by above approaches, undesirable artifacts and distorted structures, however, are still exists, especially in the results of real scenarios.

2.2 Transformer and Self-attention

Transformer is firstly applied on natural language processing (NLP) tasks, which mainly leverages self-attention mechanism to capture long-range dependencies in the input domain. The seminal work of Vaswani et al. [2017] proposes to use solely attention mechanisms for machine translation. Since then, transformer architecture has opened up a new route. Lots of popular methods have been proposed and have achieved the state-of-the-art performance in different NLP tasks. The breakthroughs achieved by transformer in NLP domain have attracted lots of interest in the computer vision community. Many studies have successfully adapted transformer models to varies computer vision tasks including image recognition, object detection, image super-resolution and several other tasks. A comprehensive overview of the vision transformer literature has been introduced by Han et al. [2020].

3 Method

Given paired training face photo-sketch samples {(xi, yi) ∈ (X, Y )}Ni=1 , our goal is to learn a mapping function G that maps images from photo domain X to sketch domain Y or learn a mapping function F that maps images from sketch domain Y to photo domain X. The pipeline of the proposed Sketch-Transformer is shown in Figure 2. It consists of three closely-related modules, including a multiscale feature and position encoder (MFP-Encoder) for patchlevel feature and position embedding, a residual self-attention module for capturing long-range spatial dependency, and a multi-scale spatially-adaptive de-normalization decoder (MSPADE-Decoder) for image reconstruction.

3.1 MFP-Encoder

The MFP-Encoder integrates convolutional architectures and a face parsing model to extract multi-scale feature embeddings and positional encodings in each local area. It consists of two paths: a feature embedding path and a position embedding path, as shown in Figure 3.

The feature embedding path utilizes a series of convolution layers (a stride-1 convolution layer and four stride-2 convolution layers) to gradually extract multi-scale features. Therefore, the feature vector of each position in the last activation (FP 5 ) represents the high-level features of a 16×16 patch in the corresponding local area of the input image. The position embedding path utilizes a face parsing model [Yu et al., 2018] to extract semantic facial labels and scale them to different spatial resolution. Denote semantic facial labels of each layer as M l , M l ∈ ℜcl×hl×wl , where cl, hl, wl denote component number, height and width of the semantic labels of the lth feature layer. Each value (0 or 1) in M l denotes whether the position belongs to the c-th component. Such semantic facial labels actually contain sufficient positional information and can indicate which semantic component the feature embeddings of each position belongs to. We concatenate the feature embeddings and position embeddings at different level to obtain the multi-scale feature and position embeddings. Then, the first four feature and position embeddings are passed to the MSPADE-Decoder as spatial information to help supplement spatial and texture information and the last one is passed to a residual self-attention model to learn long-range dependencies between the embeddings (tokens) from all positions.

3.2 Residual Self-attentionModule

Self-attention is the core component of the transformer architecture, which can capture long-range dependency between tokens. From the MFP-Encoder, we obtain the patchlevel feature and position embeddings of all positions. However, the relationships between these embeddings are neglect. Therefore, we introduce a residual self-attention module to capture their dependencies. The module consists of nine basic residual self-attention layers. The illustration of each layer is shown in Figure 4.

The intuition behind this module is to update each vector at each position of the embeddings by aggregating global information from all other positions. Through this module, we can get the revised embeddings ˆ FP 5 which have learned the long-range dependencies.

3.3 MSPADE-Decoder

We utilize the spatially-adaptive de-normalization (SPADE) block [Park et al., 2019] on multi-scale feature and position embeddings to gradually reconstruct the target image. More specifically, we utilize positional normalization (PN) [Li et al., 2019] instead of batch normalization (BN) to better preserving the structure information synthesized in prior layers. The illustration of the MSPADE-Decoder is shown in Figure 4.

Q

K

V

1x1 conv

1x1 conv

1x1 conv

𝑥𝑥

Figure 4: The illustration of the residual self-attention mudule.

3.4 Loss Function

The full loss of our model consists of two loss functions: adversarial loss and perceptual loss. For the sake of brevity, we only describe the losses for photo to sketch synthesis task. The losses for sketch to photo synthesis has the same form. For convenience of expression, we denote the SketchTransformer as G. In order to provide the adversarial loss, we utilize a 70 × 70 PatchGAN discriminator [Isola et al., 2017], which is denoted as D.

Adversarial Loss

In this work, instead of using the vanilla GAN[Goodfellow et al., 2014], we use the Least Squares GAN [Mao et al., 2017] for stable training. For the mapping function G and its discriminator D, we express the objective as:

Perceptual Loss

To ensure that the generated image and its ground truth are similar in semantic feature level, we introduce the perceptual loss [Johnson et al., 2016]:

where φj indicates feature maps of the jth layer of a pretrained VGG-19 model [Simonyan and Zisserman, 2014], Cj , Hj andWj indicate channel numbers, height and width of the feature maps, respectively.

Full Loss

By combining above losses, we can achieve our full loss:

In this work, we set λ1 = 1, λ2 = 5 to keep corresponding losses in the same order of magnitude.

4 Experiments

In this section, we first discuss the experimental settings. We will then conduct ablation study to quantify the contribution of different configurations to overall effectiveness. Finally, we will compare our results with state-of-the-art methods both qualitatively and quantitatively.

4.1 Implement Details

All models are trained on a NVIDIA Tesla V100 GPU using Adam optimizer with β1 = 0.5 and β2 = 0.99. We train all models with a fixed learning rate of 0.0002 until 300,000 iterations. The batch size is set to 1 for all experiments. Weights were initialized from a Gaussian distribution with mean 0 and standard deviation 0.02. We scaled the size of the input images to 256×256 and normalized the pixel value to the interval [-1, 1] before putting them into the model. During training, we updated G and D alternatively at every iteration.

4.2 Database

The experiments are conducted on two public databases: the CUFS database [Tang and Wang, 2009] and the CUFSF dataset [Zhang et al., 2011b]. The CUFS database consists of 188 identities from the Chinese University of Hong Kong (CUHK) student database [Tang and Wang, 2003], 123 identities from the AR database [Martinez and Benavente, 1998], and 295 identities from the XM2VTS database [Messer et al., 1999]. Each identity has a photo-sketch pair under normal light condition, and with a neutral expression. The CUFSF database has 1,194 identities from the FERET database [Phillips et al., 2000]. For each identity, there is a photo with illumination variation and a sketch with exaggerated structure. Therefore, face photo-sketch synthesis on the CUFSF database is more challenging than on the CUFS

dataset. All images are processed by aligning the center of two eyes to the fixed position and cropping to the size of 200×250. The way we divide the training set and the test set is the same as [Zhu et al., 2017b]. For the CUFS database, 88 face photo-sketch pairs in CUHK database, 80 face photosketch pairs in AR database and 100 face photo-sketch pairs in XM2VTS database are selected for training and the rest are used for testing. For the CUFSF database, 250 face photosketch pairs are selected for training and the rest are used for testing. Table 1 shows the partition settings of the databases.

4.3 Baselines

For fair comparison, we run face photo-sketch synthesis on our method and all baselines for input images of size 200 × 250 under the same partition setting. We compare our method with seven state-of-the-art methods: DGFL [Zhu et al., 2017b], FCN [Zhang et al., 2015], pix2pix [Isola et al., 2017], CycleGAN [Zhu et al., 2017a], PS2MAN [Wang et al., 2018], Wild [Chen et al., 2018] and SCAGAN [Yu et al., 2020]. Among these baselines, DGFL is traditional exemplarbased method which achieves the best performance while the others are deep learning-based methods. All results are obtained from the source codes provided by the authors except the results of FCN. We implement FCN by ourselves and get the results which are consistent with the original work. Because FCN, DGFL and Wild methods are designed for face photo → sketch synthesis task, we only compare with their

synthetic face sketches. Other methods have both synthetic face photos and face sketches that used for comparison.

4.4 EvaluationMetrics

In this paper, we use three types of evaluation metrics to evaluate the objective quality of the synthetic images: the learned perceptual image patch similarity (LPIPS) [Zhang et al., 2018], the Fr´echet Inception Distance (FID) [Heusel et al., 2017] and the feature similarity index (FSIM) [Zhang et al., 2011a]. The LPIPS takes two images (image patches) as the input, calculates the L2 distance between their normalized deep feature embeddings, and predicts the perceptual judgment score through the linear layer. A lower score indicates better quality of synthetic images. FID is designed to capture the Fr´echet difference of two Gaussians (synthetic and realworld images). We compute the FID score between the synthetic images and real ones. Lower FID score indicates better quality of synthetic images. FSIM is a commonly used metric for full-reference image quality assessment, which captures the similarity between low-level features of images. It shows higher consistency with human visual perception. We calculated the average FSIM score between synthetic images and real ones. A higher FSIM score indicates better quality of synthetic images.

4.5 Ablation Study

We compute the LPIPS (alex) score between the synthetic images and real ones on CUHK database under different configurations to quantify the contribution of different configurations to overall effectiveness. The ablation study is conducted on four configurations: (a) U-net [Ronneberger et al., 2015]

architecture; (b) Using MSPADE-Decoder to replace the origin decoder in U-net; (c) Adding residual self-attention module on the basis of (b); (d) Adding position embeddings on the basis of (c) (i.e. Full Sketch-Transformer). The evaluation scores are shown in Table 2, from which we can conclude that all the modifications are critical to the final effectiveness of the proposed method.

4.6 Comparison with Baselines

Figure 6 presents some synthetic face photos from different methods on the CUFS dataset and the CUFSF dataset. The results of pix2pix and CycleGAN have sharp edges but possess obvious artifacts and noise. PS2MAN produces less artifacts but its results are blurry. Face photos synthesized by the SCAGAN have reasonable texture and less artifacts, but still possess some structure distortions. As shown in the figure, synthetic photos of the proposed method retain considerable structural information and achieve the most reasonable texture distribution, and therefore has the best visual performance.

Some synthetic face sketches from different methods on the CUFS dataset and the CUFSF dataset are shown in Figure 7. The results of DGFL and FCN are too blurry. GANbased methods (pix2pix, CycleGAN, PS2MAN and SCAGAN) can generate sketch-like textures. However, some undesirable textures are produced in eye and hair areas. Wild Test Photo DGFL

FCN

pix2pix CycleGAN PS2MAN

Wild SCAGAN Ours Ground Truth

Figure 8: Examples of synthetic face sketches on face photos in the wild.

has stronger robustness to the environment noises but tends to generate over smooth results, and the texture distribution of its synthetic sketches is inconsistent with that of training sketches. The proposed Sketch-Transformer can generate the most sketch-like textures while maintain the global structures.

Figure 8 presents some synthetic face sketches from different methods on face photos with deformations and illumination variations. Results of DGFL are able to preserve desirable structures but lose texture details. Results of pix2pix, PS2MAN tend to lose structural information and mistake the shaded area as the hair area. CycleGAN can preserve considerable structures but its synthetic sketches are more like face photos. Wild has desirable visual performance but the texture distribution of its synthetic sketches is inconsistent with that of training sketches. Our results can preserve enough structural information while generate satisfactory textures.

Table 3 presents the evaluation scores of the synthetic face photos/sketches on the CUFS dataset and the CUFSF dataset. The proposed model obtains the best score in most cases, which indicate that it achieves the best performance.

5 Conclusion

In this paper, we investigate the application potential of transformer architecture (especially the self-attention mechanism) on face photo-sketch synthesis task. For this purpose, we propose a Sketch-Transformer network which consists of three closely-related modules: a MFP-Encoder, a self-attention module, and a MSPADE-Decoder. We compare the proposed models with recent state-of-the-art methods on two public datasets and face images in real scenarios. Both qualitative and quantitative results demonstrate that the proposed method achieves significant improvements in both retaining structural information and generating appropriate textures. In the future, we intend to further investigate the method of applying the self-attention module to multi-scale feature embeddings.

Acknowledgments

This work was supported in part by the National Key Research and Development Program of China under Grant 2018AAA0103202; in part by the National Natural Science Foundation of China under Grants 62036007, 61922066, 61876142, 61772402, and 62050175; in part bt the Natural Science Basic Research Plan in Shaanxi Province of China under Grant 2021JQ-198; in part by the Fundamental Research Funds for the Central Universities under Grant XJS210102.