Scribble-to-Painting Transformation with Multi-Task Generative Adversarial Networks

1 Introduction

Recent advancements in deep neural networks have brought tremendous successes in style transfer, which converts photos into artistic oil paintings [Gatys et al., 2016; Isola et al., 2017; Zhu et al., 2017]. Style transfer is among the first few commercialized deep learning technologies [Tanno et al., 2017]. Users are excited at these applications, which allow them to generate self-portraits and paintings as if they were painted by famous artists. However, these applications require a photo to generate an artistic painting. A more interesting task is to eliminate this requirement, giving users full opportunities to create paintings on scenes that do not exist in the real world.

In this paper, we consider a novel application, which produces artistic paintings based on user-generated scribbles (see Figure 1(a) for an example). In this way, users can create paintings as they like, far beyond the restrictions posed by real-world photos. Scribble-to-painting transformation poses novel challenges, which are not encountered in neural style transfer before. As shown in Figure 1(a), a scribble often only contains strokes that outline the objects in a scene, whose information content is much more sparse than a real-world photo. As a result, a scribble-to-painting neural network has to infer missing details, such as the colors of the surfaces or the geometric shapes of the objects, based on outlines from the scribble. This is different from the case of photo-to-painting style transfer, where the neural network is to summarize, reduce and adapt the information within a realworld photo into an artistic painting. The aforementioned differences are fundamental, precluding existing style transfer approaches such as Pix2pix [Isola et al., 2017] and Scribbler [Sangkloy et al., 2016b] to succeed in our scribble-topainting application.

We use multi-task learning to address the task of scribbleto-painting transformation. We first ask ourselves how human painters work from the basis of a scribble. Human painters first recognize each object based on the scribble, and then start to add details [Vyshedskiy and Dunn, 2015]. For example, in the first example of Figure 1, recognizing the entire object as a bus helps to color its wheels and windows. Similarly, in the second example, a human being has to recognize the rectangular object as a cell-phone before he can color the object properly.

Inspired by this, we build Dual Scribble-to-Painting Network (DSP-Net), which simultaneously solves the problems of scribble-to-painting transformation and object detection and segmentation. The whole architecture of DSP-Net is shown in Figure 2, which composes of two sub-networks. A primary network - the artist network - generates oil paintings based on scribble images, while the secondary network - the semantic network - recognizes and segments objects from the scribble image. The primary and secondary networks share the first few layers for feature extraction. The core idea is that the training of the secondary network helps the shared layers to capture a better semantic representation, therefore assisting the training of the primary network. We demonstrate that joint training on these two tasks brings in additional benefits. Similar ideas can be found in CoupleGAN [Liu and Tuzel, 2016] and S2 -GAN [Wang and Gupta, 2016] but with notable differences. CoupleGAN learns the joint distribution of multi-domain images using two parallel Generative Adversarial Networks (GANs) [Goodfellow et al., 2014] while S2 -GAN cascades two GANs in a series manner. Our secondary segmentation task is supervised directly from annotated results, which is different from the second GANs as in the previous works.

The experimental result shows that DSP-Net outperforms previous models both visually and quantitatively. Visually, as shown in Figure 3, the paintings produced by DSP-Net contains more detailed textures which match better with the styles of the target paintings. Quantitatively, compared to previous models, our model matches better to the target artistic paintings under the metrics of the content-mismatching loss and the style-mismatching loss based on a pre-trained VGG19 network. The experiments on Sketchy database [Sangkloy et al., 2016a] prove that our approach generalizes well to real human-drawn scribbles. Moreover, the human evaluation results show that the majority (62.44%) of participants picks our results as the best in terms of details and aesthetics.

We make another contribution by generating and publishing a benchmark dataset for scribble-to-painting transformation. Another problem in building a scribble-to-painting transformer is to obtain large and high-quality datasets. There are some painting datasets for classification such as the Paintings Dataset [Crowley and Zisserman, 2014] and Painting 91 [Khan et al., 2014]. However, they do not come with the corresponding scribbles and are not labeled semantically as well. The number of available oil-paintings online is also far less than the number of photos. Our insight is to rely on large image datasets which were already semantically labeled and the existing approaches for line extraction and neural style transfer. In particular, we start with the COCO dataset [Lin et al., 2014]. We extract lines of the photos to form surrogates for scribble images. Line extraction images resemble real scribbles well. In fact, we showed in the experimental section that DSP-Net trained using line extraction images can be directly applied to real scribbles with satisfactory results. We then use existing neural style transfer algorithms [Johnson et al., 2016] to transform photos in the COCO dataset to oil paintings. We collect a large dataset of more than 5000 pairs of scribble and painting in this way.

2 Scribble to Painting Transformation

Scribble-to-painting transformation is an interesting but challenging task. See Figure 1 for an example, where scribble images in (a) are transformed into stylized paintings in (b). Users can produce beautiful artistic paintings based on simple scribbles using this application, without restricting to realworld scenes captured by a camera.

Mathematically, we use xi ∈ X to denote one scribble image as input. An artistic painting as output is denoted as yi ∈ Y. The training data d = {(x1, y1), ..., (xm, ym)} is drawn from the joint space X ×Y. The objective of scribbleto-painting transformation is to find a mapping h ∈ H from X to Y, which best mimics the mappings in training dataset.

Let L be a loss function that measures the difference between h(x) and the target y. The objective mapping h could be found by minimizing the expected loss:

Scribble-to-painting transformation is related to neural style transfer, where real photos are transferred into artistic paintings. Nonetheless, it is conceivably more challenging. The main challenge comes from the sparse information content of the scribbles. See Figure 1(a). There are many blank areas in the scribbles, which contain no information. Therefore, a neural net has to infer missing details from the scribble, such as the colors of surfaces and the geometric shapes of objects, etc, in scribble-to-painting transformation. It is a process of adding information.

On the contrary, many state-of-art style transfer models work from photos, which are rich in terms of information content. The classical style transfer application is a information reduction task, where the information-rich photos are converted into information-sparse paintings. We believe that this is a fundamental difference, which precludes direct application of successful approaches in style transfer to scribbleto-painting transformation.

3 Dual Scribble-to-Painting Network

As discussed above, the sparsity of information in the scribble images poses a significant challenge in scribble-to-painting transformation. To solve this problem, we get inspired by human cognitive process: how do human painters work from the basis of a scribble? Before human painters start adding details, they first recognize each object from the scribble. The recognition process helps human painters to decide the correct colors, geometry, and textures of the artistic painting.

Inspired by this, we explore multi-task learning for scribble-to-painting transformation. We employ a secondary semantic segmentation task to assist the primary task of scribble-to-painting transformation. In this setting, besides the input scribbles xi's and the output paintings yi's of the primary task, there is an additional output zi, which is a semantic map of detected objects in the scene (See Figure 1(c)). As a result, the dataset consists of a set of triples d = {(x1, y1, z1), ..., (xm, ym, zm)} drawn from an underlying distribution Px,y,z(.) defined over X × Y × Z.

We propose the Dual Scribble-to-Painting Network (DSPNet), where both the primary and the secondary tasks are trained simultaneously (See Figure 2). The primary neural network - the artist network - is a generative adversarial network. It contains a generator G to transform the input scribble image xi to an fake artistic painting G(xi). The goal of the generator is to fake G(xi) so that it is hard for the discriminator D to tell apart from the real painting yi. The discriminator, on the other hand, tries to separate the real paintings from the fake ones. After the generator and discriminator networks reach an equilibrium, the generated paintings will look similar to the real ones.

The secondary neural network - the semantic network - share the generator network G with the first neural net. Its task is to detect and segment objects from the scribble images, therefore assisting the primary artist network to generate paintings with correct semantics. The semantic network contains a sub-network named assistant A, which maps the output of the generator G(xi) into a semantic segmentation map image A(G(xi)), which best mimics the output of true zi. In other words, we would like to minimize the distance between A(G(xi)) and zi with respect to a criterion R, such as the smooth L1 loss function [Girshick, 2015] used in our paper. Combining these two neural networks, the objective of DSP-Net can be formulated as:

Note that both the scribble x and painting y are fed to the discriminator D. This design allows the discriminator to detect abnormal "content switching" behavior of the generator. Suppose the generator G transforms a scribble of a dog into a painting of a cat. The discriminator D is able to discover this fact because the original dog scribble is also fed into D.

The artist and semantic networks cooperate with each other by sharing the parameters of G. G can be treated as a feature extraction network for the semantic task while it is the backbone of the artist network. Figure 2 illustrates the general architecture. When training the primary artist network, D tries to maximize the first two terms and G tries to minimize the second term in Eqn.2 adversarially. When training the secondary semantic network, both the assistant network A and the generator G are optimized cooperatively with respect to the third term in Eqn.2. The training of the two networks is interleaved until convergence.

3.1 Primary Artist Network

The objective of the primary artist network is to synthesize an oil painting given a scribble image. We use an adaptation of the conditional GAN introduced in [Isola et al., 2017; Radford et al., 2015] as our primary network.

The generator and discriminator of the primary network are shown in Figure 1. The generator G : X → Y is a mapping from scribbles to paintings. The discriminator D : Y → {0, 1} is a mapping from artistic paintings to a binary outcome. Generator G receives a scribble image x and try to generate a fake painting G(x) that cannot be distinguished as a real painting by the discriminator. In this paper, we use a variant of the encoder-decoder architecture in [Isola et al., 2017] as the generator, which first encodes the feature of a scribble image into a low-dimensional representation and then decodes it to an artistic painting.

Discriminator D is trained to differentiate whether a given painting is synthesized by generator or it is a real one. The fake painting should be classified as 0 and the real painting should be classified as 1. In this paper, we adapt the Markov discriminator introduced in [Li and Wand, 2016] as our architecture. The min-max objective for both the generator G and the discriminator D in the primary artist network is:

When the competitive training between the generator and the discriminator achieves an equilibrium, the generator will learn to generate relatively realistic paintings.

3.2 Secondary Semantic Segmentation Network

Leveraging multi-task learning, we introduce a secondary semantic segmentation network to detect and segment objects based on scribble images. Both the primary and secondary tasks are trained simultaneously while sharing the parameters of G. This secondary semantic segmentation task is shown using blue arrows in Figure 2.

We apply another encoder-decoder architecture for the secondary task, which is named as an assistant A. Given a scribble image x, the secondary network is trained by minimizing the segmentation loss between the generated semantic segmentation A(G(x)) and the ground truth z:

Here, R is a loss function that penalizes the difference between A(G(x)) and z. In this paper, we use smooth L1 loss proposed in [Girshick, 2015] as our loss function. Different from the case in the primary task where generator and discriminator have competing objectives, the generator G and A share the same objectives in the secondary task. Empirically, the secondary semantic segmentation network also speeds up the training because the semantic segmentation task appears to be easier to train than the primary task. The semantic task can also serve as a regularizer that helps reduce overfitting.

During training, the primary artist network and the secondary semantic segmentation network are optimized in an interleaved manner.

4 Dataset Generation

Another challenge in scribble-to-painting transformation is to obtain a large and high-quality dataset. However, the number of available oil paintings and scribbles are quite limited. It is even harder to find the pairs of scribble and painting for a supervised transformer. What is more, in order to train the semantic segmentation task simultaneously in the multi-task learning setting, the semantic images of corresponding paintings are necessary. However, currently there is no manually labeled semantic segmentation for oil paintings.

In this paper, we build a triple dataset including scribbles, paintings and semantic images based on COCO dataset [Lin et al., 2014]. COCO dataset provides the pairs of the real photo pi and its corresponding semantic image zi. We build scribble xi shown in Figure 3a from pi with a simple but effective Canny edge extracting algorithm [Canny, 1986]. The painting yi shown in Figure 3b is generated with a fast neural style network [Johnson et al., 2016] pre-trained on COCO dataset. yi is obtained by feeding real photos pi into the pretrained network. Since the photos pi contain full information, the quality of synthesized paintings is good enough for them to serve as the training ground truths. Along with the semantic image zi which is manually labeled for photo pi by COCO dataset, the triples (xi, yi, zi) are induced.

5 Experiments

In this section, we are going to evaluate the experimental performances of DSP-Net and several existing models widely used in image transformation, including Neural Style [Gatys et al., 2016], Fast Neural Style [Johnson et al., 2016], Pix2pix [Isola et al., 2017], and CycleGAN [Zhu et al., 2017]. We train the GAN based models for 200 epochs and neural style for 1000 iterations. We keep the other hyper-parameters and basic settings as the recommended value mentioned in their original code.

We use the triples of scribbles, paintings, and semantic images obtained using the methods introduced in Section 4. 5000 images from COCO dataset [Lin et al., 2014] are randomly sampled to build the dataset. We split the dataset into 4500 images for training and 500 images for a test. The oil paintings The Starry Night by Vincent van Gogh and The Scream by Edvard Munch are used as the style images. The code and dataset could be found at https://github.com/ jinningli/DSP-Net.

5.1 Visual Analysis

Figure 3 shows the painting generated by DSP-Net and other well-known models used in style transfer task. Our DSPNet model contains more detailed and real textures which match better with the target style. The first column shows the scribble images xi. Different from photos, these scribbles are greyscale images and we can find that there are many areas left blank. The second column is the ground truth painting yi synthesized from real photo images pi. The following columns are the generated images with style transfer methods and DSP-Net.

A DSP-Net without the secondary task is equivalent to a conditional GAN (Pix2pix model), the result of which is shown in Figure 3c. The Pix2pix model can generate reasonable colors and contents. However, it fails to generate a vivid style and brushwork of oil paintings. Comparing to Pix2pix, DSP-Net in Figure 3g generates more realistic patterns of oil painting and semantic contents. This is because the secondary semantic segmentation task forces the generator to generate varieties of paintings which satisfies the constraints of semantic information.

The sparsity of scribble images causes serious blurring and blank space (See CycleGAN model in Figure 3d and Fast Neural Style model in Figure 3f). Although the CycleGAN model successfully synthesizes parts of the style texture of The Starry Night and The Scream in the areas where enough information is provided, it failed to generate the patterns in sparse areas. Similar problems also happen to Fast Neural Style model. This model can only infer the color and patterns to apply within a limited distance from the drawn scribble lines. So the uncertain area is left blank.

The performance of the popular neural style algorithm in Figure 3e is also quite limited in the scribble-to-painting transformation task. It is hard to figure out the objects in the synthesized images. They look like a recombination of the patterns of The Starry Night or The Scream. The neural style algorithm relies on a pre-trained VGG-19 network to extract the features of both the scribble images and paintings. However, the VGG-19 network is pre-trained to extract the features of photos on ImageNet dataset, so that its ability to recognize the lines in scribble images is limited.

The secondary semantic segmentation network also speeds up the training process of primary networks because this task can be viewed as a sub-problem of painting synthesis which is easier to train. Based on the semantic information provided by this sub-problem, the convergence speed of the primary network will be increased.

5.2 Quantitative Analysis

We also compare our model quantitatively with some stateof-the-art style transfer models. Experimental result proves that our model performs better under the metrics of content mismatching and style mismatching (See Table 1). This means our results are closer to the objective painting images with respect to both content and style similarity.

Inspired by the idea mentioned in the neural style algorithm [Gatys et al., 2016] to use a pre-trained network as feature extractor, we propose two metrics, the content mismatching loss δc and the style mismatching loss δs, to quantitatively evaluate the results of scribble-to-painting transformation task. The mismatchings of content and style are evaluated between the synthesized paintings and real ones. We use a pre-trained very deep convolutional network [Simonyan and Zisserman, 2014] (VGG network) to extract the encoded features. We use M l (x) to denote the encoded feature map of x in layer l.

Assume h is the hypothesis of the model to be evaluated. The content loss is defined as the mean squared error between features of h(x) and y extracted from different layers of VGG network, δc(x, y, h) = 1 2 ∑l ∑i,j [M l i,j (y) - M l i,j (h(x))]2 .

In order to calculate the style mismatching loss, we use Gram matrix [Gatys et al., 2015] to capture texture information of different layers between fake painting h(x) and real painting y. The Gram matrix denoted by El i,j of an image q on the layer l is calculated as the the inner product between the i-th and j-th vectorized feature maps, El i,j (q) = ∑k Mi,k(q)lM l j,k (q). After calculating all the gram matrices of y and h(x) on different layer l, the style mismatching δs can be described as δs(x, y, h) = ∑l 1 4W 2 l H2 l ∑i,j [El i,j (y) - El i,j (h(x))]2 , where Wl and Hl are the width and height of the feature map in the layer l of VGG-19 network.

With two metrics mentioned above, we calculate the average of content loss δc and style loss δs on the test set, which contains 500 test samples. We calculate the average mismatching losses on the test dataset by ¯ δc(h) = 1m ∑m δc(xm, ym, h)) and ¯ δs(h) = 1m ∑m δs(xm, ym, h).

Assume hk is the hypothesis of the k-th model. We apply normalization to both the content and style mismatching losses by ∆c(hk) = ¯ δc(hk)/ ∑k ¯ δc(hk) and ∆s(hk) = ¯ δs(hk)/ ∑k ¯ δs(hk). The results of normalized mismatching losses of different models and the normalized standard deviation are shown in Table 1, the lower the better.

In the aspect of content mismatching, DSP-Net achieves the lowest mismatching loss of 0.175, which outperforms 5% than the rank-2 Pix2pix model. See the visual result in Figure 3, both the results of Pix2pix and DSP-Net are realistic with respect to the content distinctiveness. However, the blurring and texture mismatching limit the performance of Pix2pix. The content mismatching of Fast Neural Style is the largest, which is 0.246, since Fast Neural Style model fails to generate textures in the area where no scribble is provided.

In the aspect of style mismatching, DSP-Net outperforms other models as well. The style loss of DSP-Net is 5.4% lower than the rank-2 CycleGAN model. This result is also reasonable comparing with the visual result in Figure 3. The textures of results of DSP-Net looks the most similar to the ground truth in Figure 3b. The CycleGAN model also contains some style textures. However, it suffers from the problem of sparse input and mode collapse. Pix2pix gets the highest mismatching loss because in Figure 3c, almost the whole image remains blur and almost no texture is synthesized.

5.3 Generated Paintings with Real Scribbles

We use edge extractions as our input because they are easier to obtain in large quantity than human-drawn scribbles. The use of edge extractions to mimic scribbles are seen in multiple publications [Lu et al., 2012; Lim et al., 2013]. Recently,

• (a) Butterfly (b) Synthesized (c) Flower

(d) Synthesized SketchyGAN [Chen and Hays, 2018] has tried to augment the edge extractions with human-drawn sketches in the Sketchy Dataset [Sangkloy et al., 2016a] to improve the model generalization. However, the quality of its synthesized images from sparse sketches is still not satisfactory enough.

Table 2: Human evaluation in terms of content realism and style aesthetics. We report the percentage of votes that each model is chosen as the most realistic and aesthetic.

Even though our approach is trained on images of edge extraction, it generalizes well to human-drawn scribbles. We train DSP-Net model on our dataset derived from COCO introduced in Section 4, and test it on the Sketchy Dataset consisting of real human-drawn sketches. The results are shown in Figure 4. Both the synthesized details of objects and the style patterns are realistic.

5.4 Human Evaluation: Realism and Aesthetics

We complete a survey among students from different majors. We show every participant 6 questions. In each question, we show the participant one scribble image (randomly drawn from the testset) and the corresponding six oil paintings synthesized by five baseline models and our proposed model. The order of synthesized paintings is shuffled so that the participant do not know which painting corresponds to which model. In each question, we let participants vote for the most realistic and artistic painting. Then we count the percentage of votes each model gets. We in total receive 402 votes from 67 participants. Results in Table 2 show that the paintings synthesized by DSP-Net are chosen to be the most realistic and aesthetic in 62.44% out of all questions.

6 Conclusion

In this paper, we focus on a novel task that transforms scribbles into artistic paintings. This task is more challenging than classical image style transfer, because the scribbles contain far less information compared with photos. We use multitask learning to solve this problem. We introduce the Dual Scribble-to-Painting Network (DSP-Net), which consists of two jointly trained neural networks. The primary network is trained to generate artistic images based on scribbles and the secondary network is for semantic segmentation. Experimental results show that DSP-Net outperforms previous models both visually and quantitatively. Experiments on humandrawn scribbles and human participated surveys also demonstrate the effectiveness of DSP-Net. As an additional contribution, we publish a large dataset for scribble-to-painting transformation. Possible future directions include extending the application of the multi-task generative adversarial networks into other important tasks in computer vision and natural language processing.