From Reality to Perception: Genre-Based Neural Image Style Transfer

1 Introduction

While Vincent Van Gogh was looking out of the window of his asylum room at Saint-Rmy-de-Provence in a hot summer night, the glow of the stars and the moon were swirling against the dark blue sky in his vision. Van Gogh depicted his perception that view brought him with vigorous colors, swaying brushstrokes and distorted scenes, which made Starry Night one of the most famous art pieces of the world.

After years since Van Gogh passed away, the world start to explore and identify the unique beauty of his work. People start to study how to recreate images in his style. This problem, now being categorized under image artistic rendering field, has drawn much attention in recent years.

Development in convolutional neural networks for computer vision tasks has enlightened a neural style transfer paradigm shift in this field. Both academia and industry have witnessed the great success that neural style transfer has brought for the field. Typical neural style transfer approaches either train feed-forward networks [Johnson et al., 2016; Zhu et al., 2017a; Chen et al., 2017; Dumoulin et al., 2016], or apply iterative optimization methods [Gatys et al., 2016] to migrate visual attribute, with filter pyramid of VGG network [Simonyan and Zisserman, 2015] as representations for content and style images. These methods yield impressive results, but feature maps at different layers are directly used to capture pixel correlations of the content and the style images, which lacks of structural constraint. In view of this problem, some research [Li and Wand, 2016; Liao et al., 2017] regard this as texture or image synthesis, which enhanced the structure preservation in style migration and generated more credible results.

∗Corresponding author: Nannan Wang (nnwang@xidian.edu.cn)

However, the style of an artist or a genre includes not only color or texture patterns of one single painting, but also the underlying creation ideas which were normally neglected in previous methods. In art production, the way that artists perceive the real world and express their perceptions in the artworks are critical design aspects to distinguish one style from the other. For example, the bright colors are often adopted to express the tension of life in Van Goghs paintings, while the broken up and reassembled geometry patches symbolize cubism artworks. Although current methods are capable of capturing visual appearance of single image, the overall style of the artist or the genre can not get fully expressed in the result.

In this paper, we present a framework for genre-based image style transfer. Driven by the well-known assumption that artists express their feelings of the real world in artworks, we introduce perception guidance in genre style transfer. We collect 309 pairs of Van Goghs painting and the corresponding photo that have similar scenery, some of them were shot from the site he used to live. We also collect cubist artworks and photos with similar semantic meanings to demonstrate the generalization of our method. We train an encoderdecoder network with separate genre paintings. The output of encoder forms an embedding representation space for each genre, from which the perception-guided style representation is reconstructed. Then the style representation is transformed into neural artwork by the decoder.

The reconstruction is based on the reality-perception correspondence between semantically-related photo and painting. Solving the correspondence problem is equivalent to matching the heterogeneous domains, in our case is the photo domain and artwork domain. Recently, deep convolutional neural networks [Simonyan and Zisserman, 2015] trained for discriminative tasks have brought up striking success. An excellent property of deep CNNs is that they encode image information into hierarchical representations, with spatial consistency between adjacent layers. This property facilitates a coarse-to-fine Nearest-Neighbor Field (NNF) computing, an essential link in establishing the correspondence from the reality to perception. Our method adopts pyramid of feature maps from pre-trained CNNs for image classification as representations for both domains. We use correlation coefficient of the deep representations as similarity metric. Heterogeneous images such as photos and oil paintings, have different visual appearances even when they are semantically alike. Using correlation coefficient could eliminate the affect of the amplitude change brought by different visual appearances, and in the meanwhile reflect the semantic similarity over the variance of deep representations in local region.

The major contributions of this work are:

1. We present a novel framework for incorporating realityperception correspondence into neural image style transfer, which we show to be effective for generating artworks with distinctive genre characters.
2. We extend PatchMatch with a new similarity metric in deep feature space for heterogeneous domain matching, which proved to provide better semantic discriminability.

Neural artworks could be more than just migrating color distributions or texture patterns from one style image. It can also express the underlying design ideas as we model after real human artists by integrating their perceptions into style transfer process. We show that our method can generate artworks with distinct genre characters through extensive experiment results. We believe that the proposed framework could provide new insights for considering peoples role in image style transfer problems.

2 RelatedWork

Artistic Image Style Transfer. The target of image artistic rendering [Semmo et al., 2017] is to simulate the appearance of reference image while maintaining the content of the source image. Traditional methods [Hertzmann et al., 2001; Semmo et al., 2015] employs machine learning or statistical models to emulate the visual appearance from example pairs. However, these methods only perform low-level texture transfer, which limited its control over the design aspects of artwork creation.

Representing the style and content images had always been technical limitations in image artistic rendering problems before, but recent advancements in deep learning and convolutional neural networks (CNNs) has enlightened a paradigm shift in this field. The first attempt to generate neural artwork may be DeepDream [Mordvintsev et al., 2015]. Inspired by their work, Gayts et al. [2016] produce impressive results for transferring styles of famous paintings onto images. Their method applies different layers of pre-trained VGG network as filters to extract content and style representations. Then, the output is stylized based on pixel correlations between the representation of the style image and the content image. This idea has been further developed to painting style transfer on head portraits [Selim et al., 2016] and style image synthesis [Li and Wand, 2016]. Semantic segmentation were introduced in CNNs to turn two-bit doodles into fine art [Champandard, 2016].

Aside from the aforementioned iterative optimization approaches, more and more work start to train feed-forward networks for faster or even real-time stylization. Zhu et al. [2017a] introduced cycle-consistent adversarial networks for unpaired images and applied their method on style transfer. Johnson et al [2016] designed a transform net which is trained with custom perceptual loss functions for real-time image style transfer. Chen et al [2017] and Dumoulin et al. [2016] designed network architectures to learn and capture the representation of artistic style, and further enabled fusion of different styles.

However, among all of these methods, the style they transferred from one or more reference images are merely colors and texture patterns, but the underlying design aspects involved in actual artwork creation have not been taken into consideration. In contrast, our framework incorporated artists perceptions of the real world into style transfer process, which complements the artwork production mechanisms, and produce results with distinctive genre character. We reconstruct the perception-guided style representation in an embedding space created by the auto-encoder trained with genre dataset, as it has more genre-specific characteristic than original pixel intensities or dCNN features.

Heterogeneous Domain Transformation. The core of our method is to reconstruct the style representation of the real world photo based on the correspondence from reality to perception, which could be regarded as heterogeneous domains. Methods have been proposed for heterogeneous image transformation problems, such as photos to sketches [Wang et al., 2017], and photos to paintings [Semmo et al., 2015]. However, these methods are very target-specific and can not be easily extended to other image styles.

Early matching methods, such as optical flow [Brox et al., 2004] and PatchMatch [Barnes et al., 2009], were designed in image space to match local image statistics like brightness consistency or pixel intensities, which would easily fail under appearance variations. Then, the development of various handcrafted local region descriptors [Lowe, 2004] invoked noteworthy progress. These descriptors are robust to visual variations and local transformations. Some methods [Liu et al., 2016] combine these descriptors with dense matching to perform more reliable matching. However, these approaches are still based on low-level features and hence would fail to match semantically-related images. While other methods [Zhu et al., 2017b; Liao et al., 2017] apply dCNN representations in semantic matching, they directly use Euclidean distance between deep features as similarity measurement. However, images from heterogeneous domains are commonly different in visual appearances, even when they have similar semantic contents. As a result, the Euclidean distance between their deep representations does not serve the best as semantic similarity measurement.

3 Model

As shown in Figure 1, our framework consists of three parts: encoder E, representation reconstruction module, and decoder D. Encoder E and decoder D are combined during training and separated to perform encoding and decoding functions in stylization. The framework is designed after the process of artwork creation, where artists first perceive the real world scenes, and then create artworks with distinctive styles based on their personal perceptions.

The encoder-decoder network creates an embedding representation space, in which to reconstruct perception-guided style representation Ψr (x). To include perception guidance, we compute a NNF that establishes correspondences between dCNN features of the photo and the painting. In stylization phase, the encoder subnetwork E transforms the real world photo into pseudo content representation Ψc (x). The reconstructed style representation Ψr (x) would be optimized by minimizing its difference from Ψc (x) for preserving better structure with the real world photo. Then the decoder subnetwork D would transform the optimized style representation Ψr ′ (x) into neural artwork.

3.1 Encoder-Decoder Network

Network Architecture. Inspired by the architecture used in [Johnson et al., 2016], the encoder E contains 3 convolutional layers: one stride-1 convolution layer followed by two stride-2 convolution layers to downsample the input image. Symmetrically, the decoder D starts with two convolutional layers with stride 1/2 to upsample the style representation and concatenate with one stride 1 convolution layer. Each convolution layer is followed by an instance normalization Embedding Representation Space

Figure 2: The style representations in embedding space can be pieced out and recombined to form a new artwork.

[Ulyanov et al., 2016] and ReLU nonlinearity layer to form a block, except for the output layer. The first and last convolutional layers use 9 × 9 kernels, other layers use 3 × 3 kernels. Jonhson et al. [2016] added residual blocks for its capability of learning identity function. However, as the target of our network is to create a stable representation space for each genre, we discard residual blocks because its mechanism of adding residuals to the input would introduce noise into embedding space.

Training. We train the network with separate art collections to produce an output painting as same as possible to the input painting. Each painting is resized to 256 × 256. The identity loss Lidentity is the squared Euclidean error between input image I and output image O:

Genre-specific Representation Space. The output of the encoder forms an embedding representation space for each genre. The 256×256 paintings are encoded into 128×64×64 style representations Ψs (x), a 3D tensor with channel × width × height. From Figure 2 we can see that, the artworks are encoded into multi-layer feature maps, and when we piece out the fragments of the feature maps from different paintings, the decoder can still transform it into a plausible artwork.

In stylization phase, the encoder E and decoder D operate independently. We feed real world photo to E to generate pseudo content representation Ψc (x), the layout of which is same with the input photo while the response amplitude is identical with style representation Ψs (x). Figure 3 demonstrates some corresponding channels of Ψs (x) and Ψc (x). Each channel of the feature maps generated by a convolution kernel could be considered as one aspect of genre's style characteristic.

3.2 Style Representation Reconstruction

Given an real world photo and artwork pair xc and xs, which is different considerably in appearance but have similar semantic contents, we reconstruct a perception-guided style representation Ψr (x) for real world photo by finding the mappings from the deep feature maps of xc to xs.

Preprocessing

Our method starts by feeding the photo and the painting into VGG-19 network [Simonyan and Zisserman, 2015], which was pre-trained on ImageNet database for image classification tasks, and perform forward propagation. We obtain the pyramid of feature maps {DLc } and {DLs } (L = 1, 2, 3) for the input photo xc and painting xs. The feature maps of each scale derive from the reluL 2 layer of the Lth convolution blocks in VGG-19. The deep feature maps {DLc } and {DLs } are resized to 64 × 64, the same size of the style representation Ψs (x). Then we divide all the representations and deep feature maps into 3 × 3 patches, with stride 1. The patches are very densely sampled to achieve the best reconstruction quality.

Coarse-to-fine Nearest-Neighbor Field Search

We adopt a coarse-to-fine searching strategy for computing NNF: we estimate a quick initial NNF guess by taking PatchMatch [Barnes et al., 2009] into deep feature space, then a more precise NNF is obtained by searching the neighborhood around the initial guess. The extracted NNF together with representations from the embedding space are used to reconstruct perception-guided style representation of the real world photo.

For deep feature maps at layer L, we estimate an initial NNF represented by mapping functions f L c→s . f L c→s maps a point in deep feature map DLc to another in DLs . f L c→s is computed by minimizing the following function:

L

c s

f 

Coarse NNF

Reconstruction

Fine NNF

Reconstruction

Figure 4: Visualization of NNFs (top row), reconstructed results with coarse NNFs (middle row) and reconstructed results with fine NNFs (bottom row).

where M(p) is the patch around pixel p. The patch size is set to be 3×3. corr(DLc (x), DLs (y)) is the correlation coefficient between DLc (x) and DLs (x). For each patch around pixel p in DLc , we find its nearest neighbor position q = f L c→s (p) inDLs .

Equation (2) can be efficiently optimized with PatchMatch method [Barnes et al., 2009]. Here, we adopt PatchMatch in deep feature domain for better representation capability, and we use correlation coefficient between deep feature maps as similarity metric to eliminate the affect of visual appearances variation over semantic contents.

f L c→s obtained from optimizing Equation (2) is considered as an initial coarse NNF, and is further transformed into nearest indexes φLc→s (xi) for patches.

For the ith patch DLc (xi) in deep feature maps DLc , we find its best matching patch DLs (xNN(i)) in deep feature maps DLs within the search region around φLc→s (xi) for patches:

Then the obtained nearest patch indexes is transformed into mapping functions f L+1 c→s , and serve as guidance for coarse NNF search to limit the random search space in layer L + 1. The coarse-to-fine NNF search is repeated for layer 1 - 3 (as shown in Figure 4 ), updating correspondences with multiscale deep feature maps.

Perception-Guided Representation Reconstruction

After we obtain the final nearest indexes for patches, the perception-guided representation Ψr (xc) of the real world photo xc could be reconstructed from the embedding representation Ψs (xs) with the reality-perception correspondence. The best matches are stitched together into a complete style representation with overlapping area averaged.

L=1

L=2

L=3

3.3 Optimization

Let Ψr ′ (x) denote the modified version of Ψr (xc). Our objective is to make the reconstructed style representation be structurally similar to the real world photo. Hence Ψr ′ (x) can be obtained by minimizing the following loss function :

, where Ψc(xc) is the pseudo content representation from the encoder with real world photo as input, α is set to 0.2 in our experiment. We minimize Equation (4) using backpropagation with L-BFGS [Zhu et al., 1997]. Such an optimization is similar to updating the stylization result by pretrained VGG network described in [Gatys et al., 2016], except in our case, the target representation Ψr ′ (x) is updated through encoder E. Further, the altered representation would be transformed into neural artworks through pre-trained decoder:

4 Experimental Results and Analysis

4.1 Experimental Settings

Dataset

We conducted our experiments on our Van Gogh-photo dataset and the cubism-photo dataset. The Van Gogh-photo dataset consists of 309 Van Goghs painting and the semantically corresponding real world photo pairs. The cubismphoto dataset has 34 cubist painting and real world photo pairs. We choose these two genres on account of two reasons: Firstly, Van Goghs paintings and cubism artworks are commonly used as reference images in style transfer methods. Secondly, their paintings have distinctive genre characters, such as the bright colors in Van Goghs paintings and the broken geometry patches in cubism artworks.

Encoder-Decoder Network Training Details

The network is trained on the paintings of separate genres. All the paintings and the photos is scaled to the size of 256 × 256. We train the network with a batch size of 5 for 300 epochs. And we adopt the Adam optimization method

[Kingma and Ba, 2015] with the initial learning rate of 0.001. We do not apply weight decay as the model would not easily overfit.

All experiments are conducted with Python on Ubuntu 16.04 system, with i7-4790 3.6G CPU and 12G NVIDIA Titan X GPU. GPU was used in the network training and deep feature extraction phase. The two most time-consuming parts of our proposed method lie in style reconstruction phase: (1) nearest neighbor field computation (about 120 seconds), which need to compute correlation coefficient over hundreds of feature channels, and (2) optimizing the reconstructed style representation (about 100 seconds), which require about 400 iterations to converge for Equation (4).

4.2 Result and Comparison

Result

The main purpose of our method is to imitate the art production process and transfer the artists and genre's overall style to photo. Figure 5 shows some results of the generated neural artworks and the original real world photos in both genres. For Van Goghs style, our results possess oil painting traits: the strokes is more clear, and there are small brush-like textures resembling the actual oil painting. The colors in the generated artworks are brighter than the original photo, which is a distinct personal painting style of Van Gogh. As for cubism, of which the objects are analyzed, broken up and reassembled in an abstracted form, our results can also express this character rather than merely migrating the color patterns to the photo.

Comparison on Matching

We evaluate the matching quality of our approach and most commonly used matching methods on heterogeneous image pairs with semantically-related scenes. From Figure 6 we can see that existing matching methods ([Barnes et al., 2009; Liu et al., 2016]) is incapable of establishing semantic correspondences between images when they are vastly different in appearances. On the contrary, our matching method can produce more acceptable result.

Comparison on Style Transfer

Recent state-of-the-art techniques in neural style transfer yield very impressive results, we compare our method with these approaches on two different categories of data: (1) single painting as style image, and (2) using a collection of paintings. It is difficult to evaluate the quality of the stylized image quantitatively as there is no acknowledged quality assessment criteria in neural image artistic rendering. For comparable visual quality, we ran author-released implementations with default settings.

Single Image. Typical CNN-based style transfer methods use only one style image and one content image. As realityperception correspondence is the core of our idea, we choose one real world photo as content image, and one semanticallyrelated painting from Van Gogh's collections as style image for comparison with other methods.

In Figure 7, we compare our results with two most representative methods under iterative optimization (Neural Style [Gatys et al., 2016]) and feed-forward network (Perceptual Loss [Johnson et al., 2016]) categories. When the style image and content image are semantically corresponding, CNNbased style transfer methods can cast better color pattern of the original style image than ours, but the result lost semantic meaning to some extent. For example, in the result of Perceptual loss, the color of sky can not remain blue around the tree. And the yellow color pattern in Neural Style result reduced the contrast between flowers and leaves. In our method, styl- Real world photo Cycle GAN

Ours

Figure 8: Comparison with CycleGAN.

ization is an overall transformation tendency, like brighter colors. Our method mainly emphasize the reality-perception correspondence in genre style transfer, which makes the results more plausible and hand-painted like.

Using Collection of Paintings. In Figure 8, we compare our result with CycleGAN [Zhu et al., 2017a]. CycleGAN used unpaired images to train the network by minimizing cycle consistency loss. We use author-released Van Goghphoto datasets to train CycleGAN for Van Gogh style transfer. However, due to the absence of reality-perception correspondence in the unpaired training data and the property of generative network, their result is more similar to the input real world photo and lack of painting characteristic. In comparison, our result has distinct hand-painted feature in strokes, and the colors are brighter.

5 Discussion and Conclusion

In this paper, we proposed a novel framework for genre-based style transfer. The key insight of the framework is that we redefine the style representation of the input photo based on the reality-perception correspondence from the genre. We modeled after the actual artists' creation process, trying to incorporate the specific artists or genres overall character into style transfer. The artworks generated in Van Goghs style is much brighter in color, and the ones in cubism have similar broken and ensemble feature. We believe this method may provide a new idea in neural style transfer. It may also be proven useful for other genre-based applications, such as genre classification, and art piece identification.

There are still some interesting issues for further investigation. For example, finding correspondence from the reality to perception is not completely reliable. There are chances that an object in the real world photo has no match in the whole genre, then the semantic correspondence could not be well-established. Moreover, for paintings that are highly abstractions of the real world, the correspondence from reality to perception is hard to establish, as dCNN feature maps lack discrimination for abstracted paintings. A possible solution would be to train a regression model on real world element and its abstraction pairs to learn the correspondence rather than simply matching the dCNN features.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (under Grants 61501339, 61772402, 61671339, 61432014, U1605252), in part by Young Elite Scientists Sponsorship Program by CAST (under Grant 2016QNRC001), in part by Natural Science Basic Research Plan in Shaanxi Province of China (under Grant 2017JM6085), in part by Young Talent fund of University Association for Science and Technology in Shaanxi, China, in part by CCF-Tencent Open Fund (under Grant IAGR 20170103), in part by the Leading Talent of Technological Innovation of Ten-Thousands Talents Program under Grant CS31117200001.