Demystifying Neural Style Transfer

1 Introduction

Transferring the style from one image to another image is an interesting yet difficult problem. There have been many efforts to develop efficient methods for automatic style transfer [Hertzmann et al., 2001; Efros and Freeman, 2001; Efros and Leung, 1999; Shih et al., 2014; Kwatra et al., 2005]. Recently, Gatys et al. proposed a seminal work [Gatys et al., 2016]: It captures the style of artistic images and transfer it to other images using Convolutional Neural Networks (CNN). This work formulated the problem as finding an image that matching both the content and style statistics based on the neural activations of each layer in CNN. It achieved impressive results and several follow-up works improved upon this innovative approaches [Johnson et al., 2016; Ulyanov et al., 2016; Ruder et al., 2016; Ledig et al., 2016]. Despite the fact that this work has drawn lots of attention, the fundamental element of style representation: the Gram matrix in [Gatys et al., 2016] is not fully explained. The reason why Gram matrix can represent artistic style still remains a mystery.

∗Corresponding author

In this paper, we propose a novel interpretation of neural style transfer by casting it as a special domain adaptation [Beijbom, 2012; Patel et al., 2015] problem. We theoretically prove that matching the Gram matrices of the neural activations can be seen as minimizing a specific Maximum Mean Discrepancy (MMD) [Gretton et al., 2012a]. This reveals that neural style transfer is intrinsically a process of distribution alignment of the neural activations between images. Based on this illuminating analysis, we also experiment with other distribution alignment methods, including MMD with different kernels and a simplified moment matching method. These methods achieve diverse but all reasonable style transfer results. Specifically, a transfer method by MMD with linear kernel achieves comparable visual results yet with a lower complexity. Thus, the second order interaction in Gram matrix is not a must for style transfer. Our interpretation provides a promising direction to design style transfer methods with different visual results. To summarize, our contributions are shown as follows:

1. First, we demonstrate that matching Gram matrices in neural style transfer [Gatys et al., 2016] can be reformulated as minimizing MMD with the second order polynomial kernel.
2. Second, we extend the original neural style transfer with different distribution alignment methods based on our novel interpretation.

2 RelatedWork

In this section, we briefly review some closely related works and the key concept MMD in our interpretation.

Style Transfer Style transfer is an active topic in both academia and industry. Traditional methods mainly focus on the non-parametric patch-based texture synthesis and transfer, which resamples pixels or patches from the original source texture images [Hertzmann et al., 2001; Efros and Freeman, 2001; Efros and Leung, 1999; Liang et al., 2001]. Different methods were proposed to improve the quality of the patchbased synthesis and constrain the structure of the target image. For example, the image quilting algorithm based on dynamic programming was proposed to find optimal texture boundaries in [Efros and Freeman, 2001]. AMarkov Random Field (MRF) was exploited to preserve global texture structures in [Frigo et al., 2016]. However, these non-parametric methods suffer from a fundamental limitation that they only use the low-level features of the images for transfer.

Recently, neural style transfer [Gatys et al., 2016] has demonstrated remarkable results for image stylization. It fully takes the advantage of the powerful representation of Deep Convolutional Neural Networks (CNN). This method used Gram matrices of the neural activations from different layers of a CNN to represent the artistic style of a image. Then it used an iterative optimization method to generate a new image from white noise by matching the neural activations with the content image and the Gram matrices with the style image. This novel technique attracts many follow-up works for different aspects of improvements and applications. To speed up the iterative optimization process in [Gatys et al., 2016], Johnson et al. [Johnson et al., 2016] and Ulyanov et al. [Ulyanov et al., 2016] trained a feed-forward generative network for fast neural style transfer. To improve the transfer results in [Gatys et al., 2016], different complementary schemes are proposed, including spatial constraints [Selim et al., 2016], semantic guidance [Champandard, 2016] and Markov Random Field (MRF) prior [Li and Wand, 2016]. There are also some extension works to apply neural style transfer to other applications. Ruder et al. [Ruder et al., 2016] incorporated temporal consistence terms by penalizing deviations between frames for video style transfer. Selim et al. [Selim et al., 2016] proposed novel spatial constraints through gain map for portrait painting transfer. Although these methods further improve over the original neural style transfer, they all ignore the fundamental question in neural style transfer: Why could the Gram matrices represent the artistic style? This vagueness of the understanding limits the further research on the neural style transfer.

Domain Adaptation Domain adaptation belongs to the area of transfer learning [Pan and Yang, 2010]. It aims to transfer the model that is learned on the source domain to the unlabeled target domain. The key component of domain adaptation is to measure and minimize the difference between source and target distributions. The most common discrepancy metric is Maximum Mean Discrepancy (MMD) [Gretton et al., 2012a], which measure the difference of sample mean in a Reproducing Kernel Hilbert Space. It is a popular choice in domain adaptation works [Tzeng et al., 2014; Long et al., 2015; Long et al., 2016]. Besides MMD, Sun et al. [Sun et al., 2016] aligned the second order statistics by whitening the data in source domain and then re-correlating to the target domain. In [Li et al., 2017], Li et al. proposed a parameter-free deep adaptation method by simply modulating the statistics in all Batch Normalization (BN) layers.

MaximumMean Discrepancy Suppose there are two sets of samples X = {xi}ni=1 and Y = {yj}mj=1 where xi and yj are generated from distributions p and q, respectively. MaximumMean Discrepancy (MMD) is a popular test statistic for the two-sample testing problem, where acceptance or rejection decisions are made for a null hypothesis p = q [Gretton et al., 2012a]. Since the population MMD vanishes if and only p = q, the MMD statistic can be used to measure the difference between two distributions. Specifically, we calculates MMD defined by the difference between the mean embedding on the two sets of samples. Formally, the squared MMD is defined as:

where φ(·) is the explicit feature mapping function of MMD. Applying the associated kernel function k(x, y) = 〈φ(x), φ(y)〉, the Eq. 1 can be expressed in the form of kernel:

The kernel function k(·, ·) implicitly defines a mapping to a higher dimensional feature space.

3 Understanding Neural Style Transfer

In this section, we first theoretically demonstrate that matching Gram matrices is equivalent to minimizing a specific form of MMD. Then based on this interpretation, we extend the original neural style transfer with different distribution alignment methods.

Before explaining our observation, we first briefly review the original neural style transfer approach [Gatys et al., 2016]. The goal of style transfer is to generate a stylized image x∗ given a content image xc and a reference style image xs. The feature maps of x∗ , xc and xs in the layer l of a CNN are denoted by Fl ∈ RNl×Ml , Pl ∈ RNl×Ml and Sl ∈ RNl×Ml respectively, where Nl is the number of the feature maps in the layer l and Ml is the height times the width of the feature map.

In [Gatys et al., 2016], neural style transfer iteratively generates x∗ by optimizing a content loss and a style loss:

where α and β are the weights for content and style losses, Lcontent is defined by the squared error between the feature maps of a specific layer l for x∗ and xc:

and Lstyle is the sum of several style loss Llstyle in different layers:

where wl is the weight of the loss in the layer l and Llstyle is defined by the squared error between the features correlations expressed by Gram matrices of x∗ and xs:

where the Gram matrix Gl ∈ RNl×Nl is the inner product between the vectorized feature maps of x∗ in layer l:

and similarly Al is the Gram matrix corresponding to Sl .

3.1 Reformulation of the Style Loss

In this section, we reformulated the style loss Lstyle in Eq. 6. By expanding the Gram matrix in Eq. 6, we can get the formulation of Eq. 8, where f l ·k and sl·k is the k-th column of Fl and Sl .

By using the second order degree polynomial kernel k(x, y) = (xT y)2 , Eq. 8 can be represented as:

where F l is the feature set of x∗ where each sample is a column of Fl , and S l corresponds to the style image xs. In this way, the activations at each position of feature maps is considered as an individual sample. Consequently, the style loss ignores the positions of the features, which is desired for style transfer. In conclusion, the above reformulations suggest two important findings:

1. The style of a image can be intrinsically represented by feature distributions in different layers of a CNN.
2. The style transfer can be seen as a distribution alignment process from the content image to the style image.

3.2 Different AdaptationMethods for Neural Style Transfer

Our interpretation reveals that neural style transfer can be seen as a problem of distribution alignment, which is also at the core in domain adaptation. If we consider the style of one image in a certain layer of CNN as a "domain", style transfer can also be seen as a special domain adaptation problem. The specialty of this problem lies in that we treat the feature at each position of feature map as one individual data sample, instead of that in traditional domain adaptation problem in which we treat each image as one data sample. (e.g. The feature map of the last convolutional layer in VGG-19 model is of size 14 × 14, then we have totally 196 samples in this "domain".)

Inspired by the studies of domain adaptation, we extend neural style transfer with different adaptation methods in this subsection.

MMD with Different Kernel Functions As shown in Eq. 9, matching Gram matrices in neural style transfer can been seen as a MMD process with second order polynomial kernel. It is very natural to apply other kernel functions for MMD in style transfer. First, if using MMD statistics to measure the style discrepancy, the style loss can be defined as:

where Z lk is the normalization term corresponding to different scale of the feature map in the layer l and the choice of kernel function. Theoretically, different kernel function implicitly maps features to different higher dimensional space. Thus, we believe that different kernel functions should capture different aspects of a style. We adopt the following three popular kernel functions in our experiments:

1. Linear kernel: k(x, y) = xT y;
2. Polynomial kernel: k(x, y) = (xT y + c)d ;

For polynomial kernel, we only use the version with d = 2. Note that matching Gram matrices is equivalent to the polynomial kernel with c = 0 and d = 2. For the Gaussian kernel, we adopt the unbiased estimation of MMD [Gretton et al., 2012b], which samples Ml pairs in Eq. 10 and thus can be computed with linear complexity.

BN Statistics Matching In [Li et al., 2017], the authors found that the statistics (i.e. mean and variance) of Batch Normalization (BN) layers contains the traits of different domains. Inspired by this observation, they utilized separate BN statistics for different domain. This simple operation aligns the different domain distributions effectively. As a special domain adaptation problem, we believe that BN statistics of a certain layer can also represent the style. Thus, we construct another style loss by aligning the BN statistics (mean and standard deviation) of two feature maps between two images:

where µiF l and σiF l is the mean and standard deviation of the i-th feature channel among all the positions of the feature map

in the layer l for image x∗ :

and µiSl and σiSl correspond to the style image xs.

The aforementioned style loss functions are all differentiable and thus the style matching problem can be solved by back propagation iteratively.

4 Results

In this section, we briefly introduce some implementation details and present results by our extended neural style transfer methods. Furthermore, we also show the results of fusing different neural style transfer methods, which combine different style losses. In the following, we refer the four extended style transfer methods introduced in Sec. 3.2 as linear, poly, Gaussian and BN, respectively. The images in the experiments are collected from the public implementations of neural style transfer 123 .

Implementation Details In the implementation, we use the VGG-19 network [Simonyan and Zisserman, 2015] following the choice in [Gatys et al., 2016]. We also adopt the relu4 2 layer for the content loss, and relu1 1, relu2 1, relu3 1, relu4 1, relu5 1 for the style loss. The default weight factor wl is set as 1.0 if it is not specified. The target image x∗ is initialized randomly and optimized iteratively until the relative change between successive iterations is under 0.5%. The maximum number of iterations is set as 1000. For the method with Gaussian kernel MMD, the kernel bandwidth σ2 is fixed as the mean of squared l2 distances of the sampled pairs since it does not affect a lot on the visual results. Our implementation is based on the MXNet [Chen et al., 2016] implementation 1 which reproduces the results of original neural style transfer [Gatys et al., 2016].

1 https://github.com/dmlc/mxnet/tree/master/example/neuralstyle

2 https://github.com/jcjohnson/neural-style

3 https://github.com/jcjohnson/fast-neural-style

Since the scales of the gradients of the style loss differ for different methods, and the weights α and β in Eq. 3 affect the results of style transfer, we fix some factors to make a fair comparison. Specifically, we set α = 1 because the content losses are the same among different methods. Then, for each method, we first manually select a proper β′ such that the gradients on the x∗ from the style loss are of the same order of magnitudes as those from the content loss. Thus, we can manipulate a balance factor γ (β = γβ′ ) to make trade-off between the content and style matching.

4.1 Different Style Representations

To validate that the extended neural style transfer methods can capture the style representation of an artistic image, we first visualize the style reconstruction results of different methods only using the style loss in Fig. 1. Moreover, Fig. 1 also compares the style representations of different layers. On one hand, for a specific method (one row), the results show that different layers capture different levels of style: The textures in the top layers usually has larger granularity than those in the bottom layers. This is reasonable because each neuron in the top layers has larger receptive field and thus has the ability to capture more global textures. On the other hand, for a specific layer, Fig. 1 also demonstrates that the style captured by different methods differs. For example, in top layers, the textures captured by MMD with a linear kernel are composed by thick strokes. Contrarily, the textures captured by MMD with a polynomial kernel are more fine grained.

4.2 Result Comparisons

Effect of the Balance Factor We first explore the effect of the balance factor between the content loss and style loss by varying the weight γ. Fig. 2 shows the results of four transfer methods with various γ from 0.1 to 10.0. As intended, the global color information in the style image is successfully transfered to the content image, and the results with smaller γ preserve more content details as shown in Fig. 2(b) and Fig. 2(c). When γ becomes larger, more stylized textures are incorporated into the results. For example, Fig. 2(e) and Fig. 2(f) have much more similar illumination and textures with the style image, while Fig. 2(d) shows a balanced result between the content and style. Thus, users can make trade-off between the content and the style by varying γ.

Comparisons of Different Transfer Methods Fig. 3 presents the results of various pairs of content and style images with different transfer methods 4 . Similar to matching Gram matrices, which is equivalent to the poly method, the other three methods can also transfer satisfied styles from the specified style images. This empirically demonstrates the correctness of our interpretation of neural style transfer: Style transfer is essentially a domain adaptation problem, which aligns the feature distributions. Particularly, when the weight on the style loss becomes higher (namely, larger γ), the differences among the four methods are getting larger. This indicates that these methods implicitly capture different aspects of style, which has also been shown in Fig. 1. Since these methods have their unique properties, they could provide more choices for users to stylize the content image. For example, linear achieves comparable results with other methods, yet requires lower computation complexity.

Fusion of Different Neural Style Transfer Methods

Since we have several different neural style transfer methods, we propose to combine them to produce new transfer results. Fig. 4 demonstrates the fusion results of two combinations (linear + Gaussian and poly + BN). Each row presents the results with different balance between the two methods. For example, Fig. 4(b) in the first two rows emphasize more on BN and Fig. 4(f) emphasizes more on poly. The results in the middle columns show the interpolation between these two methods. We can see that the styles of different methods are blended well using our method.

4 More results can be found at http://www.icst.pku.edu.cn/struct/Projects/mmdstyle/result1000/show-full.html

5 Conclusion

Despite the great success of neural style transfer, the rationale behind neural style transfer was far from crystal. The vital "trick" for style transfer is to match the Gram matrices of the features in a layer of a CNN. Nevertheless, subsequent literatures about neural style transfer just directly improves upon it without investigating it in depth. In this paper, we present a timely explanation and interpretation for it. First, we theoretically prove that matching the Gram matrices is equivalent to a specific Maximum Mean Discrepancy (MMD) process. Thus, the style information in neural style transfer is intrinsically represented by the distributions of activations in a CNN, and the style transfer can be achieved by distribution alignment. Moreover, we exploit several other distribution alignment methods, and find that these methods all yield promising transfer results. Thus, we justify the claim that neural style transfer is essentially a special domain adaptation problem both theoretically and empirically. We believe this interpretation provide a new lens to re-examine the style transfer problem, and will inspire more exciting works in this research area.

Acknowledgement

This work was supported by the National Natural Science Foundation of China under Contract 61472011.