---
title: "Universal Video Style Transfer via Crystallization, Separation, and Blending"
authors:
  - "Haofei Lu"
  - "Zhizhong Wang"
doi: "10.24963/ijcai.2022/687"
url: "https://ijcai.org/proceedings/2022/687"
pdf: "https://ijcai.org/proceedings/2022/0687.pdf"
venue: "Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence"
year: 2022
pages: "4957-4965"
track: "AI and Arts"
keywords:
  - "Theory and philosophy of arts and creativity in AI systems: Autonomous creative or artistic AI"
  - "Methods and resources: Machine learning, deep learning, neural models, reinforcement learning"
---

# Universal Video Style Transfer via Crystallization, Separation, and Blending

Haofei Lu, Zhizhong Wang

*Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence*, pages 4957-4965. DOI: [10.24963/ijcai.2022/687](https://doi.org/10.24963/ijcai.2022/687)

## Abstract

Universal video style transfer aims to migrate arbitrary styles to input videos. However, how to maintain the temporal consistency of videos while achieving high-quality arbitrary style transfer is still a hard nut to crack. To resolve this dilemma, in this paper, we propose the CSBNet which involves three key modules: 1) the Crystallization (Cr) Module that generates several orthogonal crystal nuclei, representing hierarchical stability-aware content and style components, from raw VGG features; 2) the Separation (Sp) Module that separates these crystal nuclei to generate the stability-enhanced content and style features; 3) the Blending (Bd) Module to cross-blend these stability-enhanced content and style features, producing more stable and higher-quality stylized videos. Moreover, we also introduce a new pair of component enhancement losses to improve network performance. Extensive qualitative and quantitative experiments are conducted to demonstrate the effectiveness and superiority of our CSBNet. Compared with the state-of-the-art models, it not only produces temporally more consistent and stable results for arbitrary videos but also achieves higher-quality stylizations for arbitrary images.

## Full text

## 1 Introduction

Style transfer aims to render a content target with the desired style of a given style reference. Recently, the seminal work of [Gatys et al., 2016] showed that the Grammatrix (i.e., feature correlation) of features extracted from a Deep Convolutional Neural Network (DCNN) could represent the style patterns well. Since then, significant efforts have been made to improve the performance in various aspects, including efficiency [Johnson et al., 2016], quality [Wang et al., 2021; Chen et al., 2021b], generality [Huang and Belongie, 2017; Li et al., 2017], and diversity [Ulyanov et al., 2017; Wang et al., 2020b], etc. However, existing style transfer approaches are mostly designed for still images, while the stylization for dynamic videos is still an open challenge.

∗Corresponding author.

Compared with image style transfer, video style transfer is a much more challenging task. The method should consider both the temporal consistency and stylization quality of the stylized videos. Existing image style transfer methods, while achieving satisfactory results for still images, often result in significant jitters and flickers when applied to videos due to the lack of in-depth consideration into video temporal stability. To overcome this problem, [Ruder et al., 2016] introduced optical flow prediction to [Gatys et al., 2016] to maintain the video-frame consistency. However, it takes too much time to stylize a single frame. For fast video style transfer, [Chen et al., 2017] designed a feed-forward optical flow and mask estimation network, and [Gao et al., 2020] proposed a multi-style framework that estimates the light flow and uses temporal constraint. These methods benefit from optical flow prediction to maintain temporal stability but rely highly on estimation accuracy. What's more, works like [Wang et al., 2020a] have justified that optical flow not only suppresses the stylization quality but also may bring obvious video jelly effects in the results.

Recently, [Li et al., 2019] proposed a stable linear transformation that is suitable for frame-based video style transfer. Later, [Wang et al., 2020a] introduced a novel interpretation of temporal consistency without optical flow estimation and achieved consistent zero-shot video style transfer. Based on it, [Deng et al., 2021] proposed multi-channel correlation for arbitrary video style transfer. These approaches achieved the state-of-the-art video style transfer. However, their stylization quality is not satisfactory, and some slight brushstroke jitters still exist. Therefore, how to maintain the temporal consistency of videos while achieving high-quality arbitrary style transfer is still a hard nut to crack.

In order to avoid using the optical flow method and complete high-quality arbitrary style transfer while maintaining the video continuity, in this paper, we propose a novel framebased network for universal video style transfer. Our method, termed CSBNet, consists of three key modules: 1) the Crystallization (Cr) Module that uses singular value decomposition to generate several orthogonal matrices called crystal nuclei, representing hierarchical stability-aware content and style components, from raw VGG features; 2) the Separation (Sp) Module that separates these crystal nuclei to generate the stability-enhanced content and style features; 3) the Blending (Bd) Module to cross-blend these stabilityenhanced content and style features, producing more stable and higher-quality stylized videos. At the same time, we also introduce a new pair of component enhancement losses to improve network performance. Extensive qualitative and quantitative experiments demonstrate that our method can effectively suppress the generation of local artifacts and inherently maintain the smoothness of input videos while successfully keeping the content structures and style details, significantly outperforming the state-of-the-art (SOTA) models.

The main contributions of our method are:

- A novel frame-based network CSBNet consisting of a Crystallization (Cr) Module, a Separation (Sp) Module, and a Blending (Bd) Module, to achieve consistent universal video style transfer with higher stylization quality.
- A new pair of component enhancement losses to improve the performance of our network, helping produce more stable and higher-quality results.
- Extensive qualitative and quantitative experiments to demonstrate the effectiveness and superiority of our method in both image and video style transfer.

## 2 RelatedWork

Image Style Transfer. [Gatys et al., 2016] firstly introduced DCNN to artistic style transfer. In their method, the style of an image is represented by the Gram matrix (i.e., feature correlation) of the intermediate features extracted from DCNN. While achieving visually stunning results, the iterative procedure is time-consuming, and the output is sensitive to input noise. Later, [Li and Wand, 2016] improved the stability by introducing Markov Random Fields (MRFs) and proposed a patch-based method. To achieve real-time transfer, [Johnson et al., 2016] introduced a feed-forward approach by training feed-forward networks. However, the trained networks are fixed for pre-defined styles. To achieve arbitrary style transfer, AdaIN [Huang and Belongie, 2017] aligned the feature maps of content images with style ones by normalizing means and variances. Also, WCT [Li et al., 2017] orthogonalized the content features and matched the covariance matrix of the style features. However, outputs generated by AdaIN and WCT failed to represent the detailed textures sufficiently and unexpectedly brought twisted style patterns at some local areas more or less. Further, Linear [Li et al., 2019] learned a linear transformation to better preserve content affinity during style transfer. SANet [Park and Lee, 2019] employed a style-attention module to better capture local style patterns. AAMS [Yao et al., 2019] introduced self-attention methods to improve output quality. AdaAttN [Liu et al., 2021] proposed a novel attention and normalization module for arbitrary style transfer. IECAST [Chen et al., 2021a] incorporated the adversarial nets and contrastive losses to improve SANet.

Despite the recent progress in image style transfer, few of them considered the temporal consistency, resulting in obvious flickering effects when applied to videos. Moreover, there's still improvement room for existing style transfer approaches to better blend content structures with style patterns to generate higher-quality stylized results.

Video Style Transfer. Current video style transfer methods can be roughly divided into two categories: 1) methods based on optical flow prediction and 2) non-optical flow prediction methods with single frame constraints.

In order to maintain video-frame consistency, [Ruder et al., 2016] introduced optical flow prediction of the previous frame to guide the generation of the current time. However, it needs several minutes to generate a single output frame, which is unsuitable for large-scale applications. Consequently, [Chen et al., 2017] designed a feed-forward network for fast video style transfer. [Gao et al., 2020] introduced ConvLSTMs to stabilize results and FlowNet(s) to simulate optical-flow prediction. These methods benefit from optical flow prediction to maintain temporal consistency but rely highly on estimation accuracy. In addition, optical-flow guidance also limits the quality of stylization [Wang et al., 2020a].

Recently, Linear [Li et al., 2019] learned a stable linear transformation that is suitable for frame-based video style transfer. ReReVST [Wang et al., 2020a] introduced a novel interpretation of temporal consistency without optical flow estimation and achieved consistent zero-shot video style transfer via relaxation and regularization. MCCNet [Deng et al., 2021] proposed multi-channel correlation for arbitrary video style transfer. These methods achieved the state-of-theart video style transfer. However, their stylization quality is unsatisfactory, and some slight brushstroke jitters still exist.

To avoid using the optical flow estimation while obtaining high-quality stylization and maintaining the video continuity, we propose a novel frame-based universal video style transfer approach via crystallization, separation, and blending. The stability-aware content and style components of the content and style images are first crystallized and separated. Then, the stability-enhanced content feature of the content image and the style feature of the style image are cross-blended, thus producing temporally more stable and higher-quality results.

## 3 ProposedMethod

## 3.1 Network Structure

We first depict the main structure of our network from the high level, which is based on the encoder-decoder model. As shown in Fig. 1, CSBNet uses a pre-trained VGG-19 network [Simonyan and Zisserman, 2014] as the encoder and the symmetric network as the decoder. Given a content image Ic and a style image Is, we first generate their corresponding VGG features Fc = VGG(Ic) and Fs = VGG(Is).

Then, the VGG features Fc and Fs are fed into the Crystallization (Cr) Module (Sec. 3.2) to generate several stabilityaware orthogonal matrices A1,A2, ...,AN, namely crystal nuclei in our work:

<!-- formula-not-decoded -->

Next, the Separation (Sp) Module (Sec. 3.2) completes the separation of these crystal nuclei according to the empirically determined cursors Kc and Ks (which control the degree of separation), generating the stability-aware content component F content and style component F style :

<!-- formula-not-decoded -->

Figure 1: Overview framework of CSBNet. Cr: Crystallization; Sp: Separation; Bd: Blending.

![Image](https://ijcai.org/proceedings/2022/fulltext/687/image_000000_a3f11dc81b6590356c0792fa14dcaf2376240b10f677693ffdcace5f5a06a4c6.png)

In the following step, the style component F style c of Fc and the content component F content s of Fs will be removed, and we only use the content component F content c of Fc and style component F style s of Fs to generate the stability-enhanced content feature F enhance c and style feature F enhance s . They are cross-blended via the Blending (Bd) Module (Sec. 3.3):

<!-- formula-not-decoded -->

Finally, the blended feature Fcs is decoded to obtain the stylized output Ics:

<!-- formula-not-decoded -->

## 3.2 Crystallization &amp; SeparationModule

After obtaining the VGG features Fc and Fs, existing neural style transfer approaches always use these raw features directly to produce the stylized feature Fcs. However, we argue that for raw VGG features, the content components and style components are mixed. And directly using the mixed features will easily generate unnatural and flickering effects where the style components of the content images remain, or the content components of the style patterns appear in the form of ghosting artifacts. To generate these stability-aware content and style components and complete the separation of them, we propose two key modules in our CSBNet, i.e., the Crystallization (Cr) Module and Separation (Sp) Module.

The structures of Cr and Sp modules are shown in Fig. 3, where these two modules are closely related.

Crystallization (Cr) Module. Given a raw VGG feature F ∈ {Fc, Fs} ∈ RC×H×W (where C is the number of channels, H and W are height and width, respectively), we first flatten it to ˆ F ∈ RC×HW and center ˆ F by subtracting its channel-wise mean.

Then we perform orthogonal projection on ˆ F via singular value decomposition (SVD):

<!-- formula-not-decoded -->

Figure 2: Visualization of content components (top) and style components (bottom) in pixel-level crystallization and separation.

![Image](https://ijcai.org/proceedings/2022/fulltext/687/image_000001_aea470f13d61794c778a412ee815892c5101ab90f89c29310a60f26b9e5d79e8.png)

where N = min(C,HW), σi is singular value, ui ∈ RC×1 and vi ∈ RHW×1 are the ith column of U and V , respectively. We define Ai = σiuivT i ∈ RC×HW and a matrix set Φ = {A1, A2, ..., AN}. According to the properties of SVD, for every Ai, Aj (i ̸= j) in Φ, we have:

<!-- formula-not-decoded -->

which means Ai and Aj are orthogonal. In experiments, we find these orthogonal matrices can represent different content and style components hierarchically (see later "Separation (Sp) Module"). These matrices are stability-aware, e.g., if we separate and remove the style component of the content feature and the content component of the style feature, we can fundamentally enhance the transfer stability. Therefore, we design the Sp Module for this purpose. For easy identification, we call these orthogonal matrices crystal nuclei.

Separation (Sp) Module. This module is designed to separate the above orthogonal crystal nuclei into the content component F content and the style component F style :

<!-- formula-not-decoded -->

where K is an empirically determined cursor to decide the degree of separation between the content and style components. Θ is the unflatten operation to reshape the feature size to C × H × W .

Figure 3: Structures of Crystallization (Cr) Module and Separation (Sp) Module.

![Image](https://ijcai.org/proceedings/2022/fulltext/687/image_000002_0c578b856564ae8ab40f7400aa845028b3c01d7e0c29113f80ab434a1243740c.png)

To demonstrate the motivation and rationality of this separation, we visualize the pixel-level content components and style components in Fig. 2. As can be observed, with the increase of K, the content component (top row) filters out more style information and preserve more pure structural details. In contrast, the style component (bottom row) adds more detailed style information and filters out certain content information. In this way, we can further separate the content and style information to obtain more pure content and style features, which may be helpful to improve the stylization quality and transfer stability.

Specifically, we set K = Kc to separate the crystal nuclei of the content feature Fc, and K = Ks to separate the crystal nuclei of the style feature Fs. Thus we can easily control the content-style separation for the content image and style image via adjusting Kc and Ks, respectively.

After separation, the style component F style c of Fc and the content component F content s of Fs are removed, and we only use the content component F content c of Fc and the style component F style s of Fs to generate the stability-enhanced content feature F enhance c and style feature F enhance s :

<!-- formula-not-decoded -->

where f1(·), f2(·), g(·), and h(·) are 1×1 learnable depthwise convolution layers. We use depthwise convolution here in order to maintain the hierarchical independence of each channel. Besides, depthwise convolution also offers less computational complexity. As shown in Fig. 3, a double-pole switch is used to generate F enhance c and F enhance s based on whether the input feature is from the content or style image.

## 3.3 BlendingModule

We further design a novel Blending (Bd) Module to blend the stability-enhanced content feature F enhance c with the style feature F enhance s more effectively. Our Bd module is based on MCCNet [Deng et al., 2021], which considers the multichannel correlation to achieve more stable feature alignment:

<!-- formula-not-decoded -->

where f i/k ∈ R1×HW is the ith /kth channel of flattened feature F ∈ { ˆ Fc, ˆ Fs, ˆ Fcs}, C is the number of channels and wk represents the weight of the k-th style channel learned by a fully connected layer. While it has been verified that this operation can improve the stability and temporal consistency of video style transfer, we found it is easy to bring bar-like artifacts to the generated outputs (e.g., the 1st and 5th rows in Fig. 5). It is because the MCC in Eq. (9) is sub-optimal, which only learns partial information of the Gram matrix.

For explanation, let's start with the Gram matrix of Fs:

<!-- formula-not-decoded -->

It can be easily derived that the weighted item ||f k s ||2 in Eq. (9) equals f k s (f k s )T , which is the kth diagonal element of the Gram matrix in Eq. (10). This indicates that MCCNet conducts style transfer based on only C diagonal elements of Gram matrix, while C × C - C non-diagonal elements, which also represent the correlation between channels, are abandoned. Thus it may lead to degraded transfer results.

Our proposed Bd Module overcomes this problem through better capturing the associations between different style channels. As shown in Fig. 4, we first normalize and tansform the enhanced content feature F enhance c and the style feature F enhance s as follows:

<!-- formula-not-decoded -->

Figure 4: Structure of our Blending (Bd) Module.

![Image](https://ijcai.org/proceedings/2022/fulltext/687/image_000003_9018ec5acdccbf696996428907939ec7c4e8112304d47a7dea8ea195c0f8e3e8.png)

where J (·) and K(·) are 1 × 1 learnable convolution layers, and Norm(·) is channel-wise mean-variance normalization.

Then for ˜ Fs, we calculate its Gram matrix Gs = Gram(˜ Fs) and weight each row gis of Gs:

<!-- formula-not-decoded -->

where wi ∈ R1×C is the ith row of W ∈ RC×C , a parameterlearnable matrix. Based on this, we further obtain the fusion component ˜ Fcs. The ith column of ˜ Fcs is as follows:

<!-- formula-not-decoded -->

where ˜ f i ∈ R1×H×W is the ith channel of ˜ F ∈ {˜ Fcs, ˜ Fc}. The final fusion feature Fcs can be obtained as:

<!-- formula-not-decoded -->

where R(·) is a 1 × 1 learnable convolution layer.

## 3.4 Loss Function

We train our CSBNet using the following total loss function consisting of a peceptual loss Lpercp, our proposed component enhancement loss Lcomp, and a smooth loss Lsmooth:

<!-- formula-not-decoded -->

Perceptual Loss. To achieve style transfer, we use a pretrained VGG-19 to extract content and style features and compute the content and style perceptual losses similar to AdaIN [Huang and Belongie, 2017]:

<!-- formula-not-decoded -->

where λcontent and λstyle are hyper-parameters to balance the content and style performance.

Specifically, the content perceptual loss is used to minimize the content differences between the generated images Ics and content images Ic:

<!-- formula-not-decoded -->

where φi represents the ith layer in VGG-19, here we use Relu4 1 and Relu5 1 to calculate Lcontent.

The style perceptual loss minimizes the style differences between the generated images Ics and style images Is:

<!-- formula-not-decoded -->

where µ(·) and σ(·) denote the channel-wise mean and standard deviation, respectively. And we use Relu1 1, Relu2 1, Relu3 1, and Relu4 1 to calculate Lstyle.

Component Enhancement Loss. In order to enhance the quality of content-style component separation in the network, we design a novel pair of component enhancement losses, consisting of a content componet loss Lc-comp and a style component loss Ls-comp:

<!-- formula-not-decoded -->

where λc-comp and λs-comp are hyper-parameters.

We hope to minimize the content and style differences of corresponding components. Thus, we define the content component loss as the content differences between the stabilityenhanced content features F enhance c and the generated features Fcs:

<!-- formula-not-decoded -->

Similarly, the style component loss is used to minimize the style differences between the stability-enhanced style features F enhance s and the generated features Fcs:

<!-- formula-not-decoded -->

With these component enhancement losses, our network can better learn the stability-enhanced content and style information, thus effectively suppressing the generation of artifacts and improving the stylization quality as well as stability (see ablation studies in later Sec. 4.4).

Figure 5: Qualitative comparisons of universal image and video style transfer methods.

![Image](https://ijcai.org/proceedings/2022/fulltext/687/image_000004_6e645249cf5c7eb9940d83be568b82cce76e2ff1e66a8addec107b351012189a.png)

Smooth Loss. Similar to MCCNet [Deng et al., 2021], we also adopt a total variation loss Ltv [Johnson et al., 2016] and an illumination loss Lillum to smooth the results and improve the robustness to complex light conditions in input videos.

<!-- formula-not-decoded -->

where λtv and λillum are hyper-parameters. The illumination loss Lillum is defined as:

<!-- formula-not-decoded -->

where I noise cs is the results generated with noisy content image Ic + θ (θ is random Gaussian noise) and style image Is.

## 4 Experimental Results

## 4.1 Implementation Details

The hyper-parameters in our model are set to λcontent = 3, λstyle = 10, λc-comp = 3, λs-comp = 1, λtv = 0.0001, and λillum = 3000. The separation cursors are empirically set to Kc = 4, Ks = -10 (here '-' represents the inverted index of array). We choose MS-COCO [Lin et al., 2014] as content image set and WikiArt [Phillips and Mackintosh, 2011] as style image set. During the training stage, the input images are resized to 512×512 pixels and randomly cropped to 256× 256 pixels. Batch size is chosen as 4 for 160,000 iterations.

## 4.2 Qualitative Comparisons

Eight SOTA universal image and video style transfer methods are selected for comparison, including AdaIN [Huang and Belongie, 2017], WCT [Li et al., 2017], Linear [Li et al., 2019], AAMS [Yao et al., 2019], MCCNet [Deng et al., 2021], AdaAttN [Liu et al., 2021], IECAST [Chen et al., 2021a], and ReReVST [Wang et al., 2020a].

Figure 6: Qualitative comparisons of video style transfer. The first row shows the stylized video frames. The second row shows the heat maps visualizing the average differences between adjacent frames.

![Image](https://ijcai.org/proceedings/2022/fulltext/687/image_000005_171a19d510cb44d290571e6a7893d75f3e5dbed8d8387409f3484f31763b40eb.png)

Image Style Transfer. The comparisons of image stylizations are shown in Fig. 5. The over-simplified alignment of AdaIN easily leads to style artifacts and content distortion (e.g. 2nd and 5th rows). Due to global parameter transformation, WCT seriously damages the content structures in the results. Linear can better preserve content affinity, but the degree of stylization is low (e.g. 1st and 2nd rows). AAMS, MCCNet, AdaAttN, and IECAST are essentially based on the attention mechanism. However, point-like artifacts appear in AAMS (e.g. 1st, 3rd, and 5th rows), and bar-like artifacts appear in MCCNet (e.g. 1st and 5th rows). AdaAttN preserves the content structures well but still suffers from noticeable artifacts (e.g. 3rd and 5th rows) and poor transfer of style color distributions (e.g. 1st and 3rd rows). IECAST is prone to produce eye-like artifacts (e.g. 2nd and 3rd rows). ReReVST is designed for video style transfer, which often produces less stylized results for image stylizations.

Table 1: The average metrics of inputs and stylized results of different methods on 10 video clips. The first and second rows are Ec and Es, evaluating stylization quality. The third and fourth rows are LPIPS and D∗ , evaluating video consistency. The last row is the average transfer time for a single 512 × 512 frame, evaluating transfer efficiency. These metrics all follow the rule: the lower, the better.

| Metrics    | Input       | Ours        | AdaIN       | WCT          | Linear      | AAMS        | MCCNet      | AdaAttN     | IECAST      | ReReVST     |
|------------|-------------|-------------|-------------|--------------|-------------|-------------|-------------|-------------|-------------|-------------|
| Ec Es      | 0.000 3.241 | 1.874 1.130 | 2.393 1.135 | 3.541 1.560  | 1.856 1.280 | 2.119 1.142 | 2.051 1.165 | 2.158 1.249 | 1.930 1.458 | 1.988 1.523 |
| LPIPS D∗   | 0.071 30.78 | 0.108 47.50 | 0.190 80.88 | 0.432 113.71 | 0.117 63.75 | 0.297 86.48 | 0.122 51.13 | 0.219 71.46 | 0.245 91.30 | 0.137 54.80 |
| Time/(sec) | -           | 0.017       | 0.008       | 0.579        | 0.013       | 2.173       | 0.015       | 0.018       | 0.011       | 0.098       |

|      |    𝐸! |    𝐸" |    𝐷∗ |   LPIPS |
|------|-------|-------|-------|---------|
| Ours | 1.874 | 1.130 | 47.50 |   0.108 |
| (b)  | 2.353 | 1.137 | 55.38 |   0.129 |
| (c)  | 2.171 | 1.234 | 69.12 |   0.111 |
| (d)  | 2.323 | 1.159 | 63.19 |   0.118 |

Figure 7: Ablation study of Crystallization and Separation (Cr-Sp) Module.

![Image](https://ijcai.org/proceedings/2022/fulltext/687/image_000006_ec1308af4ea0f16d41b52efe83b7d05860e8c9ebcac7b687c8c9dd9c75d864c5.png)

In contrast to these SOTA approaches, the results of our CSBNet exhibit higher stylization quality. The active separation of content and style components by the Cr and Sp Modules successfully suppresses the appearance of style artifacts and better maintains the content structures. Moreover, the better utilization of style information by the Bd Module leads to a higher degree of stylization in the results.

Video Style Transfer. The comparisons of video stylizations are shown in Fig. 6. We present heatmaps to show the average differences between adjacent frames. It can be seen that the difference of our results is closest to that of the input frames, obviously outperforming other methods in terms of temporal stability and consistency. This is due to the fact that our CSBNet can effectively maintain the content structures of the video frames and suppress the generation of random artifacts causing frame-flicker.

## 4.3 Quantitative Comparisons

Stylization Quality. We leverage the content and style errors Ec and Es of [Gatys et al., 2016] to quantitatively evaluate the stylization quality. As shown in the top two rows of Tab. 1, CSBNet enjoys the lowest style error Es among 9 methods, while content error Ec is only slightly higher than Linear [Li et al., 2019], ranking the second. Metrics Ec and Es show that our method can effectively improve stylization quality while better preserving content structures, leading to comprehensively more satisfying transfer results.

Figure 8: Ablation study of component enhancement loss.

![Image](https://ijcai.org/proceedings/2022/fulltext/687/image_000007_ef9eb73135280cff7e5d2d959e17ee5e54ca491b6d5c5e2055608757ef2f7eee.png)

Video Consistency. We adopt LPIPS (Learned Perceptual Image Patch Similarity) [Zhang et al., 2018] and D∗ [Deng et al., 2021] to quantify the stability and consistency of video style transfer. Given a video clip with N frames, we compute the average perceptual distances between adjacent frames via LPIPS, and the average pixel distances via D∗ , i.e., D∗ = 1 N-1 ∑N-2 t=0 ||Ft+1-Ft||2. The results shown in the middle two rows of Tab. 1 further reflect the superiority of our CSBNet on video style transfer, which is consistent with the qualitative results in Fig. 6.

Efficiency. As shown in the last row of Tab. 1, CSBNet achieves comparable speed with SOTA methods, only slightly behind AdaIN, Linear, MCCNet, and IECAST. However, we believe it is worthwhile to sacrifice slight run time speed for higher stylization quality and improvement of consistency.

## 4.4 Ablation Studies

Crystallization and Separation (Cr-Sp) Module. Fig. 7 shows the results of video style transfer with and without CrSp module for Fc and Fs, respectively. It can be found that (b) without Cr-Sp for Fc, the unstable strip and dot artifacts are generated in the results; (c) without Cr-Sp for Fs, there will be a loss of content details (e.g., the feather and eyes of the bird) and the blue color of the content image leak into the result (e.g., the eyes of the bird). In contrast, with Cr-Sp for both Fc and Fs, our method generates smoother and more satisfying results with better-preserved content structures and

|   Kc |   Ec |   Es |    D∗ |   LPIPS |   Ks |   Ec |   Es |    D∗ |   LPIPS |
|------|------|------|-------|---------|------|------|------|-------|---------|
|    0 | 2.07 | 1.27 | 52.75 |   0.116 |   -1 | 1.94 | 1.23 | 48.45 |   0.115 |
|    2 | 1.94 | 1.61 | 48.92 |   0.111 |   -4 | 1.95 | 1.28 | 48.40 |   0.113 |
|    4 | 1.87 | 1.13 | 47.50 |   0.108 |  -10 | 1.87 | 1.13 | 47.50 |   0.108 |
|   10 | 1.90 | 1.16 | 49.78 |   0.112 |  -15 | 1.95 | 1.15 | 49.47 |   0.110 |
|   15 | 1.91 | 1.21 | 51.39 |   0.116 |  -20 | 1.99 | 1.15 | 53.80 |   0.115 |
|   20 | 1.92 | 1.33 | 51.30 |   0.118 |  -30 | 2.07 | 1.14 | 54.50 |   0.118 |
|  200 | 2.08 | 1.32 | 51.79 |   0.119 | -200 | 2.11 | 1.18 | 54.00 |   0.117 |

(a) Effects of Kc (Ks = -10)

(b) Effects of Ks (Kc = 4)

Table 2: Effects of different settings of Kc and Ks.

fewer local artifacts. This observation is consistent with the quantitative scores shown in column (e), which validates that our Cr-Sp module indeed helps improve the stylization quality and temporal stability.

Component Enhancement Loss. A pair of new losses is introduced in our framework for enhancing transfer results, and thus we further conduct ablation experiments on it. From Fig. 8, we can observe that without the proposed component enhancement loss, the edge of the generated result is prone to produce grid-like artifacts (e.g., the edges of tree and the squirrel's ear and back), which not only degrades the stylization quality, but also reduces the transfer stability. Best viewed in conjunction with the quantitative scores below.

Setting of Kc and Ks. We also conduct ablation experiments to demonstrate the effects of different settings of our separation cursors Kc and Ks. We can observe from Tab. 2 that increasing the values of |Kc| and |Ks| within a certain range not only helps to reduce the content and style errors Ec and Es, but also increases the temporal stability of output videos (lower D∗ and LPIPS scores). It is because increasing the value of |Kc| can filter out the style components of content images, thus helping improve the stylization of the results. Meanwhile, increasing the value of |Ks| can filter out the content components of style images, thus helping preserve the integrity of the content structures and improving the videos' fluency. However, too high values of |Kc| and |Ks| will reduce the stylized quality and stability of the output videos.

## 5 Conclusion

In this work, we propose a novel frame-based network, termed CSBNet, for universal video style transfer. Three innovative modules, i.e., the Crystallization (Cr) Module, Separation (Sp) Module, and Blending (Bd) Module, and a new pair of component enhancement losses are introduced to improve video consistency and achieve higher-quality stylization. Extensive qualitative and quantitative experiments are conducted to demonstrate the effectiveness and superiority of our CSBNet against state-of-the-art algorithms.

## References

1. [Chen et al., 2017] Dongdong Chen, Jing Liao, Lu Yuan, Nenghai Yu, and Gang Hua. Coherent online video style transfer. In Proceedings of the IEEE International Confer- ence on Computer Vision, pages 1105-1114, 2017.
2. [Chen et al., 2021a] Haibo Chen, Lei Zhao, Zhizhong Wang, Huiming Zhang, Zhiwen Zuo, Ailin Li, Wei Xing, and Dongming Lu. Artistic style transfer with internal-external learning and contrastive learning. In Advances in Neural Information Processing Systems, 2021.
3. [Chen et al., 2021b] Haibo Chen, Lei Zhao, Zhizhong Wang, Huiming Zhang, Zhiwen Zuo, Ailin Li, Wei Xing, and Dongming Lu. Dualast: Dual style-learning networks for artistic style transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 872-881, 2021.
4. [Deng et al., 2021] Yingying Deng, Fan Tang, Weiming Dong, Haibin Huang, Chongyang Ma, and Changsheng Xu. Arbitrary video style transfer via multi-channel corre- lation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1210-1217, 2021.
5. [Gao et al., 2020] Wei Gao, Yijun Li, Yihang Yin, and Ming- Hsuan Yang. Fast video multi-style transfer. In Proceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3222-3230, 2020.
6. [Gatys et al., 2016] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2414- 2423, 2016.
7. [Huang and Belongie, 2017] Xun Huang and Serge Be- longie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE In- ternational Conference on Computer Vision, pages 1501- 1510, 2017.
8. [Johnson et al., 2016] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pages 694-711. Springer, 2016.
9. [Li and Wand, 2016] Chuan Li and Michael Wand. Combin- ing markov random fields and convolutional neural net- works for image synthesis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2479-2486, 2016.
10. [Li et al., 2017] Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang. Universal style transfer via feature transforms. Advances in neural infor- mation processing systems, 30, 2017.
11. [Li et al., 2019] Xueting Li, Sifei Liu, Jan Kautz, and Ming- Hsuan Yang. Learning linear transformations for fast image and video style transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3809-3817, 2019.
12. [Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In European conference on com- puter vision, pages 740-755. Springer, 2014.
13. [Liu et al., 2021] Songhua Liu, Tianwei Lin, Dongliang He, Fu Li, Meiling Wang, Xin Li, Zhengxing Sun, Qian Li, and Errui Ding. Adaattn: Revisit attention mechanism in arbitrary neural style transfer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6649-6658, 2021.
14. [Park and Lee, 2019] Dae Young Park and Kwang Hee Lee. Arbitrary style transfer with style-attentional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5880-5888, 2019.
15. [Phillips and Mackintosh, 2011] Fred Phillips and Brandy Mackintosh. Wiki art gallery, inc.: A case for critical thinking. Issues in Accounting Education, 26(3):593-608, 2011.
16. [Ruder et al., 2016] Manuel Ruder, Alexey Dosovitskiy, and Thomas Brox. Artistic style transfer for videos. In German conference on pattern recognition, pages 26-36. Springer, 2016.
17. [Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
18. [Ulyanov et al., 2017] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Improved texture networks: Maximiz- ing quality and diversity in feed-forward stylization and texture synthesis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6924- 6932, 2017.
19. [Wang et al., 2020a] Wenjing Wang, Shuai Yang, Jizheng Xu, and Jiaying Liu. Consistent video style transfer via relaxation and regularization. IEEE Transactions on Im- age Processing, 29:9125-9139, 2020.
20. [Wang et al., 2020b] Zhizhong Wang, Lei Zhao, Haibo Chen, Lihong Qiu, Qihang Mo, Sihuan Lin, Wei Xing, and Dongming Lu. Diversified arbitrary style transfer via deep feature perturbation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7789-7798, 2020.
21. [Wang et al., 2021] Zhizhong Wang, Lei Zhao, Haibo Chen, Zhiwen Zuo, Ailin Li, Wei Xing, and Dongming Lu. Evaluate and improve the quality of neural style transfer. Computer Vision and Image Understanding, 207:103203, 2021.
22. [Yao et al., 2019] Yuan Yao, Jianqiang Ren, Xuansong Xie, Weidong Liu, Yong-Jin Liu, and Jun Wang. Attention- aware multi-stroke style transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1467-1475, 2019.
23. [Zhang et al., 2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.