Spatiotemporal Super-Resolution with Cross-Task Consistency and Its Semi-supervised Extension

1 Introduction

Videos with high spatiotemporal resolutions are typically spatially sharper and temporally coherent, and hence are preferable to humans. However, acquiring such videos requires higher power consumption and larger storage. To compromise between the user experience and acquiring cost, spatial SR (super-resolution) and temporal SR have drawn increasing attention in computer vision. Spatiotemporal SR involves two sub-tasks, including spatial SR and temporal SR. Specifically, the former recovers the high-resolution frame by referring to a single low-resolution one, while the latter upscales the video frame rate by synthesizing intermediate frames.

In this work, we address spatiotemporal SR to increase the frame rate and resolution simultaneously for a given video. Both spatial SR such as [Shi et al., 2016; Kim et al., 2016; Haris et al., 2018] and temporal SR such as [Liu et al., 2017; Jiang et al., 2018; Bao et al., 2019] have been investigated extensively in the literature. However, few research advancements [Shahar et al., 2011; Sharma et al., 2017] have been made on spatiotemporal SR, which is more practical for lowquality video processing and understanding. An effective and intuitive way for spatiotemporal SR is to apply the two subtasks, spatial SR and temporal SR, sequentially. We consider the two sub-tasks highly correlated. On the one hand, spatial SR offers high-resolution frame details, which are essential to temporal SR. On the other hand, temporal SR enriches motion information, which generally facilitates spatial SR. However, the correlation between the two tasks has not been well exploited, especially in the era of deep learning.

This paper addresses the aforementioned issues. We correlate spatial SR and temporal SR for spatiotemporal SR based on a key insight: The order of applying spatial and temporal SR is interchangeable, as illustrated in Figure 1. We learn a two-stream network. One stream is composed of a temporal SR module fT followed by a spatial SR module fS for spatiotemporal SR. The other stream is formed by switching the two modules. No matter whether the ground truth for spatiotemporal SR is available, the results predicted by the two streams should be consistent. Hence, we introduce a cross-stream consistency loss to enforce the similarity between the outputs of the two streams. The introduced loss correlates spatial SR and temporal SR. High-resolution details estimated by spatial SR help align pixels cross consecutive frames, which is crucial for temporal SR. High-framerate videos produced by temporal SR in turn facilitate spatial SR by regularizing it to predict temporally consistent SR results. Thereby, the two SR modules can improve each other by mutually providing supervisory signals.

More importantly, the cross-stream consistency loss does not rely on any high-resolution and high-frame-rate video for training. That is, learning from unlabeled training data, i.e., videos of low resolutions and frame rates here, is enabled. By taking abundant unlabeled videos as input, the cross-stream consistency loss serves as the objective function to guide network training in a self-taught manner. We leverage this property to carry out semi-supervised learning. It turns out that our method makes the most of training data, and can derive an effective model with few high-quality video collections.

The main contribution of this work is three-fold. First, we present the first end-to-end trainable network for spatiotemporal SR where the spatial and temporal SR modules are correlated and benefit each other. Second, the proposed method exploits cross-stream consistency and enables learning from unlabeled data, greatly reducing the cost of collecting videos of high spatiotemporal resolution, which can be very expensive in some applications like medical and satellite images. Third, evaluated intensively on four benchmark datasets including the Vimeo-90K [Xue et al., 2019], Middlebury optical flow [Baker et al., 2007], Vid4 [Liu and Sun, 2011], and DAVIS [Pont-Tuset et al., 2017] datasets, our method achieves the state-of-the-art performance.

2 RelatedWork

2.1 Spatial Super-Resolution

Spatial SR has been explored extensively. Early approaches are developed based on the sampling theory, e.g., using linear or bicubic interpolation for SR [Keys, 1981]. However, interpolation exhibits limitations in predicting detailed, realistic textures. Advanced methods aim to establish complex mapping between low-resolution (LR) and high-resolution (HR) images by using machine learning algorithms, such as neighbor embedding [Gao et al., 2012] and sparse coding [Zeyde et al., 2012].

The pioneering work SRCNN [Dong et al., 2016] employs a three-layer CNN model to approximate the non-linear mapping between LR and HR images in an end-to-end trainable manner. Based on residual learning [He et al., 2016] for deep network optimization, modern CNN-based models such as VDSR [Kim et al., 2016] and DRRN [Tai et al., 2017] further boost the performance of SR. Recently, residual-dense network (RDN) [Zhang et al., 2018] employs dense connections to learn the local representations from patches for improving SR. Likewise, DBPN [Haris et al., 2018] presents a series of densely connected upsampling and downsampling layers to represent different image degradations for enhancing SR.

2.2 Temporal Super-Resolution

Conventional methods, e.g., [Baker et al., 2011; Yu et al., 2013], for temporal SR usually estimate dense correspondences between consecutive frames, and synthesize intermediate frames by the estimated correspondences. The quality of the interpolation results is highly dependent on the quality of the estimated optical flow. However, optical flow estimation is difficult and often suffers from many issues, such as occlusions, large motion, and blur.

CNNs have shown their effectiveness for optical flow estimation. Thereby, some CNN-based methods carry out frame interpolation by optical flow estimation [Bailer et al., 2019]. However, these CNN-based methods relying on flow field prediction need training data in the form of dense correspondences, which are hard to annotate. Instead of relying on optical flow, some frame synthesis methods leverage CNNs to directly generate the intermediate frames [Niklaus et al., 2017]. They do not take dense correspondences as training data but the ground-truth intermediate frames. However, these methods still suffer from blurred results and artifacts. Liu et al. [Liu et al., 2017] address the problem of blurred results by proposing the deep voxel flow, a 3D optical flow to warp frames based on trilinear sampling. Their method makes the synthesized frames sharper, but the issue of artifacts remains unsolved. Recently, DAIN [Bao et al., 2019] is proposed to explicitly detect occlusions by exploring depth cues, based on the observation that closer objects should be preferably synthesized in intermediate frames.

2.3 Spatiotemporal Super-Resolution

Research efforts on spatial SR and temporal SR are quite extensive. However, few advancements have been made on simultaneous spatial and temporal SR, i.e., spatiotemporal SR, which upscales video frame rates and resolutions at the same time. Spatiotemporal SR is important in many vision applications such as surveillance where analyzing videos of low quality is required. In the conventional approach, Shahar et al. [Shahar et al., 2011] propose an example-based approach where spatiotemporal SR is realized by combining information from multiple spacetime patches. Sharma et al. [Sharma et al., 2017] propose the first deep-learning-based method, called coupled deep convolutional auto-encoder (CDCA), for spatiotemporal SR. CDCA generates the convolutional feature maps of the spatial patches in up-sampled LR and HR video frames using convolutional auto-encoder (CAE) and learns the relationships between these feature maps by CNNs at the same time. CDCA computes the up-sampled LR by tri-cubic interpolation, but it ignores motion correspondences between consecutive LR frames.

Unlike existing methods where the collaboration between the spatial and temporal modules is ignored, our method mitigates the aforementioned drawbacks by introducing the crossstream consistency loss to jointly learn spatial and temporal SR modules in an end-to-end trainable manner. Thus, our method has the following advantages. First, the spatial SR and temporal SR modules are correlated and trained at the same time. More realistic frames are interpolated by referring to the high-resolution details recovered by the spatial SR

module, while better high-resolution frames are produced by using the motion information estimated by the temporal SR module. Second, the proposed cross-stream consistency loss allows learning from unlabeled data and greatly reduces the requirement of high-resolution videos. In addition, the spatial and temporal SR modules are present several times in the network, but the multiple copies of each module share weights, making the number of parameters manageable. These nice properties distinguish our method from prior work.

2.4 Cycle Consistency

Exploiting cycle consistency properties to regularize structured prediction has been explored in the literature. In unsupervised domain adaptation, exploiting cross-domain invariance in the label space achieves more consistent task predictions [Chen et al., 2019]. In video frame interpolation, the cycle consistency loss enforces the similarity between the input frames and the cyclic mapped-back frames [Liu et al., 2019]. In semantic matching, object co-segmentation, and co-saliency detection, cycle or transitivity based consistency losses help regularize the network training, e.g., [Chen et al., 2015; Chen et al., 2018; Chen et al., 2020; Tsai et al., 2019]. In motion analysis, computing bi-directional optical flow is useful to infer occlusions [Zou et al., 2018] and enforce temporal consistency [Lai et al., 2018].

In this work, we show a novel and feasible way of exploiting cross-stream consistency to address spatiotemporal SR. To the best of our knowledge, this work makes the first attempt to improve spatiotemporal SR by leveraging crossstream consistency and end-to-end training. We design a twostream network with spatial and temporal SR modules. Both streams consist of the two SR modules but have them in different orders. We show that enforcing the two streams to make consistent predictions leads to substantially improved performance. Employing the cross-stream consistency loss further enables semi-supervised learning.

3 Proposed Approach

This section describes the proposed approach. First, we give the problem definition. Then, the proposed network architecture and its objective function as well as the semi-supervised extension are described. Finally, the implementation details are provided.

3.1 ProblemDefinition

Given a video with a low frame resolution and/or low frame rate, the goal of spatiotemporal SR is to generate a highquality video with a higher frame resolution and frame rate. Let I L t-1 and I L t+1 be two consecutive low-resolution (LR) frames at timestamps t - 1 and t + 1. The spatiotemporal SR model F in this work is derived to generate three highresolution (HR) frames, I H t-1 , I H t , and I H t+1 . We use the superscript and subscript to denote the frame resolution and timestamp, respectively. The model F for upsacling both the frame resolution and frame rate can be represented as

Spatiotemporal SR can be decomposed into two sub-tasks: spatial SR and temporal SR, which are associated with learnable modules fS and fT , respectively.

Sub-task spatial SR targets at recovering the HR details and producing sharper video frames. It estimates an HR frame I H ∈ RsP×sQ given an LR one I L ∈ RP×Q, where P and Q are the frame height and width respectively and s > 1 is the upscaling factor. The module fS can be expressed as

By applying fS to an frame k times, the frame resolution is increased by a factor of sk .

The other sub-task temporal SR aims at upscaling video frame rates. Given two consecutive frames, It-1 and It+1, of an arbitrary resolution, the temporal SR module fT generates the intermediate frame It between the two input frames. The module fT can be formulated as

Repeatedly applying module fT to a video k times upscales the frame rate by a factor of 2k .

To produce an HR video in both its image resolution and frame rate through Eq. (1), frames I H t-1 and I H t+1 can be obtained by applying the spatial SR module fS to I L t-1 and I L t+1 respectively, while frame I H t relies on by the collaboration of both spatial SR module fS and temporal SR module fT . The two modules can work independently. One naive way for carrying out spatiotemporal SR is the sequential combination of the spatial and temporal SR modules, namely applying fT followed by fS or its inverse. In Figure 2, FT→S and FS→T represent two network streams, each of which combines the two SR modules. Given two consecutive LR frames I L t-1 and I L t+1 , either FT→S or FS→T can generate the HR frame I H t as defined below

3.2 Network Architecture

To better accomplish spatiotemporal SR and even carry it out in a semi-supervised fashion, we present an end-to-end trainable network which is composed of two network streams FT→S and FS→T , as illustrated in Figure 2. By taking two consecutive LR frames I L t-1 and I L t-1 as input, streamFT→S feeds them to the temporal SR module fT to generate the interpolated frame I L t and passes I L t to the spatial SR module fS to synthesize the HR frame I H t . In stream FS→T , I L t-1 and I L t-1 are fed into fS to produce HR frames I H t-1 and I H t-1 , which then serve as the input to fT to obtain the interpolated HR result I H t . The multiple copies of each module share weights so that the number of learnable parameters in the proposed network is manageable. We do not make any assumption about the spatial and temporal SR modules fS and fT . Our method is general to work with existing spatial and temporal SR algorithms by adopting them as the modules fS and fT , and accomplishes spatiotemporal SR. In the inference phase, either FT→S or FS→T can obtain the result. We just need to pass through one of the two streams.

3.3 Objective Function

The objective function L for training the proposed network consists of two loss terms, including the reconstruction loss LR and cross-stream consistency loss LC. The former LR guides the two network streams, FS→T and FT→S, to perform spatiotemporal SR by making their prediction as close to the ground truth as possible. The latter LC enforces the consistency between the results predicted by the two network streams. The training objective function L is defined by

where λ is the hyperparameter used to control the relative importance between LR and LC, which are detailed below.

Reconstruction Loss LR

To guide the training of the two network streams, FT→S and FS→T , this loss enforces their predicted results to be consistent with the ground truth. For each training triplet {I H t-1 , I H t , I H t+1 }, we first down-sample I H t-1 and I H t+1 to yield I L t-1 and I L t+1 by bicubic interpolation, respectively. With I L t-1 and I L t+1 , the reconstruction loss LR is formulated as

Cross-stream Consistency Loss LC

Our key insight for cross-stream consistency loss is that while both streams FS→T and FT→S can estimate the spatiotemporal SR results, their prediction should be the same no matter if the ground truth I H t is given. We thus propose a cross-stream consistency loss LC that synchronizes the outputs of the two network streams. Specifically, this loss LC is given below

3.4 Semi-supervised Learning

As shown in Eq. (7), the cross-stream consistency loss does not rely on any high-resolution ground truth. It allows the proposed method to work with unsupervised (i.e., low-resolution here) training data and hence enables semisupervised learning. Specifically, the resulting loss function for semi-supervised spatiotemporal SR is designed as

where the labeled data DL = {(I L t-1 , I H t , I L t+1 )} are used in the reconstruction loss LR and the cycle consistency loss LC. Unlabeled data DU = {(I L t-1 , I L t+1 )} are used merely in LC since DU does not have high-resolution ground truth required in LR. By adopting the semi-supervised spatiotemporal SR, our model can be derived by using a small set of labeled data with a large amount of unlabeled data.

3.5 Implementation Details

Our method can work with existing spatial and temporal SR models. In this work, we use VDSR [Kim et al., 2016], ESPCN [Shi et al., 2016], and DBPN [Haris et al., 2018] as the spatial SR module, while adopt DVF [Liu et al., 2017], Super Slomo [Jiang et al., 2018], and DAIN [Bao et al., 2019] as the temporal SR module. Since DBPN and DAIN are released in Pytorch, our implementation regarding DBPN and DAIN is realized by Pytorch, while the rest are developed with Tensorflow. The spatial and temporal SR modules are first trained separately, then used to reconstruct the proposed network architecture, and finally fine-tuned. Although multiple spatial SR modules are present in the network, they are identical and share weights. This setting is also applied for the temporal SR module. We set the batch size, learning rate, momentum, and weight decay to 2, 10-3 , 0.9, and 5×10-4 , respectively. We train and evaluate our model on a single NVIDIA GeForce GTX 1080Ti graphics card with 11GB memory.

4 Experimental Results

In this section, we first describe the datasets used in the experiments, and then conduct ablation studies and comparisons between our method and existing methods.

4.1 Datasets

We train the proposed method for spatiotemporal SR by using the training set of the Vimeo-90k dataset [Xue et al., 2019], which is recently built for evaluating the performance of video processing tasks, such as video frame interpolation and super-resolution. The Vimeo-90k dataset contains 51, 313 samples for training and 3, 782 samples for testing. Each sample contains three high-resolution consecutive frames of resolution 256 × 448. For each training sample in the supervised setting, the middle frame serves as the ground truth while the low-resolution counterparts of the other two frames act as inputs. In our experiments, we downscale each frame side by a factor of 4 to yield the low-resolution counterparts. In the semi-supervised setting, only the low-resolution counterparts of unlabeled samples are used.

In addition to the testing set of Vimeo-90K, we use the Middlebury optical flow [Baker et al., 2007], Vid4 [Liu and Sun, 2011], and DAVIS [Pont-Tuset et al., 2017] datasets to evaluate the performance of the proposed method. For testing sets in the form of videos, we downscale the resolution of the odd-numbered frames by a factor of 4, and then remove all even-frames to generate low spatiotemporal resolution frame sequences. The downscaled consecutive oddnumbered frames serve as the inputs while the original evennumbered frames act as ground truth for evaluation. We adopt Peak Signal-to-Noise Ratio (PSNR) and structural similarity (SSIM) as the performance measure.

4.2 Ablation Studies

We conduct ablation studies for analyzing our method. Our method can work with existing spatial or temporal SR modules. We choose VDSR [Kim et al., 2016], ESPCN [Shi et al., 2016], and DBPN [Haris et al., 2018] as the spatial SR modules, and select DVF [Liu et al., 2017], Super Slomo [Jiang et al., 2018], and DAIN [Bao et al., 2019] as the temporal SR modules. Each baseline model is realized by sequentially applying the spatial and temporal models. Our method instead derives a two-stream model using the objective in Eq. (5). Both the baseline and our method are trained using the Vimeo-90K training set.

In the following, we compare our method to the baseline under different spatial and temporal module combinations. To measure the benefit of the proposed consistency loss LC, we also compare the results with and without using this loss. To analyze the proposed semi-supervised spatiotemporal SR, we perform the evaluation with various ratios of labeled data to the whole training set and different numbers of unlabeled data. Finally, we conduct the sensitivity analysis of hyperparameter λ, and assess the effect of adopting cross-stream consistency loss LC.

Figure 5: Frames synthesized by the baseline B and our method F using different amounts of labeled data.

Figure 6: Synthesized frames and the performance with different numbers of unlabeled training data.

Comparisons with the baseline. We first verify whether the proposed network architecture has better performance by jointly learning the spatial and temporal SR modules across network streams. Table 1 reports the individual and the average qualities of spatiotemporal SR from the two network streams, FS→T and FT→S, on four datasets. The results in Table 1 show that our proposed method achieves consistent performance gains over different module combinations and datasets. Figure 3 displays some spatiotemporal SR results synthesized by the baseline and the proposed method. Both the qualitative and quantitative results indicate that our method employing the two-stream network to correlate the spatial and temporal modules can produce remarkably better and sharper synthesized frames which contain less visual artifacts and exhibit characteristics more similar to the ground truth frames.

Cross-stream consistency loss LC. We verify if the crossstream consistency loss LC improves spatiotemporal SR under supervised and semi-supervised settings. Unless further specified, our method using VDSR and DVF as the spatial and temporal SR modules respectively is evaluated in the following experiments. Figure 4(e) shows the performance with and without LC under different ratios of the labeled training data to the whole training set, from 0.1% to 100%. The results confirm that this loss LC consistently improves the performance with different amounts of labeled training data.

Figure 4 visualizes the SR results with and without using LC in the case of using 100% labeled training data. As shown in Figure 4(c) and 4(d), the absence of LC leads to blurrier synthesized frames no matter which network stream, FS→T or FT→S, is used for prediction. We also notice that FS→T produces worse results. We consider that in this case of scenes with less texture and high motion, applying the spatial SR module first in stream FS→T cannot recover useful HR details. Instead, firstly applying the temporal SR in FT→S reveals this issue. However, which module should be applied first is unknown in general. As shown in Figure 4(a) and 4 (b), our method with the aid of LC can alleviate the above problems to produce consistent results of two streams by enforcing the consistence between the two streams.

Two-stream network. We explore the impact of the ratio of labeled data to unlabeled data in semi-supervised learning. We randomly select 2,048 samples from Vimeo-90K as the training set, in which k% of these samples are labeled while the rest are unlabeled where k ∈ {10, 20, ..., 100}. Figure 5(e) shows that our method with two network streams achieves much better performance than the baseline with a single stream with various ratios. It is notable that our approach can use much less labeled data to reach the same performance by the baseline. Figure 5(b) and 5(c) show that our method with 20% of labeled data produces sharper and better frames than the baseline using 100% of labeled data.

Effect of using unlabeled samples. To investigate the impact of unlabeled samples in semi-supervised spatiotemporal SR, we fix the number of labeled data to 32, and evaluate the performance of our method with different numbers of unlabeled data. Figure 6(e) shows that using the proposed cross-stream consistency loss to exploit unlabeled data improves the performance especially when more unlabeled data are provided. The visualization example in Figure 6 reveals that the synthesized frames become sharper and clearer when the number of unlabeled data increases.

Hyperparameter λ. We measure the effect of the hyperparameter λ, which controls the relative importance of the cross-stream consistency loss LC to the reconstruction loss LR in Eq. (5). We randomly select 512 samples from Vimeo90K as the training set, in which 32 of these samples are labeled while the rest are unlabeled. Figure 8 reports the performance of our method with different values of hyperparameter λ. It can be observed that the loss function LC is crucial, since the performance gain by changing λ from zero to a positive value is significant. Once the value of λ is larger than a threshold, say 8 ∼ 10 in this case, the performance decreases instead. The curve in Figure 8 is smooth. In addition, a broader range of the λ value results in the improved performance, which implies that finding a suitable value of λ to get performance gain is not difficult.

4.3 Comparisons with ExistingMethods

The literature of spatiotemporal SR is limited. We compare our method with two existing methods: One is examplebased method [Shahar et al., 2011] and the other is a deeplearning-based method [Sharma et al., 2017], called coupled deep convolutional auto-encoder (CDCA). For comparison with CDCA, we re-implement the approach. Note that the quantitative results of the method [Shahar et al., 2011] are not available. We compare our method with that in [Shahar et al., 2011] on the video data provided in its project website 1 , and show qualitative comparison in the supplementary materials and Figure 7, where the proposed approach can produce more accurate and sharper frames, such as the regions of stars on the flag. Table 2 and Figure 9 show that our method performs favorably against CDCA on all the four datasets, and our method achieves the state-of-the art performance. CDCA ignores motion correspondences between consecutive lowresolution frames, leading to more artifacts especially in the regions with large motion. In contrast, our method leverages jointly the temporal SR module to smooth content transition and the spatial SR module to recover more high-resolution details, resulting in realistic and high-quality frames. About the inference time with the input two images of size 64×112 and the output image of size 256 × 448, our method takes about 60ms, CDCA takes 18ms. Although CDCA is more efficient, our method significantly outperforms CDCA.

1 http://www.wisdom.weizmann.ac.il/∼vision/SingleVideoSR. html

5 Conclusions

We present the first end-to-end trainable network for spatiotemporal SR. By exploiting cross-stream consistency to jointly train spatial and temporal SR modules, the proposed approach allows the two modules to share the supervisory signals and benefit each other. In addition, the cross-stream consistency loss enables semi-supervised learning, and can guide network training in a self-taught manner with unlabeled data. Both quantitative and qualitative results show that our method performs favorably against the existing methods, and achieving the state-of-the-art performance.

Acknowledgements

This work was supported in part by the Ministry of Science and Technology (MOST) under grants MOST 1072628-E-009-007-MY3, MOST 109-2634-F-007-013, MOST 107-2628-E-001-001-MY3, and MOST 108-2218-E-002048, and by Qualcomm through a Taiwan University Research Collaboration Project.