Deep Optical Flow Estimation Via Multi-Scale Correspondence Structure Learning

1 Introduction

Optical flow estimation seeks for perceiving the motion information across consecutive video frames, and has a wide range of vision applications such as human action recognition and abnormal event detection. Despite the significant progress in the literature, optical flow estimation is still confronted with a number of difficulties in discriminative feature representation, correspondence structure modeling, computational flexibility, etc. In this paper, we focus on how to set up an effective learning pipeline that is capable of performing multiscale correspondence structure modeling with discriminative feature representation in a flexible end-to-end deep learning framework.

∗Corresponding author

Due to the effectiveness in statistical modeling, learning based approaches emerge as an effective tool of optical flow estimation [Dosovitskiy et al., 2015; Jason et al., 2016; Ahmadi and Patras, 2016; Zhu et al., 2017]. Usually, these approaches either just take image matching at a single scale into account, or take a divide-and-conquer strategy that copes with image matching at multiple scales layer by layer. Under the circumstances of complicated situations (e.g., large inter-image displacement or complex motion), they are often incapable of effectively capturing the interaction or dependency relationships among the multi-scale inter-image correspondence structures, which play an important role in robust optical flow estimation. Furthermore, their matching strategies are often carried out in the following two aspects. 1) Set a fixed range of correspondence at a single scale in the learning process [Dosovitskiy et al., 2015; Jason et al., 2016; Zhu et al., 2017]; and 2) update the matching range dynamically with a coarse-to-fine scheme [Ahmadi and Patras, 2016; Ranjan and Black, 2016]. In practice, since videos have timevarying dynamic properties, selecting an appropriate fixed range for matching is difficult for adapting to various complicated situations. Besides, the coarse-to-fine scheme may cause matching error propagations or accumulations from coarse scales to fine scales. Therefore, for the sake of robust optical flow estimation, correspondence structure modeling ought to be performed in an adaptive multi-scale collaborative way. Moreover, it is crucial to effectively capture the cross-scale dependency information while preserving spatial self-correlations for each individual scale in a totally datadriven fashion.

Motivated by the above observations, we propose a novel unified end-to-end optical flow estimation approach called Multi-Scale Correspondence Structure Learning (MSCSL) (as shown in Fig. 1), which jointly models the dependency of multi-scale correspondence structures by a Spatial Conv-GRU neural network model based on multi-level deep learning features. To summarize, the contributions of this work are twofold:

• We propose a multi-scale correspondence structure learning approach, which captures the multi-scale interimage-correlation correspondence structures based on the multi-level deep learning features. As a result, the task of optical flow estimation is accomplished by jointly learning the inter-image correspondence structures

at multiple scales within an end-to-end deep learning framework. Such a multi-scale correspondence structure learning approach is innovative in optical flow estimation to the best of our knowledge.

• We design a Spatial Conv-GRU neural network model to model the cross-scale dependency relationships among the multi-scale correspondence structures while preserving spatial self-correlations for each individual scale in a totally data-driven manner. As a result, adaptive multiscale matching information fusion is enabled to make optical flow estimation adapt to various complicated situations, resulting in robust estimation results.

2 Our Approach

2.1 ProblemFormulation

Let {(Xi,Oi)}Ni=1 be a set of N training samples, where Xi = (XAi ∈ RH×W×3 ,XBi ∈ RH×W×3 ) and Oi ∈ RH×W×2 represent an RGB image pair and the corresponding optical flow respectively. In this paper, our objective is to learn a model f(Xi; θ) parameterized by θ to predict the dense motion of the first image XAi . For the sake of expression, we ignore the left subscript i in the remaining parts.

In this paper, we focus on two factors, (1) computing the correlation maps between image representations at different scales and adaptively setting up the correspondence structure in a data-driven way, (2) encoding the correspondence maps into high-level feature representation for regressing the optical flow.

2.2 Multi-Scale Correspondence Structure Modelling

Multi-Scale Image Representations. To represent the input image at multiple scales, we firstly employ convolution neural networks (CNNs) to extract the deep features at a single scale parameterized by θ1 to represent the image I, as illustrated in Fig. 1:

and then model the multi-level feature representations parameterized by {θ2,l} with Y as the input, as depicted in Fig. 1:

where Fl represents the l-th level, and the size of Fl+1 is larger than that of Fl . From top to bottom (or coarse to fine), the feature representations at small scales 1 tend to learn the sematic components, which contribute to find the correspondence of semantic parts with large displacements; Furthermore, the large scale feature maps tend to learn the local representation, which can distinguish the patches with small displacements. In this paper, we use {FA,l } and {FB,l } to denote the multi-scale representations of XA and XB respectively.

Correspondence Structure Modelling. Given an image pair (XA,XB ) from a video sequence, we firstly extract their multi-level feature representations {FA,l } and {FB,l } using Eq. 1 and Eq. 2. In order to learn the correspondence structures between the image pair, we calculate the similarity between the corresponding feature representations instead. Firstly, we discuss the correlation computation proposed in [Dosovitskiy et al., 2015]:

1 In this paper, the small scale means small size; the large scale means large size

where FAi,j and FBi,j denote the feature vector at the (i, j)- th location of FA and FB respectively, and Concat{·} denotes concatenating the elements in the set {·} to a vector, S(i, j; d) denotes the (2d + 1) × (2d+1) neighborhood of location (i, j). The meaning of Eq. 3 is that given a maximum displacement d, the correlations between the location (i, j) in FA and S(i, j; d) in FB can be obtained by computing the similarities between the square patch of size (2k + 1) × (2k +1) centered at location (i, j) in FA and square patches of the same size centered at all locations of S in FB .

To model the correspondence between the (i, j)-th location in FA and its corresponding location (ˆi, ˆj) in FB , we can (1) calculate Corr(FAi,j ,FBS(i,j;d) ) in a small neighbourhood S of the (i, j)-th location in FB , or (2) calculate Corr(FAi,j ,FBS(i,j;d) ) in a large enough neighbourhood S of the (i, j)-th location in FB , or even in the whole feature map FB . But the former can not guarantee the computation of similarity between the (i, j)-th location and the corresponding (ˆi, ˆj)-th location, while the latter leads to low computational efficiency, because the complexity O(d2k2 ) of Eq. 3 exhibits quadratic growth when the value of d increases. To address that problem, we adopt correlation computation at each scale of multi-scale feature representations {FA,l } and {FB,l }:

where the maximum displacement dl varies from bottom to top.

In order to give the network more flexibility in how to deal with the correspondence maps, we add three convolutional layers to the outputs of the Correlation operation, which is the same as that proposed in [Dosovitskiy et al., 2015], to extract the high-level representations parameterized by {θ3,l}, as described in Fig. 1:

2.3 Correspondence Maps Encoding Using Spatial Conv-GRU

Cross-Scale Dependency Relationships Modelling. For the sake of combining the correlation representations {Ql } and preserving the spatial structure to estimate dense optical flow, we consider the representations as a feature map sequence, and then apply Convolutional Gated-Recurrent-Unit Recurrent Networks(Conv-GRUs) to model the cross-scale dependency relationships among the multi-scale correspondence structures. Conv-GRUs have been used to model the temporal dependencies between frames of the video sequence [Ballas et al., 2015; Siam et al., 2016]. A key advantage of ConvGRUs is that they can not only model the dependencies among a sequence, but also preserve the spatial location of each feature vector. One of significant differences between a Conv-GRU and a traditional GRU is that innerproduct operations are replaced by convolution operations.

However, because of the employed scheme similar to coarse-to-fine, the size of the (t+1)-th input in the sequence is larger than that of the t-th input. We cannot apply the standard Conv-GRU on our problem, so instead we propose a Spatial Conv-GRU in which each layer's output is upsampled as the input of the next layer. For the input sequence {Ql }, the formulation of the Spatial Conv-GRU is:

where ∗ and ⊙ denote a convolution operation and an element-wise multiplication respectively, and σ is an activation function, e.g., sigmoid., ⊛ denotes the transposed convolution. The Spatial Conv-GRU can model the transition from coarse to fine and recover the spatial topology, outputting intra-level dependency maps {Hl }.

Intra-Level Dependency Maps Combination. After getting the hidden outputs {Hl }, we upsample them to the same size, written as Pl :

where θ4 := {Wz,Uz,Wr,Ur,W,U,W↑ } are the parameters needed to be optimized. Furthermore, we concatenate the hidden outputs {Pl } with the 2nd convolutional output ˜YA of XA to get the final encoded feature representation for optical flow estimation, as depicted in Fig. 1:

where Concat represents the concatenation operation.

Finally, the proposed framework learns a function parameterized by θ5 to predict the optical flow:

2.4 Unified End-to-End Optimization

As the image representation, correspondence structure learning and correspondence maps encoding are highly related, we construct a unified end-to-end framework to optimize the three parts jointly. The loss function used in the optimization framework consists of two parts, namely, a supervised loss and an unsupervised loss (or reconstruction loss). The former is the endpoint error (EPE), which measures the Euclidean distance between the predicted flow ˆO and the ground truth O, while the latter is based on the brightness constancy assumption, which measures the Euclidean distance between the first image XA and the warped second image XBwarp .

where ˆOu and ˆOv denote the displacement in horizontal and vertical respectively, and λ is the balance parameter. XBwarp can be calculated via bilinear sampling according to ˆO, as proposed in Spatial Transform Networks[Jaderberg et al., 2015]:

Because the raw data XA and XB contain noise and illumination changes and are less discriminative, in some cases the brightness constancy assumption is not satisfied; Furthermore, in highly saturated or very dark regions, the assumption also suffers difficulties [Jason et al., 2016]. Therefore, applying Eq. 16 on the raw data directly will make the network more difficult when training. To address that issue, we apply the brightness constancy assumption on the 2nd convolutional outputs ˜YA and ˜YB of XA and XB instead of XA and XB . The training and test stages are shown in Alg. 1.

3 Experiments

3.1 Datasets

Flying Chairs [Dosovitskiy et al., 2015] is a synthetic dataset created by applying affine transformations to a real image dataset and a rendered set of 3D chair models. This dataset contains 22, 872 image pairs, and is split into 22, 232 training and 640 test pairs.

MPI Sintel [Butler et al., 2012] is created from an animated movie and contains many large displacements and provides dense ground truth. It consists of two versions: the Final version and the Clean version. The former contains motion blurs and atmospheric effects, while the latter does not include these effects. There are 1, 041 training image pairs for each version.

KITTI 2012 [Geiger et al., 2012] is created from real world scenes by using a camera and a 3D laser scanner. It consists of 194 training image pairs with sparse optical flow ground truth.

Middlebury [Baker et al., 2011] is a very small dataset, containing only 8 image pairs for training. And the displacements are typically limited to 10 pixels.

3.2 Implementation Details

Network Architecture

In this part, we introduce the network architecture briefly. We use 7×7 convolutional kernel for the first convolutional layer and 5×5 for the second and third convolutional layers. Then we use max-pooling and convolutional operations to obtain multi-scale representations, as illustrated in Fig. 1. The correlation layer is the same as that proposed in [Dosovitskiy et al., 2015], and the dl are set to 5, 5, 10, 10 from top to bottom (or from coarse to fine). And then we employ 3 × 3 kernel and 4×4 kernel for the other convolutional layers and deconvolutional layers respectively.

Data Augmentation

To avoid overfitting and improve the generalization of network, we employ the data augmentation strategy for the training by performing random online transformations, including scaling, rotation, translation, as well as additive Gaussian noise, contrast, multiplicative color changes to the RGB channels per image, gamma and additive brightness.

Training Details

We implement our architecture using Caffe [Jia et al., 2014] and use an NVIDIA TITAN X GPU to train the network. To verify our proposed framework, we conduct three comparison experiments, (1) MSCSL/wosr, this experiment does not contain both the proposed Spatial Conv-GRU and reconstruction loss, and use the refinement network proposed in [Dosovitskiy et al., 2015] to predict dense optical flow, (2) MSCSL/wor, this experiment employs the Spatial Conv-GRU, which can be implemented by unfolding the recurrent model in the prototxt file, to encode the correspondence maps for dense optical flow estimation and demonstrates the effectiveness of the Spatial Conv-GRU in comparison to MSCL/wosr, (3) MSCSL, this experiment contains all parts (Spatial ConvGRU and reconstruction loss) aforementioned.

In the MSCSL/wosr and MSCSL/wor, we train the networks on Flying Chairs training dataset using Adam optimization with β1 = 0.9 and β2 = 0.999. To tackle the gradients explosion, we adopt the same strategy as proposed in [Dosovitskiy et al., 2015]. Specifically, we firstly use a learning rate of 1e - 6 for the first 10k iterations with a batch size of 8 pairs. After that, we increase the learning rate to 1e-4 for the following 300k iterations, and then divide it by

2 every 100k iterations. We terminate the training after 600k iterations (about 116 hours).

In the MSCSL, we firstly train the MSCSL/wor for 500k iterations using the training strategy above. After that, we add the reconstruction loss with the balance parameter λ = 0.005. And then we fine-tune the network for 100k iterations with a fixed learning of 1.25e - 5.

After training the three networks on Flying Chairs training dataset respectively, we fine-tune the networks on the MPI Sintel training dataset for tens of thousands of iterations with a fixed learning rate of 1e - 6 until the networks converge. Specifically, we fine-tune the networks on the Clean version and Final version together with 1, 816 for training and 266 for validation. Since the KITTI 2012 dataset and Middlebury dataset are small and only contain sparse ground truth, we do not conduct fine-tuning on these two datasets.

3.3 Comparison to State-of-the-Art

In this section, we compare our proposed methods to recent state-of-the-art approaches, including traditional methods, such as EpicFlow [Revaud et al., 2015], DeepFlow [Weinzaepfel et al., 2013], FlowFields [Bailer et al., 2015], EPPM [Bao et al., 2014], LDOF [Brox and Malik, 2011], DenseFlow [Yang and Li, 2015], and deep learning based methods, such as FlowNetS [Dosovitskiy et al., 2015], FlowNetC [Dosovitskiy et al., 2015], SPyNet [Ranjan and Black, 2016]. Table 1 shows the performance comparison between our proposed methods and the state-of-the-art using average endpoint errors (EPE). We mainly focus on the deep learning based methods, so we only compare our proposed methods with the learning-based frameworks such as FlowNet and SpyNet.

Flying Chairs. For all three comparison experiments, We train our networks on this dataset firstly, and employ MPI Sintel dataset to fine-tune them further. Table 1 shows that MSCSL outperforms the other comparison experiments, MSCSL/wosr and MSCSL/wor. Furthermore, our proposed methods achieve better performance comparable with the state-ofthe-art methods. After fine-tuning, in most cases most learning based methods suffer from performance decay, this is mostly because of the disparity between Flying Chairs and MPI Sintel dataset. Some visual estimation results on this dataset are shown in Fig. 3.

MPI Sintel. After the training on Flying Chairs firstly, we fine-tune the trained models on this dataset. The models

Algorithm 1: Deep Optical Flow Estimation Via MSCSL

trained on Flying Chairs are evaluated on the training dataset. The results shown in Table 1 demonstrate MSCSL's and MSCSL/sor's better ability to generalize than MSCSL/wosr's and other learning based approaches'. To further verify our proposed methods, we compare our methods with FlowNetS, FlownetC and SPyNet on MPI Sintel test dataset for different velocities and distances from motion boundaries, as described in Table 2. As shown in Table 1 and Table 2, our proposed methods perform better than other deep learning based methods. However, in the regions with velocities larger than 40 pixels (smaller than 10 pixels), the proposed methods are less accurate than FlowNetC (SpyNet). Some visual results are shown in Fig. 2.

KITTI 2012 and Middlebury. These two datasets are too small, so we do not fine-tune the models on these datasets. We evaluate the trained models on KITTI 2010 training dataset, KITTI 2012 test dataset and Middlebury training dataset respectively. Table 1 shows that our proposed methods outperform other deep learning based approaches remarkably on the KITTI 2012 dataset (including training set and test set). However, in most cases, on Middlebury training dataset, mainly containing small displacements, our proposed methods do not perform well, comparison to SPyNet.

Analysis. The results of our framework are more smooth and fine-grained. Specifically, our framework is capable of capturing the motion information of fine-grained object parts, as well as preserving edge information. Meanwhile, our Spatial Conv-GRU can suppress the noises in the results of model without it. All these insights can be observed in Fig. 3 and Fig. 2. However, our proposed frameworks are incapable of effectively capturing the correspondence structure and unstable in regions where the texture is uniform (e.g., on Middlebury dataset).

Timings. In Table 1, we show the per-frame runtimes of different approaches. Traditional methods are often implemented on a single CPU, while deep learning based methods tend to run on GPU. Therefore, we only compare the runtimes with FlowNetS, FlowNetC and SPyNet. The results in Table 1 demonstrate that our proposed methods (run on NVIDIA TITAN X GPU) improve the accuracy with a comparable speed against the state-of-the-art.

4 Conclusion

In this paper, we propose a novel end-to-end multi-scale correspondence structure learning based on deep learning for optical flow estimation. The proposed MSCSL learns the correspondence structure and models the multi-scale dependency in a unified end-to-end deep learning framework. Our model outperforms the state-of-the-art approaches based on deep learning by a considerable computing efficiency. The experimental results on several datasets demonstrate the effectiveness of our proposed framework.

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant U1509206 and Grant 61472353, in part by the Alibaba-Zhejiang University Joint Institute of Frontier Technologies.