Self-Supervised Video Representation Learning with Constrained Spatiotemporal Jigsaw

1 Introduction

Self-supervised learning (SSL) has achieved tremendous successes recently for static images [He et al., 2020; Chen et al., 2020] and shown to be able to outperform supervised learning on a wide range of downstream image understanding tasks. However, such successes have not yet been reproduced for videos. Since different SSL models differ mostly on the pretext tasks employed on the unlabeled training data, designing pretext tasks more suitable for videos is the current focus for self-supervised video representation learning [Han et al., 2020a; Wang et al., 2020].

∗Corresponding author.

Spatiotemporal analysis is the key to many video content understanding tasks. A good video representation learned from the self-supervised pretext task should capture discriminative information jointly along both spatial and temporal dimensions. It is thus somewhat counter-intuitive to note that most existing SSL pretext tasks for videos do not explicitly require joint spatiotemporal video understanding. For example, some spatial pretext tasks have been borrowed from images without any modification [Jing et al., 2018], ignoring the temporal dimension. On the other hand, many recent video-specific pretext tasks typically involve speed or temporal order prediction [Lee et al., 2017; Benaim et al., 2020], i.e., operating predominately along the temporal axis.

A natural choice for a spatiotemporal pretext task is to solve 3D jigsaw puzzles, whose 2D counterpart has been successfully used for images [Noroozi and Favaro, 2016]. Indeed, solving 3D puzzles requires the learned model to understand spatiotemporal continuity, a key step towards video content understanding. However, directly solving a 3D puzzle turns out to be intractable: a puzzle of 3×3×3 pieces (the same size as a Rubik's cube) can have 27! possible permutations. Video volume even in a short clip is much larger than that. Nevertheless, the latest neural sorting model [Paumard et al., 2020; Du et al., 2020] can only handle permutations a few orders of magnitude less, so offer no solution. This is hardly surprising because such a task is daunting even for humans: Most people would struggle with a standard Rubik's cube, let alone a much larger one.

In this paper, we propose a novel Constrained Spatiotemporal Jigsaw (CSJ) pretext task for self-supervised video representation learning. The key idea is to form 3D jigsaw puzzles in a constrained manner so that it becomes solvable. This is achieved by factorizing the permutations into the three spatiotemporal dimensions and then applying them sequentially. This ensures that for a given video clip, large continuous spatiotemporal cuboids exist after the constrained shuffling to provide sufficient cues for the model to reason about spatiotemporal continuity (see Figure 1(b)(c)). Such large continuous cuboids are also vital for human understanding of video as revealed in neuroscience and visual studies [Stringer et al., 2006; Chen et al., 2019]. Even with the constrained puzzles, solving them directly could still be extremely hard. Consequently, instead of directly solving the puzzles (i.e., recovering the permutation matrix so that each piece can be put back), four surrogate tasks are carefully designed. They are more solvable but meanwhile still ensure that the learned representation is sensitive to spatiotemporal continuity at both the local and global levels. Concretely, given a video clip shuffled with our constrained permutations, we make sure that the top-2 largest continuous cuboids (LCCs) dominate the clip volume. The level of continuity in the shuffle clip as a whole is thus determined mainly by the volumes of these LCCs, and whether they are at the right order (see Fig. 1(d)(e)) both spatially and temporally. Our surrogate tasks are thus designed to locate these LCCs and predict their order so that the model learned with these tasks can be sensitive to spatiotemporal continuity both locally and globally.

Our main contributions are three-fold: (1) We introduce a new pretext task for self-supervised video representation learning called Constrained Spatiotemporal Jigsaw (CSJ). To our best knowledge, this is the first work on self-supervised video representation learning that leverages spatiotemporal jigsaw understanding. (2) We propose a novel constrained shuffling method to construct easy 3D jigsaws containing large LCCs. Four surrogate tasks are then formulated in place of the original jigsaw solving tasks. They are much more solvable yet remain effective in learning spatiotemporal discriminative representations. (3) Extensive experiments show that our approach achieves state-of-the-art on two downstream tasks across various benchmarks.

2 RelatedWork

Self-supervised Learning with Pretext Tasks. According to the transformations used by the pretext task, existing SSL methods for video presentation learning can be divided into three groups: (1) Spatial-Only Transformations: Derived from the original image domain [Gidaris et al., 2018], [Jing et al., 2018] leveraged the spatial-only transformations for selfsupervised video presentation learning. (2) Temporal-Only Transformations: [Misra et al., 2016; Lee et al., 2017] obtained shuffled video frames with the temporal-only transformations and then distinguished whether the shuffled frames are in chronological order. [Xu et al., 2019] chose to shuffle video clips instead of frames. [Benaim et al., 2020; Yao et al., 2020] exploited the speed transformation via determining whether one video clip is accelerated. (3) Spatiotemporal Transformations: There are only a few recent approaches [Ahsan et al., 2019; Kim et al., 2019] that leveraged both spatial and temporal transformations by permuting 3D spatiotemporal cuboids.

However, due to the aforementioned intractability of solving the spatiotemporal jigsaw puzzles, they only leveraged either temporal or spatial permutations as training signals, i.e., the crops are extracted from a 4-cell grid of shape 2x2x1 or 1x1x4. Therefore, no true spatiotemporal permutations have been considered in [Ahsan et al., 2019; Kim et al., 2019]. In contrast, given that both spatial appearances and temporal relations are important cues for video representation learning, the focus of this work is on investigating how to exploit the 3D permutations jointly for self-supervised video presentation learning, i.e., if we follow the statement in [Kim et al., 2019], 16 crops are actually sampled from a grid of shape 2x2x4. To that end, our CSJ presents the first spatiotemporal continuity based pretext task for video SSL, thanks to a novel constrained 3D jigsaw and four surrogate tasks to reason about the continuity in the 3D jigsaw puzzles without solving them directly.

Self-supervised Learning with Contrastive Learning. Contrastive learning is another self-supervised learning approach that has become increasingly popular in the image domain [He et al., 2020; Chen et al., 2020]. Recently, it has been incorporated into video SSL as well [Han et al., 2020b]. Contrastive learning and transformation based pretext tasks are orthogonal to each other and often combined in that different transformed versions of a data sample form the positive set used in contrastive learning. Recent works [Han et al., 2019; 2020a; Zhuang et al., 2020] leveraged features from the future frame embeddings or with the memory bank. They modeled spatiotemporal representations using only contrastive learning without transformations. Contrastive learning is also exploited in one of our surrogate pretext tasks. Different from existing works, we explore the spatiotemporal transformations in the form of CSJ and employ contrastive learning to distinguish different levels of spatiotemporal continuity in shuffled jigsaws. This enables us to learn more discriminative spatiotemporal representations.

3 Constrained Spatiotemporal Jigsaw

3.1 ProblemDefinition

The main goal of self-supervised video representation learning is to learn a video feature representation function f(·) without using any human annotations. A general approach to achieving this goal is to generate a supervisory signal y from an unlabeled video clip x and construct a pretext task P to predict y from f(x). The process of solving the pretext task P encourages f(·) to learn discriminative spatiotemporal representations from videos.

The pretext task P is constructed by applying a transformation function t(·; θ) parameterized by θ and then automatically deriving y from θ, e.g., y can be the type of the transformation. Based on this premise, P is defined as the prediction of y using the feature map of the transformed video clip f(˜ x), i.e., P : f(˜ x) → y, where ˜ x = t(x; θ). For example, in [Lee et al., 2017], t(·; θ) denotes a temporal transformation that permutes the four frames of video clip x in a temporal order θ, ˜ x = t(x; θ) is the shuffled clip, and the pseudo-label y is defined as the permutation order θ (e.g., 1324, 4312, etc.). The pretext task P is then a classification problem of 24 categories because there are 4! = 24 possible orders.

3.2 Constrained Permutations

Solving spatiotemporal video jigsaw puzzles seems to be an ideal pretext task for learning discriminative representation as it requires an understanding of spatiotemporal continuity. After shuffling the pixels in a video clip using a 3D permutation matrix, the pretext task is to recover the permutation matrix. However, as explained earlier, this task is intractable given even moderate video clip sizes. Our solution is to introduce constraints on the permutations. As a result, a new pretext task PCSJ based on Constrained Spatiotemporal Jigsaw (see Fig. 2(a)) is formulated, which is much easier to solve than a random/unconstrained jigsaw.

Specifically, our goal is to introduce constraints to the permutations so that the resultant shuffled video clip is guaranteed to have large continuous cuboids (see Fig. 2(a)). Similar to humans [Stringer et al., 2006], having large continuous cuboids is key for a model to understand a 3D jigsaw and therefore to have any chance to solve it. Formally, the volume of a shuffled video clip ˜ x are denoted as {T,H,W}, measuring its sizes along the temporal, height, and width dimensions, respectively. A cuboid is defined as a crop of ˜ x: c = ˜ xt1:t2,h1:h2,w1:w2 , where t1, t2 ∈ {1, 2, . . . , T}, h1, h2 ∈ {1, 2, . . . ,H},w1,w2 ∈ {1, 2, . . . ,W}. If all the jigsaw pieces (smallest video clip unit, e.g.a pixel or a 3D pixel block) in c keep the same relative order as they were in x (before being shuffled), we call the cuboid c as a continuous cuboid ccont . The cuboid's volume equals (t2 - t1) × (h2 - h1) × (w2 - w1), and the largest continuous cuboid (LCC) ccont max is the ccont with the largest volume.

We introduce two permutation strategies to ensure that the volumes of LCCs are large in relation to the whole video clip volume after our shuffling transformation t(·; θCSJ). First, instead of shuffling x in three spatiotemporal dimensions simultaneously, t(·; θCSJ) factorizes the permutations into the three spatiotemporal dimensions and then utilizes them sequentially to generate shuffled clips, e.g., in the order of T,W,H and only once. Note that the volume of the generated ˜ x stays the same with different permutation orders (e.g., TWH and HTW). Second, we shuffle a group of jigsaw pieces together instead of each piece individually along each dimension.

Given that a video clip has eight frames, taking the temporal dimension as an example, we make an arbitrarily grouped permutation. θCSJ could be represented as the permutation from {12345678} to {84567123}. The longest and the second-longest index ranges are: [2, 5] for coordinates {4567}, and [6, 8] for coordinates {123}. Note that we make 3 groups of frames only as an example. Since the permutations are not deterministic, in practice, different numbers of groups (e.g.., 3, 4, or 5) will be selected for different video clips. With these two permutation strategies, not only do we have large LCCs, but also they are guaranteed to have clearly separable boundaries (see Fig. 2(b)) with surrounding pieces due to the factorized and grouped permutation design. This means that they are easily detectable.

3.3 Surrogate Tasks

Having permutation constraints preserves more spatiotemporal continuity in the shuffled clip and reduces the amount of possible permutations. But exploiting them to make a neural sorting model tractable is still far from trivial. Instead of solving the jigsaw directly, our PCSJ is thus formulated as four surrogate tasks: Largest Continuous Cuboid Detection (LCCD), Clip Shuffling Pattern Classification (CSPC), Contrastive Learning over Shuffled Clips (CLSC), and Clip Continuity Measure Regression (CCMR). As illustrated in Fig. 2(b), given an unlabeled clip x, we first construct a minibatch of 8 clips {˜ x1, ˜ x2, ..., ˜ x8} by shuffling x with different but related constrained permutations (to be detailed later). These shuffled clips and the raw clip x are then fed into a 3D CNN model f(·) for spatiotemporal representation learning with a non-local operation [Wang et al., 2018]:

where NL(·, ·) denotes the non-local operator, and f(˜ xi) and f(x) denote the feature map of ˜ xi and x from the last convolutional layer of f(·), respectively. The resultant feature map fNL(˜ xi) is further passed through a spatial pooling layer followed by a separately fully-connected layer for each surrogate task. Note that the raw video feature map f(x) is used as guidance through the non-local based attention mechanism to help fulfill the tasks. This is similar to humans needing to see the completed jigsaw picture to help solve the puzzle.

We first explain how the eight permutations from the same raw clip are generated. First, the factorized and grouped permutations are applied to x to create one shuffled clip. By examining the largest and the second-largest continuous puzzle piece numbers of each dimension ({T, H, W}), we can easily identify the top-2 largest continuous cuboids (LCCs). Next, by varying the relative order of the top-2 LCCs either in the correct (original) order or the reverse order in each dimension, 2×2×2=8 permutations are obtained. By controlling the group size in permutation, we can make sure that the top2 LCCs account for a large proportion, saying 80% of the total clip volume. Our four tasks are thus centered around these two LCCs as they largely determine the overall spatiotemporal continuity of the shuffled clip.

The first task LCCD is to locate the top-2 LCCs {ccont max (j) : j = 1, 2} and formulated as a regression problem. Given a ground-truth LCC ccont max (j), a Gaussian kernel is applied to its center to depict the possibility of each pixel in ˜ x belonging to the LCC. This leads to a soft mask M j LCCD with the same size of ˜ x: M j LCCD is all 0 outside the region of ccont max (j), and exp(- ||a-ac||2 2σ2 g ) inside the region, where a, ac denote any pixel and the center point, respectively. σg is the hyperparameter which is set as 1 empirically. In the training stage, FPN [Lin et al., 2017] is used for multi-level feature fusion. LCCD is optimized using the MSE loss in each point:

where MSE(·, ·) denotes the MSE loss function, and M j LCCD(a)′ is the prediction of each pixel a.

CSPC is designed to recognize the shuffling pattern of a shuffled clip. As mentioned early, the eight shuffled clips in each mini-batch are created from the same raw clip and differ only in the relative order of the top-2 LCCs along each of the three dimensions. There are thus eight permutations depending on the order (correct or reverse) in each dimension. Based on this understanding, CSPC is formulated as a multi-class classification task to recognize each shuffled clip into one of these eight classes, which is optimized using the Cross-Entropy (CE) loss:

where CE(·, ·) is the CE loss function and l′CSPC [i] is the predicted class label of i-th shuffled clip in each mini-batch.

The two tasks above emphasize on local spatiotemporal continuity understanding. In contrast, CLSC leverages the contrastive loss to encourage global continuity understanding. In particular, since the top-2 LCCs dominate the volume of a clip, it is safe to assume that if their relative order is correct in all three dimensions, the shuffled clip largely preserve continuity compared to the original clip, while all other 7 permutations feature large discontinuity in at least one dimension. We thus form a contrastive learning task with the original video x and the most continuous shuffled video ˜ xi as a positive pair, and x and the rest ˜ xj (j ̸= i) as negative pairs. CLSC is optimized using the Noise Contrastive Estimation (NCE) [Oord et al., 2018] loss:

where sim(·, ·) is defined by the dot product: f(x)⊤f(˜ xi), and τ is the temperature hyper-parameter. Note that the nonlocal operator is not used in CLSC.

CCMR is similar to CLSC in that it also enforces global continuity understanding, but differs in that it is a regression task aimed at predicting a global continuity measure. We consider two such measures. Since the total size of the top-2 LCCs {ccont max (j) : j = 1, 2} is a good indicator of how continuous a shuffle video clip is, the first measure lld directly measures the relative total size of the top-2 LCCs: lld = v(ccont max (1))+v(ccont max (2)) v(˜ x) , where v(·) represents the volume of a clip/cuboid. The second measure lt/h/w hd examines the shuffling degree of ˜ x in each dimension, computed as the normalized hamming distance: hamming(˜ x) Nc(Nc-1)/2 , where hamming(·) denotes the hamming distance in each dimension between the original piece sequence and the permuted one, and Nc represents the number of pieces in each dimension so that Nc(Nc -1)/2 indicates the maximum possible hamming distance in the dimension. CCMR is optimized using the Mean Squared Error (MSE) loss:

where l′ ld , lt′ hd , lh′ hd , lw′ hd are the prediction of the model.

3.4 Overall Learning Objective

Our entire CSJ framework is optimized end-to-end with the learning objective defined as:

where σ1, σ2, σ3, σ4 are the weights for these losses. We deploy the adaptive weighting mechanism [Kendall et al., 2018] to weight these tasks, resulting no free hyper-parameters to tune. We also adopt curriculum learning [Bengio et al., 2009] to train our network by shuffling clips from easy to hard.

4 Experiments

4.1 Datasets and Settings

We select three benchmark datasets for performance evaluation: UCF101 [Soomro et al., 2012], HMDB51 [Kuehne et al., 2011], and Kinetics-400 (K400) [Kay et al., 2017], containing 13K/7K/306K video clips from 101/51/400 action classes, respectively. In the self-supervised pre-training stage, we utilize the first training split of UCF101/HMDB51 and the training split of K400 without using their labels. As in [Han et al., 2020a], we adopt R2D3D as the backbone network, which is modified from R3D [Hara et al., 2018] with fewer parameters. By fine-tuning the pre-trained model, we can evaluate the SSL performance on a downstream task (i.e., action classification). Following [Han et al., 2019; He et al., 2020], two evaluation protocols are used: comparisons against state-of-the-arts follow the more popular fully fine-tuning evaluation protocol, but ablation analysis takes both the linear evaluation and fully fine-tuning protocols. For the experiments on supervised learning, we report top-1 accuracy on the first test split of UCF101/HMDB51 as the standard [Han et al., 2020a].

4.2 Implementation Details

Raw videos in these datasets are decoded at a frame rate of 24-30 fps. From each raw video, we start from a randomly selected frame index and sample a consecutive 16-frame video clip with a temporal stride of 4. For data augmentation, we first resize the video frames to 128×171 pixels, from which we extract random crops of size 112×112 pixels. We also apply random horizontal flipping to the video frames during training. Random color jittering is utilized to avoid learning shortcuts.We exploit only the raw RGB video frames as input, and do not leverage optical flow or other auxiliary signals for self-supervised pre-training.

4.3 Model Evaluations

Results of Directly Solving CSJ. We first demonstrate the results of solving the CSJ task directly in Table 1. We randomly shuffle video clips into 4 × 4 × 4 jigsaw puzzles. To recognize the correct permutation, the model solve a (4! ×4! ×4!)-way classification task in the pre-training stage.

We compare the CSJ task with the joint LCCD+CCMR task under the same setting for fair comparison. Linear evaluation is adopted to show the effectiveness of different tasks. We can observe from the table that solving LCCD+CCMR jointly is more effective than solving CSJ directly.

Ablation Study. We conduct ablative experiments to validate the effectiveness of four CSJ surrogate tasks and two additional learning strategies. From Table 2, we can observe that: (1) Self-supervised learning with each of the four tasks shows better generalization than fine-tuning the network from scratch (random initialization). (2) By training over all the four tasks jointly, we can achieve large performance gains (see '+LCCD' vs. 'CCMR'). (3) Each additional learning strategy (i.e., adaptive weighting or curriculum learning) leads to a small boost to the performance by 0.3-0.5%. (4) Our full model achieves a remarkable classification accuracy of 70.4%, demonstrating the effectiveness of our proposed CSJ with only the RGB video stream (without additional optical flow, audio, or text modalities).

Visualization of LCCD Predictions. We also demonstrate the visualization of the LCCD predictions from the pretrained models in Fig. 3. We can observe that solving the LCCD task indeed enables the model to learn the locations of LCCs and understand spatiotemporal continuity, which is a key step towards video content understanding.

4.4 Main Results

Comparison in Action Recognition. A standard way to evaluate a self-supervised video representation learning model is to use it to initialize an action recognition model on a small dataset. Specifically, after self-supervised pre-training on UCF101/HMDB51/K400, we exploit the learned backbone for fully fine-tuning on UCF101 and HMDB51, following [Han et al., 2020a; Wang et al., 2020]. We consider one baseline: fully-supervised learning with pre-training on K400. Note that this baseline is commonly regarded as the upper bound of self-supervised representation learning [Alwassel et al., 2020]. From Table 3, we have the following observations: (1) Our CSJ achieves state-of-the-art performance on both UCF101 and HMDB51. Particularly, with the backbone R2D3D-18 that is weaker than R(2+1)D-18, our CSJ performs comparably w.r.t. Pace on UCF101 but achieves a 10% improvement over Pace on HMDB51. (2) By exploiting spatiotemporal transformations for self-supervised representation learning, our CSJ beats either methods with only temporal transformations (†) or methods with both spatial and temporal transformations (‡), as well as those learning spatiotemporal representations (∗) via only contrastive learning (w./o. spatiotemporal transformations). (3) Our CSJ also outperforms CBT [Sun et al., 2019], which used ten-times more massive datasets (K600 + Howto100M) and multiple modalities (RGB+Audio). (4) Our CSJ is the closest to the fullysupervised one (upper bound), validating its effectiveness in self-supervised video representation learning.

Comparison in Video Retrieval. We evaluate our CSJ method in the video retrieval task. Following [Xu et al., 2019], we extract each video clips' embeddings with the pre-training model and use each clip in the test set to query the k nearest clips in the training set. The comparative results in Table 4 show that our method outperforms all other self-supervised methods and achieves new state-of-the-art in video retrieval on UCF101. Particularly, our method beats the latest competitor PRP [Yao et al., 2020] on four out of five metrics. This indicates that our proposed CSJ is also effective for video representation learning in video retrieval.

5 Conclusion

We have introduced a novel self-supervised video representation learning method named Constrained Spatiotemporal Jigsaw (CSJ). By introducing constrained permutations, our proposed CSJ is the first to leverage spatiotemporal jigsaw in self-supervised video representation learning. We also propose four surrogate tasks based on our constrained spatiotemporal jigsaws. They are designed to encourage a video representation model to understand the spatiotemporal continuity, a key building block towards video content analysis. Extensive experiments were carried out to validate the effectiveness of each of the four CSJ tasks and also show that our approach achieves the state-of-the-art on two downstream tasks.

Acknowledgements

This work was supported in part by National Natural Science Foundation of China (61976220 and 61832017), Beijing Outstanding Young Scientist Program (BJJWZYJH012019100020098), and Open Project Program Foundation of Key Laboratory of Opto-Electronics Information Processing, CAS (OEIP-O-202006).