Self-Supervised Video Action Localization with Adversarial Temporal Transforms

1 Introduction

Temporal action localization (TAL) in untrimmed videos is one of the most challenging tasks in video understanding. In real-world applications, untrimmed videos usually contain multiple action instances and irrelevant background scenes. The aim of temporal action localization is to locate the temporal boundary and predict the action category for each action instance in a video. To learn an effective action localization model, most existing temporal action localization methods utilize fine-grained supervision, which requires manually annotated temporal boundaries and action category label for each action instance. However, labeling the temporal boundary of an action instance is time-consuming. In this paper, we focus on weakly-supervised temporal action localization, where only video-level action category labels are available.

∗Corresponding author.

To tackle the weakly-supervised TAL task, most existing works [Nguyen et al., 2018; Narayan et al., 2019; Yu et al., 2019] fall into a multiple-instance-learning framework. In specific, a video is treated as a bag of frames and fed into video-level classification networks. By enforcing the accuracy of video-level predictions, a weakly-supervised TAL model learns to generate a class activation sequence (CAS) for each video, which essentially indicates how likely each frame belongs to an action class [Shou et al., 2018]. Action instances can then be temporally localized based on the CAS. However, without supervision from temporal boundaries of actions, CAS is empirically observed to be often incomplete (tends to cover only the most discriminative part of the action) and noisy (often incorrectly activates background moments).

Figure 1 illustrates the motivation of this work. The performance of a weakly-supervised TAL model heavily relies on the quality of CAS. We aim to address a rarely-explored aspect in weakly-supervised TAL: self-supervision brought by temporal transforms on CAS. Given an untrimmed video that contains several actions, the resultant CAS can drastically change after applying some temporal transforms to a video (such as fast forwarding selected segment of the video while keeping other segments unchanged). It does not bring a severe issue for video-level prediction, since it is invariant to these transforms. However, the stability of CAS is crucial for precisely de-limiting the action temporal boundary. To obtain good performance, we would expect the CAS to be strongly equivariant with respect to numerous temporal transforms.

To mitigate the supervision gap between weakly and fully supervised temporal action localization, we propose a selfsupervised equivariant transform consistency constraint that minimizes the discrepancy between the CAS of the original / temporally-transformed videos. In particular, we design an action localization network with a Siamese architecture, comprised of two sub-networks with the identical design that read the feature sequences of the original / temporally-transformed videos, respectively. Without loss of generality, this work only investigates three types of temporal transforms including resize, window warp, and attention-based time warp. A policy network is learned to select a temporal transform at each iteration. Critically, the localization and policy networks shall operate adversarially to each other, similar to Generative Adversarial Networks (GAN). The localization network desires the two Siamese sub-networks return transform-equivariant class activation sequences, while the policy network selects a transform that potentially maximizes the inconsistency between original / temporally-transformed videos. In addition, to enhance the completeness of the TAL results, we also propose a self-refine module that utilizes both temporal and semantic contexts to generate more integral action instances.

The technical contributions of this work can be summarized as below: (1) To our best knowledge, this work represents the first attempt to explore self-supervised equivariant consistency regularization for weakly-supervised temporal action localization. To generate temporal transformed videos, we propose attention based time warp methods and design a policy network to select transform operations for each video; (2) We propose a self-refine module to explicitly leverage the temporal and semantic context information to obtain a more complete temporal interval of actions; (3) Comprehensive evaluations are conducted on two challenging video benchmarks: THUMOS14 and ActivityNet. Our method re-calibrates the state-of-the-art performance on both benchmarks by large margins.

2 RelatedWork

Weakly-supervised action localization. It learns to localize activities inside videos with only video-level action category labels available. Most of the relevant approaches adopt the multiple instance learning framework. UntrimmedNet [Wang et al., 2017a] proposes a selection module to rank clip proposals and locate action instances. CMCS [Liu et al., 2019] improves UntrimmedNet by using a multi-branch network with diversity loss to model action completeness. 3C-Net [Narayan et al., 2019] implies that multi-label center loss and action counting loss can be used to decrease intra-class variations and increase the separability of adjacent action instances. BM [Nguyen et al., 2019] proposes a background-aware loss to explicitly model background content. DGAM [Shi et al., 2020] proposes a generative attention mechanism to separate action and context frames. TSCN [Zhai et al., 2020] generates frame-level pseudo labels by combining the predictions of RGB and flow streams. Although these methods generally achieve promising results, they only process the video at a fixed temporal resolution, ignoring the considerable variation of action duration. More importantly, previous works have rarely studied the equivariant property between the temporal boundary of an action and temporal transforms.

Self supervised learning (SSL). It has received increasing attention, noting the potential of learning effective semantic feature representations without human annotations. Prior study has explored different kinds of self-supervision, such as prediction of the spatial context [Doersch et al., 2015], colorization [Zhang et al., 2016], and transition equivalence [Gidaris et al., 2018; Noroozi et al., 2017]. Recent SSL works on videos mainly focus on spatio-temporal orders of videos. For example, the frame order in videos is the main clue utilized in [Misra et al., 2016]. Other types of temporal context include its conjunction of spatial context [Wang et al., 2017b], as well as the use of statistics on spatio-temporal cooccurrence [Isola et al., 2015]. In all the works, [Wang et al., 2020] is most close to our work. It uses an attention mechanism to do self-supervised learning on semantic segmentation. However, the self-supervision used in [Wang et al., 2020] focuses on simple scale transforms in the spatial domain. Differently, our method operates temporally with learnable video transforms. We formulate it in an adversarial fashion by designing a policy network to select the most challenging temporal transform at each iteration. To our best knowledge, we are the first to explore SSL in the weaklysupervised TAL task.

3 Our Method

During training, we are provided with a training set of untrimmed videos V = {vi, yi}Mi=1 , where M is the number of videos, yi ∈ {0, 1}C denotes the video-level category label of video vi, C is the number of action categories. At test time, the output for each test video is a set of localized action instances (bj, ej, cj, qj), where bj and ej denote the start and end time, cj refers to the predicted action category, and qj is the confidence score.

3.1 Video Feature Extraction

To extract feature sequence, we first divide an untrimmed video into a set of snippets, each of which contains several consecutive frames. As in previous works [Narayan et al., 2019; Nguyen et al., 2019; Shi et al., 2020], the RGB and flow I3D [Carreira and Zisserman, 2017] models pretrained on Kinetics [Carreira and Zisserman, 2017] are utilized to extract two-stream video features for each snippet. Let X ∈ RT×D denote the RGB or flow feature sequence, where T denotes the count of snippets and D is the dimension of features.

3.2 Architecture Overview

As shown in Figure 2, we propose an action localization network with a Siamese design, containing two sub-networks Net1 and Net2 that share the same structure and parameters. The inputs of Net1 and Net2 are video feature sequences X, ˆX = o(X) extracted from a video and its temporallytransformed version, where o denotes a temporal transform operation. A policy network is included to choose one from a set of pre-defined temporal transform operations O.

For the Siamese net, let us take Net1 to describe the computational flow. The initial video feature X first goes through some additional temporal convolutions to better adapt to the current task, obtaining the embedding feature Xe. After that, Xe is fed to a classification branch and an attention branch, rendering features Xa and Xc respectively. Each branch is composed of a temporal convolution layer and a fully connected layer as the net head. The attention branch generates temporal attention weights A ∈ RT×1 . For the classification branch, its coarse output is sent to a self-refine (SR) module, getting the CAS S ∈ RT×C . Eventually, A and S are joined to predict the class probability p = softmax(AᵀS) in the video level. Similar treatment for Net2.

The learning of action localization network is guided by:

where Lcls, Ltrans cls are the classification losses of Net1 and Net2 respectively. Lcons is self-supervised transform consistency loss. α is a weighting coefficient. We will elaborate on the details of each term in following sections.

3.3 Self-refine Module

The classification branch generates the CAS for each snippet. However, the initial CAS tends to be incomplete or noisy. To remedy it, we propose a self-refine module that utilizes both temporal and semantic context information.

For a video v that contains T snippets, we first construct a fully connected graph G. Nodes in G are the T snippets of v. Let E ∈ RT×T be the adjacency matrix of G. Ei,j is the edge weight between node i and j. Intuitively, temporally neighboring snippets are more likely to have correlated content. We define the temporal weight Etem i,j of edge i, j as:

where σ is empirically set as T/10 in all experiments. For the semantic similarity of snippets i and j, we evaluate it by measuring the Cosine value:

where Xr = Conv1D(Xc) ∈ RT×D1 is an enhanced feature with learnable 1-D convolution.

The final inter-node similarity Ei,j is calculated by multiplication:

For numerical tractability, weights of the outbound edges of each node are normalized to be 1. After constructing the adjacency matrix E, a random walk routine is called to refine the coarse CAS S of the classification branch. In implementation, S is updated iteratively via S ←- E × S. We empirically find that an overall iteration of 3 times will strike a good balance of accuracy and computational complexity.

3.4 Self-supervised Equivariant Consistency

Human visual perception shows good consistency for temporal action localization tasks when they watch the video at different playback rates. For example, when we watch a video that contains a high jump action, the action category of the video will not change, yet the duration and temporal boundary of the high jump action will change as the playback rate varies. State differently, the action category of the video is invariant to different playback rates, while the temporal boundary of action is equivariant. To make CAS S have the equivariant property, this work proposes a self-supervised equivariant consistency loss Lcons.

Given a video v with action category c. Suppose the input video undergoes a temporal transform defined by o(·), namely ˆX = o(X) for the input video feature X. Recall that S, ˆ S are the final class activation sequences for the original / transformed videos. Critically, if the activation sequence S were equivariant with respect to o(·), there shall be high consistency with ˆ S. Let So = o(S) and we only slice the profile of the ground-truth class c, namely So c = softmax(So [:, c]), ˆ Sc = sof tmax( ˆ S[:, c]). Here the softmax is used to get the action class distribution along the temporal axis. The selfsupervised equivariant consistency loss Lcons is defined as:

where KL denotes the Kullback-Leibler divergence.

3.5 Adversarial Temporal Transform

To make the idea of self-supervision feasible, we first define a bank of candidate temporal transforms, which can be divided into the following categories:

1.Resize. Similar to the resize operation in image processing, the resize operation performs up-sampling or downsampling the entire video feature sequence along the time dimension uniformly.

• 2.Window warp. It divides the feature sequence into some non-overlapping sub-sequences. A sub-sequence will be chosen for either temporal stretching or contraction.

3.Attention based time warp. It re-samples the video feature map following a probability distribution induced by the temporal attention A.

To perform time warp, we first get the upsampled video feature Xup ∈ RNT×D and temporal attention Aup ∈ RNT×1 by performing linear interpolation on original X,A. N is a parameter to control the upsampling rate. We devise three different time-warp operators: (1) attention-enhanced time warp. (2) attention-complemented time warp. (3) attention gradient enhanced time warp. The attention enhanced time warp samples T locations from the upsampled NT locations. The i-th position is sampled with a probability disenh[i], where

The temporal interval highlighted by A is often the actionoccurring moment. disenh leads more dense sampling (similar to slow-motion playback) on action-related intervals and suppresses background segments (similar to fast-forwarding). In this way, we have more fine-grained view of action boundaries.

Likewise, the attention complemented time warp samples T locations following a probability discomp, where

The attention complemented time warp mainly samples temporal location that is overlooked by temporal attention A. A video usually contains multiple actions. The temporal attention usually focuses on the most discriminative part of a video, ignoring some action instances with short duration. Attention complemented time warp allows the network to mine more actions that are missed by current temporal attention.

The attention gradient enhanced time warp samples T locations from disga, where

Temporal locations with large gradients often belong to the boundary of the action. By intensive sampling nearby the boundary, the start and end time of the actions can be determined more accurately.

Inspired by GANs, we design a policy network to select a transform operation o from the transform bank O, which maximizes training loss of the localization network. In practice, the policy network consists of one temporal convolution layer and two fully-connected layers. Let θ denote the parameters of policy network. Taking video feature sequence as input, policy network output the distribution pθ(o) which defines a probability of operation o being selected. Formally, the goal of policy network is to maximize J(θ) = ∑o∈O pθ(o)r(o), where r(o) = Ltrans cls + αLcons is a reward corresponding to transform operation o.

3.6 Optimization

Since some temporal transform operations are nondifferentiable, it is intractable to train the localization / policy networks jointly. Instead, we train them by alternately executing the following steps: (1) Update the action localization network with loss L as in Eqn. 1. (2) Utilize the REINFORCE [Williams, 1992] algorithm to train the policy network.

3.7 Action Localization

During testing, only Net1 is required for action localization. Given a test video feature sequence X, we first use Net1 to obtain its CAS S and class probability distribution p. We utilize a two-stage threshold method to generate action localization results. First, we filter out action categories whose probabilities are below a threshold τ. For a remaining category c, we use a set of threshold values [α0, ..., αr] to threshold on S[:, c] respectively and generate localization proposals. Let (bi, ei, c, qi) denote the i-th proposal, where bi is the start time, ei is the end time, c is the action category and qi is the proposal score. As in [Liu et al., 2019], qi is calculated by:

where inner denotes the region (bi, ei), and outer denotes the surrounding region (bi - (ei - bi)/4, bi) ∪ (ei, ei + (ei - bi)/4). The category probability is combined with the weight γ. Non-maximum suppression(NMS) is used to remove duplicate proposals and generate the final localization results.

4 Evaluations

4.1 Data Description and Evaluation Protocol

To evaluate our method, we conduct experiments on two video benchmarks: THUMOS14 [Idrees et al., 2017] and ActivityNet [Heilbron et al., 2015].

THUMOS14. It contains untrimmed videos with temporal annotations from 20 action classes. There are 200 videos in the validation set and 212 videos in the testing set. Following the settings in previous works [Chao et al., 2018; Liu et al., 2019], we train our model on the validation set and evaluate on the test set.

ActivityNet. To facilitate comparisons, we conduct experiments on both ActivityNet-1.2 and ActivityNet-1.3. ActivityNet-1.3 contains about 20,000 videos from 200 activity classes. ActivityNet-1.2 is a subset of ActivityNet-1.3, which has videos from 100 activity classes. This dataset is divided into training, validation and testing sets with a ratio of 2:1:1. Since the annotations of testing set are withheld, we use the training set to train our model and evaluate on the validation set as in previous work [Paul et al., 2018; Narayan et al., 2019].

Evaluation protocol. We report the traditional mean Average Precision (mAP) at different temporal intersection over union (IoU) thresholds. The average mAP with IoU thresholds [0.5:0.95:0.05] is used to compare different methods on ActivityNet. On THUMOS14, the IoU thresholds are from 0.1 to 0.7 with a stride of 0.1.

4.2 Implementation Details

We implement our model in PyTorch. To extract video features, we utilize I3D [Carreira and Zisserman, 2017] models pre-trained on Kinetics [Carreira and Zisserman, 2017]. For each snippet, the corresponding optical flow is generated using the TV-L1 algorithm. The input of I3D is 16 stacked RGB or optical flow frames. The output is a 1024-D feature for each stream. Two separate models are trained for RGB and flow streams respectively. Then outputs of RGB and optical flow streams are combined by late fusion to generate the action localization results.

The action localization model is trained with batch size 24 and optimized by Adam. The learning rate of localization model is 0.001 on ActivityNet and 0.0001 on THUMOS14. The policy network is optimized by Adam with 0.0001 learning rate on ActivityNet and 0.00001 learning rate on THUMOS14. α in Eqn. 1 is 0.5. For action localization, classes whose video-level probabilities below 0.1 are filtered out. For the remaining class c, a set of threshold values ranging from [0.1 : 1.0 : 0.1] × mean(S[:, c]) is used to generate action proposals. γ is set to 0.1 when scoring proposals.

4.3 Comparisons with State-of-the-art

Table 1 summarizes comparisons with existing weaklysupervised and fully-supervised methods on THUMOS14. Our method outperforms other weakly-supervised methods when IoU varies from 0.1 to 0.6. Specifically, for mAP@0.5, our method improves the performance by 2.8%. Furthermore, the results achieved by our weakly-supervised method are comparable to the results obtained by several fully-supervised methods, indicating the effectiveness of our method.

Tables 2 and 3 present the results on benchmarks ActivityNet-1.2 and ActivityNet-1.3 respectively. On both versions of ActivityNet, our method significantly outperforms other state-of-the-art weakly-supervised methods in terms of average mAP. The performance of our method is slightly lower than other methods when IoU=0.95. Since the difference between actions surrounding context and action instances is small, temporal boundaries of actions have an intrinsic ambiguity. We observe that our method sometimes generates overly complete proposals containing surrounding context, leading to false positives when IoU is high. Nonetheless, we improve the average mAP on ActivityNet-1.3 from the previous state-of-the-art 22.5% to 26.0%.

4.4 Ablation Studies

To analyze how each component contributes to the overall performance, we conduct ablation studies on the THUMOS14 test set. We start with an action localization model without Siamese architecture and self-refine module. The model is trained only by Lcls. Adding Ltrans cls indicates the Siamese architecture is used, and the model is supervised by the classification loss of original and transformed videos. The self-supervised consistency loss Lcons is further included to validate the effectiveness of self-supervision. Finally, the self-refine module is added to get our full model.

Table 4 demonstrates the results by considering one more component at each stage. Adding Ltrans cls to the baseline model improves the mAP by 1.6%. Self-supervision consistency loss contributes a significant increase of 4.6%, demonstrating the effectiveness of the proposed equivariant constraint. The self-refine module further improves the mAP from 28.8% to 32.9%, showing that temporal and semantic context is critical to generate accurate CAS.

Table 5 shows the results of different temporal transform operations. To validate the effectiveness of adversarial temporal transformation, we compare the following five settings: 1) only use resize type of transform operations (Resize); 2) only use window warp type of transform operations (Window Warp); 3) only use attention based time warp type of transform operations (Attention Based Time Warp); 4) use all three type of transform operations, and randomly select one operation from all operations at each training step (Random Select); 5) use all three type of transform operations. The policy network is used to select one operation from all operations at each training step (Adversarial Select). The result shows that when single type of operation is used, the performance of attention based time warp methods is better than other two types, which proves its effectiveness. In specific, compared to Resize operation, attention based time warp improves the mAP from 27.8% to 31.1%. Compared with the random select, adversarial select improves the mAP by 1.6%, verifying the effectiveness of policy network. When using all types of operations, the performance is better than using single type of operation, indicating different types of operations are complementary.

5 Conclusions

This paper explores self-supervised equivariant consistency regulation for weakly-supervised temporal action localization. We design a set of temporal transform operations and utilize a policy network to select an operation in an adversarial manner. On THUMOS14 and ActivityNet, our methods re-calibrate the state-of-the-are performance on weaklysupervised temporal action localization.

Acknowledgements

This work is supported by National Natural Science Foundation of China (61772037), Beijing Natural Science Foundation (Z190001) and Tencent AI Lab Rhino-Bird Focused Research Program (JR202021).