{"type":"ScholarlyArticle","title":"Self-Supervised Video Representation Learning with Constrained Spatiotemporal Jigsaw","authors":[{"name":"Yuqi Huo","slug":"yuqi-huo","given":"Yuqi","family":"Huo","affiliation":"School of Information , Renmin University of China , Beijing , China; Gaoling School of Artificial Intelligence , Renmin University of China , Beijing , China"},{"name":"Mingyu Ding","slug":"mingyu-ding","given":"Mingyu","family":"Ding","affiliation":"The University of Hong Kong , Pokfulam , Hong Kong , China"},{"name":"Haoyu Lu","slug":"haoyu-lu","given":"Haoyu","family":"Lu","affiliation":"School of Information , Renmin University of China , Beijing , China"},{"name":"Ziyuan Huang","slug":"ziyuan-huang","given":"Ziyuan","family":"Huang","affiliation":"National University of Singapore , Singapore"},{"name":"Mingqian Tang","slug":"mingqian-tang","given":"Mingqian","family":"Tang","affiliation":"Alibaba Group , Hangzhou , China"},{"name":"Zhiwu Lu","slug":"zhiwu-lu","given":"Zhiwu","family":"Lu","affiliation":"Gaoling School of Artificial Intelligence , Renmin University of China , Beijing , China"},{"name":"Tao Xiang","slug":"tao-xiang","given":"Tao","family":"Xiang","affiliation":"University of Surrey , Surrey , UK"}],"abstract":"This paper proposes a novel pretext task for self-supervised video representation learning by exploiting spatiotemporal continuity in videos. It is motivated by the fact that videos are spatiotemporal by nature and a representation learned by detecting spatiotemporal continuity/discontinuity is thus beneficial for downstream video content analysis tasks. A natural choice of such a pretext task is to construct spatiotemporal (3D) jigsaw puzzles and learn to solve them. However, as we demonstrate in the experiments, this task turns out to be intractable. We thus propose Constrained Spatiotemporal Jigsaw (CSJ) whereby the 3D jigsaws are formed in a constrained manner to ensure that large continuous spatiotemporal cuboids exist. This provides sufficient cues for the model to reason about the continuity. Instead of solving them directly, which could still be extremely hard, we carefully design four surrogate tasks that are more solvable. The four tasks aim to learn representations sensitive to spatiotemporal continuity at both the local and global levels. Extensive experiments show that our CSJ achieves state-of-the-art on various benchmarks.","doi":"10.24963/ijcai.2021/104","url":"https://ijcai.org/proceedings/2021/104","pdf":"https://ijcai.org/proceedings/2021/0104.pdf","markdown":"https://ijcai.org/proceedings/2021/104.md","bibtex":"https://ijcai.org/proceedings/2021/bibtex/104","volume":{"slug":"2021","title":"Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2021","url":"https://ijcai.org/proceedings/2021/","year":2021,"location":"Montreal","dates":"19-27 August 2021","editors":"Prof. Zhi-Hua Zhou","isbn":"978-0-9992411-9-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2021"},"section":"Computer Vision I","track":"Main Track","number":104,"pages":{"first":751,"last":757},"keywords":["Computer Vision: Action Recognition","Machine Learning: Unsupervised Learning"],"videos":[],"references":[{"position":1,"raw":"[Ahsan et al., 2019] Unaiza Ahsan, Rishi Madhok, and Irfan Essa. Video jigsaw: Unsupervised learning of spatiotemporal context for video action recognition. In WACV, pages 179-189, 2019.","title":"Video jigsaw: Unsupervised learning of spatiotemporal context for video action recognition","authors":"Ahsan","year":"2019","venue":"WACV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alwassel et al., 2020] Humam Alwassel, Dhruv Mahajan, Loren- zo Torresani, Bernard Ghanem, and Du Tran. Self-supervised learning by cross-modal audio-video clustering. In NeurIPS, 2020.","title":"Self-supervised learning by cross-modal audio-video clustering","authors":"Humam Alwassel, Dhruv Mahajan, Lorenzo Torresani, Bernard Ghanem, Du Tran","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Benaim et al., 2020] Sagie Benaim, Ariel Ephrat, Oran Lang, In- bar Mosseri, William T. Freeman, Michael Rubinstein, Michal Irani, and Tali Dekel. Speednet: Learning the speediness in videos. In CVPR, pages 9922-9931, 2020.","title":"Speednet: Learning the speediness in videos","authors":"Sagie Benaim, Ariel Ephrat, Oran Lang, Inbar Mosseri, William T Freeman, Michael Rubinstein, Michal Irani, Tali Dekel","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bengio et al., 2009] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICMLE, pages 41-48, 2009.","title":"Curriculum learning","authors":"Yoshua Bengio, Jérôme Louradour, Ronan Collobert, Jason Weston","year":"2009","venue":"ICMLE","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2019] Yue Chen, Yalong Bai, Wei Zhang, and Tao Mei. Destruction and construction learning for fine-grained im- age recognition. In CVPR, pages 5157-5166, 2019.","title":"Destruction and construction learning for fine-grained image recognition","authors":"Yue Chen, Yalong Bai, Wei Zhang, Tao Mei","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2020] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for con- trastive learning of visual representations. In ICML, pages 10709-10719, 2020.","title":"A simple framework for contrastive learning of visual representations","authors":"Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Du et al., 2020] Ruoyi Du, Dongliang Chang, Ayan Kumar Bhu- nia, Jiyang Xie, Yi-Zhe Song, Zhanyu Ma, and Jun Guo. Fine- grained visual classification via progressive multi-granularity training of jigsaw patches. arXiv preprint arXiv:2003.03836, 2020.","title":"Finegrained visual classification via progressive multi-granularity training of jigsaw patches","authors":"Ruoyi Du, Dongliang Chang, Ayan Kumar Bhunia, Jiyang Xie, Yi-Zhe Song, Zhanyu Ma, Jun Guo","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gidaris et al., 2018] Spyros Gidaris, Praveer Singh, and Nikos Ko- modakis. Unsupervised representation learning by predicting im- age rotations. ICLR, 2018.","title":"Unsupervised representation learning by predicting image rotations","authors":"Spyros Gidaris, Praveer Singh, Nikos Komodakis","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Han et al., 2019] Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding. In IC- CV Workshop, pages 1483-1492, 2019.","title":"Video representation learning by dense predictive coding","authors":"Han","year":"2019","venue":"IC-CV Workshop","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Han et al., 2020a] Tengda Han, Weidi Xie, and Andrew Zisser- man. Memory-augmented dense predictive coding for video rep- resentation learning. In ECCV, pages 312-329, 2020.","title":"Memory-augmented dense predictive coding for video representation learning","authors":"Han","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Han et al., 2020b] Tengda Han, Weidi Xie, and Andrew Zisser- man. Self-supervised co-training for video representation learn- ing. In NeurIPS, 2020.","title":"Self-supervised co-training for video representation learning","authors":"Han","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hara et al., 2018] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cnns retrace the history of 2d c- nns and imagenet? In CVPR, pages 6546-6555, 2018.","title":"Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?","authors":"Hara","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2020] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pages 9729-9738, 2020.","title":"Momentum contrast for unsupervised visual representation learning","authors":"Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jing et al., 2018] Longlong Jing, Xiaodong Yang, Jingen Liu, and Yingli Tian. Self-supervised spatiotemporal feature learning vi- a video rotation prediction. arXiv preprint arXiv:1811.11387, 2018.","title":"Self-supervised spatiotemporal feature learning via video rotation prediction","authors":"Longlong Jing, Xiaodong Yang, Jingen Liu, Yingli Tian","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kay et al., 2017] Will Kay, João Carreira, Karen Simonyan, Bri- an Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950, 2017.","title":"The kinetics human action video dataset","authors":"Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, Andrew Zisserman","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kendall et al., 2018] Alex Kendall, Yarin Gal, and Roberto Cipol- la. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In CVPR, pages 7482-7491, 2018.","title":"Multi-task learning using uncertainty to weigh losses for scene geometry and semantics","authors":"Alex Kendall, Yarin Gal, Roberto Cipolla","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kim et al., 2019] Dahun Kim, Donghyeon Cho, and In So Kweon. Self-supervised video representation learning with space-time cubic puzzles. In AAAI, pages 8545-8552, 2019.","title":"Self-supervised video representation learning with space-time cubic puzzles","authors":"Kim","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kuehne et al., 2011] Hildegard Kuehne, Hueihan Jhuang, Estibal- iz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: A large video database for human motion recognition. In ICCV, pages 2556-2563, 2011.","title":"Hmdb: A large video database for human motion recognition","authors":"Hildegard Kuehne, Hueihan Jhuang, Estibaliz Garrote, Tomaso Poggio, Thomas Serre","year":"2011","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lee et al., 2017] Hsin-Ying Lee, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. Unsupervised representation learning by sorting sequences. In ICCV, pages 667-676, 2017.","title":"Unsupervised representation learning by sorting sequences","authors":"Hsin-Ying Lee, Jia-Bin Huang, Maneesh Singh, Ming-Hsuan Yang","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lin et al., 2017] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaim- ing He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, pages 2117-2125, 2017.","title":"Feature pyramid networks for object detection","authors":"Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, Serge Belongie","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Misra et al., 2016] Ishan Misra, C. Lawrence Zitnick, and Martial Hebert. Shuffle and learn: Unsupervised learning using temporal order verification. In ECCV, pages 527-544, 2016.","title":"Shuffle and learn: Unsupervised learning using temporal order verification","authors":"Ishan Misra, C Lawrence Zitnick, Martial Hebert","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Noroozi and Favaro, 2016] Mehdi Noroozi and Paolo Favaro. Un- supervised learning of visual representations by solving jigsaw puzzles. In ECCV, pages 69-84, 2016.","title":"Unsupervised learning of visual representations by solving jigsaw puzzles","authors":"Mehdi Noroozi, Paolo Favaro","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Oord et al., 2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive cod- ing. arXiv preprint arXiv:1807.03748, 2018.","title":"Representation learning with contrastive predictive coding","authors":"Aaron Van Den Oord, Yazhe Li, Oriol Vinyals","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Paumard et al., 2020] Marie-Morgane Paumard, David Picard, and Hedi Tabia. Deepzzle: Solving visual jigsaw puzzles with deep learning and shortest path optimization. TIP, 29:3569-3581, 2020.","title":"Deepzzle: Solving visual jigsaw puzzles with deep learning and shortest path optimization","authors":"Marie-Morgane Paumard, David Picard, Hedi Tabia","year":"2020","venue":"TIP","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Soomro et al., 2012] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012.","title":"Ucf101: A dataset of 101 human actions classes from videos in the wild","authors":"Khurram Soomro, Mubarak Amir Roshan Zamir, Shah","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Stringer et al., 2006] Simon M. Stringer, Gavin Perry, Edmund T. Rolls, and J. H. Proske. Learning invariant object recognition in the visual system with continuous transformations. Biological Cybernetics, 94(2):128-142, 2006.","title":"Learning invariant object recognition in the visual system with continuous transformations","authors":"M Simon, Gavin Stringer, Edmund T Perry, J H Rolls, Proske","year":"2006","venue":"Biological Cybernetics","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sun et al., 2019] Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Learning video representations us- ing contrastive bidirectional transformer. arXiv preprint arX- iv:1906.05743, 2019.","title":"Learning video representations using contrastive bidirectional transformer","authors":"Chen Sun, Fabien Baradel, Kevin Murphy, Cordelia Schmid","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2018] Xiaolong Wang, Ross Girshick, Abhinav Gup- ta, and Kaiming He. Non-local neural networks. In CVPR, pages 7794-7803, 2018.","title":"Non-local neural networks","authors":"Xiaolong Wang, Ross Girshick, Abhinav Gupta, Kaiming He","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2020] Jiangliu Wang, Jianbo Jiao, and Yun-Hui Liu. Self-supervised video representation learning by pace prediction. In ECCV, pages 504-521, 2020.","title":"Self-supervised video representation learning by pace prediction","authors":"Jiangliu Wang, Jianbo Jiao, Yun-Hui Liu","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Xu et al., 2019] Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di X- ie, and Yueting Zhuang. Self-supervised spatiotemporal learning via video clip order prediction. In CVPR, pages 10334-10343, 2019.","title":"Self-supervised spatiotemporal learning via video clip order prediction","authors":"Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, Yueting Zhuang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yao et al., 2020] Yuan Yao, Chang Liu, Dezhao Luo, Yu Zhou, and Qixiang Ye. Video playback rate perception for self-supervised spatio-temporal representation learning. In CVPR, pages 6548- 6557, 2020.","title":"Video playback rate perception for self-supervised spatio-temporal representation learning","authors":"Yuan Yao, Chang Liu, Dezhao Luo, Yu Zhou, Qixiang Ye","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhuang et al., 2020] Chengxu Zhuang, Tianwei She, Alex Ando- nian, Max Sobol Mark, and Daniel Yamins. Unsupervised learn- ing from video with deep neural embeddings. In CVPR, pages 9563-9572, 2020.","title":"Unsupervised learning from video with deep neural embeddings","authors":"Zhuang","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}