{"type":"ScholarlyArticle","title":"Spatiotemporal Super-Resolution with Cross-Task Consistency and Its Semi-supervised Extension","authors":[{"name":"Han-Yi Lin","slug":"han-yi-lin","given":"Han-Yi","family":"Lin","affiliation":"Department of Computer Science and Information Engineering , National Taiwan University , Taiwan"},{"name":"Pi-Cheng Hsiu","slug":"pi-cheng-hsiu","given":"Pi-Cheng","family":"Hsiu","affiliation":"Research Center for Information Technology Innovation , Academia Sinica , Taiwan; Department of Computer Science and Information Engineering , National Chi Nan University , Taiwan"},{"name":"Tei-Wei Kuo","slug":"tei-wei-kuo","given":"Tei-Wei","family":"Kuo","affiliation":"Department of Computer Science and Information Engineering , National Taiwan University , Taiwan; Department of Computer Science and College of Engineering , City University of Hong Kong , Hong Kong"},{"name":"Yen-Yu Lin","slug":"yen-yu-lin","given":"Yen-Yu","family":"Lin","affiliation":"Research Center for Information Technology Innovation , Academia Sinica , Taiwan; Department of Computer Science , National Chiao Tung University , Taiwan"}],"abstract":"Spatiotemporal super-resolution (SR) aims to upscale both the spatial and temporal dimensions of input videos, and produces videos with higher frame resolutions and rates. It involves two essential sub-tasks: spatial SR and temporal SR. We design a two-stream network for spatiotemporal SR in this work. One stream contains a temporal SR module followed by a spatial SR module, while the other stream has the same two modules in the reverse order. Based on the interchangeability of performing the two sub-tasks, the two network streams are supposed to produce consistent spatiotemporal SR results. Thus, we present a cross-stream consistency to enforce the similarity between the outputs of the two streams. In this way, the training of the two streams is correlated, which allows the two SR modules to share their supervisory signals and improve each other. In addition, the proposed cross-stream consistency does not consume labeled training data and can guide network training in an unsupervised manner. We leverage this property to carry out semi-supervised spatiotemporal SR. It turns out that our method makes the most of training data, and can derive an effective model with few high-resolution and high-frame-rate videos, achieving the state-of-the-art performance. The source code of this work is available at https://hankweb.github.io/STSRwithCrossTask/.","doi":"10.24963/ijcai.2020/86","url":"https://ijcai.org/proceedings/2020/86","pdf":"https://ijcai.org/proceedings/2020/0086.pdf","markdown":"https://ijcai.org/proceedings/2020/86.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/86","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":86,"pages":{"first":615,"last":622},"keywords":["Machine Learning: Semi-Supervised Learning","Computer Vision: Other","Machine Learning: Deep Learning: Convolutional networks"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24296/file_24296.m3u8","length_sec":281,"provider":"hls"}],"references":[{"position":1,"raw":"[Bailer et al., 2019] Christian Bailer, Bertram Taetz, and Di- dier Stricker. Flow fields: Dense correspondence fields for highly accurate large displacement optical flow estimation. TPAMI, 2019.","title":"Flow fields: Dense correspondence fields for highly accurate large displacement optical flow estimation","authors":"Bailer","year":"2019","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baker et al., 2007] Simon Baker, Daniel Scharstein, J. P. Lewis, Stefan Roth, Michael J. Black, and Richard Szeliski. A database and evaluation methodology for opti- cal flow. In ICCV, 2007.","title":"A database and evaluation methodology for optical flow","authors":"Simon Baker, J P Daniel Scharstein, Stefan Lewis, Michael J Roth, Richard Black, Szeliski","year":"2007","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Baker et al., 2011] Simon Baker, Daniel Scharstein, J. P. Lewis, Stefan Roth, Michael J. Black, and Richard Szeliski. A database and evaluation methodology for opti- cal flow. IJCV, 2011.","title":"A database and evaluation methodology for optical flow","authors":"Simon Baker, J P Daniel Scharstein, Stefan Lewis, Michael J Roth, Richard Black, Szeliski","year":"2011","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bao et al., 2019] Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang. Depth-aware video frame interpolation. In CVPR, 2019.","title":"Depth-aware video frame interpolation","authors":"Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, Ming-Hsuan Yang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2015] Hsin-Yi Chen, Yen-Yu Lin, and Bing-Yu Chen. Co-segmentation guided hough transform for robust feature matching. TPAMI, 2015.","title":"Co-segmentation guided hough transform for robust feature matching","authors":"Hsin-Yi Chen, Yen-Yu Lin, Bing-Yu Chen","year":"2015","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2018] Yun-Chun Chen, Po-Hsiang Huang, Li- Yu Yu, Jia-Bin Huang, Ming-Hsuan Yang, and Yen-Yu Lin. Deep semantic matching with foreground detection and cycle-consistency. In ACCV, 2018.","title":"Deep semantic matching with foreground detection and cycle-consistency","authors":"Yun-Chun Chen, Po-Hsiang Huang, Li-Yu Yu, Jia-Bin Huang, Ming-Hsuan Yang, Yen-Yu Lin","year":"2018","venue":"ACCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2019] Yun-Chun Chen, Yen-Yu Lin, Ming- Hsuan Yang, and Jia-Bin Huang. Crdoco: Pixel-level do- main transfer with cross-domain consistency. In CVPR, 2019.","title":"Crdoco: Pixel-level domain transfer with cross-domain consistency","authors":"Yun-Chun Chen, Yen-Yu Lin, Ming-Hsuan Yang, Jia-Bin Huang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2020] Yun-Chun Chen, Yen-Yu Lin, Ming- Hsuan Yang, and Jia-Bin Huang. Show, match and segment: Joint weakly supervised learning of semantic matching and object co-segmentation. TPAMI, 2020.","title":"Show, match and segment: Joint weakly supervised learning of semantic matching and object co-segmentation","authors":"Yun-Chun Chen, Yen-Yu Lin, Ming-Hsuan Yang, Jia-Bin Huang","year":"2020","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dong et al., 2016] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. TPAMI, 2016.","title":"Image super-resolution using deep convolutional networks","authors":"Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang","year":"2016","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gao et al., 2012] Xinbo Gao, Kaibing Zhang, Dacheng Tao, and Xuelong Li. Image super-resolution with sparse neigh- bor embedding. TIP, 2012.","title":"Image super-resolution with sparse neighbor embedding","authors":"Xinbo Gao, Kaibing Zhang, Dacheng Tao, Xuelong Li","year":"2012","venue":"TIP","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Haris et al., 2018] Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita. Deep back- projection networks for super-resolution. In CVPR, 2018.","title":"Deep backprojection networks for super-resolution","authors":"Haris","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jiang et al., 2018] Huaizu Jiang, Deqing Sun, Varun Jam- pani, Ming-Hsuan Yang, Erik G. Learned-Miller, and Jan Kautz. Super slomo: High quality estimation of multi- ple intermediate frames for video interpolation. In CVPR, 2018.","title":"Super slomo: High quality estimation of multiple intermediate frames for video interpolation","authors":"Jiang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Keys, 1981] Robert G Keys. Cubic convolution interpola- tion for digital image processing. TASSP, 1981.","title":"Cubic convolution interpolation for digital image processing","authors":"G Robert, Keys","year":"1981","venue":"TASSP","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kim et al., 2016] Jiwon Kim, Jung Kwon Lee, and Ky- oung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In CVPR, 2016.","title":"Accurate image super-resolution using very deep convolutional networks","authors":"Jiwon Kim, Jung Kwon Lee, Kyoung Mu, Lee","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lai et al., 2018] Wei-Sheng Lai, Jia-Bin Huang, Oliver Wang, Eli Shechtman, Ersin Yumer, and Ming-Hsuan Yang. Learning blind video temporal consistency. In ECCV, 2018.","title":"Learning blind video temporal consistency","authors":"Wei-Sheng Lai, Jia-Bin Huang, Oliver Wang, Eli Shechtman, Ersin Yumer, Ming-Hsuan Yang","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu and Sun, 2011] Ce Liu and Deqing Sun. A bayesian ap- proach to adaptive video super resolution. In CVPR, 2011.","title":"A bayesian approach to adaptive video super resolution","authors":"Ce Liu, Deqing Sun","year":"2011","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2017] Ziwei Liu, Raymond A. Yeh, Xiaoou Tang, Yiming Liu, and Aseem Agarwala. Video frame synthesis using deep voxel flow. In ICCV, 2017.","title":"Video frame synthesis using deep voxel flow","authors":"Ziwei Liu, Raymond A Yeh, Xiaoou Tang, Yiming Liu, Aseem Agarwala","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2019] Yu-Lun Liu, Yi-Tung Liao, Yen-Yu Lin, and Yung-Yu Chuang. Deep video frame interpolation us- ing cyclic frame generation. In AAAI, 2019.","title":"Deep video frame interpolation using cyclic frame generation","authors":"Yu-Lun Liu, Yi-Tung Liao, Yen-Yu Lin, Yung-Yu Chuang","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Niklaus et al., 2017] Simon Niklaus, Long Mai, and Feng Liu. Video frame interpolation via adaptive convolution. In CVPR, 2017.","title":"Video frame interpolation via adaptive convolution","authors":"Simon Niklaus, Long Mai, Feng Liu","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pont-Tuset et al., 2017] Jordi Pont-Tuset, Federico Per- azzi, Sergi Caelles, Pablo Arbelaez, Alexander Sorkine- Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv:1704.00675, 2017.","title":"Jordi Pont-Tuset, Federico Perazzi","authors":"Pont-Tuset","year":"2017","venue":"The 2017 davis challenge on video object segmentation","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Shahar et al., 2011] Oded Shahar, Alon Faktor, and Michal Irani. Space-time super-resolution from a single video. In CVPR, 2011.","title":"Space-time super-resolution from a single video","authors":"Oded Shahar, Alon Faktor, Michal Irani","year":"2011","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sharma et al., 2017] Manoj Sharma, Santanu Chaudhury, and Brejesh Lall. Space-time super-resolution using deep learning based framework. In PReMI, 2017.","title":"Space-time super-resolution using deep learning based framework","authors":"Manoj Sharma, Santanu Chaudhury, Brejesh Lall","year":"2017","venue":"PReMI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shi et al., 2016] Wenzhe Shi, Jose Caballero, Ferenc Huszar, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single im- age and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR, 2016.","title":"Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network","authors":"Shi","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tai et al., 2017] Ying Tai, Jian Yang, and Xiaoming Liu. Image super-resolution via deep recursive residual net- work. In CVPR, 2017.","title":"Image super-resolution via deep recursive residual network","authors":"Ying Tai, Jian Yang, Xiaoming Liu","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tsai et al., 2019] Chung-Chi Tsai, Weizhi Li, Kuang-Jui Hsu, Xiaoning Qian, and Yen-Yu Lin. Image co-saliency detection and co-segmentation via progressive joint opti- mization. TIP, 2019.","title":"Image co-saliency detection and co-segmentation via progressive joint optimization","authors":"Chung-Chi Tsai, Weizhi Li, Kuang-Jui Hsu, Xiaoning Qian, Yen-Yu Lin","year":"2019","venue":"TIP","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xue et al., 2019] Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T. Freeman. Video enhance- ment with task-oriented flow. IJCV, 2019.","title":"Video enhancement with task-oriented flow","authors":"Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, William T Freeman","year":"2019","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yu et al., 2013] Zhefei Yu, Houqiang Li, Zhangyang Wang, Zeng Hu, and Chang Wen Chen. Multi-level video frame interpolation: Exploiting the interaction among different levels. TCSVT, 2013.","title":"Multi-level video frame interpolation: Exploiting the interaction among different levels","authors":"Yu","year":"2013","venue":"TCSVT","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zeyde et al., 2012] Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse- representations. In Curves and Surfaces, 2012.","title":"On single image scale-up using sparserepresentations","authors":"Zeyde","year":"2012","venue":"Curves and Surfaces","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2018] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. In CVPR, 2018.","title":"Residual dense network for image super-resolution","authors":"Zhang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zou et al., 2018] Yuliang Zou, Zelun Luo, and Jia-Bin Huang. Df-net: Unsupervised joint learning of depth and flow using cross-task consistency. In ECCV, 2018.","title":"Df-net: Unsupervised joint learning of depth and flow using cross-task consistency","authors":"Yuliang Zou, Zelun Luo, Jia-Bin Huang","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}