{"type":"ScholarlyArticle","title":"Video Frame Interpolation Based on Deformable Kernel Region","authors":[{"name":"Haoyue Tian","slug":"haoyue-tian","given":"Haoyue","family":"Tian","affiliation":"College of Computer Science and Technology , Nanjing University of Aeronautics and Astronautics"},{"name":"Pan Gao","slug":"pan-gao","given":"Pan","family":"Gao","affiliation":"College of Computer Science and Technology , Nanjing University of Aeronautics and Astronautics"},{"name":"Xiaojiang Peng","slug":"xiaojiang-peng","given":"Xiaojiang","family":"Peng","affiliation":"College of Big Data and Internet , Shenzhen Technology University"}],"abstract":"Video frame interpolation task has recently become more and more prevalent in the computer vision field. At present, a number of researches based on deep learning have achieved great success. Most of them are either based on optical flow information, or interpolation kernel, or a combination of these two methods. However, these methods have ignored that there are grid restrictions on the position of kernel region during synthesizing each target pixel. These limitations result in that they cannot well adapt to the irregularity of object shape and uncertainty of motion, which may lead to irrelevant reference pixels used for interpolation. In order to solve this problem, we revisit the deformable convolution for video interpolation, which can break the fixed grid restrictions on the kernel region, making the distribution of reference points more suitable for the shape of the object, and thus warp a more accurate interpolation frame. Experiments are conducted on four datasets to demonstrate the superior performance of the proposed model in comparison to the state-of-the-art alternatives.","doi":"10.24963/ijcai.2022/188","url":"https://ijcai.org/proceedings/2022/188","pdf":"https://ijcai.org/proceedings/2022/0188.pdf","markdown":"https://ijcai.org/proceedings/2022/188.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/188","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":188,"pages":{"first":1349,"last":1355},"keywords":["Computer Vision: Machine Learning for Vision","Computer Vision: Video analysis and understanding"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987019","length_sec":86,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985162","length_sec":657,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bao et al., 2019a] Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang. Depth-aware video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 3703-3712, 2019.","title":"Depth-aware video frame interpolation","authors":"Bao","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bao et al., 2019b] Wenbo Bao, Wei-Sheng Lai, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang. Memc-net: Motion estimation and motion compensation driven neural network for video interpolation and enhancement. IEEE transactions on pattern analysis and machine intelligence, 2019.","title":"Memc-net: Motion estimation and motion compensation driven neural network for video interpolation and enhancement","authors":"Bao","year":"2019","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Butler et al., 2012] Daniel J Butler, Jonas Wulff, Garrett B Stanley, and Michael J Black. A naturalistic open source movie for optical flow evaluation. In European conference on computer vision, pages 611-625. Springer, 2012.","title":"A naturalistic open source movie for optical flow evaluation","authors":"J Daniel, Jonas Butler, Garrett B Wulff, Michael J Stanley, Black","year":"2012","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dai et al., 2017] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. De- formable convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 764-773, 2017.","title":"Deformable convolutional networks","authors":"Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, Yichen Wei","year":"2017","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ilg et al., 2017] Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2462- 2470, 2017.","title":"Flownet 2.0: Evolution of optical flow estimation with deep networks","authors":"Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, Thomas Brox","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Jiang et al., 2018] Huaizu Jiang, Deqing Sun, Varun Jam- pani, Ming-Hsuan Yang, Erik Learned-Miller, and Jan Kautz. Super slomo: High quality estimation of multi- ple intermediate frames for video interpolation. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9000-9008, 2018.","title":"Super slomo: High quality estimation of multiple intermediate frames for video interpolation","authors":"Jiang","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Lee et al., 2020] Hyeongmin Lee, Taeoh Kim, Tae-young Chung, Daehyun Pak, Yuseok Ban, and Sangyoun Lee. Adacof: Adaptive collaboration of flows for video frame interpolation. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 5316-5325, 2020.","title":"Adacof: Adaptive collaboration of flows for video frame interpolation","authors":"Lee","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Liu et al., 2019] Yu-Lun Liu, Yi-Tung Liao, Yen-Yu Lin, and Yung-Yu Chuang. Deep video frame interpolation us- ing cyclic frame generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8794-8802, 2019.","title":"Deep video frame interpolation using cyclic frame generation","authors":"Yu-Lun Liu, Yi-Tung Liao, Yen-Yu Lin, Yung-Yu Chuang","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Niklaus et al., 2017a] Simon Niklaus, Long Mai, and Feng Liu. Video frame interpolation via adaptive convolution. In Proceedings of the IEEE Conference on Computer Vi- sion and Pattern Recognition, pages 670-679, 2017.","title":"Video frame interpolation via adaptive convolution","authors":"Niklaus","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Niklaus et al., 2017b] Simon Niklaus, Long Mai, and Feng Liu. Video frame interpolation via adaptive separable con- volution. In Proceedings of the IEEE International Con- ference on Computer Vision, pages 261-270, 2017.","title":"Video frame interpolation via adaptive separable convolution","authors":"Niklaus","year":"2017","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Niklaus et al., 2017c] Simon Niklaus, Long Mai, and Feng Liu. Video frame interpolation via adaptive separable con- volution. In Proceedings of the IEEE International Con- ference on Computer Vision, pages 261-270, 2017.","title":"Video frame interpolation via adaptive separable convolution","authors":"Niklaus","year":"2017","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Niklaus et al., 2021] Simon Niklaus, Long Mai, and Oliver Wang. Revisiting adaptive convolutions for video frame interpolation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1099-1109, 2021.","title":"Revisiting adaptive convolutions for video frame interpolation","authors":"Niklaus","year":"2021","venue":"Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Paliwal and Kalantari, 2020] Avinash Paliwal and Nima Khademi Kalantari. Deep slow motion video reconstruction with hybrid imaging system. IEEE trans- actions on pattern analysis and machine intelligence, 42(7):1557-1569, 2020.","title":"Deep slow motion video reconstruction with hybrid imaging system","authors":"Avinash Paliwal, Nima Khademi, Kalantari","year":"2020","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Perazzi et al., 2016] Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung. A benchmark dataset and evaluation methodology for video object segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 724-732, 2016.","title":"A benchmark dataset and evaluation methodology for video object segmentation","authors":"Federico Perazzi, Jordi Pont-Tuset, Brian Mcwilliams, Luc Van Gool, Markus Gross, Alexander Sorkine-Hornung","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ranjan and Black, 2017] Anurag Ranjan and Michael J Black. Optical flow estimation using a spatial pyramid net- work. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4161-4170, 2017.","title":"Optical flow estimation using a spatial pyramid network","authors":"Anurag Ranjan, Michael J Black","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ronneberger et al., 2015] Olaf Ronneberger, Philipp Fis- cher, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pages 234-241. Springer, 2015.","title":"Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation","authors":"Ronneberger","year":"2015","venue":"International Conference on Medical image computing and computerassisted intervention","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shi et al., 2021] Zhihao Shi, Xiaohong Liu, Kangdi Shi, Linhui Dai, and Jun Chen. Video frame interpolation via generalized deformable convolution. IEEE Transactions on Multimedia, 2021.","title":"Video frame interpolation via generalized deformable convolution","authors":"Zhihao Shi, Xiaohong Liu, Kangdi Shi, Linhui Dai, Jun Chen","year":"2021","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Soomro et al., 2012] Khurram Soomro, Amir Roshan Za- mir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012.","title":"Ucf101: A dataset of 101 human actions classes from videos in the wild","authors":"Khurram Soomro, Mubarak Amir Roshan Zamir, Shah","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sun et al., 2018] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz. Pwc-net: Cnns for optical flow us- ing pyramid, warping, and cost volume. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8934-8943, 2018.","title":"Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume","authors":"Deqing Sun, Xiaodong Yang, Ming-Yu Liu, Jan Kautz","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xu et al., 2019] Xiangyu Xu, Li Siyao, Wenxiu Sun, Qian Yin, and Ming-Hsuan Yang. Quadratic video interpola- tion. arXiv preprint arXiv:1911.00627, 2019.","title":"Quadratic video interpolation","authors":"Xiangyu Xu, Li Siyao, Wenxiu Sun, Qian Yin, Ming-Hsuan Yang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Xue et al., 2019] Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman. Video enhancement with task-oriented flow. International Journal of Com- puter Vision, 127(8):1106-1125, 2019.","title":"Video enhancement with task-oriented flow","authors":"Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, William T Freeman","year":"2019","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhu et al., 2019] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. Deformable convnets v2: More deformable, better results. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 9308-9316, 2019.","title":"Deformable convnets v2: More deformable, better results","authors":"Xizhou Zhu, Han Hu, Stephen Lin, Jifeng Dai","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}