{"type":"ScholarlyArticle","title":"DFVSR: Directional Frequency Video Super-Resolution via Asymmetric and Enhancement Alignment Network","authors":[{"name":"Shuting Dong","slug":"shuting-dong","given":"Shuting","family":"Dong","affiliation":"Tsinghua Shenzhen International Graduate School , Tsinghua University; Peng Cheng Laboratory"},{"name":"Feng Lu","slug":"feng-lu","given":"Feng","family":"Lu","affiliation":"Tsinghua Shenzhen International Graduate School , Tsinghua University; Peng Cheng Laboratory"},{"name":"Zhe Wu","slug":"zhe-wu","given":"Zhe","family":"Wu","affiliation":"Peng Cheng Laboratory"},{"name":"Chun Yuan","slug":"chun-yuan","given":"Chun","family":"Yuan","affiliation":"Tsinghua Shenzhen International Graduate School , Tsinghua University; Peng Cheng Laboratory"}],"abstract":"Recently, techniques utilizing frequency-based methods have gained significant attention, as they exhibit exceptional restoration capabilities for detail and structure in video super-resolution tasks. However, most of these frequency-based methods mainly have three major limitations: 1) insufficient exploration of object motion information, 2) inadequate enhancement for high-fidelity regions, and 3) loss of spatial information during convolution. In this paper, we propose a novel network, Directional Frequency Video Super-Resolution (DFVSR), to address these limitations. Specifically,  we reconsider object motion from a new perspective and propose Directional Frequency Representation (DFR), which not only borrows the property of frequency representation of detail and structure information but also contains the direction information of the object motion that is extremely significant in videos. Based on this representation,  we propose a Directional Frequency-Enhanced Alignment (DFEA) to use double enhancements of task-related information for ensuring the retention of high-fidelity frequency regions to generate the high-quality alignment feature. Furthermore, we design a novel Asymmetrical U-shaped network architecture to progressively fuse these alignment features and output the final output. This architecture enables the intercommunication of the same level of resolution in the encoder and decoder to achieve the supplement of spatial information. Powered by the above designs, our method achieves superior performance over state-of-the-art models on both quantitative and qualitative evaluations.","doi":"10.24963/ijcai.2023/76","url":"https://ijcai.org/proceedings/2023/76","pdf":"https://ijcai.org/proceedings/2023/0076.pdf","markdown":"https://ijcai.org/proceedings/2023/76.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/76","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":76,"pages":{"first":681,"last":689},"keywords":["Computer Vision: CV: Other","Computer Vision: CV: Image and video retrieval"],"videos":[],"references":[{"position":1,"raw":"[Cao et al., 2021] Jiezhang Cao, Yawei Li, Kai Zhang, and Luc Van Gool. Video super-resolution transformer. arXiv preprint arXiv:2106.06847, 2021.","title":"Video super-resolution transformer","authors":"Cao","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chan et al., 2021a] Kelvin CK Chan, Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. Basicvsr: The search for essential components in video super-resolution and be- yond. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4947- 4956, 2021.","title":"Basicvsr: The search for essential components in video super-resolution and beyond","authors":"Chan","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chan et al., 2021b] Kelvin CK Chan, Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. Understanding de- formable alignment in video super-resolution. In Proceed- ings of the AAAI conference on artificial intelligence, vol- ume 35, pages 973-981, 2021.","title":"Understanding deformable alignment in video super-resolution","authors":"Chan","year":"2021","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chan et al., 2022] Kelvin CK Chan, Shangchen Zhou, Xi- angyu Xu, and Chen Change Loy. Basicvsr++: Improv- ing video super-resolution with enhanced propagation and alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5972- 5981, 2022.","title":"Basicvsr++: Improving video super-resolution with enhanced propagation and alignment","authors":"Shangchen Kelvin Ck Chan, Xiangyu Zhou, Chen Change Xu, Loy","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dai et al., 2017] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. De- formable convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 764-773, 2017.","title":"Deformable convolutional networks","authors":"Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, Yichen Wei","year":"2017","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fuoli et al., 2019] Dario Fuoli, Shuhang Gu, and Radu Tim- ofte. Efficient video super-resolution through recurrent latent space propagation. In 2019 IEEE/CVF Interna- tional Conference on Computer Vision Workshop (IC- CVW), pages 3476-3485. IEEE, 2019.","title":"Efficient video super-resolution through recurrent latent space propagation","authors":"Dario Fuoli, Shuhang Gu, Radu Timofte","year":"2019","venue":"IEEE/CVF International Conference on Computer Vision Workshop (IC-CVW)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Haris et al., 2019] Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita. Recurrent back-projection network for video super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3897-3906, 2019.","title":"Recurrent back-projection network for video super-resolution","authors":"Haris","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2017] Yan Huang, Wei Wang, and Liang Wang. Video super-resolution via bidirectional recur- rent convolutional networks. IEEE transactions on pat- tern analysis and machine intelligence, 40(4):1015-1028, 2017.","title":"Video super-resolution via bidirectional recurrent convolutional networks","authors":"Yan Huang, Wei Wang, Liang Wang","year":"2017","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jo et al., 2018] Younghyun Jo, Seoung Wug Oh, Jaeyeon Kang, and Seon Joo Kim. Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pages 3224-3232, 2018.","title":"Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation","authors":"Jo","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lai et al., 2017] Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. Deep laplacian pyramid networks for fast and accurate super-resolution. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 624-632, 2017.","title":"Deep laplacian pyramid networks for fast and accurate super-resolution","authors":"Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, Ming-Hsuan Yang","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Leng et al., 2022] Jiaxu Leng, Jia Wang, Xinbo Gao, Bo Hu, Ji Gan, and Chenqiang Gao. Icnet: Joint alignment and reconstruction via iterative collaboration for video super-resolution. In Proceedings of the 30th ACM Interna- tional Conference on Multimedia, pages 6675-6684, 2022.","title":"Icnet: Joint alignment and reconstruction via iterative collaboration for video super-resolution","authors":"Jiaxu Leng, Jia Wang, Xinbo Gao, Bo Hu, Ji Gan, Chenqiang Gao","year":"2022","venue":"Proceedings of the 30th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu and Sun, 2013] Ce Liu and Deqing Sun. On bayesian adaptive video super resolution. IEEE transactions on pattern analysis and machine intelligence, 36(2):346-360, 2013.","title":"On bayesian adaptive video super resolution","authors":"Ce Liu, Deqing Sun","year":"2013","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2021] Shaoli Liu, Chengjian Zheng, Kaidi Lu, Si Gao, Ning Wang, Bofei Wang, Diankai Zhang, Xi- aofeng Zhang, and Tianyu Xu. Evsrnet: Efficient video super-resolution with neural architecture search. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2480-2485, 2021.","title":"Evsrnet: Efficient video super-resolution with neural architecture search","authors":"Shaoli Liu, Chengjian Zheng, Kaidi Lu, Si Gao, Ning Wang, Bofei Wang, Diankai Zhang, Xiaofeng Zhang, Tianyu Xu","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2022] Chengxu Liu, Huan Yang, Jianlong Fu, and Xueming Qian. Learning trajectory-aware transformer for video super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5687-5696, 2022.","title":"Learning trajectory-aware transformer for video super-resolution","authors":"Chengxu Liu, Huan Yang, Jianlong Fu, Xueming Qian","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nah et al., 2019] Seungjun Nah, Sungyong Baik, Seokil Hong, Gyeongsik Moon, Sanghyun Son, Radu Timofte, and Kyoung Mu Lee. Ntire 2019 challenge on video de- blurring and super-resolution: Dataset and study. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0-0, 2019.","title":"Ntire 2019 challenge on video deblurring and super-resolution: Dataset and study","authors":"Nah","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Qiu et al., 2022] Zhongwei Qiu, Huan Yang, Jianlong Fu, and Dongmei Fu. Learning spatiotemporal frequency- transformer for compressed video super-resolution. In Computer Vision-ECCV 2022: 17th European Confer- ence, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVIII, pages 257-273. Springer, 2022.","title":"Learning spatiotemporal frequencytransformer for compressed video super-resolution","authors":"Zhongwei Qiu, Huan Yang, Jianlong Fu, Dongmei Fu","year":"2022","venue":"Computer Vision-ECCV 2022: 17th European Conference","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sajjadi et al., 2018] Mehdi SM Sajjadi, Raviteja Vemula- palli, and Matthew Brown. Frame-recurrent video super- resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6626- 6634, 2018.","title":"Frame-recurrent video superresolution","authors":"S M Mehdi, Raviteja Sajjadi, Matthew Vemulapalli, Brown","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tian et al., 2020] Yapeng Tian, Yulun Zhang, Yun Fu, and Chenliang Xu. Tdan: Temporally-deformable alignment network for video super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3360-3369, 2020.","title":"Tdan: Temporally-deformable alignment network for video super-resolution","authors":"Tian","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2019] Xintao Wang, Kelvin CK Chan, Ke Yu, Chao Dong, and Chen Change Loy. Edvr: Video restora- tion with enhanced deformable convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0-0, 2019.","title":"Edvr: Video restoration with enhanced deformable convolutional networks","authors":"Wang","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xiao et al., 2021] Zeyu Xiao, Xueyang Fu, Jie Huang, Zhen Cheng, and Zhiwei Xiong. Space-time distillation for","title":"Space-time distillation for","authors":"Zeyu Xiao, Xueyang Fu, Jie Huang, Zhen Cheng, Zhiwei Xiong","year":null,"venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}