{"type":"ScholarlyArticle","title":"Dual Enhancement in ODI Super-Resolution: Adapting Convolution and Upsampling to Projection Distortion","authors":[{"name":"Xiang Ji","slug":"xiang-ji","given":"Xiang","family":"Ji","affiliation":"State Key Laboratory of Networking and Switching Technology , Beijing University of Posts and Telecommunications"},{"name":"Changqiao Xu","slug":"changqiao-xu","given":"Changqiao","family":"Xu","affiliation":"State Key Laboratory of Networking and Switching Technology , Beijing University of Posts and Telecommunications"},{"name":"Lujie Zhong","slug":"lujie-zhong","given":"Lujie","family":"Zhong","affiliation":"Information Engineering College , Capital Normal University"},{"name":"Shujie Yang","slug":"shujie-yang","given":"Shujie","family":"Yang","affiliation":"State Key Laboratory of Networking and Switching Technology , Beijing University of Posts and Telecommunications"},{"name":"Han Xiao","slug":"han-xiao","given":"Han","family":"Xiao","affiliation":"State Key Laboratory of Networking and Switching Technology , Beijing University of Posts and Telecommunications"},{"name":"Gabriel-Miro Muntean","slug":"gabriel-miro-muntean","given":"Gabriel-Miro","family":"Muntean","affiliation":"School of Electronic Engineering , Dublin City University"}],"abstract":"Omnidirectional images (ODIs) demand considerably higher resolution to ensure high quality across all viewports. Traditional convolutional neural networks (CNN)-based single-image super-resolution (SISR) networks, however, are not effective for spherical ODIs. This is due to the uneven pixel density distribution and varying texture complexity in different regions that arise when projecting from a sphere to a plane.  Additionally, the computational and memory costs associated with large-sized ODIs present a challenge for real-world application.  To address these issues, we propose an efficient distortion-adaptive super-resolution network (ODA-SRN).  Specifically, ODA-SRN employs a series of specially designed Distortion Attention Block Groups (DABG) as its backbone. Our Distortion Attention Blocks (DABs) utilize multi-segment parameterized convolution to generate dynamic filters, which compensate for distortion and texture fading during feature extraction. Moreover, we introduce an upsampling scheme that accounts for the dependence of pixel position and distortion degree to achieve pixel-level distortion offset. A comprehensive set of results demonstrates that our ODA-SRN significantly improves the super-resolution performance for ODIs, both quantitatively and qualitatively, when compared to other state-of-the-art methods.","doi":"10.24963/ijcai.2024/100","url":"https://ijcai.org/proceedings/2024/100","pdf":"https://ijcai.org/proceedings/2024/0100.pdf","markdown":"https://ijcai.org/proceedings/2024/100.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/100","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":100,"pages":{"first":902,"last":910},"keywords":["Agent-based and Multi-agent Systems: MAS: Human-agent interaction","Computer Vision: CV: Applications","Computer Vision: CV: Structural and model-based approaches, knowledge representation and reasoning","Computer Vision: CV: 3D computer vision"],"videos":[],"references":[{"position":1,"raw":"[Anwar et al., 2021] Saeed Anwar, Salman Khan, and Nick Barnes. A deep journey into super-resolution: A survey. ACM Comput. Surv., 53(3):34, 2021.","title":"A deep journey into super-resolution: A survey","authors":"Saeed Anwar, Salman Khan, Nick Barnes","year":"2021","venue":"ACM Comput. Surv","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brandon Yang, 2020] Quoc V. Le Jiquan Ngiam Bran- don Yang, Gabriel Bender. Condconv: Conditionally pa- rameterized convolutions for efficient inference. arxiv, 2020.","title":"Condconv: Conditionally parameterized convolutions for efficient inference","authors":"V Quoc, Jiquan Le, Brandon Ngiam, Gabriel Yang, Bender","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2021] Jingye Chen, Bin Li, and Xiangyang Xue. Scene text telescope: Text-focused scene im- age super-resolution. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12021-12030, 2021.","title":"Scene text telescope: Text-focused scene image super-resolution","authors":"Chen","year":"2021","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2022] Guang Chen, Haitao Wang, Kai Chen, Zhijun Li, Zida Song, Yinlong Liu, Wenkai Chen, and Alois Knoll. A survey of the four pillars for small ob- ject detection: Multiscale representation, contextual in- formation, super-resolution, and region proposal. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 52(2):936-953, 2022.","title":"A survey of the four pillars for small object detection: Multiscale representation, contextual information, super-resolution, and region proposal","authors":"Guang Chen, Haitao Wang, Kai Chen, Zhijun Li, Zida Song, Yinlong Liu, Wenkai Chen, Alois Knoll","year":"2022","venue":"IEEE Transactions on Systems, Man, and Cybernetics: Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dai et al., 2019] Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang. Second-order attention net- work for single image super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11065-11074, 2019.","title":"Second-order attention network for single image super-resolution","authors":"Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, Lei Zhang","year":"2019","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Deng et al., 2021] Xin Deng, Hao Wang, Mai Xu, Yichen Guo, Yuhang Song, and Li Yang. Lau-net: Latitude adap- tive upscaling network for omnidirectional image super- resolution. 2021 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 9185-9194, 2021.","title":"Lau-net: Latitude adaptive upscaling network for omnidirectional image superresolution","authors":"Xin Deng, Hao Wang, Mai Xu, Yichen Guo, Yuhang Song, Li Yang","year":"2021","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Deng et al., 2023] Xin Deng, Hao Wang, Mai Xu, Li Li, and Zulin Wang. Omnidirectional image super-resolution via latitude adaptive network. IEEE Transactions on Multime- dia, pages 4108-4120, 2023.","title":"Omnidirectional image super-resolution via latitude adaptive network","authors":"Xin Deng, Hao Wang, Mai Xu, Li Li, Zulin Wang","year":"2023","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dong et al., 2014] Chao Dong, Change Loy Chen, He Kaiming, and Xiaoou Tang. Learning a deep convolutional network for image super-resolution. ECCV. Springer International Publishing, 2014, pages 184-199, 2014.","title":"Learning a deep convolutional network for image super-resolution","authors":"Chao Dong, Change Loy Chen, He Kaiming, Xiaoou Tang","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpass- ing human-level performance on imagenet classification. 2015 IEEE International Conference on Computer Vision (ICCV), pages 1026-1034, 2015.","title":"Delving deep into rectifiers: Surpassing human-level performance on imagenet classification","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2015","venue":"IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 2015 Inter- national Conference on Learning Representations (ICLR), 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"2015 International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2020] Shiyuan Li, Chunyu Lin, Kang Liao, Yao Zhao, and Xue Zhang. Panoramic image quality- enhancement by fusing neural textures of the adaptive ini- tial viewport. 2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), pages 816-817, 2020.","title":"Panoramic image qualityenhancement by fusing neural textures of the adaptive initial viewport","authors":"Shiyuan Li, Chunyu Lin, Kang Liao, Yao Zhao, Xue Zhang","year":"2020","venue":"IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2022] Wenbo Li, Kun Zhou, Lu Qi, Liying Lu, and Jiangbo Lu. Best-buddy gans for highly detailed image super-resolution. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 1412-1420, 2022.","title":"Best-buddy gans for highly detailed image super-resolution","authors":"Wenbo Li, Kun Zhou, Lu Qi, Liying Lu, Jiangbo Lu","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lim et al., 2017] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep resid- ual networks for single image super-resolution. 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1132-1140, 2017.","title":"Enhanced deep residual networks for single image super-resolution","authors":"Lim","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2020] Jie Liu, Wenjie Zhang, Yuting Tang, Jie Tang, and Gangshan Wu. Residual feature aggregation network for image super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2359-2368, 2020.","title":"Residual feature aggregation network for image super-resolution","authors":"Jie Liu, Wenjie Zhang, Yuting Tang, Jie Tang, Gangshan Wu","year":"2020","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ma et al., 2020] Dizhi Ma, Andrew Lumsdaine, and Wen- hui Zhou. Flexible spatial and angular light field super resolution. 2020 IEEE International Conference on Image Processing (ICIP), pages 2970-2974, 2020.","title":"Flexible spatial and angular light field super resolution","authors":"Ma","year":"2020","venue":"IEEE International Conference on Image Processing (ICIP)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mei et al., 2020] Yiqun Mei, Yuchen Fan, Yuqian Zhou, Lichao Huang, Thomas S Huang, and Honghui Shi. Im- age super-resolution with cross-scale non-local attention and exhaustive self-exemplars mining. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5690-5699, 2020.","title":"Image super-resolution with cross-scale non-local attention and exhaustive self-exemplars mining","authors":"Yiqun Mei, Yuchen Fan, Yuqian Zhou, Lichao Huang, Thomas S Huang, Honghui Shi","year":"2020","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ozcinar et al., 2019] Cagri Ozcinar, Aakanksha Rana, and Aljosa Smolic. Super-resolution of omnidirectional im- ages using adversarial learning. 2019 IEEE 21st In- ternational Workshop on Multimedia Signal Processing (MMSP), pages 1-6, 2019.","title":"Super-resolution of omnidirectional images using adversarial learning","authors":"Cagri Ozcinar, Aakanksha Rana, Aljosa Smolic","year":"2019","venue":"IEEE 21st International Workshop on Multimedia Signal Processing (MMSP)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Paszke et al., 2017] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De- Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. Work- shop Autodiff Submission in Conference on Neural Infor- mation Processing Systems (NIPS-W), 2017.","title":"Automatic differentiation in pytorch","authors":"Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De-Vito, Zeming Lin, Alban Desmaison, Luca Antiga, Adam Lerer","year":"2017","venue":"Workshop Autodiff Submission in Conference on Neural Information Processing Systems (NIPS-W)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Senmao et al., 2023] Tian Senmao, Lu Ming, Liu Jiaming, Guo Yandong, Chen Yurong, and Zhang Shunli. Cabm: Content-aware bit mapping for single image super- resolution network with large input. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.","title":"Cabm: Content-aware bit mapping for single image superresolution network with large input","authors":"Tian Senmao, Lu Ming, Liu Jiaming, Guo Yandong, Chen Yurong, Zhang Shunli","year":"2023","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shen et al., 2022] Zhijie Shen, Chunyu Lin, Kang Liao, Lang Nie, Zishuo Zheng, and Yao Zhao. Panoformer: Panorama transformer for indoor 360 depth estimation. ECCV, pages 23-27, 2022.","title":"Panoformer: Panorama transformer for indoor 360 depth estimation","authors":"Zhijie Shen, Chunyu Lin, Kang Liao, Lang Nie, Zishuo Zheng, Yao Zhao","year":"2022","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Su and Grauman, 2017] Yu-Chuan Su and Kristen Grau- man. Making 360 video watchable in 2d: Learning videog- raphy for click free viewing. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1368-1376. IEEE, 2017.","title":"Making 360 video watchable in 2d: Learning videography for click free viewing","authors":"Yu-Chuan Su, Kristen Grauman","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Su and Kristen, 2017] Yuchuan Su and Grauman. Kristen. Learning spherical convolution for fast features from 360 imagery. NIPS, 2017.","title":"Yuchuan Su and Grauman. Kristen. Learning spherical convolution for fast features from 360 imagery","authors":"Kristen Su","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sun et al., 2016] Yule Sun, Ang Lu, and Lu Yu. Ahg8: Ws- psnr for 360 video objective quality evaluation. Joint Video Exploration Team of ITU-T SG16 WP3 and ISO/IEC, JTC1/SC29/WG11(JVET-D0040), 2016.","title":"Ahg8: Wspsnr for 360 video objective quality evaluation","authors":"Yule Sun, Ang Lu, Lu Yu","year":"2016","venue":"Joint Video Exploration Team of ITU-T SG16 WP3 and ISO/IEC","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sun et al., 2021] Yuqi Sun, Ri Cheng, Bo Yan, and Shili Zhou. Space-angle super-resolution for multi-view im- ages. Proceedings of the 29th ACM International Con- ference on Multimedia, page 750-759, 2021.","title":"Space-angle super-resolution for multi-view images","authors":"Yuqi Sun, Ri Cheng, Bo Yan, Shili Zhou","year":"2021","venue":"Proceedings of the 29th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tai et al., 2017] Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu. Memnet: A persistent memory network for image restoration. 2017 IEEE International Conference on Computer Vision (ICCV), pages 4549-4557, 2017.","title":"Memnet: A persistent memory network for image restoration","authors":"Ying Tai, Jian Yang, Xiaoming Liu, Chunyan Xu","year":"2017","venue":"IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2021] Hu Wang, Peng Chen, Bohan Zhuang, and Chunhua Shen. Fully quantized image super- resolution networks. Proceedings of the 29th ACM Inter- national Conference on Multimedia, page 639-647, 2021.","title":"Fully quantized image superresolution networks","authors":"Hu Wang, Peng Chen, Bohan Zhuang, Chunhua Shen","year":"2021","venue":"Proceedings of the 29th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xiao et al., 2012] Jianxiong Xiao, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Recognizing scene viewpoint using panoramic place representation. 2012 IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 2695-2702, 2012.","title":"Recognizing scene viewpoint using panoramic place representation","authors":"Jianxiong Xiao, Krista A Ehinger, Aude Oliva, Antonio Torralba","year":"2012","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yoon et al., 2022] Youngho. Yoon, Inchul. Chung, Lin Wang, and Kuk-Jin Yoon. Spheresr: 360 image super- resolution with arbitrary projection via continuous spheri- cal image representation. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5667-5676, 2022.","title":"Spheresr: 360 image superresolution with arbitrary projection via continuous spherical image representation","authors":"Yoon","year":"2022","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yu et al., 2023] Fanghua Yu, Xintao Wang, Mingdeng Cao, Gen Li, Ying Shan, and Chao Dong. Osrt: Omnidirec- tional image super-resolution with distortion-aware trans- former. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.","title":"Osrt: Omnidirectional image super-resolution with distortion-aware transformer","authors":"Yu","year":"2023","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2018] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super- resolution using very deep residual channel attention net- works. ECCV 2018: Computer Vision, pages 294-310, 2018.","title":"Image superresolution using very deep residual channel attention networks","authors":"Zhang","year":"2018","venue":"ECCV 2018: Computer Vision","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2021] Yulun Zhang, Donglai Wei, Can Qin, Huan Wang, Hanspeter Pfister, and Yun Fu. Context rea- soning attention network for image super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4278-4287, 2021.","title":"Context reasoning attention network for image super-resolution","authors":"Zhang","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2023] Yunjian Zhang, Yanwei Liu, Jinxia Liu, Antonios Argyriou, Liming Wang, Zhen Xu, and Xi- angyang Ji. Social vr platform: Building 360-degree shared vr spaces. IEEE Transactions on Image Process- ing, 32:2552-2567, 2023.","title":"Social vr platform: Building 360-degree shared vr spaces","authors":"Yunjian Zhang, Yanwei Liu, Jinxia Liu, Antonios Argyriou, Liming Wang, Zhen Xu, Xiangyang Ji","year":"2023","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}