{"type":"ScholarlyArticle","title":"A Novel Learnable Interpolation Approach for Scale-Arbitrary Image Super-Resolution","authors":[{"name":"Jiahao Chao","slug":"jiahao-chao","given":"Jiahao","family":"Chao","affiliation":"Shanghai Key Lab of Trustworthy Computing , East China Normal University , Shanghai , China;"},{"name":"Zhou Zhou","slug":"zhou-zhou","given":"Zhou","family":"Zhou","affiliation":"Shanghai Key Lab of Trustworthy Computing , East China Normal University , Shanghai , China;"},{"name":"Hongfan Gao","slug":"hongfan-gao","given":"Hongfan","family":"Gao","affiliation":"Shanghai Key Lab of Trustworthy Computing , East China Normal University , Shanghai , China;"},{"name":"Jiali Gong","slug":"jiali-gong","given":"Jiali","family":"Gong","affiliation":"Shanghai Key Lab of Trustworthy Computing , East China Normal University , Shanghai , China;"},{"name":"Zhenbing Zeng","slug":"zhenbing-zeng","given":"Zhenbing","family":"Zeng","affiliation":"Shanghai University of Finance and Economics Zhejiang College , Jinhua , China"},{"name":"Zhengfeng Yang","slug":"zhengfeng-yang","given":"Zhengfeng","family":"Yang","affiliation":"Shanghai Key Lab of Trustworthy Computing , East China Normal University , Shanghai , China;"}],"abstract":"Deep convolutional neural networks (CNNs) have achieved unprecedented success in single image super-resolution over the past few years. Meanwhile, there is an increasing demand for single image super-resolution with arbitrary scale factors in real-world scenarios. Many approaches adopt scale-specific multi-path learning to cope with multi-scale super-resolution with a single network. However, these methods require a large number of parameters. To achieve a better balance between the reconstruction quality and parameter amounts, we proposes a learnable interpolation method that leverages the advantages of neural networks and interpolation methods to tackle the scale-arbitrary super-resolution task. The scale factor is treated as a function parameter for generating the kernel weights for the learnable interpolation. We demonstrate that the learnable interpolation builds a bridge between neural networks and traditional interpolation methods. Experiments show that the proposed learnable interpolation requires much fewer parameters and outperforms state-of-the-art super-resolution methods.","doi":"10.24963/ijcai.2023/63","url":"https://ijcai.org/proceedings/2023/63","pdf":"https://ijcai.org/proceedings/2023/0063.pdf","markdown":"https://ijcai.org/proceedings/2023/63.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/63","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":63,"pages":{"first":564,"last":572},"keywords":["Computer Vision: CV: Other","Machine Learning: ML: Convolutional networks"],"videos":[],"references":[{"position":1,"raw":"[Aghighi, 2015] Hossein Aghighi. Markov random field models for classification of remote sensing data and super- resolution mapping. PhD thesis, University of New South Wales, Sydney, Australia, 2015.","title":"Markov random field models for classification of remote sensing data and superresolution mapping","authors":"Hossein Aghighi","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ahn et al., 2018] Namhyuk Ahn, Byungkon Kang, and Kyung-Ah Sohn. Fast, accurate, and lightweight super- resolution with cascading residual network. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision -ECCV 2018 -15th Eu- ropean Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part X, volume 11214 of Lecture Notes in Computer Science, pages 256-272. Springer, 2018.","title":"Fast, accurate, and lightweight superresolution with cascading residual network","authors":"Ahn","year":"2018","venue":"Computer Vision -ECCV 2018 -15th European Conference","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Akhtar et al., 2015] Naveed Akhtar, Faisal Shafait, and Aj- mal S. Mian. Bayesian sparse representation for hyper- spectral image super resolution. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 3631-3640. IEEE Computer Society, 2015.","title":"Bayesian sparse representation for hyperspectral image super resolution","authors":"Naveed Akhtar, Faisal Shafait, Ajmal S Mian","year":"2015","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bevilacqua et al., 2012] Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie-Line Alberi-Morel. Low- complexity single-image super-resolution based on non- negative neighbor embedding. In Richard Bowden, John P. Collomosse, and Krystian Mikolajczyk, editors, British Machine Vision Conference, BMVC 2012, Surrey, UK, September 3-7, 2012, pages 1-10. BMVA Press, 2012.","title":"Lowcomplexity single-image super-resolution based on nonnegative neighbor embedding","authors":"Marco Bevilacqua, Aline Roumy, Christine Guillemot, Marie-Line Alberi-Morel","year":"2012","venue":"British Machine Vision Conference, BMVC 2012","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2021] Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with lo- cal implicit image function. In IEEE Conference on Com- puter Vision and Pattern Recognition, CVPR 2021, vir- tual, June 19-25, 2021, pages 8628-8638. Computer Vi- sion Foundation / IEEE, 2021.","title":"Learning continuous image representation with local implicit image function","authors":"Chen","year":"2021","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[De Boor, 1962] Carl De Boor. Bicubic spline interpola- tion. Journal of mathematics and physics, 41(1-4):212- 218, 1962.","title":"Carl De Boor. Bicubic spline interpolation","authors":"De Boor","year":"1962","venue":"Journal of mathematics and physics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dong et al., 2015] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. CoRR, abs/1501.00092, 2015.","title":"Image super-resolution using deep convolutional networks","authors":"Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Guo et al., 2021] Mantang Guo, Jing Jin, Hui Liu, and Junhui Hou. Learning dynamic interpolation for ex- tremely sparse light fields with wide baselines. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 2430-2439. IEEE, 2021.","title":"Learning dynamic interpolation for extremely sparse light fields with wide baselines","authors":"Guo","year":"2021","venue":"IEEE/CVF International Conference on Computer Vision, ICCV 2021","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hu et al., 2019] Xuecai Hu, Haoyuan Mu, Xiangyu Zhang, Zilei Wang, Tieniu Tan, and Jian Sun. Meta-sr: A magnification-arbitrary network for super-resolution. In IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 1575-1584. Computer Vision Foundation / IEEE, 2019.","title":"Meta-sr: A magnification-arbitrary network for super-resolution","authors":"Hu","year":"2019","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2015] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-exemplars. In IEEE Conference on Com- puter Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 5197-5206. IEEE Com- puter Society, 2015.","title":"Single image super-resolution from transformed self-exemplars","authors":"Jia-Bin Huang, Abhishek Singh, Narendra Ahuja","year":"2015","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"ICLR (Poster)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kong et al., 2021] Xiangtao Kong, Xina Liu, Jinjin Gu, Yu Qiao, and Chao Dong. Reflash dropout in image super- resolution. CoRR, abs/2112.12089, 2021.","title":"Reflash dropout in image superresolution","authors":"Kong","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liang et al., 2021] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In IEEE/CVF International Conference on Computer Vision Workshops, ICCVW 2021, Montreal, BC, Canada, October 11-17, 2021, pages 1833-1844. IEEE, 2021.","title":"Swinir: Image restoration using swin transformer","authors":"Liang","year":"2021","venue":"IEEE/CVF International Conference on Computer Vision Workshops, ICCVW 2021","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lim et al., 2017] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep resid- ual networks for single image super-resolution. In CVPR Workshops, pages 1132-1140. IEEE Computer Society, 2017.","title":"Enhanced deep residual networks for single image super-resolution","authors":"Lim","year":"2017","venue":"CVPR Workshops","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lin et al., 2008] Chung-Chi Lin, Ming-Hwa Sheu, Huann- Keng Chiang, Chishyan Liaw, and Zeng-Chuan Wu. The efficient VLSI design of BI-CUBIC convolution interpo- lation for digital image processing. In International Sym- posium on Circuits and Systems (ISCAS 2008), 18-21 May 2008, Sheraton Seattle Hotel, Seattle, Washington, USA, pages 480-483. IEEE, 2008.","title":"The efficient VLSI design of BI-CUBIC convolution interpolation for digital image processing","authors":"Chung-Chi Lin, Ming-Hwa Sheu, Huann-Keng Chiang, Chishyan Liaw, Zeng-Chuan Wu","year":"2008","venue":"International Symposium on Circuits and Systems (ISCAS 2008)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In 2021 IEEE/CVF International Con- ference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 9992-10002. IEEE, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo","year":"2021","venue":"IEEE/CVF International Conference on Computer Vision, ICCV 2021","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Maas et al., 2013] Andrew L Maas, Awni Y Hannun, An- drew Y Ng, et al. Rectifier nonlinearities improve neu- ral network acoustic models. In Proc. icml, volume 30, page 3. Citeseer, 2013.","title":"Rectifier nonlinearities improve neural network acoustic models","authors":"Andrew L Maas, Andrew Y Awni Y Hannun, Ng","year":"2013","venue":"Proc. icml","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Martin et al., 2001] David R. Martin, Charless C. Fowlkes, Doron Tal, and Jitendra Malik. A database of human seg- mented natural images and its application to evaluating segmentation algorithms and measuring ecological statis- tics. In ICCV, pages 416-425. IEEE Computer Society, 2001.","title":"A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics","authors":"David R Martin, Charless C Fowlkes, Doron Tal, Jitendra Malik","year":"2001","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[NVIDIA et al., 2020] NVIDIA, Péter Vingelmann, and Frank H.P. Fitzek. Cuda, release: 10.2.89, 2020.","title":"NVIDIA, Péter Vingelmann","authors":"Nvidia","year":"2020","venue":"Cuda","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Okuta et al., 2017] Ryosuke Okuta, Yuya Unno, Daisuke Nishino, Shohei Hido, and Crissman Loomis. Cupy: A numpy-compatible library for nvidia gpu calculations. In Proceedings of Workshop on Machine Learning Systems (LearningSys) in The Thirty-first Annual Conference on Neural Information Processing Systems (NIPS), 2017.","title":"Cupy: A numpy-compatible library for nvidia gpu calculations","authors":"Ryosuke Okuta, Yuya Unno, Daisuke Nishino, Shohei Hido, Crissman Loomis","year":"2017","venue":"Proceedings of Workshop on Machine Learning Systems (LearningSys) in The Thirty-first Annual Conference on Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Park et al., 2020] Seobin Park, Jinsu Yoo, Donghyeon Cho, Jiwon Kim, and Tae Hyun Kim. Fast adaptation to super- resolution networks via meta-learning. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, ed- itors, Computer Vision -ECCV 2020 -16th European Con- ference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVII, volume 12372 of Lecture Notes in Computer Science, pages 754-769. Springer, 2020.","title":"Fast adaptation to superresolution networks via meta-learning","authors":"Park","year":"2020","venue":"Computer Vision -ECCV 2020 -16th European Conference","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Paszke et al., 2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché- Buc, Emily B. Fox, and Roman Garnett, editors, Ad- vances in Neural Information Processing Systems 32: An- nual Conference on Neural Information Processing Sys- tems 2019, NeurIPS 2019, December 8-14, 2019, Vancou- ver, BC, Canada, pages 8024-8035, 2019.","title":"Pytorch: An imperative style, highperformance deep learning library","authors":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z Yang, Zachary Devito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, Soumith Chintala","year":"2019","venue":"Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Shi et al., 2016] Wenzhe Shi, Jose Caballero, Ferenc Huszar, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single im- age and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR, pages 1874- 1883. IEEE Computer Society, 2016.","title":"Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network","authors":"Shi","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Simonyan et al., 2014] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Yoshua Bengio and Yann LeCun, edi- tors, 2nd International Conference on Learning Represen- tations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings, 2014.","title":"Deep inside convolutional networks: Visualising image classification models and saliency maps","authors":"Karen Simonyan, Andrea Vedaldi, Andrew Zisserman","year":"2014","venue":"editors, 2nd International Conference on Learning Representations, ICLR 2014","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Timofte et al., 2017] Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Yang, Lei Zhang, Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee et al. NTIRE 2017 challenge on single image super-resolution: Methods and results. In CVPR Work- shops, pages 1110-1121. IEEE Computer Society, 2017.","title":"NTIRE 2017 challenge on single image super-resolution: Methods and results","authors":"Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Yang, Lei Zhang, Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, Kyoung Mu, Lee","year":"2017","venue":"CVPR Workshops","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1706.03762, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2004] Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process., 13(4):600-612, 2004.","title":"Image quality assessment: from error visibility to structural similarity","authors":"Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli","year":"2004","venue":"IEEE Trans. Image Process","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2018a] Xintao Wang, Ke Yu, Kelvin C.K. Chan, Chao Dong, and Chen Change Loy. BasicSR: Open source image and video restoration toolbox. https: //github.com/xinntao/BasicSR, 2018.","title":"BasicSR: Open source image and video restoration toolbox","authors":"Wang","year":"2018","venue":null,"doi":null,"url":"https://github.com/xinntao/BasicSR,2018","internal_url":null},{"position":29,"raw":"[Wang et al., 2018b] Yifan Wang, Federico Perazzi, Brian McWilliams, Alexander Sorkine-Hornung, Olga Sorkine- Hornung, and Christopher Schroers. A fully progressive approach to single-image super-resolution. In 2018 IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 864-873. Computer Vision Foundation / IEEE Computer Society, 2018.","title":"A fully progressive approach to single-image super-resolution","authors":"Wang","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2018","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2021] Longguang Wang, Yingqian Wang, Zaiping Lin, Jungang Yang, Wei An, and Yulan Guo. Learning A single network for scale-arbitrary super- resolution. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 4781-4790. IEEE, 2021.","title":"Learning A single network for scale-arbitrary superresolution","authors":"Wang","year":"2021","venue":"IEEE/CVF International Conference on Computer Vision, ICCV 2021","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zeyde et al., 2010] Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse- representations. In Curves and Surfaces, volume 6920 of Lecture Notes in Computer Science, pages 711-730.","title":"On single image scale-up using sparserepresentations","authors":"Zeyde","year":"2010","venue":"Curves and Surfaces","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2018a] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super- resolution using very deep residual channel attention net- works. In Vittorio Ferrari, Martial Hebert, Cristian Smin- chisescu, and Yair Weiss, editors, Computer Vision - ECCV 2018 -15th European Conference, Munich, Ger- many, September 8-14, 2018, Proceedings, Part VII, vol- ume 11211 of Lecture Notes in Computer Science, pages 294-310. Springer, 2018.","title":"Image superresolution using very deep residual channel attention networks","authors":"Zhang","year":"2018","venue":"Computer Vision -ECCV 2018 -15th European Conference","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2018b] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 2472- 2481. Computer Vision Foundation / IEEE Computer So- ciety, 2018.","title":"Residual dense network for image super-resolution","authors":"Zhang","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}