{"type":"ScholarlyArticle","title":"Lightweight Bimodal Network for Single-Image Super-Resolution via Symmetric CNN and Recursive Transformer","authors":[{"name":"Guangwei Gao","slug":"guangwei-gao","given":"Guangwei","family":"Gao","affiliation":"Nanjing University of Posts and Telecommunications , Nanjing , China; National Institute of Informatics , Tokyo , Japan"},{"name":"Zhengxue Wang","slug":"zhengxue-wang","given":"Zhengxue","family":"Wang","affiliation":"Nanjing University of Posts and Telecommunications , Nanjing , China"},{"name":"Juncheng Li","slug":"juncheng-li","given":"Juncheng","family":"Li","affiliation":"The Chinese University of Hong Kong , Hong Kong , China"},{"name":"Wenjie Li","slug":"wenjie-li","given":"Wenjie","family":"Li","affiliation":"Nanjing University of Posts and Telecommunications , Nanjing , China"},{"name":"Yi Yu","slug":"yi-yu","given":"Yi","family":"Yu","affiliation":"National Institute of Informatics , Tokyo , Japan"},{"name":"Tieyong Zeng","slug":"tieyong-zeng","given":"Tieyong","family":"Zeng","affiliation":"The Chinese University of Hong Kong , Hong Kong , China"}],"abstract":"Single-image super-resolution (SISR) has achieved significant breakthroughs with the development of deep learning. However, these methods are difficult to be applied in real-world scenarios since they are inevitably accompanied by the problems of computational and memory costs caused by the complex operations. To solve this issue, we propose a Lightweight Bimodal Network (LBNet) for SISR. Specifically, an effective Symmetric CNN is designed for local feature extraction and coarse image reconstruction. Meanwhile, we propose a Recursive Transformer to fully learn the long-term dependence of images thus the global information can be fully used to further refine texture details. Studies show that the hybrid of CNN and Transformer can build a more efficient model. Extensive experiments have proved that our LBNet achieves more prominent performance than other state-of-the-art methods with a relatively low computational cost and memory consumption. The code is available at https://github.com/IVIPLab/LBNet.","doi":"10.24963/ijcai.2022/128","url":"https://ijcai.org/proceedings/2022/128","pdf":"https://ijcai.org/proceedings/2022/0128.pdf","markdown":"https://ijcai.org/proceedings/2022/128.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/128","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":128,"pages":{"first":913,"last":919},"keywords":["Computer Vision: Representation Learning","Machine Learning: Feature Extraction, Selection and Dimensionality Reduction","Computer Vision: Machine Learning for Vision","Computer Vision: Computational photography"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986599","length_sec":90,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984742","length_sec":333,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Ahn et al., 2018] Namhyuk Ahn, Byungkon Kang, and Kyung. Fast, accurate, and lightweight super-resolution with cascading residual network. In ECCV, 2018.","title":"Fast, accurate, and lightweight super-resolution with cascading residual network","authors":"Ahn","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bevilacqua et al., 2012] Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie Line Alberi-Morel. Low- complexity single-image super-resolution based on non- negative neighbor embedding. In BMVC, 2012.","title":"Lowcomplexity single-image super-resolution based on nonnegative neighbor embedding","authors":"Marco Bevilacqua, Aline Roumy, Christine Guillemot, Marie Line Alberi-Morel","year":"2012","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2021] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In CVPR, 2021.","title":"Pre-trained image processing transformer","authors":"Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, Wen Gao","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dong et al., 2014] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning a deep convolutional net- work for image super-resolution. In ECCV, 2014.","title":"Learning a deep convolutional network for image super-resolution","authors":"Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gao et al., 2022] Guangwei Gao, Wenjie Li, Juncheng Li, Fei Wu, Huimin Lu, and Yi Yu. Feature distillation in- teraction weighting network for lightweight image super- resolution. In AAAI, 2022.","title":"Feature distillation interaction weighting network for lightweight image superresolution","authors":"Gao","year":"2022","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Huang et al., 2015] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-exemplars. In CVPR, 2015.","title":"Single image super-resolution from transformed self-exemplars","authors":"Jia-Bin Huang, Abhishek Singh, Narendra Ahuja","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2017] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.","title":"Densely connected convolutional networks","authors":"Gao Huang, Zhuang Liu, Laurens Van Der Maaten, Kilian Q Weinberger","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hui et al., 2018] Zheng Hui, Xiumei Wang, and Xinbo Gao. Fast and accurate single image super-resolution via infor- mation distillation network. In CVPR, 2018.","title":"Fast and accurate single image super-resolution via information distillation network","authors":"Zheng Hui, Xiumei Wang, Xinbo Gao","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hui et al., 2019] Zheng Hui, Xinbo Gao, Yunchu Yang, and Xiumei Wang. Lightweight image super-resolution with information multi-distillation network. In ACM MM, 2019.","title":"Lightweight image super-resolution with information multi-distillation network","authors":"Zheng Hui, Xinbo Gao, Yunchu Yang, Xiumei Wang","year":"2019","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kim et al., 2016a] Jiwon Kim, Jung Kwon Lee, and Ky- oung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In CVPR, 2016.","title":"Accurate image super-resolution using very deep convolutional networks","authors":"Kim","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kim et al., 2016b] Jiwon Kim, Jung Kwon Lee, and Ky- oung Mu Lee. Deeply-recursive convolutional network for image super-resolution. In CVPR, 2016.","title":"Deeply-recursive convolutional network for image super-resolution","authors":"Kim","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lan et al., 2020] Rushi Lan, Long Sun, Zhenbing Liu, Huimin Lu, Cheng Pang, and Xiaonan Luo. Madnet: A fast and lightweight network for single-image super reso- lution. IEEE TCYB, 51(3):1443-1453, 2020.","title":"Madnet: A fast and lightweight network for single-image super resolution","authors":"Rushi Lan, Long Sun, Zhenbing Liu, Huimin Lu, Cheng Pang, Xiaonan Luo","year":"2020","venue":"IEEE TCYB","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2021a] Juncheng Li, Zehua Pei, and Tieyong Zeng. From beginner to master: A survey for deep learning- based single-image super-resolution. arXiv preprint arXiv:2109.14335, 2021.","title":"From beginner to master: A survey for deep learningbased single-image super-resolution","authors":"Li","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2021b] Wenbo Li, Kun Zhou, Lu Qi, Nianjuan Jiang, Jiangbo Lu, and Jiaya Jia. Lapar: Linearly- assembled pixel-adaptive regression network for single image super-resolution and beyond. In NeurIPS, 2021.","title":"Lapar: Linearlyassembled pixel-adaptive regression network for single image super-resolution and beyond","authors":"Li","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liang et al., 2021] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In ICCV, 2021.","title":"Swinir: Image restoration using swin transformer","authors":"Liang","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lim et al., 2017] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In CVPRW, 2017.","title":"Enhanced deep residual networks for single image super-resolution","authors":"Lim","year":"2017","venue":"CVPRW","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lu et al., 2022] Zhisheng Lu, Juncheng Li, Hong Liu, Chaoyan Huang, Linlin Zhang, and Tieyong Zeng. Trans- former for single image super-resolution. In CVPRW, 2022.","title":"Transformer for single image super-resolution","authors":"Zhisheng Lu, Juncheng Li, Hong Liu, Chaoyan Huang, Linlin Zhang, Tieyong Zeng","year":"2022","venue":"CVPRW","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Martin et al., 2001] David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluat- ing segmentation algorithms and measuring ecological statistics. In ICCV, 2001.","title":"A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics","authors":"David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik","year":"2001","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Matsui et al., 2017] Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. Sketch-based manga retrieval using manga109 dataset. MTAP, 76(20):21811-21838, 2017.","title":"Sketch-based manga retrieval using manga109 dataset","authors":"Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, Kiyoharu Aizawa","year":"2017","venue":"MTAP","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tai et al., 2017] Ying Tai, Jian Yang, and Xiaoming Liu. Image super-resolution via deep recursive residual net- work. In CVPR, 2017.","title":"Image super-resolution via deep recursive residual network","authors":"Ying Tai, Jian Yang, Xiaoming Liu","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2019] Chaofeng Wang, Zheng Li, and Jun Shi. Lightweight image super-resolution with adap- tive weighted learning network. arXiv preprint arXiv:1904.02358, 2019.","title":"Lightweight image super-resolution with adaptive weighted learning network","authors":"Chaofeng Wang, Zheng Li, Jun Shi","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2021] Longguang Wang, Xiaoyu Dong, Yingqian Wang, Xinyi Ying, Zaiping Lin, Wei An, and Yulan Guo. Exploring sparsity in image super-resolution for efficient inference. In CVPR, 2021.","title":"Exploring sparsity in image super-resolution for efficient inference","authors":"Wang","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xiao et al., 2021] Jun Xiao, Qian Ye, Rui Zhao, Kin-Man Lam, and Kao Wan. Self-feature learning: An efficient deep lightweight network for image super-resolution. In ACM MM, 2021.","title":"Self-feature learning: An efficient deep lightweight network for image super-resolution","authors":"Jun Xiao, Qian Ye, Rui Zhao, Kin-Man Lam, Kao Wan","year":"2021","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zeyde et al., 2010] Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse- representations. In ICCS, 2010.","title":"On single image scale-up using sparserepresentations","authors":"Zeyde","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2018] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super- resolution using very deep residual channel attention net- works. In ECCV, 2018.","title":"Image superresolution using very deep residual channel attention networks","authors":"Zhang","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2021a] Dongyang Zhang, Changyu Li, Ning Xie, Guoqing Wang, and Jie Shao. Pffn: Progres- sive feature fusion network for lightweight image super- resolution. In ACM MM, 2021.","title":"Pffn: Progressive feature fusion network for lightweight image superresolution","authors":"Zhang","year":"2021","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2021b] Xinyan Zhang, Peng Gao, Sunxiangyu Liu, Kongya Zhao, Guitao Li, Liuguo Yin, and Chang Wen Chen. Accurate and efficient image super-resolution via global-local adjusting dense network. IEEE TMM, 23:1924-1937, 2021.","title":"Accurate and efficient image super-resolution via global-local adjusting dense network","authors":"Zhang","year":"2021","venue":"IEEE TMM","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}