{"type":"ScholarlyArticle","title":"Hundred-Kilobyte Lookup Tables for Efficient Single-Image Super-Resolution","authors":[{"name":"Binxiao Huang","slug":"binxiao-huang","given":"Binxiao","family":"Huang","affiliation":"The University of Hong Kong"},{"name":"Jason Chun Lok Li","slug":"jason-chun-lok-li","given":"Jason Chun Lok","family":"Li","affiliation":"The University of Hong Kong"},{"name":"Jie Ran","slug":"jie-ran","given":"Jie","family":"Ran","affiliation":"The University of Hong Kong"},{"name":"Boyu Li","slug":"boyu-li","given":"Boyu","family":"Li","affiliation":"The University of Hong Kong"},{"name":"Jiajun Zhou","slug":"jiajun-zhou","given":"Jiajun","family":"Zhou","affiliation":"The University of Hong Kong"},{"name":"Dahai Yu","slug":"dahai-yu","given":"Dahai","family":"Yu","affiliation":"TCL Corporate Research"},{"name":"Ngai Wong","slug":"ngai-wong","given":"Ngai","family":"Wong","affiliation":"The University of Hong Kong"}],"abstract":"Conventional super-resolution (SR) schemes make heavy use of convolutional neural networks (CNNs), which involve intensive multiply-accumulate (MAC) operations, and require specialized hardware such as graphics processing units. This contradicts the regime of edge AI that often runs on devices strained by power, computing, and storage resources. Such a challenge has motivated a series of lookup table (LUT)-based SR schemes that employ simple LUT readout and largely elude CNN computation. Nonetheless, the multi-megabyte LUTs in existing methods still prohibit on-chip storage and necessitate off-chip memory transport. This work tackles this storage hurdle and innovates hundred-kilobyte LUT (HKLUT) models amenable to on-chip cache. Utilizing an asymmetric two-branch multistage network coupled with a suite of specialized kernel patterns, HKLUT demonstrates an uncompromising performance and superior hardware efficiency over existing LUT schemes. Our implementation is publicly available at: https://github.com/jasonli0707/hklut.","doi":"10.24963/ijcai.2024/95","url":"https://ijcai.org/proceedings/2024/95","pdf":"https://ijcai.org/proceedings/2024/0095.pdf","markdown":"https://ijcai.org/proceedings/2024/95.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/95","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":95,"pages":{"first":857,"last":865},"keywords":["Computer Vision: CV: Applications","Computer Vision: CV: Image and video synthesis and generation"],"videos":[],"references":[{"position":1,"raw":"[Agustsson and Timofte, 2017] Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super- resolution: Dataset and study. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 126-135, 2017.","title":"Ntire 2017 challenge on single image superresolution: Dataset and study","authors":"Eirikur Agustsson, Radu Timofte","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition workshops","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ahn et al., 2018] Namhyuk Ahn, Byungkon Kang, and Kyung-Ah Sohn. Fast, accurate, and lightweight super- resolution with cascading residual network. In Pro- ceedings of the European conference on computer vision (ECCV), pages 252-268, 2018.","title":"Fast, accurate, and lightweight superresolution with cascading residual network","authors":"Ahn","year":"2018","venue":"Proceedings of the European conference on computer vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bevilacqua et al., 2012] Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie Line Alberi-Morel. Low- complexity single-image super-resolution based on non- negative neighbor embedding. 2012.","title":"Lowcomplexity single-image super-resolution based on nonnegative neighbor embedding","authors":"Marco Bevilacqua, Aline Roumy, Christine Guillemot, Marie Line Alberi-Morel","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Du et al., 2022] Zongcai Du, Ding Liu, Jie Liu, Jie Tang, Gangshan Wu, and Lean Fu. Fast and memory-efficient network towards efficient image super-resolution. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 853-862, 2022.","title":"Fast and memory-efficient network towards efficient image super-resolution","authors":"Zongcai Du, Ding Liu, Jie Liu, Jie Tang, Gangshan Wu, Lean Fu","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Glorot et al., 2011] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 315-323. JMLR Workshop and Conference Proceedings, 2011.","title":"Deep sparse rectifier neural networks","authors":"Xavier Glorot, Antoine Bordes, Yoshua Bengio","year":"2011","venue":"Proceedings of the fourteenth international conference on artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Huang et al., 2015] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-exemplars. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5197-5206, 2015.","title":"Single image super-resolution from transformed self-exemplars","authors":"Jia-Bin Huang, Abhishek Singh, Narendra Ahuja","year":"2015","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hui et al., 2019] Zheng Hui, Xinbo Gao, Yunchu Yang, and Xiumei Wang. Lightweight image super-resolution with information multi-distillation network. In Proceedings of the 27th acm international conference on multimedia, pages 2024-2032, 2019.","title":"Lightweight image super-resolution with information multi-distillation network","authors":"Zheng Hui, Xinbo Gao, Yunchu Yang, Xiumei Wang","year":"2019","venue":"Proceedings of the 27th acm international conference on multimedia","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jo and Kim, 2021] Younghyun Jo and Seon Joo Kim. Prac- tical single-image super-resolution using look-up table. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 691-700, 2021.","title":"Practical single-image super-resolution using look-up table","authors":"Jo, Kim ;, Younghyun Jo, Seon Joo, Kim","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Keys, 1981] Robert Keys. Cubic convolution interpolation for digital image processing. IEEE transactions on acous- tics, speech, and signal processing, 29(6):1153-1160, 1981.","title":"Cubic convolution interpolation for digital image processing","authors":"Robert Keys","year":"1981","venue":"IEEE transactions on acoustics, speech, and signal processing","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kirkland and Kirkland, 2010] Earl J Kirkland and Earl J Kirkland. Bilinear interpolation. Advanced Computing in Electron Microscopy, pages 261-263, 2010.","title":"Bilinear interpolation","authors":"Kirkland ; ] Kirkland, J Earl, Earl J Kirkland, Kirkland","year":"2010","venue":"Advanced Computing in Electron Microscopy","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lai et al., 2017] Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. Deep laplacian pyramid networks for fast and accurate super-resolution. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 624-632, 2017.","title":"Deep laplacian pyramid networks for fast and accurate super-resolution","authors":"Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, Ming-Hsuan Yang","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lai et al., 2018] Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. Fast and accurate image super-resolution with deep laplacian pyramid networks. IEEE transactions on pattern analysis and machine intel- ligence, 41(11):2599-2613, 2018.","title":"Fast and accurate image super-resolution with deep laplacian pyramid networks","authors":"Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, Ming-Hsuan Yang","year":"2018","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2022] Jiacheng Li, Chang Chen, Zhen Cheng, and Zhiwei Xiong. Mulut: Cooperating multiple look-up ta- bles for efficient image super-resolution. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVIII, pages 238-256. Springer, 2022.","title":"Mulut: Cooperating multiple look-up tables for efficient image super-resolution","authors":"Jiacheng Li, Chang Chen, Zhen Cheng, Zhiwei Xiong","year":"2022","venue":"Computer Vision-ECCV 2022: 17th European Conference","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2023] Guandu Liu, Yukang Ding, Mading Li, Ming Sun, Xing Wen, and Bin Wang. Reconstructed con- volution module based look-up tables for efficient image super-resolution. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 12217- 12226, 2023.","title":"Reconstructed convolution module based look-up tables for efficient image super-resolution","authors":"Guandu Liu, Yukang Ding, Mading Li, Ming Sun, Xing Wen, Bin Wang","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Luo et al., 2016] Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. Advances in neural information processing systems, 29, 2016.","title":"Understanding the effective receptive field in deep convolutional neural networks","authors":"Wenjie Luo, Yujia Li, Raquel Urtasun, Richard Zemel","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ma et al., 2022] Cheng Ma, Jingyi Zhang, Jie Zhou, and Ji- wen Lu. Learning series-parallel lookup tables for effi- cient image super-resolution. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, Octo- ber 23-27, 2022, Proceedings, Part XVII, pages 305-321. Springer, 2022.","title":"Learning series-parallel lookup tables for efficient image super-resolution","authors":"Cheng Ma, Jingyi Zhang, Jie Zhou, Jiwen Lu","year":"2022","venue":"Computer Vision-ECCV 2022: 17th European Conference","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Martin et al., 2001] David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluat- ing segmentation algorithms and measuring ecological statistics. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 2, pages 416-423. IEEE, 2001.","title":"A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics","authors":"David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik","year":"2001","venue":"Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Matsui et al., 2017] Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. Sketch-based manga retrieval using manga109 dataset. Multimedia Tools and Applications, 76:21811-21838, 2017.","title":"Sketch-based manga retrieval using manga109 dataset","authors":"Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, Kiyoharu Aizawa","year":"2017","venue":"Multimedia Tools and Applications","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shi et al., 2016a] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single im- age and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1874-1883, 2016.","title":"Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network","authors":"Shi","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shi et al., 2016b] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single im- age and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1874-1883, 2016.","title":"Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network","authors":"Shi","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Song et al., 2021] Dehua Song, Yunhe Wang, Hanting Chen, Chang Xu, Chunjing Xu, and DaCheng Tao. Addersr: Towards energy efficient image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15648-15657, 2021.","title":"Addersr: Towards energy efficient image super-resolution","authors":"Dehua Song, Yunhe Wang, Hanting Chen, Chang Xu, Chunjing Xu, Dacheng Tao","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Timofte et al., 2013] Radu Timofte, Vincent De Smet, and Luc Van Gool. Anchored neighborhood regression for fast example-based super-resolution. In Proceedings of the IEEE international conference on computer vision, pages 1920-1927, 2013.","title":"Anchored neighborhood regression for fast example-based super-resolution","authors":"Radu Timofte, Vincent De Smet, Luc Van Gool","year":"2013","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Timofte et al., 2015] Radu Timofte, Vincent De Smet, and Luc Van Gool. A+: Adjusted anchored neighborhood re- gression for fast super-resolution. In Computer Vision- ACCV 2014: 12th Asian Conference on Computer Vision, Singapore, Singapore, November 1-5, 2014, Revised Se- lected Papers, Part IV 12, pages 111-126. Springer, 2015.","title":"A+: Adjusted anchored neighborhood regression for fast super-resolution","authors":"Radu Timofte, Vincent De Smet, Luc Van Gool","year":"2014","venue":"Computer Vision-ACCV 2014: 12th Asian Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transac- tions on image processing, 13(4):600-612, 2004.","title":"Image quality assessment: from error visibility to structural similarity","authors":"Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli","year":"2004","venue":"IEEE transactions on image processing","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2018] Xintao Wang, Ke Yu, Shixiang Wu, Jin- jin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution genera- tive adversarial networks. In Proceedings of the European conference on computer vision (ECCV) workshops, pages 0-0, 2018.","title":"Esrgan: Enhanced super-resolution generative adversarial networks","authors":"Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, Chen Change Loy","year":"2018","venue":"Proceedings of the European conference on computer vision (ECCV) workshops","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zeyde et al., 2012] Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse- representations. In Curves and Surfaces: 7th International Conference, Avignon, France, June 24-30, 2010, Revised Selected Papers 7, pages 711-730. Springer, 2012.","title":"On single image scale-up using sparserepresentations","authors":"Zeyde","year":"2010","venue":"Curves and Surfaces: 7th International Conference","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}