{"type":"ScholarlyArticle","title":"MetaISP: Efficient RAW-to-sRGB Mappings with Merely 1M Parameters","authors":[{"name":"Zigeng Chen","slug":"zigeng-chen","given":"Zigeng","family":"Chen","affiliation":"National University of Singapore"},{"name":"Chaowei Liu","slug":"chaowei-liu","given":"Chaowei","family":"Liu","affiliation":"National University of Singapore"},{"name":"Yuan Yuan","slug":"yuan-yuan","given":"Yuan","family":"Yuan","affiliation":"Huawei Technologies Ltd"},{"name":"Michael Bi Mi","slug":"michael-bi-mi","given":"Michael","family":"Bi Mi","affiliation":"Huawei Technologies Ltd"},{"name":"Xinchao Wang","slug":"xinchao-wang","given":"Xinchao","family":"Wang","affiliation":"National University of Singapore"}],"abstract":"State-of-the-art deep ISP models alleviate the dilemma of limited generalization capabilities across heterogeneous inputs by increasing the size and complexity of the network,  which inevitably leads to considerable growth in parameter counts and FLOPs. To address this challenge, this paper presents MetaISP - a streamlined model that achieves superior reconstruction quality by adaptively modulating its parameters and architecture in response to diverse inputs. Our rationale revolves around obtaining corresponding spatial and channel-wise correction matrices for various inputs within distinct feature spaces, which assists in assigning optimal attention. This is achieved by predicting dynamic weights for each input image and combining these weights with multiple learnable basis matrices to construct the correction matrices. The proposed MetaISP makes it possible to obtain best performance while being computationally efficient. SOTA results are achieved on two large-scale datasets, e.g. 23.80dB PSNR on ZRR, exceeding the previous SOTA 0.19dB with only 9.2% of its parameter count and 10.6% of its FLOPs; 25.06dB PSNR on MAI21, exceeding the previous SOTA 0.17dB with only 0.9% of its parameter count and 2.7% of its FLOPs.","doi":"10.24963/ijcai.2024/76","url":"https://ijcai.org/proceedings/2024/76","pdf":"https://ijcai.org/proceedings/2024/0076.pdf","markdown":"https://ijcai.org/proceedings/2024/76.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/76","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":76,"pages":{"first":686,"last":694},"keywords":["Computer Vision: CV: Computational photography","Computer Vision: CV: Applications"],"videos":[],"references":[{"position":1,"raw":"[Ba et al., 2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Ge- offrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.","title":"Layer normalization","authors":"Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E Hinton","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2021] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12299-12310, 2021.","title":"Pre-trained image processing transformer","authors":"Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, Wen Gao","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dai et al., 2020] Linhui Dai, Xiaohong Liu, Chengqi Li, and Jun Chen. Awnet: Attentive wavelet network for image isp. In Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pages 185-201. Springer, 2020.","title":"Awnet: Attentive wavelet network for image isp","authors":"Dai","year":"2020","venue":"Computer Vision-ECCV 2020 Workshops","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hendrycks and Gimpel, 2016] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.","title":"Gaussian error linear units","authors":"Gimpel Hendrycks, Dan Hendrycks, Kevin Gimpel","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hsyu et al., 2021] Ming-Chun Hsyu, Chih-Wei Liu, Chao- Hung Chen, Chao-Wei Chen, and Wen-Chia Tsai. Csanet: High speed channel spatial attention network for mobile isp. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 2486-2493, 2021.","title":"Csanet: High speed channel spatial attention network for mobile isp","authors":"Ming-Chun Hsyu, Chih-Wei Liu, Chao-Hung Chen, Chao-Wei Chen, Wen-Chia Tsai","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2018] Jie Hu, Li Shen, and Gang Sun. Squeeze- and-excitation networks. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pages 7132-7141, 2018.","title":"Squeezeand-excitation networks","authors":"Jie Hu, Li Shen, Gang Sun","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ignatov et al., 2020a] Andrey Ignatov, Radu Timofte, Zhilu Zhang, Ming Liu, Haolin Wang, Wangmeng Zuo, Jiawei Zhang, Ruimao Zhang, Zhanglin Peng, Sijie Ren, et al. Aim 2020 challenge on learned image signal processing pipeline. In Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pages 152-170. Springer, 2020.","title":"Aim 2020 challenge on learned image signal processing pipeline","authors":"Ignatov","year":"2020","venue":"Computer Vision-ECCV 2020 Workshops","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ignatov et al., 2020b] Andrey Ignatov, Luc Van Gool, and Radu Timofte. Replacing mobile camera isp with a single deep learning model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 536-537, 2020.","title":"Replacing mobile camera isp with a single deep learning model","authors":"Ignatov","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ignatov et al., 2022a] Andrey Ignatov, Grigory Malivenko, Radu Timofte, Yu Tseng, Yu-Syuan Xu, Po-Hsiang Yu, Cheng-Ming Chiang, Hsien-Kai Kuo, Min-Hung Chen, Chia-Ming Cheng, et al. Pynet-v2 mobile: Efficient on-device photo processing with neural networks. In 2022 26th International Conference on Pattern Recogni- tion (ICPR), pages 677-684. IEEE, 2022.","title":"Pynet-v2 mobile: Efficient on-device photo processing with neural networks","authors":"Ignatov","year":"2022","venue":"26th International Conference on Pattern Recognition (ICPR)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ignatov et al., 2022b] Andrey Ignatov, Radu Timofte, Shuai Liu, Chaoyu Feng, Furui Bai, Xiaotao Wang, Lei Lei, Ziyao Yi, Yan Xiang, Zibin Liu, et al. Learned smartphone isp on mobile gpus with deep learning, mobile ai & aim 2022 challenge: Report. arXiv preprint arXiv:2211.03885, 2022.","title":"Learned smartphone isp on mobile gpus with deep learning, mobile ai & aim 2022 challenge","authors":"Ignatov","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ignatov et al., 2023a] Andrey Ignatov, Anastasia Sycheva, Radu Timofte, Yu Tseng, Yu-Syuan Xu, Po-Hsiang Yu, Cheng-Ming Chiang, Hsien-Kai Kuo, Min-Hung Chen, Chia-Ming Cheng, et al. Microisp: Processing 32mp pho- tos on mobile devices with deep learning. In Computer Vision-ECCV 2022 Workshops: Tel Aviv, Israel, Octo- ber 23-27, 2022, Proceedings, Part II, pages 729-746. Springer, 2023.","title":"Microisp: Processing 32mp photos on mobile devices with deep learning","authors":"Ignatov","year":"2022","venue":"Computer Vision-ECCV 2022 Workshops","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ignatov et al., 2023b] Andrey Ignatov, Radu Timofte, Shuai Liu, Chaoyu Feng, Furui Bai, Xiaotao Wang, Lei Lei, Ziyao Yi, Yan Xiang, Zibin Liu, et al. Learned smart- phone isp on mobile gpus with deep learning, mobile ai & aim 2022 challenge: report. In Computer Vision-ECCV 2022 Workshops: Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part III, pages 44-70. Springer, 2023.","title":"Learned smartphone isp on mobile gpus with deep learning, mobile ai & aim 2022 challenge: report","authors":"Ignatov","year":"2022","venue":"Computer Vision-ECCV 2022 Workshops","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Johnson et al., 2016] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 694- 711. Springer, 2016.","title":"Perceptual losses for real-time style transfer and super-resolution","authors":"Justin Johnson, Alexandre Alahi, Li Fei-Fei","year":"2016","venue":"Computer Vision-ECCV 2016: 14th European Conference","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kim et al., 2020] Byung-Hoon Kim, Joonyoung Song, Jong Chul Ye, and JaeHyun Baek. Pynet-ca: enhanced pynet with channel attention for end-to-end mobile image signal processing. In Computer Vision-ECCV 2020 Work- shops: Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pages 202-212. Springer, 2020.","title":"Pynet-ca: enhanced pynet with channel attention for end-to-end mobile image signal processing","authors":"Byung-Hoon Kim, Joonyoung Song, Jong Chul Ye, Jaehyun Baek","year":"2020","venue":"Computer Vision-ECCV 2020 Workshops","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liang et al., 2021a] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF international conference on computer vi- sion, pages 1833-1844, 2021.","title":"Swinir: Image restoration using swin transformer","authors":"Liang","year":"2021","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liang et al., 2021b] Zhetong Liang, Jianrui Cai, Zisheng Cao, and Lei Zhang. Cameranet: A two-stage framework for effective camera isp learning. IEEE Transactions on Image Processing, 30:2248-2262, 2021.","title":"Cameranet: A two-stage framework for effective camera isp learning","authors":"Liang","year":"2021","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2018] Pengju Liu, Hongzhi Zhang, Kai Zhang, Liang Lin, and Wangmeng Zuo. Multi-level wavelet-cnn for image restoration. In Proceedings of the IEEE con- ference on computer vision and pattern recognition work- shops, pages 773-782, 2018.","title":"Multi-level wavelet-cnn for image restoration","authors":"Liu","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition workshops","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Paszke et al., 2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural in- formation processing systems, 32, 2019.","title":"Pytorch: An imperative style, highperformance deep learning library","authors":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Raimundo et al., 2022] Daniel Wirzberger Raimundo, An- drey Ignatov, and Radu Timofte. Lan: Lightweight attention-based network for raw-to-rgb smartphone image processing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 808- 816, 2022.","title":"Lan: Lightweight attention-based network for raw-to-rgb smartphone image processing","authors":"Andrey Daniel Wirzberger Raimundo, Radu Ignatov, Timofte","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Schwartz et al., 2018] Eli Schwartz, Raja Giryes, and Alex M Bronstein. Deepisp: Toward learning an end-to- end image processing pipeline. IEEE Transactions on Im- age Processing, 28(2):912-923, 2018.","title":"Deepisp: Toward learning an end-toend image processing pipeline","authors":"Schwartz","year":"2018","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sharma et al., 2005] Gaurav Sharma, Wencheng Wu, and Edul N Dalal. The ciede2000 color-difference formula: Implementation notes, supplementary test data, and math- ematical observations. Color Research & Application: En- dorsed by Inter-Society Color Council, The Colour Group (Great Britain), Canadian Society for Color, Color Sci- ence Association of Japan, Dutch Society for the Study of Color, The Swedish Colour Centre Foundation, Colour Society of Australia, Centre Franc ¸ais de la Couleur, 30(1):21-30, 2005.","title":"Inter-Society Color Council, The Colour Group (Great Britain), Canadian Society for Color, Color Science Association of Japan, Dutch Society for the Study of Color, The Swedish Colour Centre Foundation, Colour Society of Australia","authors":"Gaurav Sharma, Wencheng Wu, Edul N Dalal","year":"2005","venue":"Color Research & Application: Endorsed by","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sun et al., 2018] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz. Pwc-net: Cnns for optical flow us- ing pyramid, warping, and cost volume. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8934-8943, 2018.","title":"Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume","authors":"Deqing Sun, Xiaodong Yang, Ming-Yu Liu, Jan Kautz","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transac- tions on image processing, 13(4):600-612, 2004.","title":"Image quality assessment: from error visibility to structural similarity","authors":"Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli","year":"2004","venue":"IEEE transactions on image processing","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2022] Zhendong Wang, Xiaodong Cun, Jian- min Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer: A general u-shaped transformer for im- age restoration. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 17683-17693, 2022.","title":"Uformer: A general u-shaped transformer for image restoration","authors":"Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, Houqiang Li","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zamir et al., 2022] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5728-5739, 2022.","title":"Restormer: Efficient transformer for high-resolution image restoration","authors":"Aditya Syed Waqas Zamir, Salman Arora, Munawar Khan, Fahad Hayat, Ming-Hsuan Shahbaz Khan, Yang","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2018] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super- resolution using very deep residual channel attention net- works. In Proceedings of the European conference on computer vision (ECCV), pages 286-301, 2018.","title":"Image superresolution using very deep residual channel attention networks","authors":"Zhang","year":"2018","venue":"Proceedings of the European conference on computer vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2021] Zhilu Zhang, Haolin Wang, Ming Liu, Ruohao Wang, Jiawei Zhang, and Wangmeng Zuo. Learn- ing raw-to-srgb mappings with inaccurately aligned su- pervision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4348-4358, 2021.","title":"Learning raw-to-srgb mappings with inaccurately aligned supervision","authors":"Zhang","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}