{"type":"ScholarlyArticle","title":"LGPConv: Learnable Gaussian Perturbation Convolution for Lightweight Pansharpening","authors":[{"name":"Chen-Yu Zhao","slug":"chen-yu-zhao","given":"Chen-Yu","family":"Zhao","affiliation":"University of Electronic Science and Technology of China , Chengdu , 611731"},{"name":"Tian-Jing Zhang","slug":"tian-jing-zhang","given":"Tian-Jing","family":"Zhang","affiliation":"University of Electronic Science and Technology of China , Chengdu , 611731"},{"name":"Ran Ran","slug":"ran-ran","given":"Ran","family":"Ran","affiliation":"University of Electronic Science and Technology of China , Chengdu , 611731"},{"name":"Zhi-Xuan Chen","slug":"zhi-xuan-chen","given":"Zhi-Xuan","family":"Chen","affiliation":"University of Electronic Science and Technology of China , Chengdu , 611731"},{"name":"Liang-Jian Deng","slug":"liang-jian-deng","given":"Liang-Jian","family":"Deng","affiliation":"University of Electronic Science and Technology of China , Chengdu , 611731"}],"abstract":"Pansharpening is a crucial and challenging task that aims to obtain a high spatial resolution image by merging a multispectral (MS) image and a panchromatic (PAN) image. Current methods use CNNs with standard convolution, but we've observed strong correlation among channel dimensions in the kernel, leading to computational burden and redundancy. To address this, we propose Learnable Gaussian Perturbation Convolution (LGPConv), surpassing standard convolution. LGPConv leverages two properties of standard convolution kernels: 1) correlations within channels, learning a premier kernel as a base to reduce parameters and training difficulties caused by redundancy; 2) introducing Gaussian noise perturbations to simulate randomness and enhance nonlinear representation within channels. We incorporate LGPConv into a well-designed pansharpening network and demonstrate its superiority through extensive experiments, achieving state-of-the-art performance with minimal parameters (27K). Code is available on the GitHub page of the authors.","doi":"10.24963/ijcai.2023/517","url":"https://ijcai.org/proceedings/2023/517","pdf":"https://ijcai.org/proceedings/2023/0517.pdf","markdown":"https://ijcai.org/proceedings/2023/517.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/517","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":517,"pages":{"first":4647,"last":4655},"keywords":["Computer Vision: CV: Machine learning for vision","Machine Learning: ML: Convolutional networks","Machine Learning: ML: Applications"],"videos":[],"references":[{"position":1,"raw":"[Aiazzi et al., 2006] Bruno Aiazzi, L Alparone, Stefano Baronti, Andrea Garzelli, and Massimo Selva. Mtf- tailored multiscale fusion of high-resolution ms and pan imagery. Photogrammetric Engineering & Remote Sensing, 72(5):591-596, 2006.","title":"Mtftailored multiscale fusion of high-resolution ms and pan imagery","authors":"Bruno Aiazzi, Stefano Alparone, Andrea Baronti, Massimo Garzelli, Selva","year":"2006","venue":"Photogrammetric Engineering & Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cao et al., 2022] Xiangyong Cao, Xueyang Fu, Danfeng Hong, Zongben Xu, and Deyu Meng. Pancsc-net: A model- driven deep unfolding method for pansharpening. IEEE Transactions on Geoscience and Remote Sensing, 60:1-13, 2022.","title":"Pancsc-net: A modeldriven deep unfolding method for pansharpening","authors":"Cao","year":"2022","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deng et al., 2021] Liang-Jian Deng, Gemine Vivone, Cheng Jin, and Jocelyn Chanussot. Detail injection-based deep convolutional neural networks for pansharpening. IEEE Transactions on Geoscience and Remote Sensing, 59(8):6995-7010, 2021.","title":"Detail injection-based deep convolutional neural networks for pansharpening","authors":"Liang-Jian Deng, Gemine Vivone, Jin Cheng, Jocelyn Chanussot","year":"2021","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fu et al., 2019] Xueyang Fu, Zihuang Lin, Yue Huang, and Xinghao Ding. A variational pan-sharpening with local gradient constraints. In CVPR, pages 10265-10274, 2019.","title":"A variational pan-sharpening with local gradient constraints","authors":"Fu","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fu et al., 2020] Xueyang Fu, Wu Wang, Yue Huang, Xing- hao Ding, and John Paisley. Deep multiscale detail net- works for multiband spectral image sharpening. IEEE Transactions on Neural Networks and Learning Systems, 32(5):2090-2104, 2020.","title":"Deep multiscale detail networks for multiband spectral image sharpening","authors":"Xueyang Fu, Wu Wang, Yue Huang, Xinghao Ding, John Paisley","year":"2020","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Garzelli et al., 2007] Andrea Garzelli, Filippo Nencini, and Luca Capobianco. Optimal mmse pan sharpening of very high resolution multispectral images. IEEE Transactions on Geoscience and Remote Sensing, 46(1):228-236, 2007.","title":"Optimal mmse pan sharpening of very high resolution multispectral images","authors":"Andrea Garzelli, Filippo Nencini, Luca Capobianco","year":"2007","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Guo et al., 2020a] Penghao Guo, Peixian Zhuang, and Yecai Guo. Bayesian pan-sharpening with multiorder gradient- based deep network constraints. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 13:950-962, 2020.","title":"Bayesian pan-sharpening with multiorder gradientbased deep network constraints","authors":"Guo","year":"2020","venue":"IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Guo et al., 2020b] Penghao Guo, Peixian Zhuang, and Yecai Guo. Bayesian pan-sharpening with multiorder gradient- based deep network constraints. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 13:950-962, 2020.","title":"Bayesian pan-sharpening with multiorder gradientbased deep network constraints","authors":"Guo","year":"2020","venue":"IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Haase and Amthor, 2020] Daniel Haase and Manuel Amthor. Rethinking depthwise separable convolutions: How intra- kernel correlations lead to improved mobilenets. In IEEE Conf. Comput. Vis. Pattern Recog. (CVPR), pages 14600- 14609, 2020.","title":"Rethinking depthwise separable convolutions: How intrakernel correlations lead to improved mobilenets","authors":"Amthor ; ] Daniel Haase, Manuel Haase, Amthor","year":"2020","venue":"IEEE Conf. Comput. Vis. Pattern Recog. (CVPR)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2019] Lin He, Yizhou Rao, Jun Li, Jocelyn Chanus- sot, Antonio Plaza, Jiawei Zhu, and Bo Li. Pansharpening via detail injection based convolutional neural networks. IEEE Journal of Selected Topics in Applied Earth Observa- tions and Remote Sensing, 12(4):1188-1204, 2019.","title":"Pansharpening via detail injection based convolutional neural networks","authors":"Lin He, Yizhou Rao, Jun Li, Jocelyn Chanussot, Antonio Plaza, Jiawei Zhu, Bo Li","year":"2019","venue":"IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hein and Andriushchenko, 2017] Matthias Hein and Maksym Andriushchenko. Formal guarantees on the robustness of a classifier against adversarial manipulation. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Adv. Neural Inform. Process. Syst. (NIPS), volume 30, 2017.","title":"Formal guarantees on the robustness of a classifier against adversarial manipulation","authors":"Matthias Hein, Maksym Andriushchenko","year":"2017","venue":"Adv. Neural Inform. Process. Syst. (NIPS)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Howard et al., 2017] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.","title":"Mobilenets: Efficient convolutional neural networks for mobile vision applications","authors":"Menglong Andrew G Howard, Bo Zhu, Dmitry Chen, Weijun Kalenichenko, Tobias Wang, Marco Weyand, Hartwig Andreetto, Adam","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jin et al., 2022a] Cheng Jin, Liang-Jian Deng, Ting-Zhu Huang, and Gemine Vivone. Laplacian pyramid networks: A new approach for multispectral pansharpening. Informa- tion Fusion, 78:158-170, 2022.","title":"Laplacian pyramid networks: A new approach for multispectral pansharpening","authors":"Jin","year":"2022","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jin et al., 2022b] Zi-Rong Jin, Tian-Jing Zhang, Tai-Xiang Jiang, Gemine Vivone, and Liang-Jian Deng. Lagconv: Local-context adaptive convolution kernels with global harmonic bias for pansharpening. AAAI Conference on Artificial Intelligence (AAAI), 36(1):1113-1121, 2022.","title":"Lagconv: Local-context adaptive convolution kernels with global harmonic bias for pansharpening","authors":"Jin","year":"2022","venue":"AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2018] Xuanqing Liu, Minhao Cheng, Huan Zhang, and Cho-Jui Hsieh. Towards robust neural networks via random self-ensemble. In ECCV, pages 369-385, 2018.","title":"Towards robust neural networks via random self-ensemble","authors":"Liu","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Masi et al., 2016a] Giuseppe Masi, Davide Cozzolino, Luisa Verdoliva, and Giuseppe Scarpa. Pansharpening by convo- lutional neural networks. Remote Sensing, 8(7):594, 2016.","title":"Pansharpening by convolutional neural networks","authors":"Masi","year":"2016","venue":"Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Masi et al., 2016b] Giuseppe Masi, Davide Cozzolino, Luisa Verdoliva, and Giuseppe Scarpa. Pansharpening by convo- lutional neural networks. Remote Sensing, 8(7):594, 2016.","title":"Pansharpening by convolutional neural networks","authors":"Masi","year":"2016","venue":"Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Meyer, 2000] Carl Meyer. Matrix Analysis and Applied Lin- ear Algebra Book and Solutions Manual. 01 2000.","title":"Matrix Analysis and Applied Linear Algebra Book and Solutions Manual","authors":"Carl Meyer","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[T.-J. Zhang and Vivone, 2022] T.-Z. Huang J. Chanussot T.- J. Zhang, L.-J. Deng and G. Vivone. A triple-double convo- lutional neural network for panchromatic sharpening. IEEE Transactions on Neural Networks and Learning Systems, 2022.","title":"A triple-double convolutional neural network for panchromatic sharpening","authors":"T.-Z Huang, J Chanussot, T.-J Zhang, L.-J Deng, G Vivone","year":"2022","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Vivone et al., 2020] Gemine Vivone, Mauro Dalla Mura, An- drea Garzelli, Rocco Restaino, Giuseppe Scarpa, Magnus O Ulfarsson, Luciano Alparone, and Jocelyn Chanussot. A new benchmark based on recent advances in multispectral pansharpening: Revisiting pansharpening with classical and emerging pansharpening methods. IEEE Geoscience and Remote Sensing Magazine, 9(1):53-81, 2020.","title":"A new benchmark based on recent advances in multispectral pansharpening: Revisiting pansharpening with classical and emerging pansharpening methods","authors":"Vivone","year":"2020","venue":"IEEE Geoscience and Remote Sensing Magazine","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Weng et al., 2018] Tsui-Wei Weng, Huan Zhang, Pin-Yu Chen, Jinfeng Yi, Dong Su, Yupeng Gao, Cho-Jui Hsieh, and Luca Daniel. Evaluating the robustness of neural net- works: An extreme value theory approach, 2018.","title":"Evaluating the robustness of neural networks: An extreme value theory approach","authors":"Huan Tsui-Wei Weng, Pin-Yu Zhang, Jinfeng Chen, Dong Yi, Yupeng Su, Cho-Jui Gao, Luca Hsieh, Daniel","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yan et al., 2022a] Keyu Yan, Man Zhou, Jie Huang, Feng Zhao, Chengjun Xie, Chongyi Li, and Danfeng Hong. Panchromatic and multispectral image fusion via alter- nating reverse filtering network. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.","title":"Panchromatic and multispectral image fusion via alternating reverse filtering network","authors":"Yan","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yan et al., 2022b] Yinsong Yan, Junmin Liu, Shuang Xu, Yicheng Wang, and Xiangyong Cao. Md³net: Integrating model-driven and data-driven approaches for pansharpen- ing. IEEE Transactions on Geoscience and Remote Sensing, 60:1-16, 2022.","title":"Md³net: Integrating model-driven and data-driven approaches for pansharpening","authors":"Yan","year":"2022","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yang et al., 2017] Junfeng Yang, Xueyang Fu, Yuwen Hu, Yue Huang, Xinghao Ding, and John Paisley. Pannet: A deep network architecture for pan-sharpening. In Int. Conf. Comput. Vis. (ICCV), pages 5449-5457, 2017.","title":"Pannet: A deep network architecture for pan-sharpening","authors":"Junfeng Yang, Xueyang Fu, Yuwen Hu, Yue Huang, Xinghao Ding, John Paisley","year":"2017","venue":"Int. Conf. Comput. Vis. (ICCV)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yang et al., 2020] Yong Yang, Chenxu Wan, Shuying Huang, Hangyuan Lu, and Weiguo Wan. Pansharpening based on low-rank fuzzy fusion and detail supplement. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 13:5466-5479, 2020.","title":"Pansharpening based on low-rank fuzzy fusion and detail supplement","authors":"Yong Yang, Chenxu Wan, Shuying Huang, Hangyuan Lu, Weiguo Wan","year":"2020","venue":"IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yuan et al., 2018] Qiangqiang Yuan, Yancong Wei, Xi- angchao Meng, Huanfeng Shen, and Liangpei Zhang. A multiscale and multidepth convolutional neural network for remote sensing imagery pan-sharpening. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 11(3):978-989, 2018.","title":"A multiscale and multidepth convolutional neural network for remote sensing imagery pan-sharpening","authors":"Yuan","year":"2018","venue":"IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2019] Yongjun Zhang, Chi Liu, Mingwei Sun, and Yangjun Ou. Pan-sharpening using an efficient bidirec- tional pyramid network. IEEE Transactions on Geoscience and Remote Sensing, 57(8):5549-5563, 2019.","title":"Pan-sharpening using an efficient bidirectional pyramid network","authors":"Yongjun Zhang, Chi Liu, Mingwei Sun, Yangjun Ou","year":"2019","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhou et al., 2022] Man Zhou, Jie Huang, Keyu Yan, Hu Yu, Xueyang Fu, Aiping Liu, Xian Wei, and Feng Zhao. Spatial- frequency domain information integration fornbsp;pan- sharpening. page 274-291, Berlin, Heidelberg, 2022. Springer-Verlag.","title":"Spatialfrequency domain information integration fornbsp;pansharpening","authors":"Zhou","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhuang et al., 2019] Peixian Zhuang, Qingshan Liu, and Xinghao Ding. Pan-ggf: A probabilistic method for pan- sharpening with gradient domain guided image filtering. Signal Processing, 156:177-190, 2019.","title":"Pan-ggf: A probabilistic method for pansharpening with gradient domain guided image filtering","authors":"Zhuang","year":"2019","venue":"Signal Processing","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}