{"type":"ScholarlyArticle","title":"Bidirectional Dilation Transformer for Multispectral and Hyperspectral Image Fusion","authors":[{"name":"Shangqi Deng","slug":"shangqi-deng","given":"Shangqi","family":"Deng","affiliation":"University of Electronic Science and Technology of China"},{"name":"Liang-Jian Deng","slug":"liang-jian-deng","given":"Liang-Jian","family":"Deng","affiliation":"University of Electronic Science and Technology of China"},{"name":"Xiao Wu","slug":"xiao-wu","given":"Xiao","family":"Wu","affiliation":"University of Electronic Science and Technology of China"},{"name":"Ran Ran","slug":"ran-ran","given":"Ran","family":"Ran","affiliation":""},{"name":"Rui Wen","slug":"rui-wen","given":"Rui","family":"Wen","affiliation":"University of Electronic Science and Technology of China"}],"abstract":"Transformer-based methods have proven to be effective in achieving long-distance modeling, capturing the spatial and spectral information, and exhibiting strong inductive bias in various computer vision tasks. Generally, the Transformer model includes two common modes of multi-head self-attention (MSA): spatial MSA (Spa-MSA) and spectral MSA (Spe-MSA). However, Spa-MSA is computationally efficient but limits the global spatial response within a local window. On the other hand, Spe-MSA can calculate channel self-attention to accommodate high-resolution images, but it disregards the crucial local information that is essential for low-level vision tasks. In this study, we propose a bidirectional dilation Transformer (BDT) for multispectral and hyperspectral image fusion (MHIF), which aims to leverage the advantages of both MSA and the latent multiscale information specific to MHIF tasks. The BDT consists of two designed modules: the dilation Spa-MSA (D-Spa), which dynamically expands the spatial receptive field through a given hollow strategy, and the grouped Spe-MSA (G-Spe), which extracts latent features within the feature map and learns local data behavior. Additionally, to fully exploit the multiscale information from both inputs with different spatial resolutions, we employ a bidirectional hierarchy strategy in the BDT, resulting in improved performance. Finally, extensive experiments on two commonly used datasets, CAVE and Harvard, demonstrate the superiority of BDT both visually and quantitatively. Furthermore, the related code will be available at the GitHub page of the authors.","doi":"10.24963/ijcai.2023/404","url":"https://ijcai.org/proceedings/2023/404","pdf":"https://ijcai.org/proceedings/2023/0404.pdf","markdown":"https://ijcai.org/proceedings/2023/404.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/404","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":404,"pages":{"first":3633,"last":3641},"keywords":["Machine Learning: ML: Multi-modal learning","Computer Vision: CV: Machine learning for vision","Machine Learning: ML: Attention models"],"videos":[],"references":[{"position":1,"raw":"[Cao et al., 2020] Xiangyong Cao, Jing Yao, Zongben Xu, and Deyu Meng. Hyperspectral image classification with convolutional neural network and active learning. IEEE Trans. Geosci. Remote Sens., 58(7):4604-4616, 2020.","title":"Hyperspectral image classification with convolutional neural network and active learning","authors":"Cao","year":"2020","venue":"IEEE Trans. Geosci. Remote Sens","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dian and Li, 2019] Renwei Dian and Shutao Li. Hyper- spectral image super-resolution via subspace-based low tensor multi-rank regularization. IEEE Trans. Image Pro- cess., 28(10):5135-5146, 2019.","title":"Renwei Dian and Shutao Li. Hyperspectral image super-resolution via subspace-based low tensor multi-rank regularization","authors":"Dian, Li","year":"2019","venue":"IEEE Trans. Image Process","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dian et al., 2019] Renwei Dian, Shutao Li, and Leyuan Fang. Learning a low tensor-train rank representation for hyperspectral image super-resolution. IEEE Trans. Neural Netw. Learn. Syst., 30(9):2672-2683, 2019.","title":"Learning a low tensor-train rank representation for hyperspectral image super-resolution","authors":"Dian","year":"2019","venue":"IEEE Trans. Neural Netw. Learn. Syst","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dong et al., 2021] Weisheng Dong, Chen Zhou, Fangfang Wu, Jinjian Wu, Guangming Shi, and Xin Li. Model- guided deep hyperspectral image super-resolution. IEEE Trans. Image Process., 30:5754-5768, 2021.","title":"Modelguided deep hyperspectral image super-resolution","authors":"Weisheng Dong, Chen Zhou, Fangfang Wu, Jinjian Wu, Guangming Shi, Xin Li","year":"2021","venue":"IEEE Trans. Image Process","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Feng and Sun, 2012] Yaoze Feng and Dawen Sun. Applica- tion of hyperspectral imaging in food safety inspection and control: a review. Crit. Rev. Food Sci. Nutr., 52(11):1039- 1058, 2012.","title":"Yaoze Feng and Dawen Sun. Application of hyperspectral imaging in food safety inspection and control: a review","authors":"Sun Feng","year":"2012","venue":"Crit. Rev. Food Sci. Nutr","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gao et al., 2006] Bocai Gao, C Davis, and A Goetz. A re- view of atmospheric correction techniques for hyperspec- tral remote sensing of land surfaces and ocean color. In IEEE International Symposium on Geoscience and Re- mote Sensing, pages 1979-1981. IEEE, 2006.","title":"A review of atmospheric correction techniques for hyperspectral remote sensing of land surfaces and ocean color","authors":"Gao","year":"2006","venue":"IEEE International Symposium on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gardner and Dorling, 1998] Matt W Gardner and SR Dor- ling. Artificial neural networks (the multilayer percep- tron)-a review of applications in the atmospheric sci- ences. Atmos. Environ., 32(14-15):2627-2636, 1998.","title":"Artificial neural networks (the multilayer perceptron)-a review of applications in the atmospheric sciences","authors":"W Matt, S R Gardner, Dorling","year":"1998","venue":"Atmos. Environ","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Guo et al., 2020] Penghao Guo, Peixian Zhuang, and Yecai Guo. Bayesian pan-sharpening with multiorder gradient- based deep network constraints. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., 13:950-962, 2020.","title":"Bayesian pan-sharpening with multiorder gradientbased deep network constraints","authors":"Guo","year":"2020","venue":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Han et al., 2021] Qi Han, Zejia Fan, Qi Dai, Lei Sun, Ming- Ming Cheng, Jiaying Liu, and Jingdong Wang. On the con- nection between local attention and dynamic depth-wise convolution. In ICLR, 2021.","title":"On the connection between local attention and dynamic depth-wise convolution","authors":"Han","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hong et al., 2021] Danfeng Hong, Zhu Han, Jing Yao, Lianru Gao, Bing Zhang, Antonio Plaza, and Jocelyn Chanussot. Spectralformer: Rethinking hyperspectral im- age classification with transformers. IEEE Trans. Geosci. Remote Sens., 60:1-15, 2021.","title":"Spectralformer: Rethinking hyperspectral image classification with transformers","authors":"Danfeng Hong, Zhu Han, Jing Yao, Lianru Gao, Bing Zhang, Antonio Plaza, Jocelyn Chanussot","year":"2021","venue":"IEEE Trans. Geosci. Remote Sens","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hu et al., 2018] Jie Hu, Li Shen, and Gang Sun. Squeeze- and-excitation networks. In CVPR, June 2018.","title":"Squeezeand-excitation networks","authors":"Jie Hu, Li Shen, Gang Sun","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hu et al., 2021] Jinfan Hu, Tingzhu Huang, Liangjian Deng, Taixiang Jiang, Gemine Vivone, and Jocelyn Chanussot. Hyperspectral image super-resolution via deep spatiospectral attention convolutional neural networks. IEEE Trans. Neural Netw. Learn. Syst., 2021.","title":"Hyperspectral image super-resolution via deep spatiospectral attention convolutional neural networks","authors":"Hu","year":"2021","venue":"IEEE Trans. Neural Netw. Learn. Syst","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hu et al., 2022] Jinfan Hu, Tingzhu Huang, Liangjian Deng, Hongxia Dou, Danfeng Hong, and Gemine Vivone. Fusformer: A transformer-based fusion network for hy- perspectral image super-resolution. IEEE Geosci. Remote Sens. Lett., 19:1-5, 2022.","title":"Fusformer: A transformer-based fusion network for hyperspectral image super-resolution","authors":"Hu","year":"2022","venue":"IEEE Geosci. Remote Sens. Lett","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Huang et al., 2022] Tao Huang, Weisheng Dong, Jinjian Wu, Leida Li, Xin Li, and Guangming Shi. Deep hy- perspectral image fusion network with iterative spatio- spectral regularization. IEEE Trans. Comput Imaging., 8:201-214, 2022.","title":"Deep hyperspectral image fusion network with iterative spatiospectral regularization","authors":"Huang","year":"2022","venue":"IEEE Trans. Comput Imaging","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jiao et al., 2023] Jiayu Jiao, Yu-Ming Tang, Kun-Yu Lin, Yipeng Gao, Jinhua Ma, Yaowei Wang, and Wei-Shi Zheng. Dilateformer: Multi-scale dilated transformer for visual recognition. IEEE Transactions on Multimedia, pages 1-14, 2023.","title":"Dilateformer: Multi-scale dilated transformer for visual recognition","authors":"Jiayu Jiao, Yu-Ming Tang, Kun-Yu Lin, Yipeng Gao, Jinhua Ma, Yaowei Wang, Wei-Shi Zheng","year":"2023","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kolesnikov et al., 2021] Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weissenborn, Georg Heigold, Jakob Uszkoreit, Lucas Beyer, Matthias Minderer, Mostafa De- hghani, Neil Houlsby, Sylvain Gelly, Thomas Unterthiner, and Xiaohua Zhai. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weissenborn, Georg Heigold, Jakob Uszkoreit, Lucas Beyer, Matthias Minderer, Mostafa Dehghani, Neil Houlsby, Sylvain Gelly, Thomas Unterthiner, Xiaohua Zhai","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2018a] Shutao Li, Renwei Dian, Leyuan Fang, and José M Bioucas-Dias. Fusing hyperspectral and multispec- tral images via coupled sparse tensor factorization. IEEE Trans. Image Process., 27(8):4118-4130, 2018.","title":"Fusing hyperspectral and multispectral images via coupled sparse tensor factorization","authors":"Li","year":"2018","venue":"IEEE Trans. Image Process","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2018b] Yuhong Li, Xiaofan Zhang, and Deming Chen. Csrnet: Dilated convolutional neural networks for understanding the highly congested scenes. In CVPR, pages 1091-1100, 2018.","title":"Csrnet: Dilated convolutional neural networks for understanding the highly congested scenes","authors":"Li","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2021] Duo Li, Jie Hu, Changhu Wang, Xiangtai Li, Qi She, Lei Zhu, Tong Zhang, and Qifeng Chen. Invo- lution: Inverting the inherence of convolution for visual recognition. In CVPR, pages 12321-12330, 2021.","title":"Involution: Inverting the inherence of convolution for visual recognition","authors":"Duo Li, Jie Hu, Changhu Wang, Xiangtai Li, Qi She, Lei Zhu, Tong Zhang, Qifeng Chen","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liang et al., 2021] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In ICCV, pages 1833-1844, 2021.","title":"Swinir: Image restoration using swin transformer","authors":"Liang","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liu et al., 2020] Xiangyu Liu, Qingjie Liu, and Yunhong Wang. Remote sensing image fusion based on two-stream fusion network. Inf. Fusion., 55:1-15, 2020.","title":"Remote sensing image fusion based on two-stream fusion network","authors":"Xiangyu Liu, Qingjie Liu, Yunhong Wang","year":"2020","venue":"Inf. Fusion","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Liu et al., 2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012-10022, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Lu et al., 2020] Bing Lu, Phuong D Dao, Jiangui Liu, Yuhong He, and Jiali Shang. Recent advances of hyper- spectral imaging technology and applications in agricul- ture. Remote Sens. (Basel)., 12(16):2659, 2020.","title":"Recent advances of hyperspectral imaging technology and applications in agriculture","authors":"Bing Lu, Phuong D Dao, Jiangui Liu, Yuhong He, Jiali Shang","year":"2020","venue":"Remote Sens. (Basel)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ma et al., 2021] Qing Ma, Junjun Jiang, Xianming Liu, and Jiayi Ma. Learning a 3d-cnn and transformer prior for hyperspectral image super-resolution. arXiv preprint arXiv:2111.13923, 2021.","title":"Learning a 3d-cnn and transformer prior for hyperspectral image super-resolution","authors":"Qing Ma, Junjun Jiang, Xianming Liu, Jiayi Ma","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Meng et al., 2022] Xiangchao Meng, Nan Wang, Feng Shao, and Shutao Li. Vision transformer for pansharp- ening. IEEE Trans. Geosci. Remote Sens., 60:1-11, 2022.","title":"Vision transformer for pansharpening","authors":"Xiangchao Meng, Nan Wang, Feng Shao, Shutao Li","year":"2022","venue":"IEEE Trans. Geosci. Remote Sens","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Piqueras et al., 2011] S Piqueras, L Duponchel, R Tauler, and A De Juan. Resolution and segmentation of hyperspectral biomedical images by multivariate curve resolution-alternating least squares. Anal. Chim. Acta., 705(1-2):182-192, 2011.","title":"Resolution and segmentation of hyperspectral biomedical images by multivariate curve resolution-alternating least squares","authors":"L Piqueras, R Duponchel, Tauler, Juan De","year":"2011","venue":"Anal. Chim. Acta","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Selva et al., 2015] Massimo Selva, Bruno Aiazzi, Francesco Butera, Leandro Chiarantini, and Stefano Baronti. Hyper-sharpening: A first approach on sim-ga data. IEEE J. Sel. Top Appl. Earth Obs. Remote Sens., 8(6):3008-3024, 2015.","title":"Hyper-sharpening: A first approach on sim-ga data","authors":"Massimo Selva, Bruno Aiazzi, Francesco Butera, Leandro Chiarantini, Stefano Baronti","year":"2015","venue":"IEEE J. Sel. Top Appl. Earth Obs. Remote Sens","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2019] Wu Wang, Weihong Zeng, Yue Huang, Xinghao Ding, and John Paisley. Deep blind hyperspectral image fusion. In ICCV, October 2019.","title":"Deep blind hyperspectral image fusion","authors":"Wu Wang, Weihong Zeng, Yue Huang, Xinghao Ding, John Paisley","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wu et al., 2011] Jian Wu, Dao-Li Peng, et al. Advances in researches on hyperspectral remote sensing forestry information-extracting technology. Spectrosc. Spect. Anal., 31(9):2305-2312, 2011.","title":"Advances in researches on hyperspectral remote sensing forestry information-extracting technology","authors":"Jian Wu, Dao-Li Peng","year":"2011","venue":"Spectrosc. Spect. Anal","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Xu et al., 2022] Ting Xu, Tingzhu Huang, Liangjian Deng, and Naoto Yokoya. An iterative regularization method based on tensor subspace representation for hyperspec- tral image super-resolution. IEEE Trans. Geosci. Remote Sens., 60:1-16, 2022.","title":"An iterative regularization method based on tensor subspace representation for hyperspectral image super-resolution","authors":"Ting Xu, Tingzhu Huang, Liangjian Deng, Naoto Yokoya","year":"2022","venue":"IEEE Trans. Geosci. Remote Sens","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yan et al., 2022] Keyu Yan, Man Zhou, Jie Huang, Feng Zhao, Chengjun Xie, Chongyi Li, and Danfeng Hong. Panchromatic and multispectral image fusion via alternat- ing reverse filtering network. NeurIPS, 2022.","title":"Panchromatic and multispectral image fusion via alternating reverse filtering network","authors":"Keyu Yan, Man Zhou, Jie Huang, Feng Zhao, Chengjun Xie, Chongyi Li, Danfeng Hong","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yang et al., 2020a] Yong Yang, Chenxu Wan, Shuying Huang, Hangyuan Lu, and Weiguo Wan. Pansharpening based on low-rank fuzzy fusion and detail supplement. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., 13:5466- 5479, 2020.","title":"Pansharpening based on low-rank fuzzy fusion and detail supplement","authors":"Yang","year":"2020","venue":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yang et al., 2020b] Yong Yang, Lei Wu, Shuying Huang, Weiguo Wan, Wei Tu, and Hangyuan Lu. Multiband remote sensing image pansharpening based on dual- injection model. IEEE J. Sel. Top. Appl. Earth Obs. Re- mote Sens., 13:1888-1904, 2020.","title":"Multiband remote sensing image pansharpening based on dualinjection model","authors":"Yang","year":"2020","venue":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zamir et al., 2022] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Minghsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In CVPR, pages 5728- 5739, 2022.","title":"Restormer: Efficient transformer for high-resolution image restoration","authors":"Aditya Syed Waqas Zamir, Salman Arora, Munawar Khan, Fahad Hayat, Minghsuan Shahbaz Khan, Yang","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhang et al., 2020] Xueting Zhang, Wei Huang, Qi Wang, and Xuelong Li. Ssr-net: Spatial-spectral reconstruc- tion network for hyperspectral and multispectral image fusion. IEEE Trans. Geosci. Remote Sens., 59(7):5953- 5965, 2020.","title":"Ssr-net: Spatial-spectral reconstruction network for hyperspectral and multispectral image fusion","authors":"Zhang","year":"2020","venue":"IEEE Trans. Geosci. Remote Sens","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhou et al., 2021] Man Zhou, Xueyang Fu, Jie Huang, Feng Zhao, Aiping Liu, and Rujing Wang. Effective pan- sharpening with transformer and invertible neural network. IEEE Trans. Geosci. Remote Sens., 60, 2021.","title":"Effective pansharpening with transformer and invertible neural network","authors":"Zhou","year":"2021","venue":"IEEE Trans. Geosci. Remote Sens","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhou et al., 2022] Man Zhou, Jie Huang, Keyu Yan, Hu Yu, Xueyang Fu, Aiping Liu, Xian Wei, and Feng Zhao. Spatial-frequency domain information integration for pan- sharpening. In ECCV, pages 274-291. Springer, 2022.","title":"Spatial-frequency domain information integration for pansharpening","authors":"Zhou","year":"2022","venue":"ECCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}