{"type":"ScholarlyArticle","title":"Unsupervised Misaligned Infrared and Visible Image Fusion via Cross-Modality Image Generation and Registration","authors":[{"name":"Di Wang","slug":"di-wang","given":"Di","family":"Wang","affiliation":"School of Software Technology , Dalian University of Technology , China"},{"name":"Jinyuan Liu","slug":"jinyuan-liu","given":"Jinyuan","family":"Liu","affiliation":"School of Software Technology , Dalian University of Technology , China"},{"name":"Xin Fan","slug":"xin-fan","given":"Xin","family":"Fan","affiliation":"International School of Information Science Engineering , Dalian University of Technology , China"},{"name":"Risheng Liu","slug":"risheng-liu","given":"Risheng","family":"Liu","affiliation":"International School of Information Science Engineering , Dalian University of Technology , China"}],"abstract":"Recent learning-based image fusion methods have marked numerous progress in pre-registered multi-modality data, but suffered serious ghosts dealing with misaligned multi-modality data, due to the spatial deformation and the difficulty narrowing cross-modality discrepancy.\r\nTo overcome the obstacles, in this paper, we present a robust cross-modality generation-registration paradigm for unsupervised misaligned infrared and visible image fusion (IVIF).\r\nSpecifically, we propose a Cross-modality Perceptual Style Transfer Network (CPSTN) to generate a pseudo infrared image taking a visible image as input.\r\nBenefiting from the favorable geometry preservation ability of the CPSTN, the generated pseudo infrared image embraces a sharp structure, which is more conducive to transforming cross-modality image alignment into mono-modality registration coupled with the structure-sensitive of the infrared image. \r\nIn this case, we introduce a Multi-level Refinement Registration Network (MRRN) to predict the displacement vector field between distorted and pseudo infrared images and reconstruct registered infrared image under the mono-modality setting.\r\nMoreover, to better fuse the registered infrared images and visible images, we present a feature Interaction Fusion Module (IFM) to adaptively select more meaningful features for fusion in the Dual-path Interaction Fusion Network (DIFN).\r\nExtensive experimental results suggest that the proposed method performs superior capability on misaligned cross-modality image fusion.","doi":"10.24963/ijcai.2022/487","url":"https://ijcai.org/proceedings/2022/487","pdf":"https://ijcai.org/proceedings/2022/0487.pdf","markdown":"https://ijcai.org/proceedings/2022/487.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/487","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":487,"pages":{"first":3508,"last":3515},"keywords":["Machine Learning: Multi-modal learning","Machine Learning: Unsupervised Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987043","length_sec":92,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985186","length_sec":633,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Arar et al., 2020] Moab Arar, Yiftach Ginger, Dov Danon, Amit H. Bermano, and Daniel Cohen-Or. Unsupervised multi-modal image registration via geometry preserving image-to-image translation. In CVPR, pages 13407- 13416, 2020.","title":"Unsupervised multi-modal image registration via geometry preserving image-to-image translation","authors":"Arar","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Balakrishnan et al., 2018] Guha Balakrishnan, Amy Zhao, Mert R. Sabuncu, John V. Guttag, and Adrian V. Dalca. An unsupervised learning model for deformable medical image registration. In CVPR, pages 9252-9260, 2018.","title":"An unsupervised learning model for deformable medical image registration","authors":"Guha Balakrishnan, Amy Zhao, Mert R Sabuncu, John V Guttag, Adrian V Dalca","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cao et al., 2018] Xiaohuan Cao, Jianhuan Yang, Li Wang, Zhong Xue, Qian Wang, and Dinggang Shen. Deep learn- ing based inter-modality image registration supervised by intra-modality similarity. In MLMI, pages 55-63, 2018.","title":"Deep learning based inter-modality image registration supervised by intra-modality similarity","authors":"Xiaohuan Cao, Jianhuan Yang, Li Wang, Zhong Xue, Qian Wang, Dinggang Shen","year":"2018","venue":"MLMI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ghosh et al., 2019] Sanjay Ghosh, Ruturaj G. Gavaskar, and Kunal N. Chaudhury. Saliency guided image detail enhancement. In NCC, pages 1-6, 2019.","title":"Saliency guided image detail enhancement","authors":"Sanjay Ghosh, G Ruturaj, Kunal N Gavaskar, Chaudhury","year":"2019","venue":"NCC","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Han et al., 2013] Yu Han, Yunze Cai, Yin Cao, and Xi- aoming Xu. A new image fusion performance metric based on visual information fidelity. Information Fusion, 14(2):127-135, 2013.","title":"A new image fusion performance metric based on visual information fidelity","authors":"Yu Han, Yunze Cai, Yin Cao, Xiaoming Xu","year":"2013","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ilg et al., 2017] Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In CVPR, pages 1647-1655, 2017.","title":"Flownet 2.0: Evolution of optical flow estimation with deep networks","authors":"Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, Thomas Brox","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Isola et al., 2017] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with con- ditional adversarial networks. In CVPR, pages 5967-5976, 2017.","title":"Image-to-image translation with conditional adversarial networks","authors":"Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A Efros","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jaderberg et al., 2015] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. In NeurIPS, pages 2017-2025, 2015.","title":"Spatial transformer networks","authors":"Jaderberg","year":"2015","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lai et al., 2019] Weisheng Lai, Jiabin Huang, Narendra Ahuja, and Ming-Hsuan Yang. Fast and accurate image super-resolution with deep laplacian pyramid networks. IEEE Transactions on Pattern Analysis and Machine In- telligence, 41(11):2599-2613, 2019.","title":"Fast and accurate image super-resolution with deep laplacian pyramid networks","authors":"Weisheng Lai, Jiabin Huang, Narendra Ahuja, Ming-Hsuan Yang","year":"2019","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li and Wu, 2019] Hui Li and Xiao-Jun Wu. Densefuse: A fusion approach to infrared and visible images. IEEE Transactions on Image Processing, 28(5):2614-2623, 2019.","title":"Densefuse: A fusion approach to infrared and visible images","authors":"Hui Li, Xiao-Jun Wu","year":"2019","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2021] Hui Li, Xiao-Jun Wu, and Josef Kittler. Rfn- nest: An end-to-end residual fusion network for infrared and visible images. Information Fusion, 73:72-86, 2021.","title":"Rfnnest: An end-to-end residual fusion network for infrared and visible images","authors":"Hui Li, Xiao-Jun Wu, Josef Kittler","year":"2021","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2020a] Risheng Liu, Zi Li, Yuxi Zhang, Xin Fan, and Zhongxuan Luo. Bi-level probabilistic feature learn- ing for deformable image registration. In IJCAI, pages 723-730, 2020.","title":"Bi-level probabilistic feature learning for deformable image registration","authors":"Liu","year":"2020","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2020/101"},{"position":13,"raw":"[Liu et al., 2020b] Risheng Liu, Jinyuan Liu, Zhiying Jiang, Xin Fan, and Zhongxuan Luo. A bilevel integrated model with data-driven layer ensemble for multi-modality image fusion. IEEE Transactions on Image Processing, 30:1261- 1274, 2020.","title":"A bilevel integrated model with data-driven layer ensemble for multi-modality image fusion","authors":"Liu","year":"2020","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2021a] Jinyuan Liu, Xin Fan, Ji Jiang, Risheng Liu, and Zhongxuan Luo. Learning a deep multi-scale feature ensemble and an edge-attention guidance for im- age fusion. IEEE Transactions on Circuits and Systems for Video Technology, 2021.","title":"Learning a deep multi-scale feature ensemble and an edge-attention guidance for image fusion","authors":"Liu","year":"2021","venue":"IEEE Transactions on Circuits and Systems for Video Technology","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2021b] Jinyuan Liu, Yuhui Wu, Zhanbo Huang, Risheng Liu, and Xin Fan. Smoa: Searching a modality- oriented architecture for infrared and visible image fusion. IEEE Signal Processing Letters, 28:1818-1822, 2021.","title":"Smoa: Searching a modalityoriented architecture for infrared and visible image fusion","authors":"Liu","year":"2021","venue":"IEEE Signal Processing Letters","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2021c] Risheng Liu, Jiaxin Gao, Jin Zhang, Deyu Meng, and Zhouchen Lin. Investigating bi-level optimiza- tion for learning and vision from a unified perspective: A survey and beyond. IEEE Transactions on Pattern Analy- sis and Machine Intelligence, 2021.","title":"Investigating bi-level optimization for learning and vision from a unified perspective: A survey and beyond","authors":"Liu","year":"2021","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2021d] Risheng Liu, Zi Li, Xin Fan, Chenying Zhao, Hao Huang, and Zhongxuan Luo. Learning de- formable image registration from optimization: Perspec- tive, modules, bilevel training and beyond. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 2021.","title":"Learning deformable image registration from optimization: Perspective, modules, bilevel training and beyond","authors":"Liu","year":"2021","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2021e] Risheng Liu, Zhu Liu, Jinyuan Liu, and Xin Fan. Searching a hierarchically aggregated fusion ar- chitecture for fast multi-modality image fusion. In ACM MM, pages 1600-1608, 2021.","title":"Searching a hierarchically aggregated fusion architecture for fast multi-modality image fusion","authors":"Liu","year":"2021","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2022a] Jinyuan Liu, Xin Fan, Zhangbo Huang, Guanyao Wu, Risheng Liu, Wei Zhong, and Zhongxuan Luo. Target-aware dual adversarial learning and a multi- scenario multi-modality benchmark to fuse infrared and visible for object detection. In CVPR, 2022.","title":"Target-aware dual adversarial learning and a multiscenario multi-modality benchmark to fuse infrared and visible for object detection","authors":"Liu","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu et al., 2022b] Jinyuan Liu, Jingjie Shang, Risheng Liu, and Xin Fan. Attention-guided global-local adversarial learning for detail-preserving multi-exposure image fu- sion. IEEE Transactions on Circuits and Systems for Video Technology, 2022.","title":"Attention-guided global-local adversarial learning for detail-preserving multi-exposure image fusion","authors":"Liu","year":"2022","venue":"IEEE Transactions on Circuits and Systems for Video Technology","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liu et al., 2022c] Risheng Liu, Pan Mu, Xiaoming Yuan, Shangzhi Zeng, and Jin Zhang. A general descent aggre- gation framework for gradient-based bi-level optimization. IEEE Transactions on Pattern Analysis and Machine Intel- ligence, 2022.","title":"A general descent aggregation framework for gradient-based bi-level optimization","authors":"Liu","year":"2022","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ma et al., 2019] Jiayi Ma, Wei Yu, Pengwei Liang, Chang Li, and Junjun Jiang. Fusiongan: A generative adversarial network for infrared and visible image fusion. Information Fusion, 48:11-26, 2019.","title":"Fusiongan: A generative adversarial network for infrared and visible image fusion","authors":"Jiayi Ma, Wei Yu, Pengwei Liang, Chang Li, Junjun Jiang","year":"2019","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Mahapatra et al., 2018] Dwarikanath Mahapatra, Zongyuan Ge, Suman Sedai, and Rajib Chakravorty. Joint registra- tion and segmentation of xray images using generative ad- versarial networks. In MICCAI, pages 73-80, 2018.","title":"Joint registration and segmentation of xray images using generative adversarial networks","authors":"Mahapatra","year":"2018","venue":"MICCAI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sajjadi et al., 2017] Mehdi S. M. Sajjadi, Bernhard Schölkopf, and Michael Hirsch. Enhancenet: Single im- age super-resolution through automated texture synthesis. In ICCV, pages 4501-4510, 2017.","title":"Enhancenet: Single image super-resolution through automated texture synthesis","authors":"S M Mehdi, Bernhard Sajjadi, Michael Schölkopf, Hirsch","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tang et al., 2020] Hao Tang, Zechao Li, Zhimao Peng, and Jinhui Tang. Blockmix: Meta regularization and self- calibrated inference for metric-based meta-learning. In ACM MM, pages 610-618, 2020.","title":"Blockmix: Meta regularization and selfcalibrated inference for metric-based meta-learning","authors":"Hao Tang, Zechao Li, Zhimao Peng, Jinhui Tang","year":"2020","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tang et al., 2022] Hao Tang, Chengcheng Yuan, Zechao Li, and Jinhui Tang. Learning attention-guided pyramidal fea- tures for few-shot fine-grained recognition. Pattern Recog- nition, 2022.","title":"Learning attention-guided pyramidal features for few-shot fine-grained recognition","authors":"Hao Tang, Chengcheng Yuan, Zechao Li, Jinhui Tang","year":"2022","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2004] Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Trans- actions on Image Processing, 13(4):600-612, 2004.","title":"Image quality assessment: from error visibility to structural similarity","authors":"Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli","year":"2004","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2019a] Guan'an Wang, Tianzhu Zhang, Jian Cheng, Si Liu, Yang Yang, and Zengguang Hou. Rgb- infrared cross-modality person re-identification via joint pixel and feature alignment. In ICCV, pages 3622-3631, 2019.","title":"Rgbinfrared cross-modality person re-identification via joint pixel and feature alignment","authors":"Wang","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2019b] Zhixiang Wang, Zheng Wang, Yin- qiang Zheng, Yung-Yu Chuang, and Shin'ichi Satoh. Learning to reduce dual-level discrepancy for infrared- visible person re-identification. In CVPR, pages 618-626, 2019.","title":"Learning to reduce dual-level discrepancy for infraredvisible person re-identification","authors":"Wang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Xu et al., 2022] Han Xu, Jiayi Ma, Junjun Jiang, Xiaojie Guo, and Haibin Ling. U2fusion: A unified unsupervised image fusion network. IEEE Transactions of Pattern Anal- ysis and Machine Intelligence, 44(1):502-518, 2022.","title":"U2fusion: A unified unsupervised image fusion network","authors":"Han Xu, Jiayi Ma, Junjun Jiang, Xiaojie Guo, Haibin Ling","year":"2022","venue":"IEEE Transactions of Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yang et al., 2020] Qianye Yang, Nannan Li, Zixu Zhao, Xingyu Fan, Eric I-Chao Chang, and Yan Xu. Mri cross- modality image-to-image translation. Scientific reports, 10(1):1-18, 2020.","title":"Mri crossmodality image-to-image translation","authors":"Qianye Yang, Nannan Li, Zixu Zhao, Xingyu Fan, I-Chao Eric, Yan Chang, Xu","year":"2020","venue":"Scientific reports","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2021a] Hao Zhang, Jiteng Yuan, Xin Tian, and Jiayi Ma. GAN-FM: infrared and visible image fusion us- ing GAN with full-scale skip connection and dual marko- vian discriminators. IEEE Transactions on Computational Imaging, 7:1134-1147, 2021.","title":"GAN-FM: infrared and visible image fusion using GAN with full-scale skip connection and dual markovian discriminators","authors":"Zhang","year":"2021","venue":"IEEE Transactions on Computational Imaging","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhang et al., 2021b] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(7):2480-2495, 2021.","title":"Residual dense network for image restoration","authors":"Zhang","year":"2021","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhao et al., 2020] Zixiang Zhao, Shuang Xu, Chunxia Zhang, Junmin Liu, Jiangshe Zhang, and Pengfei Li. Did- fuse: Deep image decomposition for infrared and visible image fusion. In IJCAI, pages 970-976, 2020.","title":"Didfuse: Deep image decomposition for infrared and visible image fusion","authors":"Zixiang Zhao, Shuang Xu, Chunxia Zhang, Junmin Liu, Jiangshe Zhang, Pengfei Li","year":"2020","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2020/135"},{"position":36,"raw":"[Zhu et al., 2017] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image transla- tion using cycle-consistent adversarial networks. In ICCV, pages 2242-2251, 2017.","title":"Unpaired image-to-image translation using cycle-consistent adversarial networks","authors":"Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A Efros","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}