Unsupervised Misaligned Infrared and Visible Image Fusion via Cross-Modality Image Generation and Registration

1 Introduction

Image fusion is to extract the complementary information from different modality images and aggregate them to generate richer and more meaningful feature representations. Typically, Infrared and Visible Image Fusion (IVIF) enjoys the merit and presents conduciveness to practical applications such as autonomous driving and video surveillance.

∗Corresponding author

Most existing IVIF methods [Liu et al., 2020b; 2021e; 2022b] are designed for the hand-crafted pre-registered images, which are very labor-intensive and time-sensitive, despite numerous progress. And, they are sensitive to the discrepancy of intensity of the misaligned infrared and visible images, resulting in serious ghosting artifacts on fused images, once slight offset and deformation exist. The intrinsic reason is that the large cross-modality variation between infrared and visible images makes it impractical to straightly bridge the domain gap between them in a shared feature space. Coupled with the absence of cross-modality similarity constraints, few attempts have ever been made to fuse the misaligned infrared and visible images. The primary obstacle is cross-modality image alignment.

More often than not, existing widely-used image alignment methods [Ilg et al., 2017; Balakrishnan et al., 2018] perform pixel-wise and feature-level alignment by explicitly estimating the deformation field between the distorted image and its reference. However, they are worked under only a single modality setting [Tang et al., 2020] due to their high dependence on the similarity of distribution and appearance on either synthetic or real data with neighborhood reference. Moreover, the designs of the similarity measures [Tang et al., 2022] to optimize the cross-modality alignment process have been proven to be quite challenging. These obstacles gave rise to the development of cross-modality medical image translation [Yang et al., 2020; Liu et al., 2020a; 2021d] and cross-modality Re-ID tasks [Wang et al., 2019a; 2019b]. The former leverages the cGAN [Isola et al., 2017] to perform NeuroImage-to-NeuroImage translation, while the latter learns the cross-modality match in pedestrian images of a person across disjoint camera views by an RGB-to-IR (infrared) translation. A recent study [Arar et al., 2020] presents a multi-modality image registration method by training an image-to-image translation network on the two input modalities. The basic idea of the above-mentioned approaches is using the image-to-image translation to implement cross-modality transformation to narrow the large variation between different modalities.

Motivated by this idea, we propose a specialized crossmodality generation-registration paradigm (CGRP) for unsupervised misaligned infrared and visible image fusion. An inherent characteristic of infrared images is the emphasis on sharp geometry structures over texture details. Generating a pseudo infrared image preserving sharp structure information is more conducive to implementing mono-modality registration between it with the corresponding distorted infrared image. To better preserve geometry structure during generating pseudo infrared images from visible images, we proposed a Cross-modality Perceptual Style Transfer Network (CPSTN), which inherits the basic cycle consistent learning manner of the CycleGAN [Zhu et al., 2017], while developing a perceptual style transfer constraint and cross regularizations across two-cycle learning path to further guide the CPSTN to generate sharper structures. Without them, the generation capability of our CPSTN will degrade to that in [Arar et al., 2020]. Doing this lays the foundation for the mono-modality registration of infrared images. We utilize a Multi-level Refinement Registration Network (MRRN) to predict the deformation field from coarse to fine between distorted and pseudo infrared images and reconstruct the registered infrared image. In this case, we further perform the fusion of the registered infrared image and visible image. To enable the fusion network to focus more on faithful texture details, while avoiding feature smoothing caused by some unsophisticated fusion rules (e.g., concatenation, weighted average), we develop an Interaction Fusion Module (IFM) to adaptively select meaningful features from infrared and visible images to achieve fused results with sharp textures. We evaluate the proposed method on manually processed unaligned datasets and demonstrate its strengths through comprehensive analysis. The main contributions are summarized as follows:

• We propose a highly-robust unsupervised infrared and visible image fusion framework, which is more focused on mitigating ghosting artifacts caused by the fusion of misaligned image pairs compared with learning-based fusion methods specialized for pre-registered images.
• Considering the difficulty of the cross-modality image alignment, we exploit a specialized cross-modality generation-registration paradigm to bridge the large discrepancy between modalities, thus achieving effective infrared and visible image alignment.
• An interaction fusion module is developed to adaptively fuse multi-modality features, which avoids feature smoothing caused by unsophisticated fusion rules and emphasizes faithful texture details.

The extensive experimental results demonstrate that the proposed method performs superior capability on misaligned cross-modality image fusion. The code will be available at https://github.com/wdhudiekou/UMF-CMGR.

2 Method

2.1 Motivation

Different imaging pipelines and heat dissipation within the sensor result in misalignment between observed infrared and visible images, presenting as shift and deformation. By observing, it is found that straightly fusing misaligned infrared and visible images often suffer serious ghosts. Inspired by [Wang et al., 2019a; 2019b] that reduce cross-modality variation via image-to-image translation, and catering to the inherent characteristic that infrared image "emphasizes structure over texture", we propose a specialized cross-modality generation-registration paradigm to reduce spatial offsets and alleviate the ghost during misaligned infrared and visible image fusion.

2.2 Cross-modality Perceptual Style Transfer

The first part of the proposed cross-modality generationregistration paradigm (CGRP) is image translation. Considering that infrared images are prone to distortion due to thermal radiation, we proposed a Cross-modality Perceptual Style Transfer Network (CPSTN) to translate the visible image Ivis to its infrared counterpart I ˜ ir . Together, a pseudo infrared image pair (Iir, I˜ ir ) is formed to provide a unified representation. As shown in Figure 1, CPSTN is a UNet-like generator, and its bottom obtains 9 resnet blocks. Different from CycleGAN [Zhu et al., 2017], we tend to design a specific learning strategy controlled by a perceptual style transfer constraint and establish the inter-path correlation between the twocycle generative routes to further optimize [Liu et al., 2022c; 2021c] the generation of sharp structure of the pseudo infrared image. The pseudo infrared image is generated by I ˜ ir = Tθ(Iir), where Tθ denotes the CSPTN with network parameter θ. This optimization process is illustrated in the left dotted box in Figure 1. Note that the G(A) corresponds to our CPSTN, and the perceptual style transfer constraint and the regularization of inter-path correlation are explained in Section 2.5.

2.3 Multi-level Refinement Registration

Thanks to the reduction of cross-modality discrepancy by CPSTN, the registration of infrared images under the monomodality settings becomes the other core part of the CGRP. As shown in Figure 1, we exploit a Multi-level Refinement Registration Network (MMRN) to predict deformation field between distorted and pseudo infrared images and reconstruct the registered infrared image. The MRRN consists of a shared multi-level feature extractor (SM-FE), two Coarse-toFine deformation field estimation (C2F-DFE) modules, and a resampler layer. In each C2F-DFE, a coarse DFE module MC and a refined DFE moduleMR are included. Then, the coarse deformation field is first predicted as

the refined deformation field is estimated by

where F k denotes the k-th level of the SM-FM. Suppose the SM-FE contains K levels, when k = K, the final deformation field -ϕ = φKr is estimated. Finally, we use the resampler layer similar to the STN [Jaderberg et al., 2015] to reconstruct the registered infrared image by

the operator ◦ denotes the spatial transform for registration.

2.4 Dual-path Interation Fusion

To fuse registered infrared image I reg ir and visible image Ivis, we present a dual-path interaction fusion network (DIFN). It is composed of a dual-path feature extraction module and a feature interaction fusion module (IFM). Here, the structure of the dual-path feature extraction module inherits the residual dense network [Zhang et al., 2021b], and the extracted features are presented by

where MθE is the feature extraction module, and θE is its parameter.

Interaction Fusion Module. We utilize the IFM to adaptively select features from infrared and visible images to fuse, shown in small dashed box of Figure 1. To focus on more significant information, the feature responses are recalibrated by

and the infrared and visible features are activated as

then we obtain the final fused image by

where S is the Sigmoid function, and ⊗ denotes the elementwise multiplication operation.

2.5 Loss Functions

Perceptual style transfer loss. To generate more realistic pseudo infrared images, we introduce a perceptual style transfer (PST) loss to control the cycle consistency of CPSTN. The PST loss consists of two terms known as perceptual loss Lpcp and style loss Lsty. First, the Lpcp is defined as

where ψj is the j-th layer of the VGG-19 [Simonyan and Zisserman, 2015] model and j ∈ [2, 7, 12, 21, 30] along with weight ω ∈ [ 1 32 , 1 16 , 18 , 1, 1]. These features are also used to compute the Lsty, which is defined as

where G is Gram matrix [Sajjadi et al., 2017], used to combat "checkerboard" artifacts. The overall PST loss is

where λp is set to 1.0, λs is set to 100.

Cross regularization loss. Inventively, we present a cross regularization between two-cycle paths during training CPSTN to establish the inter-path correlation. It contains both content term Lcon and edge term Ledge, which are defined as

where ∇ denotes laplacian gradient operator. We compute Ledge using the Charbonnier Loss [Lai et al., 2019]. The overall cross regularization is computed by

where, λc = λe = 8.0.

Registration loss. We adopt a bidirectional similarity loss to constrain the registration between distorted and pseudo infrared images in feature space, which is defined as

the first term is forward, while the second term is backward with weight λrev = 0.2, in which the reverse deformation field ϕ is used to distort the pseudo infrared image I˜ ir and make it approach to the distorted input Iir. To ensure a smooth deformation field, we define a smooth loss as

And then, the overall registration loss is computed by

where λsm is set to 10 in our work.

Fusion loss. In the fuison phase, we utilize MS-SSIM loss function Lms ssim to maintain sharper intensity distribution of the fused image, which is defined as

To encourage the restoration of texture details, we model the gradient distribution and develop a joint gradient loss as

For clearer textures, the gradient of the fused image is forced to approach the maximum value between the infrared and visible image gradients. In addition, to retain the saliency targets from the two images, we leverage self-visual saliency maps inspried by [Ghosh et al., 2019; Liu et al., 2021b; 2022a] to build a new loss as

where S denotes saliency matrix, ωvis and ωir denote the weight maps for infrared and visible images, respectively. The overall fusion loss is computed by

where λssim, λJG, and λsvs are set to 1.0, 20.0, and 5.0.

Overall loss. We train our network by minimizing the following overall loss function

Note that the LGAN inherits that of CycleGAN to discriminate whether pseudo infrared image is real or fake.

3 Experiments

3.1 Dataset and Implement Details

Datasets. Our misalignment cross-modality image fusion are conducted on two widely-used datasets: TNO 1 , and RoadScene 2 , which are pre-registered. To meet the requirement of misaligned image pairs, we first generate several deformation fields by performing different degrees of affine and elastic transformations and then apply them to infrared images to obtain distorted images. We use all 221 images from RoadScene dataset as training samples. Due to the limited amount of the TNO dataset, we only use it as a testing set. In addition, we randomly select 55% images from RoadScene dataset for testing since our method is full-unsupervised.

Implement Details. The code of our method is implemented using the PyTorch framework with an NVIDIA 2080Ti GPU. We randomly select 8 image patches of size 256×256 to form a batch. The Adam optimizer (β1 = 0.9, and β2 = 0.999) is used to optimize our model. The initial learning rate is set to 0.001 and remains unchanged throughout the training phase that goes through 300 epochs.

1 http://figshare.com/articles/TNO Image Fusion Dataset/1008029.

2 https://github.com/hanna-xu/RoadScene.

3.2 Evaluation in IR-VIS Image Registration

We evaluate the middle registered results using three common metrics including MSE, Mutual Information (MI) [Mahapatra et al., 2018], and Normalized Cross Correlation (NCC) [Cao et al., 2018]. Considering that multi-modality image alignment methods are extremely rare, for a fair comparison with ours, we choose two typical image alignment algorithms based on deformation field, which are FlowNet [Ilg et al., 2017] and VoxelMorph [Balakrishnan et al., 2018]. Note that, FlowNet only predicts the deformation field for input image pairs, so the spatial transform layer (STN) [Jaderberg et al., 2015] is employed to generate registered images. As reported by Table 1, using directly these two algorithms to perform cross-modality alignment of the IR-VIS images yields negligible improvement, even worse than the misaligned inputs. In contrast, the proposed CGRP gains about 0.10 and 0.23 improvments in MI on TNO and RoadScene datasets, respectively. It is demonstrated that the proposed CGRP is more effective than existing popular image alignment methods.

3.3 Comparing with SOTA in IVIF

We perform quantitative and qualitative evaluations for the proposed method against several state-of-the-art IVIF methods, including DENSE [Li and Wu, 2019], DIDFuse [Zhao et al., 2020], FGAN [Ma et al., 2019], GAN-FM [Zhang et al., 2021a], MFEIF [Liu et al., 2021a], RFN [Li et al., 2021], and U2F [Xu et al., 2022], equipped with FlowNet+STN and VoxelMorph, two typical image alignment algorithms used to cross-modality image registration in our work. For fair comparisons, we use the same testing samples (121 and 24 images from RoadScene and TNO datasets) to evaluate them.

Quantitative evaluation. We report the quantitative fusion results on TNO and RoadScene datasets in Table 2. It can be seen that our methods numerically outperforms existing IVIF methods by large margins and ranks first across all three metrics, including Cross Correlation (CC), Visual Information Fidelity (VIF) [Han et al., 2013], and Structural Similarity Index (SSIM) [Wang et al., 2004]. Especially, compared with GAN-FM, a recent research, our method gains 0.33 and 0.46 improvements in VIF on TNO and RoadScene datasets, which demonstrate the superiority of the proposed method. Qualitative evaluation. Due to space limitation, Figure 2 only presents qualitative comparisons using FlowNet+STN as the basic registration model. By observing the local enlarged areas, we can find that prior works fail to achieve desirable alignment and ghost elimination. In contrast, the proposed method shows favorable alignment and fusion capability while preserving sharp structures.

Model efficiency. To examine model efficiency, we report the statistics of Params(M) and RunTime(s) in Table 3. We conduct the measures for a paired images with size of 64×64 on a single 2080Ti GPU. Noted that, our results are obtained by jointly measuring registration and fusion networks, while other methods only have a single fusion network. The results show that our method is competitive at least in running time.

3.4 Ablation Studies

CPSTN. We plug the CPSTN into FlowNet and VoxelMorph as their improved versions, then adopt them to conduct crossmodality IR-VIS image registration. As shown in Table 1, their quantitative results improve a large margin in collaboration with CPSTN than the original versions. Accordingly, the visual comparisons provided in Figure 3 suggest the effectiveness of the CPSTN. We observe that the registered results generated by the FlowNet model equipped with the CPSTN eliminate evident distortion. Besides, we verify the effectiveness of the CPSTN from the perspective of image fusion. As quantified in Table 4, improvements of 0.14 and 0.22 in VIF are obtained using the CPSTN on TNO and RoadScene datasets, respectively. The corresponding qualitative comparisons in Figure 4(b) and (c) show that the CPSTN contributes to favorable fusion results with negligible ghosts for misaligned infrared and visible images. The above results comprehensively reveal the effectiveness of CPSTN from registration and fusion perspectives.

MRRN. Suppose that in the absence of CPSTN, we investigate the effect of MRRN on misaligned IVIF tasks. Note that MRRN performs cross-modality image registration taking the paired infrared and visible images as inputs under the same settings. As reported in Table 4, we compare the results of the first two experiments on the TNO and RoadScene datasets, which show that using MRRN to implement crossmodality registration also improves the performance of IVIF to a certain extent, compared with straightly fusing the misaligned infrared and visible images.

Loss Functions in CPSTN. We analyze the effect of Lpst and Lcross used in the CPSTN. As shown in Figure 5, without using the constraints from Lpst and Lcross, the generated pseudo infrared image suffers serious structural degradation (Figure 5(c)) compared with the reference image (Figure 5(a)). Using Lcross, the model preserves general structural information, while subtle structures are not maintained well enough and "checkerboard artifacts" are introduced obviously (Figure 5(d)). In contrast, the pseudo infrared image generated by our model (Figure 5(e)) has a sharper geometry structure, which caters to the common sense that infrared image "emphasizes structure over texture".

Interaction Fusion Module. We replace the IFM in the DIFN by concatenation operation to examine its effectiveness for the IVIF. As observed in Figure 6, the model without using the IFM tends to generate smooth textures with low contrast, while the result using the IFM has richer and sharper textures with positively high contrast. It is indicated that adaptively selecting features to be fused is more effective for improving fused image quality.

3.5 Additional Analysis

To verify the generalization of the proposed CPSTN, we plug it into two typical image alignment algorithms (i.e., FlowNet+STN and VoxelMorph) implement mono-modality image registration between distorted and pseudo infrared images, and further to implement fusion of the registered infrared image and visible image on eight existing IVIF methods. As shown in Figure 7, the first box of each method is the result of the model without using the CPSTN, and the second box is that of the model using the CPSTN. It can be observed that: i) These alignment models equipped with the CPSTN can promote effectively the fusion performance of existing DENSE DIDFuse FGAN GAN-FM MFEIF

Figure 2: Visualization of the IVIF results from different methods on the TNO and RoadScene datasets when using the FlowNet+STN

algorithm as the basic registration model.

Figure 3: Ablation analysis of the CPSTN from the registration per-

spective on RoadScene dataset.

(a) w/o T ,R

(c) Ours

Figure 4: Ablation analysis of the CPSTN (T ) and MRRN (R) on RoadScene dataset.

IVIF methods for misaligned images, which reveals the favorable generalization ability of the CPSTN for misaligned IVIF tasks. ii) Although conditioned by i), the proposed method still outperforms these improved IVIF methods on misaligned cross-modality images, which explains that the superiority of our overall framework relies on the close cooperation of each component.

4 Conclusion

In this work, we proposed a highly-robust unsupervised misaligned infrared and visible image fusion framework for mitigating ghosts of the fused images. We exploited a generationregistration paradigm to simplify the cross-modality image alignment into a mono-modality registration. A feature interaction fusion module was developed to adaptively select meaningful features from infrared and visible images to fuse, avoiding feature smoothing and emphasizing faithful textures. Extensive experimental results demonstrated the superior capability of our method on misaligned cross-modality image fusion. Significantly, our generation-

(a) Reference

(b) VIS

registration paradigm can be well extended to existing IVIF methods to improve their fusion performance on misaligned cross-modality images.

(c) w/o Lcross,Lpst

Acknowledgments

This work is partially supported by the National Natural Science Foundation of China (Nos. 61922019, 61733002 and 62027826), and the Fundamental Research Funds for the Central Universities.