--- title: "Dynamic Brightness Adaptation for Robust Multi-modal Image Fusion" authors: - "Yiming Sun" - "Bing Cao" - "Pengfei Zhu" - "Qinghua Hu" doi: "10.24963/ijcai.2024/146" url: "https://ijcai.org/proceedings/2024/146" pdf: "https://ijcai.org/proceedings/2024/0146.pdf" venue: "Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence" year: 2024 pages: "1317-1325" track: "Main Track" keywords: - "Computer Vision: CV: Applications" - "Computer Vision: CV: Multimodal learning" --- # Dynamic Brightness Adaptation for Robust Multi-modal Image Fusion Yiming Sun, Bing Cao, Pengfei Zhu, Qinghua Hu *Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence*, pages 1317-1325. DOI: [10.24963/ijcai.2024/146](https://doi.org/10.24963/ijcai.2024/146) ## Abstract Infrared and visible image fusion aim to integrate modality strengths for visually enhanced, informative images. Visible imaging in real-world scenarios is susceptible to dynamic environmental brightness fluctuations, leading to texture degradation. Existing fusion methods lack robustness against such brightness perturbations, significantly compromising the visual fidelity of the fused imagery. To address this challenge, we propose the Brightness Adaptive multimodal dynamic fusion framework (BA-Fusion), which achieves robust image fusion despite dynamic brightness fluctuations. Specifically, we introduce a Brightness Adaptive Gate (BAG) module, which is designed to dynamically select features from brightness-related channels for normalization, while preserving brightness-independent structural information within the source images. Furthermore, we propose a brightness consistency loss function to optimize the BAG module. The entire framework is tuned via alternating training strategies. Extensive experiments validate that our method surpasses state-of-the-art methods in preserving multi-modal image information and visual fidelity, while exhibiting remarkable robustness across varying brightness levels. Our code is available: https://github.com/SunYM2020/BA-Fusion. ## Full text ## Dynamic Brightness Adaptation for Robust Multi-modal Image Fusion ## Yiming Sun , Bing Cao∗ , Pengfei Zhu and Qinghua Hu Tianjin University Tianjin Key Lab of Machine Learning Haihe Lab of ITAI Engineering Research Center of the Ministry of Education on Urban Intelligence and Digital Governance {sunyiming1895,caobing,zhupengfei,huqinghua}@tju.edu.cn Abstract Infrared and visible image fusion aim to inte- grate modality strengths for visually enhanced, in- formative images. Visible imaging in real-world scenarios is susceptible to dynamic environmental brightness fluctuations, leading to texture degra- dation. Existing fusion methods lack robustness against such brightness perturbations, significantly compromising the visual fidelity of the fused im- agery. To address this challenge, we propose the Brightness Adaptive multimodal dynamic fu- sion framework (BA-Fusion), which achieves ro- bust image fusion despite dynamic brightness fluc- tuations. Specifically, we introduce a Brightness Adaptive Gate (BAG) module, which is designed to dynamically select features from brightness- related channels for normalization, while preserv- ing brightness-independent structural information within the source images. Furthermore, we propose a brightness consistency loss function to optimize the BAG module. The entire framework is tuned via alternating training strategies. Extensive ex- periments validate that our method surpasses state- of-the-art methods in preserving multi-modal im- age information and visual fidelity, while exhibit- ing remarkable robustness across varying bright- ness levels. Our code is available: https://github. [com/SunYM2020/BA-Fusion.](https://github.com/SunYM2020/BA-Fusion) 1 Introduction Intelligent unmanned systems are susceptible to a decrease in perception ability due to the interference of real dynamic environments [Birk, 2021]. Configuring multimodal sen- sors can effectively enhance their perception ability in com- plex environments [Sun et al., 2022b]. Infrared and visi- ble cameras, as a typical set of multimodal sensors, have been widely applied in casualty searching, surveillance mis- sions, etc. However, due to the limitations of hardware de- vices and imaging mechanisms, visible or infrared cameras can usually only capture partial information of the scene and ∗Corresponding author BA-Fusion (Ours) ![Image](https://ijcai.org/proceedings/2024/fulltext/146/image_000000_488cea84dc0213b27729ebee2077621ce6af2cc5a2216170d5fda5f1687c3f1b.png) Figure 1: Visual comparisons on fused results and pixel histograms under dynamic brightness conditions. Our method keeps robust performance under varying levels of brightness. cannot fully represent the entire scene. Therefore, multimodal image fusion [Ma et al., 2019a; Zhang et al., 2021; Xu et al., 2022] can aggregate the significant contrast information of the infrared modality and the texture detail information of the visible modality, generating fusion images with sufficient information and good visual effects. In the past decades, how to design advanced fusion methods has attracted a lot of research attention. Existing infrared and visible fusion methods can be divided into two categories: traditional methods represented by image decomposition [Li et al., 2020] and sparse representation [Zhang et al., 2018], and methods based on deep learning. Among them, deep learning-based methods can be further categorized into methods based on pretrained autoencoder [Li and Wu, 2018; Zhao et al., 2020; Li et al., 2021], methods based on GAN [Ma et al., 2019b; Ma et al., 2020], methods based on CNN [Tang et al., 2022b; Sun et al., 2022a; Cao et al., 2023], methods based on diffusion models [Zhao et al., 2023], and methods based on Transformer [Wang et al., 2022; Tang et al., 2022c]. However, most of these methods directly combined the texture details and object contrasts of different modalities using a fixed correlation pattern, ignoring the dynamic changes in reality. This makes it difficult for the models to achieve dynamically robust adaptive fusion effects when facing fluctuations in environmental brightness. As shown in Fig. 1, the visible image is affected by brightness interference, resulting in overexposed or dark images. Existing methods lack the dynamic adaptive capability to handle brightness changes, inevitably leading to fluctuations in the fusion image quality with varying environmental brightness, thereby reducing the visual fidelity of the fusion image. In practical applications, infrared and visible image fusion models should possess robust adaptive capabilities to handle variations in environmental brightness to avoid fusion results being compromised due to brightness fluctuations. To fill this gap, we propose an adaptive dynamic fusion framework that can adapt to changes in brightness and achieve robust fusion of multimodal images under dynamic variations in environmental brightness. In Fig. 1, our method shows a more consistent histogram distribution under different brightness conditions, demonstrating the proposed method effectively balances the impact of brightness fluctuations on the model's learning of texture details and contrast information of different modalities, thus achieving the most robust fusion effect. Specifically, we propose a dynamic image fusion framework with brightness adaptive gating, termed BA-Fusion, which consists of two parts: Brightness Adaptive Gate (BAG) and multimodal fusion backbone network. The multimodal backbone network is composed of an encoder and a decoder, which are used to extract features from the infrared and visible modalities and generate the fusion image, respectively. The BAG module guides the model to dynamically select the most relevant feature channels with respect to brightness variations in a data-driven manner. It performs brightness normalization on these channels to eliminate the impact of brightness, while the brightness-independent channel features continue to be reused to preserve structural detail information. To train the BAG module, we designed a brightness consistency loss function, which serves as a constraint by ensuring the frequency domain brightness representation of fusion results under different brightness perturbations is consistent with that of normal fusion results. The BAG module gradually establishes the connection between brightness variations and feature channels through alternating training strategies. In this way, the proposed BA-Fusion has the capability of brightness-adaptive robust multimodal fusion. The main contributions of this paper are summarized as follows: - We propose a brightness adaptive dynamic image fusion framework, which effectively mitigates the instability issue in fusion effect caused by environmental brightness fluctuations, enabling robust fusion of infrared and visible images under dynamic brightness conditions. - We introduce a brightness adaptive gate module that establishes the correspondence between input image brightness and channel feature representation under the constraint of the brightness consistency loss function. - The proposed model dynamically balances the advan- tages of visible and infrared modalities in terms of texture details and contrast. Extensive experiments on multiple infrared-visible datasets clearly demonstrate our superiority from both quantitative and qualitative perspectives. ## 2 RelatedWorks ## 2.1 Infrared and Visible Image Fusion The infrared and visible image fusion task aims to generate fused images containing richer information by learning the multimodal superiority information [Ma et al., 2016]. DenseFuse [Li and Wu, 2018] and DIDFuse [Zhao et al., 2020] use autoencoder to extract multimodal features and combine them according to predefined rules for fusion. Recently, some task-driven fusion methods [Sun et al., 2022a; Liu et al., 2022; Tang et al., 2022a] establish a bridge between low-level image fusion and high-level visual tasks. Additionally, GAN-based methods [Ma et al., 2019b; Ma et al., 2020], Transformer-based methods [Wang et al., 2022; Tang et al., 2022c] and Diffusion models-based method [Zhao et al., 2023] have also gained widespread attention. Both DIVFusion [Tang et al., 2023] and L2Fusion [Gao et al., 2023] tackle the low-light fusion problem through the approach of low-light enhancement, without considering the fluctuations in fusion effect caused by dynamic changes in brightness (overexposure or underexposure). PIAFusion [Tang et al., 2022b] introduces an illumination-aware sub-network, but its predicted results are only used for the weights of the infrared intensity loss and visible intensity loss during the training phase. Different from coarse-grained imagelevel illumination-aware networks, our proposed BA-Fusion constructs a channel-level brightness adaptive framework to achieve dynamic normalization of brightness changes at a finer granularity, ensuring stable learning of rich texture details in the visible modality and achieving robust image fusion under dynamic illumination. ## 2.2 Brightness Correction Brightness correction helps to improve image contrast and visual appeal. Low-light image enhancement and exposure correction as typical brightness correction tasks have attracted extensive attention from researchers [Yang et al., 2023; Yao et al., 2023; Afifi et al., 2021]. Low-light image enhancement [Li et al., 2018; Zhang et al., 2022] aims to improve image visibility and quality under low-light conditions. Exposure correction [Huang et al., 2022b; Huang et al., 2022c] aims to adjust brightness components (such as illumination and reflectance) from overexposed versions to normal versions. STAN [Zhang et al., 2022] adopts a divide-andconquer strategy to model the structural representation and texture representation of low-light images separately. FECNet [Huang et al., 2022b] explores the information represented by the amplitude and phase of the Fourier Transform in exposure correction and designs an interactive module in both frequency and spatial domains to achieve brightness correction. Multimodal image fusion tasks aiming to work roundthe-clock are naturally challenged by dynamic changes in illumination. There is still a gap in how to achieve adaptive ro- ![Image](https://ijcai.org/proceedings/2024/fulltext/146/image_000001_b570b53fc4cf78ce1cc39eeadd5d72c57da78cb09565e0407d8d55da8a707a46.png) 𝑰𝑰′𝑭𝑭 Figure 2: The architecture of BA-Fusion. BA-Fusion consists of a Brightness Adaptive Gate (BAG), and the multimodal fusion backbone network. bust multimodal fusion based on image inputs with different brightness levels. For the first time, we establish the dynamic correspondence between channel-level features and brightness in the multimodal fusion task, achieving brightnessadaptive robust fusion. ## 3 Method ## 3.1 Overall Architecture In this paper, we propose a dynamic image fusion framework with brightness adaptive gating, termed BA-Fusion, which contains a Brightness Adaptive Gate (BAG) module and the multimodal fusion backbone network. Fig. 2 illustrates the architecture of the proposed BA-Fusion. The BAG module dynamically selects original and normalized features along the channel dimensions in a data-driven manner and then combines them to be passed to the decoder. The multimodal backbone network is composed of an encoder and a decoder, which are used to extract features from the infrared and visible modalities and generate the fusion image, respectively. The structure of the backbone network follows [Chen et al., 2022]. To optimize the BAG module, we also propose an alternating training strategy with a brightness consistency loss function to force the gating module to select the brightnessrelated channel, which is driven by performance stability under the brightness jitter operation. In Fig. 2, a pair of infrared image II ∈ RH×W×1 and visible image IV ∈ RH×W×3 are fed into the fusion encoders EncF to extract the multi-modal features. Among them, the infrared and visible modalities share the weights of the encoder EncF. The output of the encoder has two parts: the infrared and visible feature maps (xI and xV ). Then we feed the visible feature map xV to the BAG module to extract the channel-reorganized feature map xVg . We fuse xVg with the infrared feature map xI and feed them to the fusion decoder DecF, which generates the fused image IF ∈ RH×W×3 . ## 3.2 Brightness Adaptive Gate As shown in Fig. 2, the BAG module contains two key components, brightness normalization for eliminating brightnessrelated information and a dynamic gating module for adaptively selecting brightness-related channels. The BAG module selects the feature channels that are most relevant to the brightness change in a data-driven manner guided by dynamic gating. Brightness normalization is then performed on these channels to remove the effects of brightness while continuing to reuse the brightness-independent channel features to preserve structural detail information. ## Brightness Normalization Normalization has been proven to eliminate brightnessrelated components, retain robust structural representations, and effectively reduce the impact of brightness variations on model learning [Huang et al., 2022a]. We perform channelwise normalization of the output features xV of the encoder EncF, the formula of normalization is as follows, where µ(xV ) and σ(xV ) are the mean and standard deviation calculated independently on the spatial dimension of each channel and instance, respectively. γ, β ∈ RC are learnable parameters. The normalized feature xVnorm has a robust representation irrelevant to brightness changes, which helps the model learn robustly. However, normalization is also a double-edged sword [Yao et al., 2023], as it inevitably loses image statistical information and affects image reconstruction accuracy. ## Dynamic Gating Module In Fig. 2, we propose a dynamic gate module that takes visible feature maps xV as inputs and passes through a set of neural networks to output a routing with a set of binary indicators. This routing is used to guide the model in selecting brightness-related channels, thus achieving dynamic channel selection. Based on the routing selection result, we select only the routed channels from the xVnorm, while retaining the remaining channels of the original feature xV . Finally, these channel features are recombined to obtain the output feature xVg . This design effectively mitigates the information loss caused by normalization. The formula of dynamic gate module is as follows, where w represents the binary indicators across the channel dimension, and ⊙ is the channel-wise multiplication. The dynamic gating module contains a global average pooling (GAP) layer with 2 Conv-ReLu layers, and the output of this module is converted to a binary indicator by the binarization function G(α) to indicate the activation or deactivation of the channel. The G(α) is formalized as, where ϵ is a small positive number. This function transforms α to a value close to one or zero, resulting in an on-off switch gate without requiring additional manual threshold design. The introduction of the dynamic gating module enables dynamic normalization of the channels, allowing the recombined features to eliminate the effects of brightness while retaining the basic structural information of the features. ## 3.3 Alternating Training Strategy To train the BAG, we propose an alternating optimization strategy with brightness consistency loss to drive the dynamic gating module to adaptively select brightness-related channels and retain the brightness-independent channels. As illustrated in Fig. 2, in the first stage, we train the BAFusion model based on the original visible and infrared images. During this stage, the network weights of the BAG module are frozen, and the fusion loss Lfusion is used to optimize the entire multimodal fusion framework. In the second stage, we perform random brightness jitter on the visible images IV and input the jittered images I ′V along with the infrared images II into the BA-Fusion network to generate jittered fusion images I ′F . In this process, to encourage the BAG module to locate and filter out the brightness-related channels, we freeze the model parameters of the fusion encoder EncF and decoder DecF and only optimize the network weights of the BAG module. In the second stage, we propose a brightness consistency loss function, which constrains the consistency of the brightness and structural features of the fusion results under different brightness perturbations with the feature representation of normal fusion results in the first stage. This strategy gradually establishes the connection between brightness variations and feature channels in the BAG module, enabling the fusion model to possess dynamic fusion capability with brightness adaptation. During alternate training, all model weights are shared. ## Fusion Loss According to the alternating training strategy, we divide the network parameters into two groups based on whether they belong to the BAG module and update them using different loss functions. In the first step, we input the original visible and infrared images and update the parameters outside of the BAG module using the fusion loss function. The fusion loss Lfus formula is as follows, Among them, the pixel loss Lpixel is defined as, where W and H are the width and height. We expect the fused image to preserve the richest texture details of the images from both modalities. So the gradient loss Lgrad is formulated as, where ∇ denotes the Sobel gradient operator, which measures the texture detail information of an image. |·| stands for the absolute operation. ## Brightness Consistency Loss In the second step, we apply brightness jitter to the visible images. During this step, we freeze the parameters outside the BAG module and only update the internal parameters of the BAG using the proposed brightness consistency loss. Considering that image brightness is related to the amplitude in the frequency domain [Huang et al., 2022b], we introduce frequency domain amplitude information to design the brightness consistency loss. This encourages the network to pay closer attention to the brightness information and effectively select the channels that are related to brightness. The brightness consistency loss Lbcl is formulated as, where I ′F is the generated jittered fusion images, and A reprensents the amplitude information in frequency domain. This enables BAG to adaptively select the brightness-related channels to keep the performance on jittered fusion images. The two steps are alternately optimized by the Lfus and Lbcl, so the overall loss function is formulated as, ## 4 Experiments ## 4.1 Experimental Setting Datasets and Partition Protocol. We conducted experiments on two publicly available datasets: (M3FD [Liu et al., 2022] and LLVIP [Jia et al., 2021]). M3FD: It contains 4, 200 infrared-visible image pairs captured by on-board cameras. We used 3, 900 pairs of images for training and the remaining 300 pairs for evaluation. LLVIP: The LLVIP dataset contains 15, 488 aligned infrared-visible image pairs, which is captured by the surveillance cameras in different street scenes. We trained the model with 12, 025 image pairs and evaluated 3, 463 image pairs. Competing Methods. We compared the 9 state-of-the-art methods on two publicly available datasets (M3FD [Liu et al., 2022] and LLVIP [Jia et al., 2021]). In these comparison methods, DenseFuse [Li and Wu, 2018] and RFNNest [Li et al., 2021] are the autoencoder-based methods, PIAFusion [Tang et al., 2022b], DIVFusion [Tang et al., 2023], and IFCNN [Zhang et al., 2020] are the CNN-based methods, TarDAL [Liu et al., 2022] is the GAN-based methods. DIDFuse [Zhao et al., 2020] and DeFusion [Liang et al., 2022] are the deep learning-based decomposition methods. YDTR [Tang et al., 2022c] is the Transformer-based method. Implementation Details. We performed experiments on a computing platform with four NVIDIA GeForce RTX 3090 GPUs. We used Adam Optimization to update the overall network parameters with the learning rate of 1.0×10-4 . The training epoch is set to 60 and the batch size is 8. Evaluation Metrics. We evaluated the performance of the proposed method based on qualitative and quantitative results. The qualitative evaluation is mainly based on the visual effect of the fused image. A good fused image needs to have complementary information of multi-modal images. The quantitative evaluation mainly uses quality evaluation metrics to measure the performance of image fusion. We selected 6 popular metrics, including the spatial frequency (SF) [Eskicioglu and Fisher, 1995], standard deviation (SD), mutual information (MI) [Qu et al., 2002], visual information fidelity (VIF) [Han et al., 2013], average gradient (AG) [Cui et al., 2015], and gradient-based similarity measurement (Qabf ) [Xydeas and Petrovic, 2000]. Moreover, we also conducted task-oriented object detection evaluation. Table 1: Quantitative comparison of our BA-Fusion with 9 state-ofthe-art methods on the LLVIP dataset. Bold red indicates the best, Bold blue indicates the second best, and Bold cyan indicates the third best. | Method | SF | SD | MI | VIF | AG | Qabf | |--------------------------------|--------|--------|--------|--------|--------|--------| | DenseFuse [Li andWu, 2018] | 0.0426 | 9.3800 | 2.6764 | 0.6894 | 3.2640 | 0.3093 | | IFCNN [Zhang et al., 2020] | 0.0688 | 9.7633 | 2.9479 | 0.7797 | 5.4136 | 0.5845 | | DIDFuse [Zhao et al., 2020] | 0.0550 | 7.8074 | 2.5137 | 0.5054 | 3.4474 | 0.2436 | | RFN-Nest [Li et al., 2021] | 0.0300 | 9.7184 | 2.5042 | 0.7294 | 2.7853 | 0.2287 | | PIAFusion [Tang et al., 2022b] | 0.0787 | 9.7320 | 3.3690 | 0.8860 | 6.0846 | 0.5789 | | YDTR [Tang et al., 2022c] | 0.0474 | 8.8701 | 2.9152 | 0.6322 | 3.2043 | 0.2907 | | TarDAL [Liu et al., 2022] | 0.0647 | 9.7676 | 3.4655 | 0.7769 | 4.6094 | 0.4431 | | DeFusion [Liang et al., 2022] | 0.0425 | 9.7210 | 3.2228 | 0.7901 | 3.4194 | 0.3641 | | DIVFusion [Tang et al., 2023] | 0.0659 | 9.7579 | 2.2851 | 0.8772 | 5.7428 | 0.3386 | | BA-Fusion | 0.0812 | 9.8696 | 3.4454 | 0.9596 | 6.0597 | 0.6843 | ## 4.2 Evaluation on the LLVIP Dataset Quantitative Comparisons. The quantitative results of the different methods on the LLVIP dataset are reported in Table 1. Our method outperforms all the compared methods on 4 metrics and achieves the second best results on the remaining 2 metrics, respectively. Specifically, the highest SF we achieved indicates that the proposed method preserves richer texture details in the multi-modal images. As well as the highest SD also indicates that our fusion results can contain the highest contrast information. Qabf denotes the complementary information and edge information transferred frommulti-modal images to a fused image, respectively, and our highest results on the metrics indicate that our method can learn more valuable information from multi-modal images. Moreover, the highest VIF also means that our method can generate the most appealing fused images that are more suitable for human vision. These qualitative results demonstrate that BA-Fusion achieves the most superior fusion performance due to its ability to adapt dynamically to brightness. Qualitative Comparisons. According to Fig. 3, this depicts an actual nighttime road scene with various brightness regions. Our proposed BA-Fusion, with its ability to adapt dynamically to brightness levels, generates the fusion image with the best visual effects. Specifically, for the road traffic markings in the image, our fusion result presents the clearest texture details. On the other hand, the DIVFusion method, which possesses low-light enhancement capability, suffers from severe overexposure in this particular scene, causing blurred ground texture and weak learning of the thermal information of the car body. Furthermore, all competitive methods exhibit some degree of performance degradation in this example. Qualitative comparisons indicate that our method can dynamically fuse images based on their illumination, effectively preserving the advantages of both visible and infrared images, resulting in the fusion image with the highest information content and best visual effects. Table 2: Quantitative comparison of our BA-Fusion with 9 state-ofthe-art methods on the M3FD dataset. Bold red indicates the best, Bold blue indicates the second best, and Bold cyan indicates the third best. | Method | SF | SD | MI | VIF | AG | Qabf | |--------------------------------|--------|---------|--------|--------|--------|--------| | DenseFuse [Li andWu, 2018] | 0.0364 | 8.5987 | 2.9524 | 0.6572 | 3.0700 | 0.3838 | | IFCNN [Zhang et al., 2020] | 0.0599 | 9.2456 | 2.9954 | 0.7522 | 5.0932 | 0.5755 | | DIDFuse [Zhao et al., 2020] | 0.0420 | 9.3409 | 2.9955 | 0.7382 | 3.5668 | 0.4342 | | RFN-Nest [Li et al., 2021] | 0.0345 | 9.2984 | 2.9301 | 0.7806 | 3.1698 | 0.3772 | | PIAFusion [Tang et al., 2022b] | 0.0707 | 10.1228 | 3.8337 | 0.8447 | 5.6560 | 0.5540 | | YDTR [Tang et al., 2022c] | 0.0496 | 9.2631 | 3.2128 | 0.7276 | 3.8951 | 0.4812 | | TarDAL [Liu et al., 2022] | 0.0528 | 9.6820 | 3.2853 | 0.8347 | 4.1998 | 0.3858 | | DeFusion [Liang et al., 2022] | 0.0347 | 8.9771 | 2.9505 | 0.6752 | 2.8871 | 0.3310 | | DIVFusion [Tang et al., 2023] | 0.0656 | 9.7279 | 2.8612 | 0.9569 | 5.5956 | 0.4302 | | BA-Fusion | 0.0687 | 10.1264 | 3.8431 | 0.9786 | 5.6246 | 0.6652 | ## 4.3 Evaluation on the M3FDDataset Quantitative Comparisons. Table 2 presents the results of the quantitative evaluation on the M3FD dataset, where our method achieves the best in 4 metrics and the second best performance in the remaining metrics, respectively. In particular, it shows overwhelming advantages on MI, VIF, and Qabf , which indicates that our fusion results contain more valuable information and are more beneficial to the visual perception effect of human eyes. The highest SD and the second-best two metrics (SF & AG) also show that our fusion results retain sufficient spatial frequency, texture details, and contrast. Benefiting from the dynamic learning of brightness adaptation, our method can learn sufficient texture details and contrast information from both infrared and visible source images, avoiding the interference of visible brightness on the fusion results. Quantitative experiments on M3FD also demonstrate the state-of-the-art fusion performance of our proposed framework by dynamic adaptation of brightness. Qualitative Comparisons. As shown in Fig. 4, in complex nighttime scenes, visible images suffer from imaging blur and uneven brightness issues, making it difficult for most fusion methods to generate satisfactory results in such scenes. Traditional autoencoder-based methods, such as DenseFuse, DIDFuse, and RFN-Nest, exhibit severe contrast weakening. Additionally, most methods struggle to accurately restore the texture details of cars in the images, as seen in methods like DeFusion, YDTR, and PIAFusion. DIVFusion blindly applies low-light enhancement in this type of scene, leading to severe overexposure issues and fusion failure. Encouragingly, our method achieves the optimal fusion result by leveraging the advantage of adaptive brightness. Our approach effectively preserves the contrast information of pedestrians and cars from the infrared modality while learning all the texture information from the visible images. These qualitative results fully demonstrate that our method dynamically achieves adaptive brightness fusion by establishing the correspondence between brightness and channel features, thereby achieving optimal fusion performance. ## 4.4 Ablation Study We construct ablation studies on the LLVIP dataset, which are mainly used to verify the effectiveness of the proposed three TarDAL ![Image](https://ijcai.org/proceedings/2024/fulltext/146/image_000002_19448e17fd240b2fedcf35d8f886d55696a7b7c78e0dcf2215f941d796fdc4d3.png) IFCNN PIAFusion RFN-Nest DIVFusion BA-Fusion (Ours) Figure 3: Qualitative comparisons of various methods on representative images selected from the LLVIP dataset. Figure 4: Qualitative comparisons of various methods on representative images selected from the M3FD dataset. ![Image](https://ijcai.org/proceedings/2024/fulltext/146/image_000003_7fe5edf81da0a2ef54ab91e1f69c7261408058c38be1fd8b25f3be11c1445b5f.png) Table 3: Ablation studies on LLVIP datasets. BAG denotes brightness adaptive gating module, BC-Loss denotes brightness consistency loss function, and ATS denotes alternate training strategy. | Method | SF | SD | MI | VIF | AG | Qabf | |-------------|--------|--------|--------|--------|--------|--------| | w/o BAG | 0.0810 | 9.8555 | 3.4241 | 0.9553 | 6.0504 | 0.6825 | | w/o BC-Loss | 0.0811 | 9.8553 | 3.4352 | 0.9573 | 6.0585 | 0.6841 | | w/o ATS | 0.0810 | 9.8545 | 3.4232 | 0.9556 | 6.0472 | 0.6832 | | BA-Fusion | 0.0812 | 9.8696 | 3.4454 | 0.9596 | 6.0597 | 0.6843 | core components(brightness adaptive gating module, brightness consistency loss function, and alternate training strategy). The results of the ablation studies are shown in Table 3. Brightness AdaptiveGating. To verify the effectiveness of Brightness Adaptive Gating (BAG), we removed the brightness adaptive gating module from BA-Fusion and replaced it with a common instance normalization operation, leaving other components unchanged. The experimental results are shown in Table 3. Without brightness adaptive gating, all evaluation metrics failed to match the performance of BAFusion, indicating that normalizing all features may damage texture structure information, leading to a decrease in performance. This also demonstrates that the proposed BAG module is the key to the excellent performance of BA-Fusion. Brightness Consistency Loss. To verify the effectiveness of the Brightness Consistency Loss (BC-Loss), we remove the brightness consistency loss in the alternate training phase, and accordingly, we train the brightness adaptive gating in the second stage using only the fusion loss consistent with the first stage. The experimental results show that removing the brightness consistency loss results in a corresponding overall decrease in the performance of the model, but the magnitude of the decrease is less compared to the other components. This is mainly because brightness constraints can be constructed even if only the fusion loss is used, it is just that these constraints are not as direct and effective as our proposed brightness consistency loss. The experimental results fully demonstrate the effectiveness of the proposed brightness consistency loss. Alternate Training Strategy. To verify the effectiveness of the alternating training strategy, we keep the other components unchanged while canceling the two-step training, we no longer freeze the weights of the different components in BAFusion, and we train all components directly. We include the images generated by brightness jitter as part of the dataset and participate in the training of the whole model online, while Visible image inputs Brightness-related channel Brightness-independent channel People Car Bus Motor Lamp Truck mAP Figure 5: Channel visualization on dynamic brightness conditions. ![Image](https://ijcai.org/proceedings/2024/fulltext/146/image_000004_bbfd0a87f0b36776d4a8d84d84da3ebf541f5fcbe9ea6fdfb304fa15026bc8d2.png) 68.4 91.0 91.3 73.7 81.3 79.0 80.8 80.8 90.1 90.1 67.7 67.7 69.0 77.5 DenseFuse [Li andWu, 2018] 83.9 91.5 90.3 75.6 82.0 74.7 83.0 83.3 91.9 90.5 74.9 86.4 76.9 84.0 82.3 90.9 90.1 75.3 83.2 76.1 83.0 77.7 90.6 91.4 72.0 79.8 74.0 80.9 PIAFusion [Tang et al., 2022b] 83.5 92.9 91.3 71.9 85.4 80.0 84.2 82.3 92.2 90.9 73.8 87.0 77.4 83.9 80.9 91.6 91.5 73.5 81.1 78.7 82.9 DeFusion [Liang et al., 2022] 83.2 90.8 89.0 72.4 76.7 73.7 81.0 DIVFusion [Tang et al., 2023] 76.4 90.5 88.4 69.1 81.8 78.4 80.8 85.0 93.2 91.7 79.3 87.1 80.4 86.1 Table 4: Task-oriented object detection evaluation of our BA-Fusion with 9 state-of-the-art methods on the M3FD dataset. Bold red indicates the best, Bold blue indicates the second best, and Bold cyan indicates the third best. | Method | People | Car | |--------------------------------|----------|-------| | Visible modality | 68.4 | 91.0 | | Infrared modality | 80.8 | 90.1 | | DenseFuse [Li andWu, 2018] | 83.9 | 91.5 | | IFCNN [Zhang et al., 2020] | 83.3 | 91.9 | | DIDFuse [Zhao et al., 2020] | 82.3 | 90.9 | | RFN-Nest [Li et al., 2021] | 77.7 | 90.6 | | PIAFusion [Tang et al., 2022b] | 83.5 | 92.9 | | YDTR [Tang et al., 2022c] | 82.3 | 92.2 | | TarDAL [Liu et al., 2022] | 80.9 | 91.6 | | DeFusion [Liang et al., 2022] | 83.2 | 90.8 | | DIVFusion [Tang et al., 2023] | 76.4 | 90.5 | | BA-Fusion | 85.0 | 93.2 | applying the fusion loss and the brightness consistency loss to the different training image sources. The experimental results show that this mixed online training approach does not achieve the same fusion performance as alternating training, mainly because the fluctuation of image brightness confuses the learning of the model, making it difficult for the model to construct the relationship between brightness and channel features, leading to a degradation of the fusion performance, which also verifies the effectiveness of the proposed alternating training strategy. ## 4.5 Visualization We visualize the channel features selected by the BAG module under different brightness conditions. As shown in Fig. 5, the proposed BAG module effectively selects the brightnessrelated channels and retains the brightness-independent channels. It demonstrates the effectiveness of our method in dynamically perceiving and correcting brightness variations in different regions of the image. ## 4.6 Discussion To further verify that our BA-Fusion can realize dynamic robust fusion with brightness adaptation, we use color jitter to generate brighter and darker inputs. As shown in Fig. 6, our method can keenly perceive the change of brightness and maintains a consistent fusion effect under varying levels of brightness. The proposed method exhibits compatibility with downstream high-level vision tasks, particularly object detection. We train the YOLOv5 [Jocher, 2020] model and evaluate the detection performance using mAP as metrics. The task BA-Fusion (Ours) ![Image](https://ijcai.org/proceedings/2024/fulltext/146/image_000005_795b9d7ab1e6ccf4b559369a3b5319bdd139ce4c8ed1eddb94478f1ab6c1bfde.png) Figure 6: Visual effects on fused result under dynamic brightness conditions. Our brightness adaptive framework (BA-Fusion) keeps the robust performance under varying levels of brightness. evaluation results provided in Table 4 show that our method achieves the best performance on the object detection task. ## 5 Conclusion In this paper, we propose a brightness-adaptive dynamic image fusion framework, BA-Fusion, which consists of a Brightness Adaptive Gate (BAG) module and a multi-modal image fusion backbone network. The proposed BAG module effectively mitigates the interference of environmental brightness changes on multi-modal image fusion while preserving the structural information of the features. Through alternating training strategies, this module establishes a correspondence between image brightness and channel feature representation under the constraints of the proposed brightness consistency loss function, enabling robust multi-modal image fusion under dynamic illumination conditions. BA-Fusion can dynamically balance the advantages of the two modalities in terms of texture details and contrast, maintaining robust fusion performance in the face of brightness variations. Experimental results on several challenging datasets demonstrate that the proposed BA-Fusion outperforms state-of-the-art methods in terms of visual effects and quantitative metrics. ## Acknowledgments This work was supported in part by the National Key R&D Program of China 2022ZD0116500, in part by the National Natural Science Foundation of China under Grant 62222608, 62106171, and 61925602, in part by the Haihe Lab of ITAI under Grant 22HHXCJC00002, and in part by Tianjin Natural Science Funds for Distinguished Young Scholar under Grant 23JCJQJC00270. This work was also sponsored by CAAICANN Open Fund, developed on OpenI Community. ## References 1. [Afifi et al., 2021] Mahmoud Afifi, Konstantinos G Derpa- nis, Bjorn Ommer, and Michael S Brown. Learning multi- scale photo exposure correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9157-9167, 2021. 2. [Birk, 2021] Andreas Birk. Seeing through the forest and the trees with drones. Science Robotics, 6(55):3947, 2021. 3. [Cao et al., 2023] Bing Cao, Yiming Sun, Pengfei Zhu, and Qinghua Hu. Multi-modal gated mixture of local-to-global experts for dynamic image fusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23555-23564, 2023. 4. [Chen et al., 2022] Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun. Simple baselines for image restora- tion. In European Conference on Computer Vision, pages 17-33. Springer, 2022. 5. [Cui et al., 2015] Guangmang Cui, Huajun Feng, Zhi hai Xu, Qi Li, and Yue ting Chen. Detail preserved fusion of visi- ble and infrared images using regional saliency extraction and multi-scale image decomposition. Optics Communi- cations, 341:199-209, 2015. 6. [Eskicioglu and Fisher, 1995] Ahmet M. Eskicioglu and Paul S. Fisher. Image quality measures and their perfor- mance. IEEE Trans. Commun., 43:2959-2965, 1995. 7. [Gao et al., 2023] Xiang Gao, Guohua Lv, Aimei Dong, Zhonghe Wei, and Jinyong Cheng. L2fusion: Low-light oriented infrared and visible image fusion. In 2023 IEEE International Conference on Image Processing (ICIP), pages 2405-2409, 2023. 8. [Han et al., 2013] Yu Han, Yunze Cai, Yin Cao, and Xiaom- ing Xu. A new image fusion performance metric based on visual information fidelity. Information Fusion, 14:127- 135, 2013. 9. [Huang et al., 2022a] Jie Huang, Yajing Liu, Xueyang Fu, Man Zhou, Yang Wang, Feng Zhao, and Zhiwei Xiong. Exposure normalization and compensation for multiple- exposure correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6043-6052, 2022. 10. [Huang et al., 2022b] Jie Huang, Yajing Liu, Feng Zhao, Keyu Yan, Jinghao Zhang, Yukun Huang, Man Zhou, and Zhiwei Xiong. Deep fourier-based exposure correction network with spatial-frequency interaction. In European Conference on Computer Vision, pages 163-180. Springer, 2022. 11. [Huang et al., 2022c] Jie Huang, Man Zhou, Yajing Liu, Mingde Yao, Feng Zhao, and Zhiwei Xiong. Exposure- consistency representation learning for exposure correc- tion. In Proceedings of the 30th ACM International Con- ference on Multimedia, pages 6309-6317, 2022. 12. [Jia et al., 2021] Xinyu Jia, Chuang Zhu, Minzhen Li, Wenqi Tang, and Wenli Zhou. Llvip: A visible-infrared paired dataset for low-light vision. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pages 3496-3504, 2021. 13. [Jocher, 2020] Glenn Jocher. ultralytics/yolov5. https:// github.com/ultralytics/yolov5, October 2020. 14. [Li and Wu, 2018] Hui Li and Xiao-Jun Wu. Densefuse: A fusion approach to infrared and visible images. IEEE Transactions on Image Processing, 28(5):2614-2623, 2018. 15. [Li et al., 2018] Mading Li, Jiaying Liu, Wenhan Yang, Xi- aoyan Sun, and Zongming Guo. Structure-revealing low-light image enhancement via robust retinex model. IEEE Transactions on Image Processing, 27(6):2828- 2841, 2018. 16. [Li et al., 2020] Hui Li, Xiaojun Wu, and Josef Kittler. Md- latlrr: A novel decomposition method for infrared and vis- ible image fusion. IEEE Transactions on Image Process- ing, 29:4733-4746, 2020. 17. [Li et al., 2021] Hui Li, Xiao-Jun Wu, and Josef Kittler. Rfn- nest: An end-to-end residual fusion network for infrared and visible images. Information Fusion, 73:72-86, 2021. 18. [Liang et al., 2022] Pengwei Liang, Junjun Jiang, Xianming Liu, and Jiayi Ma. Fusion from decomposition: A self- supervised decomposition approach for image fusion. In European Conference on Computer Vision, pages 719- 735. Springer, 2022. 19. [Liu et al., 2022] Jinyuan Liu, Xin Fan, Zhanbo Huang, Guanyao Wu, Risheng Liu, Wei Zhong, and Zhongx- uan Luo. Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5802-5811, 2022. 20. [Ma et al., 2016] Jiayi Ma, Chen Chen, Chang Li, and Jun Huang. Infrared and visible image fusion via gradient transfer and total variation minimization. Information Fu- sion, 31:100-109, 2016. 21. [Ma et al., 2019a] Jiayi Ma, Yong Ma, and Chang Li. In- frared and visible image fusion methods and applications: A survey. Information Fusion, 45:153-178, 2019. 22. [Ma et al., 2019b] Jiayi Ma, Wei Yu, Pengwei Liang, Chang Li, and Junjun Jiang. Fusiongan: A generative adversarial network for infrared and visible image fusion. Information Fusion, 48:11-26, 2019. 23. [Ma et al., 2020] Jiayi Ma, Han Xu, Junjun Jiang, Xi- aoguang Mei, and Xiao-Ping Zhang. Ddcgan: A dual- discriminator conditional generative adversarial network for multi-resolution image fusion. IEEE Transactions on Image Processing, 29:4980-4995, 2020. 24. [Qu et al., 2002] Guihong Qu, Dali Zhang, and P. Yan. In- formation measure for performance of image fusion. Elec- tronics Letters, 38:313-315, 2002. 25. [Sun et al., 2022a] Yiming Sun, Bing Cao, Pengfei Zhu, and Qinghua Hu. Detfusion: A detection-driven infrared and visible image fusion network. In Proceedings of the 30th ACM International Conference on Multimedia, pages 4003-4011, 2022. 26. [Sun et al., 2022b] Yiming Sun, Bing Cao, Pengfei Zhu, and Qinghua Hu. Drone-based rgb-infrared cross-modality vehicle detection via uncertainty-aware learning. IEEE Transactions on Circuits and Systems for Video Technol- ogy, 32:6700-6713, 2022. 27. [Tang et al., 2022a] Linfeng Tang, Jiteng Yuan, and Jiayi Ma. Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fu- sion network. Information Fusion, 82:28-42, 2022. 28. [Tang et al., 2022b] Linfeng Tang, Jiteng Yuan, Hao Zhang, Xingyu Jiang, and Jiayi Ma. Piafusion: A progressive in- frared and visible image fusion network based on illumi- nation aware. Information Fusion, 83-84:79-92, 2022. 29. [Tang et al., 2022c] Wei Tang, Fazhi He, and Yu Liu. Ydtr: Infrared and visible image fusion via y-shape dynamic transformer. IEEE Transactions on Multimedia, pages 1- 16, 2022. 30. [Tang et al., 2023] Linfeng Tang, Xinyu Xiang, Hao Zhang, Meiqi Gong, and Jiayi Ma. Divfusion: Darkness-free infrared and visible image fusion. Information Fusion, 91:477-493, 2023. 31. [Wang et al., 2022] Zhishe Wang, Yanlin Chen, Wenyu Shao, Hui Li, and Lei Zhang. Swinfuse: A residual swin transformer fusion network for infrared and visible im- ages. IEEE Transactions on Instrumentation and Measure- ment, pages 1-1, 2022. 32. [Xu et al., 2022] Han Xu, Jiayi Ma, Junjun Jiang, Xiaojie Guo, and Haibin Ling. U2fusion: A unified unsuper- vised image fusion network. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44:502-518, 2022. 33. [Xydeas and Petrovic, 2000] Costas S. Xydeas and Vladimir S. Petrovic. Objective image fusion per- formance measure. Electronics Letters, 36:308-309, 2000. 34. [Yang et al., 2023] Kai-Fu Yang, Cheng Cheng, Shi-Xuan Zhao, Hong-Mei Yan, Xian-Shi Zhang, and Yong-Jie Li. Learning to adapt to light. International Journal of Com- puter Vision, 131(4):1022-1041, 2023. 35. [Yao et al., 2023] Mingde Yao, Jie Huang, Xin Jin, Ruikang Xu, Shenglong Zhou, Man Zhou, and Zhiwei Xiong. Gen- eralized lightness adaptation with channel selective nor- malization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10668- 10679, October 2023. 36. [Zhang et al., 2018] Qiang Zhang, Yi Liu, Rick S Blum, Jungong Han, and Dacheng Tao. Sparse representation based multi-sensor image fusion for multi-focus and multi- modality images: A review. Information Fusion, 40:57- 75, 2018. 37. [Zhang et al., 2020] Yu Zhang, Yu Liu, Peng Sun, Han Yan, Xiaolin Zhao, and Li Zhang. Ifcnn: A general image fu- sion framework based on convolutional neural network. Information Fusion, 54:99-118, 2020. 38. [Zhang et al., 2021] Hao Zhang, Han Xu, Xin Tian, Junjun Jiang, and Jiayi Ma. Image fusion meets deep learning: A survey and perspective. Information Fusion, 76:323-336, 2021. 39. [Zhang et al., 2022] Jinghao Zhang, Jie Huang, Mingde Yao, Man Zhou, and Feng Zhao. Structure-and texture-aware learning for low-light image enhancement. In Proceedings of the 30th ACM International Conference on Multimedia, pages 6483-6492, 2022. 40. [Zhao et al., 2020] Zixiang Zhao, Shuang Xu, Chunxia Zhang, Junmin Liu, Jiangshe Zhang, and Pengfei Li. Did- fuse: Deep image decomposition for infrared and visible image fusion. In Proceedings of the Twenty-Ninth Interna- tional Conference on International Joint Conferences on Artificial Intelligence, pages 970-976. ijcai.org, 2020. 41. [Zhao et al., 2023] Zixiang Zhao, Haowen Bai, Yuanzhi Zhu, Jiangshe Zhang, Shuang Xu, Yulun Zhang, Kai Zhang, Deyu Meng, Radu Timofte, and Luc Van Gool. Ddfm: denoising diffusion model for multi-modality im- age fusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.