MLLM-ITM: Multimodal Large Language Model Promotes Inverse Tone Mapping

1 Introduction

High dynamic range (HDR) imaging addresses the limitations of 8-bit standard dynamic range (SDR) capture by significantly extending the luminance range that a camera can record [Wang and Yoon, 2022]. By preserving detail in both brightly illuminated and deeply shadowed areas, HDR imaging allows for a more accurate reproduction of real-world lighting conditions across professional photography, computer vision, remote sensing, and emerging mixed-reality scenarios. This capability has grown increasingly relevant as more and more vision-related applications appear, enabling significant improvements in visual recognition, semantic understanding, and immersive media experiences, particularly under challenging illumination conditions.

* Corresponding author.

Current HDR capture methods typically employ multiexposure HDR or inverse tone mapping. Multi-exposure HDR imaging involves capturing multiple images at different exposure levels and subsequently fusing them to reconstruct HDR content [Lecouat et al., 2022]. However, this approach is susceptible to ghosting artefacts due to object movement and requires longer capture times, constraining its usability in real-world applications [Tel et al., 2023]. On the other hand, inverse tone-mapping operators (ITMOs) reconstruct HDR content from a single SDR exposure, which reduces capture time compared with multi-exposure HDR. Nevertheless, ITMOs inherently struggle to restore information lost in severely clipped or underexposed regions, especially since most ITMOs lack semantic understanding, limiting their effectiveness. Recently, hybrid imaging methods fusing complementary sensor modalities, such as visible with infrared [Peng et al., 2026a; Chen et al., 2026] or with eventbased cameras [Wang et al., 2019], have emerged as promising solutions for addressing these limitations. Nonetheless, such hybrid approaches generally require specialized hardware and precise cross-sensor calibration, presenting practical challenges in widespread adoption.

When only SDR images are available (usually when the existing content needs to be converted into HDR content or the hardware is limited), the only viable approach is through ITMOs. However, most existing ITMOs learn an SDR-to-HDR mapping from paired HDR/SDR data, effectively performing a locally pixel-to-pixel operation that disregards the contentdependent nature of luminance expansion. Since identical luminance values can correspond to significantly different real-world radiances depending on scene semantics, such as whether the pink-shaded regions in Fig. 2 correspond to the sky, foliage, or an artificial light source, these pixel-to-pixel mappings frequently fail. Fig. 1 contrasts a pixel-wise ITMO with our proposed approach: the former demonstrates halo artefacts along high-contrast edges and structural distortions around artificial lighting sources (highlighted by red boxes), indicative of inappropriate mapping due to the absence of semantic context.

To address this limitation, multi-modal large language models (MLLMs) provide an effective means of incorporating semantic context. MLLMs have been trained on extensive image-text datasets, learning rich, joint embedding spaces capable of encoding object identities, materials, and illumination conditions. These embeddings have successfully functioned as versatile priors for various vision tasks, including object detection [Huang et al., 2025], image style transfer [Peng et al., 2026b], and so on. Notably, LLM-HDR [Barua et al., 2025] used GPT-4o to generate a self-supervised guide during training, while lacking during inference.

Motivated by the above analysis, this paper introduces MLLM-ITM, an innovative framework integrating an MLLM front-end with an inverse tone-mapping backbone, as shown in Fig. 1. By injecting semantic priors, our pipeline achieves semantic-aware dynamic-range expansion, effectively eliminating halo artefacts and preserving structural integrity even in challenging areas where traditional pixel-wise ITMOs fail. In summary, our contributions are summarized below:

Figure 3: Typical architecture of MLLMs.

1. To the best of our knowledge, MLLM-ITM is the first method that employs an MLLM as part of an end-to-end backbone, harnessing its joint visual-language reasoning during both training and inference to directly perform inverse tone mapping, rather than invoking an MLLM only as an external tutor.
2. The proposed framework adopts a decoupled integration strategy that treats the MLLM as an interchangeable, frozen module. This design allows other MLLMs to be plugged in without redesigning the framework, enhancing the portability.
3. Experiments show that by injecting semantic priors learned from billions of image-text pairs, the proposed MLLM-ITM achieves state-of-the-art performance on the public HDR dataset.

2 RelatedWork

This section first presents HDR imaging methods and then introduces MLLMs.

2.1 High Dynamic Range Imaging

Traditional HDR pipelines recover saturated highlights and suppressed shadows by capturing a bracketed stack of exposures and fusing them into a single radiance map, commonly known as multi-exposure HDR [Lecouat et al., 2022]. Considerable effort has gone into suppressing ghosting artifacts that arise when objects or the camera itself move, using optical-flow-based alignment and exposure-time optimisation [Kalantari and Ramamoorthi, 2017]. For video, researchers have explored hardware and sampling variants: multiplying the number of sensors [Tocci et al., 2011], cycling the exposure per frame [Kalantari and Ramamoorthi, 2019], spatially varying exposure across the sensor [Serrano et al., 2016], or exploiting coded optics [Metzler et al., 2020]. While these strategies broaden temporal or spatial coverage, they also increase capture latency, calibration complexity, and the probability of motion artifacts or spatial misregistration, limiting their practicality in dynamic scenes.

Inverse tone mapping, on the other hand, seeks to reconstruct an HDR image directly from a single SDR image, turning HDR reconstruction into a learned mapping. Early work applied global linear or non-linear expansions [Masia et al., 2009; Bist et al., 2017] and content-aware guidance maps [Banterle et al., 2006; Rempel et al., 2007;

Kovaleski and Oliveira, 2014], but struggled to reconstruct heavily clipped regions [Wang et al., 2007; Didyk et al., 2008]. Deep learning has markedly improved performance: U-Net-based models [Eilertsen et al., 2017] augmented with attention mechanisms [Yu et al., 2021a] and edge-aware decompositions [Zhang and Aydin, 2021], multi-branch architectures that balance quality and efficiency [Marnerides et al., 2018], 3D-UNets that predict missing exposure stacks [Endo et al., 2017], GAN-driven refinement [Lee et al., 2018], and transformer-guided exposure selection [Zhang et al., 2023] all advance reconstruction fidelity. Nevertheless, because of a lack of semantic understanding, ITMOs remain vulnerable to color shifts and unrealistic textures. This becomes more serious when severe over- or underexposure occurs, motivating research into auxiliary modalities or priors to stabilize the expansion.

2.2 Multimodal Large Language Models

Early MLLMs adopted a modular architecture that bridges a frozen vision encoder with a frozen or lightly tuned large language model (LLM) via a compact adapter module. BLIP-2 [Li et al., 2023], for instance, aligns EVA-CLIP [Sun et al., 2023] visual features with OPT [Zhang et al., 2022] or FlanT5 using a Query-Former mechanism, enabling zero-shot image captioning and visual question answering. Similarly, MiniGPT-4 demonstrates that even a single linear projection trained on instruction-style image-text pairs can improve conversational multimodal capabilities [Zhu et al., 2024]. These bridge-based approaches significantly reduce training cost and resource demands but inherit the resolution limitations and modality gaps of their fixed backbone components.

To overcome these constraints, recent research has explored end-to-end fine-tuning of all parameters, giving rise to modern MLLMs. DeepSeek-VL initially tunes adapter layers locally before transitioning to joint optimization [Lu et al., 2024]. A notable example is Qwen3-VL [Bai et al., 2025], which provides a family of dense and mixture-of-experts vision-language models and supports strong multimodal perception, reasoning, and long-context understanding. These comprehensive strategies consistently improve performance across a wide range of tasks, including caption generation, image grounding, and video-based question answering. However, they also introduce significant computational demands.

Most modern MLLMs follow a two-stream architecture [Li et al., 2025], as shown in Fig. 3. First, each modality is preprocessed independently: textual prompts are tokenized and converted to fixed-width vectors by a text embedding layer, while raw images undergo a preprocessor (resize, pad, normalize) before a vision backbone transforms them into a sequence of patch features. Then, the two streams are fed into the language stack, where cross-modal attention fuses semantics during the decoding process. This unified architecture enables the decoupling of the downstream model from MLLMs.

3 Methodology

This section first provides an overview of the proposed MLLM-ITM and then introduces its main modules.

3.1 Overall Architecture

The overall structure of the proposed MLLM-ITM framework is shown in Fig. 4. MLLM-ITM integrates an MLLM as an encoder, a progressive cross-modal decoder designed for dynamic-range expansion, and an output head responsible for HDR radiance reconstruction. We adopt Qwen3-VL-8BInstruct [Bai et al., 2025] as our off-the-shelf frozen MLLM encoder.

First, a brief textual description of the input scene is generated using the prompt template "Please describe this image" (gray box, upper right in Fig. 4). This combined caption is processed alongside the input SDR image by the MLLM backbone. The text encoder converts the prompt into text tokens. During this process, three parallel feature streams are extracted from the vision encoder and preserved: Pix feat, low-level visual features extracted from the original input image while preserving full spatial resolution; Sem patch, the feature from the high-level MLLM encoder that contains semantically rich information; and Skip patch, the feature from the early-layer MLLM encoder preserving local detail. All parameters of the MLLM remain frozen, allowing the HDR backbone to benefit from large-scale pretraining and remain compatible with other MLLM variants.

Then, three streams are fed to the progressive cross-modal semantic interaction decoder (PSCID), which is shown in the red box of Fig. 4. PSCID contains a Text-Image Interaction Block, which injects language tokens into Sem patch and Skip patch via cross-attention, aligning visual features with scene semantics; a cascade of Context-Aware Transformer Blocks that iteratively propagate information, enabling the fusion of long-range contextual cues with fine-grained local details for more faithful HDR image reconstruction; and a Gate-MLP that adaptively balances semantics-driven and pixel-driven cues, preventing either from dominating.

Finally, a shallow convolutional head upsamples the fused representation and predicts the HDR radiance map. Since task-specific learning is confined to the HDR decoder, the overall design delivers strong performance with minimal training while retaining plug-and-play flexibility for future MLLMs.

In our Qwen3-VL-8B implementation, the frozen Qwen backbone contains 8.77B parameters, while the PSCID decoder contains only 43.29M parameters, making the taskspecific decoder about 202.5× smaller than the frozen MLLM backbone. The lightweight claim therefore refers to the module-level decoder/head overhead relative to the frozen backbone rather than the full end-to-end latency. To curb caption-driven artifacts, 20% of training captions are intentionally replaced with mismatched or nonsense text, desensitizing the model to potential hallucinations from the frozen MLLM.

3.2 Text-Image Interaction Block

The Text-Image Interaction Block serves as a key component for aligning semantic priors from the language modality with spatially resolved visual features, as illustrated in Fig. 5. Let T ∈ RNt×d represent the language tokens produced by the text encoder, and let Ps,Pk ∈ RNp×d denote the Sem patch and Skip patch tokens emitted by the visual encoder. These two sets of tokens capture complementary visual cues: Sem patch encodes high-level semantics, while Skip patch retains low-level appearance and structural detail. To enable precise alignment without introducing modality imbalance, we employ an asymmetric cross-attention (XAttn) mechanism in which the image tokens act as queries and the language tokens serve as keys and values:

where Ps,k =Ps©Pk denotes the concatenated sequence of semantic and skip tokens, and Q(·),K(·),V(·) are learnable linear projections. This step ensures that each visual token selectively attends to the most relevant textual cues in a spatially localized fashion, avoiding the global semantic drift that naive concatenation can introduce. The updated tokens are then computed as:

This semantically enriched visual representation is subsequently fed into a cascade of Context-Aware Transformer blocks, adapted from CA-ViT [Liu et al., 2022], which have been shown to be effective for HDR reconstruction in their original work. By incorporating language-derived semantics at this early stage, the network becomes capable of performing content-aware dynamic range expansion. This results in more faithful reconstruction, particularly in challenging regions with high contrast or subtle texture, while effectively suppressing artifacts such as halos.

3.3 Gate-MLP

Following the Context-Aware Transformer Blocks, the model possesses two distinct but complementary token representations: (i) context-aware tokens C, which incorporate global semantic reasoning and language-conditioned guidance, and (ii) pixel-centric tokens P, which retain low-level spatial detail but may be susceptible to noise and artifacts. To reconcile these two modalities, we propose a lightweight gating mechanism termed Gate-MLP, designed to adaptively balance the semantic richness with the spatial fidelity of pixel features.

As shown in Fig. 6, the Gate-MLP module consists of a layer normalization LN followed by a linear projection W and a sigmoid activation σ:

g = σ(W(LN(C))), g ∈ [0, 1]N×1 , (3) The resulting scalar gate determines the relative importance of the pixel-driven and semantic-driven streams on a pertoken basis. To combine the two representations, g is broadcast across the feature dimension and used to modulate the fusion:

with ⊙ the Hadamard product. Because g depends solely on the MLLM output, the network can upweigh language-driven guidance around critical scene elements while suppressing raw pixel features. This gating mechanism discourages trivial solutions (directly generating HDR from Pix feat), yielding more faithful HDR reconstructions at a small parameter overhead of 0.49% of the frozen backbone size.

4 Experiments

This section first introduces the experimental settings, dataset, and evaluation metrics. We subsequently compare our MLLM-ITM with other methods. Finally, ablation studies are presented to highlight the contributions of the main components.

4.1 Implementation Details

We used the public HDRT dataset [Peng et al., 2025] to evaluate the proposed MLLM-ITM. HDRT is a real-world dataset that includes paired HDR and SDR images. Since HDRT was originally proposed for infrared-guided HDR reconstruction, it contains poorly exposed SDR images that are difficult to convert into HDR content without supplementary infrared information. We only use the well-exposed SDR images to generate HDR images. The training set included 8,000 images, while the validation set contained 2,000 images. The dataset is partitioned following the official splits of the original HDRT benchmark.

The experiments were implemented and evaluated using PyTorch on an Intel Xeon W-2295 CPU paired with an NVIDIA Quadro RTX 8000 GPU. We trained 200 epochs using the Adam optimizer, with an initial learning rate of 4e-5, adjusted by a MultiStepLR scheduler that halved the learning rate every 20,000 steps. The loss function is the same as HDRT [Peng et al., 2025]:

where α = 1.0 and β = 1e-5. This loss formulation follows the benchmark framework provided by [Peng et al., 2025], which has been extensively validated in prior work.

For evaluating image quality, we adopt perceptually aligned variants of three standard metrics, PSNR, SSIM, and VSI. In addition, we include HDR-VDP-3 [Mantiuk et al., 2023], a perceptual metric specifically designed for HDR content. Since linear HDR color values do not correspond linearly to human visual perception, we encoded the HDR

linear color values into perceptually uniform representations to compute PSNR, SSIM, and VSI.

4.2 Comparison with Other Methods

We benchmark our MLLM-ITM against representative ITMOs on the HDRT dataset. The competitors can be divided into three categories, early deep CNNs with direct SDR to HDR regression (DrTMO [Endo et al., 2017], Deep Recursive HDRI [Lee et al., 2018]); multi-branch or frequencyaware CNNs (LaNet [Yu et al., 2021b], HDRCNN [Eilertsen et al., 2017], Deep-HDR Reconstruction [Santos et al., 2020], ExpandNet [Marnerides et al., 2018], ICTCPNet [Huang et al., 2023], HDRUNet [Chen et al., 2021a]); and the crossmodal fusion model HDRTNet [Peng et al., 2025], which leverages IR data. Quantitative results are shown in Tab. 1.

Our method attains the best score in PSNR, SSIM, and HDR-VDP-3, while remaining competitive in VSI. Specifically, MLLM-ITM delivers 23.524 dB PSNR, surpassing the strongest baseline (HDRCNN, 17.451dB) by +6.073dB. It also improves SSIM from 0.916 (HDRTNet) to 0.928 and achieves an HDR-VDP-3 score of 7.775, outperforming Deep Recursive HDRI by 0.036. Although the VSI score is slightly lower than the previous best value of 0.985, MLLM-ITM still achieves a competitive VSI of 0.977 while obtaining the best PSNR, SSIM, and HDR-VDP-3.

Traditional pixel-centric ITMOs struggle to disambiguate identical luminance values belonging to semantically different regions, leading to halo artifacts and color drift at strong contrast edges; this is reflected in their lower SSIM and HDRVDP-3 scores. By contrast, our MLLM-ITM leverages semantic understanding priors learned from billions of imagetext pairs, enabling content-aware dynamic range expansion and more accurate reconstruction of the entire image. The Gate-MLP further prevents over-smoothing, preserving fine texture that translates into higher VSI.

4.3 Visual Performance

Fig. 7 provides qualitative results that complement our quantitative findings, visually comparing the proposed MLLMITM against several state-of-the-art inverse tone-mapping methods across seven challenging scenes. For consistent visual comparison, every HDR result presented in this paper is tone-mapped using the Durand operator [Durand and Dorsey, 2002].

In bright daylight scenes with high dynamic contrast (rows 1-2), existing methods such as ICTCPNet and ExpandNet typically clip highlights in the sky region, while methods like Deep Recursive HDRI produce pronounced halo artifacts around high-contrast edges. Conversely, our MLLM-ITM effectively reconstructs cloud textures and preserves structural details in building facades, yielding results visually indistinguishable from the ground truth.

Night-time urban scenes (rows 3-4) and the backlit building scenario in row 5 illustrate another common shortcoming of current ITMOs. Methods such as LaNet and HDRCNN produce overly dim and desaturated images, while ExpandNet and Deep Recursive HDRI excessively amplify streetlights and lamps, causing overexposed regions and characteristic "ring" artefacts. Similarly, DrTMO and ICTCPNet tend to overly brighten the entire scene, deviating substantially from the ground truth appearance. Benefiting from semantic priors, MLLM-ITM generates realistic lighting conditions, maintaining readable signage and accurate color fidelity without introducing artefacts.

Finally, in severely under-exposed scenarios (rows 6-7), MLLM-ITM better recovers mid-tone structural details, such as bridge trusses and neon patterns, along with subtle shadow textures. In contrast, HDRTNet continues to exhibit mild banding and residual noise under these challenging conditions. These visual examples strongly corroborate the quantitative performance improvements detailed earlier in Tab. 1, highlighting the superior effectiveness and general applicability of our proposed MLLM-ITM framework.

4.4 Ablation Study

Tab. 2 reports a step-wise ablation on the HDRT test set, where we progressively enable or disable the four feature streams introduced in Sec. 3.1. The comparison reveals the complementary roles of the individual cues and their combined effect on reconstruction quality.

With only Pix feat, the baseline model performs a purely local regression and attains 18.396dB PSNR. Lacking any semantic guidance, it behaves as a typical pixel-wise ITMO. No Skip patch. Removing Skip patch deprives the network of low-level, high-frequency structural details. As a result, fine textures in foliage and masonry become noticeably blurry and over-smoothed (boxes in Fig. 8), PSNR drops by 0.355 dB, and VSI drops by 0.005 relative to the full model. This confirms that Skip patch is beneficial for recovering crisp local detail.

No Sem patch. Suppressing the high-level semantic tokens forces the model to extrapolate tone mapping solely from local context and low-level pixels. The absence of semantic guidance manifests as pronounced halo-box artefacts at highcontrast boundaries, especially around lamp heads (boxes in Fig. 9). Structure score decreases (SSIM 0.907) despite the presence of language cues, indicating that high-level visual semantics are indispensable for robust brightness mapping.

No Text. When explicit language tokens are removed, the network retains visual semantics but loses high-level scene descriptors such as "street-light" or "decorative bulbs". This semantic gap causes inconsistent tone allocation: the sky surrounding the lamp inherits a warm cast taken from the light source, while the other regions remain cool (Fig. 10). The perceptual metric HDR-VDP-3 degrades, underscoring the role of language priors in disambiguating object categories with similar brightness profiles.

No Pix feat. Pix feat provides the direct low-level spatial anchor from the raw input image. Disabling Pix feat forces the decoder to rely exclusively on patch- and text-level representations that have been resized and padded by the MLLM front-end. This leads to a severe degradation, with PSNR dropping from 23.524dB to 14.842dB, SSIM from 0.928 to 0.617, VSI from 0.977 to 0.743, and HDR-VDP-3 from 7.775 to 0.379. The loss of accurate pixel coordinates causes color bleeding, edge softness, and unstable local reconstruction. This result demonstrates that the direct convolutional features extracted from the raw input are indispensable as a model-independent spatial anchor for plug-and-play MLLM integration.

5 Conclusion

In this paper, we introduced MLLM-ITM, a novel inverse tone-mapping framework that leverages semantic priors provided by MLLMs. By injecting language and visual tokens into a task-specific HDR decoder, our approach achieves semantic-aware dynamic-range expansion, effectively reducing halo artefacts and preserving structural accuracy in areas where traditional pixel-based ITMOs struggle. The decoupled architecture, where the MLLM remains frozen and easily interchangeable, allows future advancements in MLLMs to be seamlessly incorporated with minimal redesign.

Extensive experiments conducted on the public HDRT dataset validate the effectiveness of our method: MLLM-ITM consistently outperforms current state-of-the-art ITMOs by substantial margins in terms of PSNR, SSIM, and HDR-VDP3. Ablation studies further demonstrate the complementary benefits of the different features extracted from the MLLM.

Future work will explore further integration of larger-scale MLLMs to enhance HDR reconstruction capabilities. Additionally, future work will extend our framework to other image processing scenarios and integrate it into other computer vision tasks.

Acknowledgments

This work was supported by the Royal Society International Exchanges 2023 Cost Share under Grant IEC\NSFC\233809.