{"type":"ScholarlyArticle","title":"Bi-level Dynamic Learning for Jointly Multi-modality Image Fusion and Beyond","authors":[{"name":"Zhu Liu","slug":"zhu-liu","given":"Zhu","family":"Liu","affiliation":"School of Software Technology , Dalian University of Technology , China"},{"name":"Jinyuan Liu","slug":"jinyuan-liu","given":"Jinyuan","family":"Liu","affiliation":"School of Mechanical Engineering , Dalian University of Technology , China"},{"name":"Guanyao Wu","slug":"guanyao-wu","given":"Guanyao","family":"Wu","affiliation":"School of Software Technology , Dalian University of Technology , China"},{"name":"Long Ma","slug":"long-ma","given":"Long","family":"Ma","affiliation":"School of Software Technology , Dalian University of Technology , China"},{"name":"Xin Fan","slug":"xin-fan","given":"Xin","family":"Fan","affiliation":"International School of Information Science Engineering , Dalian University of Technology , China"},{"name":"Risheng Liu","slug":"risheng-liu","given":"Risheng","family":"Liu","affiliation":"International School of Information Science Engineering , Dalian University of Technology , China"}],"abstract":"Recently, multi-modality scene perception tasks, e.g.,  image fusion and scene understanding, have attracted widespread attention for intelligent vision systems. However, early efforts always consider boosting a single task unilaterally and neglecting others, seldom investigating their underlying connections for joint promotion. To overcome these limitations, we establish the hierarchical dual tasks-driven deep model to bridge these tasks. Concretely, we firstly construct an image fusion module to fuse complementary characteristics and cascade dual task-related modules, including a discriminator for visual effects and a semantic network for feature measurement. \r\nWe provide a  bi-level perspective to formulate image fusion and follow-up downstream tasks. To incorporate distinct task-related responses for image fusion, we consider image fusion as a primary goal and dual modules as learnable constraints. Furthermore, we develop an efficient first-order approximation to compute corresponding gradients and present dynamic weighted aggregation to balance the gradients for fusion learning. Extensive experiments demonstrate the superiority of our method, which not only produces visually pleasant fused results but also realizes significant promotion for detection and segmentation than the state-of-the-art approaches.","doi":"10.24963/ijcai.2023/138","url":"https://ijcai.org/proceedings/2023/138","pdf":"https://ijcai.org/proceedings/2023/0138.pdf","markdown":"https://ijcai.org/proceedings/2023/138.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/138","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":138,"pages":{"first":1240,"last":1248},"keywords":["Computer Vision: CV: Recognition (object detection, categorization)","Computer Vision: CV: Scene analysis and understanding"],"videos":[],"references":[{"position":1,"raw":"[Chen et al., 2018] Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In ICML, pages 794-803.","title":"Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks","authors":"Vijay Zhao Chen, Chen-Yu Badrinarayanan, Andrew Lee, Rabinovich","year":null,"venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"PMLR, 2018.","title":null,"authors":"Pmlr","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Gulrajani et al., 2017] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. NeurlPS, 30, 2017.","title":"Improved training of wasserstein gans","authors":"Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Aaron C Vincent Dumoulin, Courville","year":"2017","venue":"NeurlPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ha et al., 2017] Qishen Ha, Kohei Watanabe, Takumi Kara- sawa, Yoshitaka Ushiku, and Tatsuya Harada. Mfnet: To- wards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes. In IROS, pages 5108- 5115. IEEE, 2017.","title":"Mfnet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes","authors":"Ha","year":"2017","venue":"IROS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Huang et al., 2022] Zhanbo Huang, Jinyuan Liu, Xin Fan, Risheng Liu, Wei Zhong, and Zhongxuan Luo. Re- conet: Recurrent correction network for fast and efficient multi-modality image fusion. In ECCV, pages 539-555. Springer, 2022.","title":"Reconet: Recurrent correction network for fast and efficient multi-modality image fusion","authors":"Huang","year":"2022","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Isola et al., 2017] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with con- ditional adversarial networks. In CVPR, pages 1125-1134, 2017.","title":"Image-to-image translation with conditional adversarial networks","authors":"Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A Efros","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jiang et al., 2022] Zhiying Jiang, Zengxi Zhang, Xin Fan, and Risheng Liu. Towards all weather and unobstructed multi-spectral image stitching: Algorithm and benchmark. In ACM MM, pages 3783-3791, 2022.","title":"Towards all weather and unobstructed multi-spectral image stitching: Algorithm and benchmark","authors":"Zhiying Jiang, Zengxi Zhang, Xin Fan, Risheng Liu","year":"2022","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Li and Wu, 2018] Hui Li and Xiao-Jun Wu. Densefuse: A fusion approach to infrared and visible images. IEEE TIP, 28(5):2614-2623, 2018.","title":"Densefuse: A fusion approach to infrared and visible images","authors":"Hui Li, Xiao-Jun Wu","year":"2018","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2022] Gongyang Li, Yike Wang, Zhi Liu, Xinpeng Zhang, and Dan Zeng. Rgb-t semantic segmentation with location, activation, and sharpening. IEEE TCSVT, 2022.","title":"Rgb-t semantic segmentation with location, activation, and sharpening","authors":"Gongyang Li, Yike Wang, Zhi Liu, Xinpeng Zhang, Dan Zeng","year":"2022","venue":"IEEE TCSVT","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lin et al., 2022] Baijiong Lin, YE Feiyang, Yu Zhang, and Ivor Tsang. Reasonable effectiveness of random weight- ing: A litmus test for multi-task learning. TMLR, 2022.","title":"Reasonable effectiveness of random weighting: A litmus test for multi-task learning","authors":"Lin","year":"2022","venue":"TMLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2019] Shikun Liu, Edward Johns, and Andrew J Davison. End-to-end multi-task learning with attention. In IEEE CVPR, pages 1871-1880, 2019.","title":"End-to-end multi-task learning with attention","authors":"Shikun Liu, Edward Johns, Andrew J Davison","year":"2019","venue":"IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2020] Risheng Liu, Jinyuan Liu, Zhiying Jiang, Xin Fan, and Zhongxuan Luo. A bilevel integrated model with data-driven layer ensemble for multi-modality image fusion. IEEE TIP, 30:1261-1274, 2020.","title":"A bilevel integrated model with data-driven layer ensemble for multi-modality image fusion","authors":"Risheng Liu, Jinyuan Liu, Zhiying Jiang, Xin Fan, Zhongxuan Luo","year":"2020","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2021a] Jinyuan Liu, Xin Fan, Ji Jiang, Risheng Liu, and Zhongxuan Luo. Learning a deep multi-scale feature ensemble and an edge-attention guidance for im- age fusion. IEEE TCSVT, 2021.","title":"Learning a deep multi-scale feature ensemble and an edge-attention guidance for image fusion","authors":"Liu","year":"2021","venue":"IEEE TCSVT","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2021b] Risheng Liu, Jiaxin Gao, Jin Zhang, Deyu Meng, and Zhouchen Lin. Investigating bi-level optimiza- tion for learning and vision from a unified perspective: A survey and beyond. IEEE TPAMI, 2021.","title":"Investigating bi-level optimization for learning and vision from a unified perspective: A survey and beyond","authors":"Liu","year":"2021","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2021c] Risheng Liu, Zhu Liu, Jinyuan Liu, and Xin Fan. Searching a hierarchically aggregated fusion ar- chitecture for fast multi-modality image fusion. In ACM MM, pages 1600-1608, 2021.","title":"Searching a hierarchically aggregated fusion architecture for fast multi-modality image fusion","authors":"Liu","year":"2021","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2022a] Jinyuan Liu, Xin Fan, Zhanbo Huang, Guanyao Wu, Risheng Liu, Wei Zhong, and Zhongxuan Luo. Target-aware dual adversarial learning and a multi- scenario multi-modality benchmark to fuse infrared and visible for object detection. In IEEE CVPR, pages 5802- 5811, 2022.","title":"Target-aware dual adversarial learning and a multiscenario multi-modality benchmark to fuse infrared and visible for object detection","authors":"Liu","year":"2022","venue":"IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2022b] Risheng Liu, Jiaxin Gao, Xuan Liu, and Xin Fan. Revisiting gans by best-response constraint: Per- spective, methodology, and application. arXiv preprint arXiv:2205.10146, 2022.","title":"Revisiting gans by best-response constraint: Perspective, methodology, and application","authors":"Liu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2023] Jinyuan Liu, Guanyao Wu, Junsheng Luan, Zhiying Jiang, Risheng Liu, and Xin Fan. Holoco: Holistic and local contrastive learning network for multi-exposure image fusion. Information Fusion, 95:237-249, 2023.","title":"Holoco: Holistic and local contrastive learning network for multi-exposure image fusion","authors":"Jinyuan Liu, Guanyao Wu, Junsheng Luan, Zhiying Jiang, Risheng Liu, Xin Fan","year":"2023","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ma et al., 2017] Jinlei Ma, Zhiqiang Zhou, Bo Wang, and Hua Zong. Infrared and visible image fusion based on vi- sual saliency map and weighted least square optimization. Infrared Physics & Technology, 82:8-17, 2017.","title":"Infrared and visible image fusion based on visual saliency map and weighted least square optimization","authors":"Jinlei Ma, Zhiqiang Zhou, Bo Wang, Hua Zong","year":"2017","venue":"Infrared Physics & Technology","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ma et al., 2022] Long Ma, Tengyu Ma, Risheng Liu, Xin Fan, and Zhongxuan Luo. Toward fast, flexible, and robust low-light image enhancement. IEEE CVPR, pages 5637- 5646, 2022.","title":"Toward fast, flexible, and robust low-light image enhancement","authors":"Long Ma, Tengyu Ma, Risheng Liu, Xin Fan, Zhongxuan Luo","year":"2022","venue":"IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sun et al., 2022] Yiming Sun, Bing Cao, Pengfei Zhu, and Qinghua Hu. Detfusion: A detection-driven infrared and visible image fusion network. In ACM MM, pages 4003- 4011, 2022.","title":"Detfusion: A detection-driven infrared and visible image fusion network","authors":"Yiming Sun, Bing Cao, Pengfei Zhu, Qinghua Hu","year":"2022","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Takumi et al., 2017] Karasawa Takumi, Kohei Watanabe, Qishen Ha, Antonio Tejero-De-Pablos, Yoshitaka Ushiku, and Tatsuya Harada. Multispectral object detection for au- tonomous vehicles. In ACM MM, pages 35-43, 2017.","title":"Multispectral object detection for autonomous vehicles","authors":"Karasawa Takumi, Kohei Watanabe, Qishen Ha, Antonio Tejero-De-Pablos, Yoshitaka Ushiku, Tatsuya Harada","year":"2017","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tang et al., 2022] Linfeng Tang, Jiteng Yuan, and Jiayi Ma. Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network. Information Fusion, 82:28-42, 2022.","title":"Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network","authors":"Tang","year":"2022","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tian et al., 2019] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: Fully convolutional one-stage object de- tection. In IEEE ICCV, pages 9627-9636, 2019.","title":"Fcos: Fully convolutional one-stage object detection","authors":"Chunhua Zhi Tian, Hao Shen, Tong Chen, He","year":"2019","venue":"IEEE ICCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2022] Di Wang, Jinyuan Liu, Xin Fan, and Risheng Liu. Unsupervised misaligned infrared and visi- ble image fusion via cross-modality image generation and registration. IJCAI, 2022.","title":"Unsupervised misaligned infrared and visible image fusion via cross-modality image generation and registration","authors":"Di Wang, Jinyuan Liu, Xin Fan, Risheng Liu","year":"2022","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2022/487"},{"position":26,"raw":"[Wu et al., 2022] Yuhui Wu, Zhu Liu, Jinyuan Liu, Xin Fan, and Risheng Liu. Breaking free from fusion rule: A fully semantic-driven infrared and visible image fusion. arXiv preprint arXiv:2211.12286, 2022.","title":"Breaking free from fusion rule: A fully semantic-driven infrared and visible image fusion","authors":"Yuhui Wu, Zhu Liu, Jinyuan Liu, Xin Fan, Risheng Liu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xie et al., 2021] Enze Xie, Wenhai Wang, Zhiding Yu, An- ima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic seg- mentation with transformers. NeurlPS, 34:12077-12090, 2021.","title":"Segformer: Simple and efficient design for semantic segmentation with transformers","authors":"Xie","year":"2021","venue":"NeurlPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xu et al., 2019] Han Xu, Pengwei Liang, Wei Yu, Junjun Jiang, and Jiayi Ma. Learning a generative model for fus- ing infrared and visible images via conditional generative adversarial network with dual discriminators. In IJCAI, pages 3954-3960, 2019.","title":"Learning a generative model for fusing infrared and visible images via conditional generative adversarial network with dual discriminators","authors":"Han Xu, Pengwei Liang, Wei Yu, Junjun Jiang, Jiayi Ma","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/549"},{"position":29,"raw":"[Xu et al., 2022] Han Xu, Jiayi Ma, Junjun Jiang, Xiaojie Guo, and Haibin Ling. U2fusion: A unified unsupervised image fusion network. IEEE TPAMI, 2022.","title":"U2fusion: A unified unsupervised image fusion network","authors":"Han Xu, Jiayi Ma, Junjun Jiang, Xiaojie Guo, Haibin Ling","year":"2022","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yan et al., 2019] Qingsen Yan, Dong Gong, Qinfeng Shi, Anton van den Hengel, Chunhua Shen, Ian Reid, and Yan- ning Zhang. Attention-guided network for ghost-free high dynamic range imaging. In IEEE CVPR, pages 1751- 1760, 2019.","title":"Attention-guided network for ghost-free high dynamic range imaging","authors":"Yan","year":"2019","venue":"IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang and Ma, 2021] Hao Zhang and Jiayi Ma. Sdnet: A versatile squeeze-and-decomposition network for real- time image fusion. IJCV, 129(10):2761-2785, 2021.","title":"Sdnet: A versatile squeeze-and-decomposition network for realtime image fusion","authors":"Hao Zhang, Jiayi Ma","year":"2021","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2021] Qiang Zhang, Shenlu Zhao, Yongjiang Luo, Dingwen Zhang, Nianchang Huang, and Jungong Han. Abmdrnet: Adaptive-weighted bi-directional modal- ity difference reduction network for rgb-t semantic seg- mentation. In IEEE CVPR, pages 2633-2642, 2021.","title":"Abmdrnet: Adaptive-weighted bi-directional modality difference reduction network for rgb-t semantic segmentation","authors":"Qiang Zhang, Shenlu Zhao, Yongjiang Luo, Dingwen Zhang, Nianchang Huang, Jungong Han","year":"2021","venue":"IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhao et al., 2020] Zixiang Zhao, Shuang Xu, Chunxia Zhang, Junmin Liu, Pengfei Li, and Jiangshe Zhang. Did- fuse: Deep image decomposition for infrared and visible image fusion. IJCAI, 2020.","title":"Didfuse: Deep image decomposition for infrared and visible image fusion","authors":"Zixiang Zhao, Shuang Xu, Chunxia Zhang, Junmin Liu, Pengfei Li, Jiangshe Zhang","year":"2020","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2020/135"},{"position":34,"raw":"[Zhao et al., 2021] Zixiang Zhao, Shuang Xu, Jiangshe Zhang, Chengyang Liang, Chunxia Zhang, and Junmin Liu. Efficient and model-based infrared and visible image fusion via algorithm unrolling. IEEE TCSVT, 2021.","title":"Efficient and model-based infrared and visible image fusion via algorithm unrolling","authors":"Zixiang Zhao, Shuang Xu, Jiangshe Zhang, Chengyang Liang, Chunxia Zhang, Junmin Liu","year":"2021","venue":"IEEE TCSVT","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhao et al., 2022] Zixiang Zhao, Jiangshe Zhang, Shuang Xu, Zudi Lin, and Hanspeter Pfister. Discrete cosine trans- form network for guided depth map super-resolution. In IEEE CVPR, pages 5687-5697. IEEE, 2022.","title":"Discrete cosine transform network for guided depth map super-resolution","authors":"Zhao","year":"2022","venue":"IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhao et al., 2023a] Zixiang Zhao, Haowen Bai, Jiangshe Zhang, Yulun Zhang, Shuang Xu, Zudi Lin, Radu Tim- ofte, and Luc Van Gool. Cddfuse: Correlation-driven dual-branch feature decomposition for multi-modality im- age fusion. IEEE CVPR, 2023.","title":"Cddfuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusion","authors":"Zhao","year":"2023","venue":"IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhao et al., 2023b] Zixiang Zhao, Haowen Bai, Yuanzhi Zhu, Jiangshe Zhang, Shuang Xu, Yulun Zhang, Kai Zhang, Deyu Meng, Radu Timofte, and Luc Van Gool. DDFM: denoising diffusion model for multi-modality im- age fusion. CoRR, abs/2303.06840, 2023.","title":"DDFM: denoising diffusion model for multi-modality image fusion","authors":"Zhao","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhou et al., 2021a] Man Zhou, Jie Xiao, Yifan Chang, Xueyang Fu, Aiping Liu, Jinshan Pan, and Zheng-Jun Zha. Image de-raining via continual learning. In IEEE CVPR, pages 4907-4916, 2021.","title":"Image de-raining via continual learning","authors":"Zhou","year":"2021","venue":"IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zhou et al., 2021b] Wujie Zhou, Shaohua Dong, Caie Xu, and Yaguan Qian. Edge-aware guidance fusion network for rgb thermal scene parsing. AAAI, 2021.","title":"Edge-aware guidance fusion network for rgb thermal scene parsing","authors":"Zhou","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}