{"type":"ScholarlyArticle","title":"Learning a Generative Model for Fusing Infrared and Visible Images via Conditional Generative Adversarial Network with Dual Discriminators","authors":[{"name":"Han Xu","slug":"han-xu","given":"Han","family":"Xu","affiliation":"Electronic Information School , Wuhan University , Wuhan 430072 , China"},{"name":"Pengwei Liang","slug":"pengwei-liang","given":"Pengwei","family":"Liang","affiliation":"Electronic Information School , Wuhan University , Wuhan 430072 , China"},{"name":"Wei Yu","slug":"wei-yu","given":"Wei","family":"Yu","affiliation":"Electronic Information School , Wuhan University , Wuhan 430072 , China"},{"name":"Junjun Jiang","slug":"junjun-jiang","given":"Junjun","family":"Jiang","affiliation":"School of Computer Science and Technology , Harbin Institute of Technology , Harbin 150001 , China"},{"name":"Jiayi Ma","slug":"jiayi-ma","given":"Jiayi","family":"Ma","affiliation":"Electronic Information School , Wuhan University , Wuhan 430072 , China"}],"abstract":"In this paper, we propose a new end-to-end model, called dual-discriminator conditional generative adversarial network (DDcGAN), for fusing infrared and visible images of different resolutions. Unlike the pixel-level methods and existing deep learning-based methods, the fusion task is accomplished through the adversarial process between a generator and two discriminators, in addition to the specially designed content loss. The generator is trained to generate real-like fused images to fool discriminators. The two discriminators are trained to calculate the JS divergence between the probability distribution of downsampled fused images and infrared images, and the JS divergence between the probability distribution of gradients of fused images and gradients of visible images, respectively. Thus, the fused images can compensate for the features that are not constrained by the single content loss. Consequently, the prominence of thermal targets in the infrared image and the texture details in the visible image can be preserved or even enhanced in the fused image simultaneously. Moreover, by constraining and distinguishing between the downsampled fused image and the low-resolution infrared image, DDcGAN can be preferably applied to the fusion of different resolution images. Qualitative and quantitative experiments on publicly available datasets demonstrate the superiority of our method over the state-of-the-art.","doi":"10.24963/ijcai.2019/549","url":"https://ijcai.org/proceedings/2019/549","pdf":"https://ijcai.org/proceedings/2019/0549.pdf","markdown":"https://ijcai.org/proceedings/2019/549.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/549","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":549,"pages":{"first":3954,"last":3960},"keywords":["Machine Learning: Deep Learning","Machine Learning Applications: Applications of Unsupervised Learning"],"videos":[],"references":[{"position":1,"raw":"[Beck and Teboulle, 2009] Amir Beck and Marc Teboulle. Fast gradient-based algorithms for constrained total vari- ation image denoising and deblurring problems. IEEE Transactions on Image Processing, 18(11):2419-2434, 2009.","title":"Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems","authors":"Amir Beck, Marc Teboulle","year":"2009","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Du et al., 2018] Qinglei Du, Han Xu, Yong Ma, Jun Huang, and Fan Fan. Fusing infrared and visible images of d- ifferent resolutions via total variation model. Sensors, 18(11):3827, 2018.","title":"Fusing infrared and visible images of different resolutions via total variation model","authors":"Du","year":"2018","venue":"Sensors","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In Advances in Neural Informa- tion Processing Systems, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Jin et al., 2017] Xin Jin, Qian Jiang, Shaowen Yao, Dong- ming Zhou, Rencan Nie, Jinjin Hai, and Kangjian He. A survey of infrared and visual image fusion methods. In- frared Physics & Technology, 85:478-501, 2017.","title":"A survey of infrared and visual image fusion methods","authors":"Xin Jin, Qian Jiang, Shaowen Yao, Dongming Zhou, Rencan Nie, Jinjin Hai, Kangjian He","year":"2017","venue":"Infrared Physics & Technology","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Li and Wu, 2019] Hui Li and Xiao-Jun Wu. Densefuse: A fusion approach to infrared and visible images. IEEE Transactions on Image Processing, 28(5):2614-2623, 2019.","title":"Densefuse: A fusion approach to infrared and visible images","authors":"Hui Li, Xiao-Jun Wu","year":"2019","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Li et al., 2017] Shutao Li, Xudong Kang, Leyuan Fang, Jianwen Hu, and Haitao Yin. Pixel-level image fusion: A survey of the state of the art. Information Fusion, 33:100- 112, 2017.","title":"Pixel-level image fusion: A survey of the state of the art","authors":"Li","year":"2017","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Li et al., 2018] Hui Li, Xiao-Jun Wu, and Josef Kittler. In- frared and visible image fusion using a deep learning framework. arXiv preprint arXiv:1804.06992, 2018.","title":"Infrared and visible image fusion using a deep learning framework","authors":"Hui Li, Xiao-Jun Wu, Josef Kittler","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Liu et al., 2017] Yu Liu, Xun Chen, Juan Cheng, and Hu Peng. A medical image fusion method based on con- volutional neural networks. In Proceedings of the Inter- national Conference on Information Fusion, pages 1-7. IEEE, 2017.","title":"A medical image fusion method based on convolutional neural networks","authors":"Yu Liu, Xun Chen, Juan Cheng, Hu Peng","year":"2017","venue":"Proceedings of the International Conference on Information Fusion","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lu et al., 2016] Wenqi Lu, Jinming Duan, Zhaowen Qiu, Zhenkuan Pan, Ryan Wen Liu, and Li Bai. Implementation of high-order variational models made easy for image pro- cessing. Mathematical Methods in the Applied Sciences, 39(14):4208-4233, 2016.","title":"Implementation of high-order variational models made easy for image processing","authors":"Wenqi Lu, Jinming Duan, Zhaowen Qiu, Zhenkuan Pan, Ryan Wen Liu, Li Bai","year":"2016","venue":"Mathematical Methods in the Applied Sciences","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ma et al., 2016] Jiayi Ma, Chen Chen, Chang Li, and Jun Huang. Infrared and visible image fusion via gradien- t transfer and total variation minimization. Information Fusion, 31:100-109, 2016.","title":"Infrared and visible image fusion via gradient transfer and total variation minimization","authors":"Jiayi Ma, Chen Chen, Chang Li, Jun Huang","year":"2016","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ma et al., 2017] Jinlei Ma, Zhiqiang Zhou, Bo Wang, and Hua Zong. Infrared and visible image fusion based on vi- sual saliency map and weighted least square optimization. Infrared Physics & Technology, 82:8-17, 2017.","title":"Infrared and visible image fusion based on visual saliency map and weighted least square optimization","authors":"Jinlei Ma, Zhiqiang Zhou, Bo Wang, Hua Zong","year":"2017","venue":"Infrared Physics & Technology","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ma et al., 2019a] Jiayi Ma, Yong Ma, and Chang Li. In- frared and visible image fusion methods and applications: a survey. Information Fusion, 45:153-178, 2019.","title":"Infrared and visible image fusion methods and applications: a survey","authors":"Ma","year":"2019","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ma et al., 2019b] Jiayi Ma, Wei Yu, Pengwei Liang, Chang Li, and Junjun Jiang. Fusiongan: A generative adversarial network for infrared and visible image fusion. Information Fusion, 48:11-26, 2019.","title":"Fusiongan: A generative adversarial network for infrared and visible image fusion","authors":"Ma","year":"2019","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mirza and Osindero, 2014] Mehdi Mirza and Simon Osin- dero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014.","title":"Conditional generative adversarial nets","authors":"Osindero Mirza","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Naidu, 2014] VPS Naidu. Hybrid ddct-pca based multi sen- sor image fusion. Journal of Optics, 43(1):48-61, 2014.","title":"VPS Naidu. Hybrid ddct-pca based multi sensor image fusion","authors":"Naidu","year":"2014","venue":"Journal of Optics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Paramanandham and Rajendiran, 2018] Nirmala Para- manandham and Kishore Rajendiran. Multi sensor image fusion for surveillance applications using hybrid image fusion algorithm. Multimedia Tools and Applications, pages 1-32, 2018.","title":"Multi sensor image fusion for surveillance applications using hybrid image fusion algorithm","authors":"Nirmala Paramanandham, Kishore Rajendiran","year":"2018","venue":"Multimedia Tools and Applications","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Prabhakar et al., 2017] K Ram Prabhakar, V Sai Srikar, and R Venkatesh Babu. Deepfuse: A deep unsupervised ap- proach for exposure fusion with extreme exposure image pairs. In Proceedings of the IEEE International Confer- ence on Computer Vision, pages 4724-4732, 2017.","title":"Deepfuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs","authors":"Ram Prabhakar, V Sai Srikar, R Venkatesh, Babu","year":"2017","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ronneberger et al., 2015] Olaf Ronneberger, Philipp Fisch- er, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Proceedings of the In- ternational Conference on Medical Image Computing and Computer-Assisted Intervention, pages 234-241. Springer, 2015.","title":"U-net: Convolutional networks for biomedical image segmentation","authors":"Ronneberger","year":"2015","venue":"Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Zhang et al., 2017] He Zhang, Vishwanath Sindagi, and Vishal M Patel. Image de-raining using a condition- al generative adversarial network. arXiv preprint arX- iv:1701.05957, 2017.","title":"Image de-raining using a conditional generative adversarial network","authors":"Zhang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zhang et al., 2018] Qiang Zhang, Yi Liu, Rick S Blum, Jungong Han, and Dacheng Tao. Sparse representation based multi-sensor image fusion for multi-focus and multi- modality images: A review. Information Fusion, 40:57- 75, 2018.","title":"Sparse representation based multi-sensor image fusion for multi-focus and multimodality images: A review","authors":"Qiang Zhang, Yi Liu, Rick S Blum, Jungong Han, Dacheng Tao","year":"2018","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhou et al., 2016] Zhiqiang Zhou, Bo Wang, Sun Li, and Mingjie Dong. Perceptual fusion of infrared and visible images through a hybrid multi-scale decomposition with gaussian and bilateral filters. Information Fusion, 30:15- 26, 2016.","title":"Perceptual fusion of infrared and visible images through a hybrid multi-scale decomposition with gaussian and bilateral filters","authors":"Zhiqiang Zhou, Bo Wang, Sun Li, Mingjie Dong","year":"2016","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhu et al., 2018] Zhiqin Zhu, Hongpeng Yin, Yi Chai, Yanxia Li, and Guanqiu Qi. A novel multi-modality image fusion method based on image decomposition and sparse representation. Information Sciences, 432:516-529, 2018.","title":"A novel multi-modality image fusion method based on image decomposition and sparse representation","authors":"Zhu","year":"2018","venue":"Information Sciences","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}