{"type":"ScholarlyArticle","title":"Who Looks like Me: Semantic Routed Image Harmonization","authors":[{"name":"Jinsheng Sun","slug":"jinsheng-sun","given":"Jinsheng","family":"Sun","affiliation":"Beijing Advanced Innovation Center for Materials Genome Engineering , School of Intelligence Science and Technology , University of Science and Technology Beijing , Beijing 100083 , China"},{"name":"Chao Yao","slug":"chao-yao","given":"Chao","family":"Yao","affiliation":"School of Computer and Communication Engineering , University of Science and Technology Beijing , Beijing 100083 , China"},{"name":"Xiaokun Wang","slug":"xiaokun-wang","given":"Xiaokun","family":"Wang","affiliation":"Beijing Advanced Innovation Center for Materials Genome Engineering , School of Intelligence Science and Technology , University of Science and Technology Beijing , Beijing 100083 , China"},{"name":"Yu Guo","slug":"yu-guo","given":"Yu","family":"Guo","affiliation":"Beijing Advanced Innovation Center for Materials Genome Engineering , School of Intelligence Science and Technology , University of Science and Technology Beijing , Beijing 100083 , China"},{"name":"Yalan Zhang","slug":"yalan-zhang","given":"Yalan","family":"Zhang","affiliation":"Beijing Advanced Innovation Center for Materials Genome Engineering , School of Intelligence Science and Technology , University of Science and Technology Beijing , Beijing 100083 , China"},{"name":"Xiaojuan Ban","slug":"xiaojuan-ban","given":"Xiaojuan","family":"Ban","affiliation":"Beijing Advanced Innovation Center for Materials Genome Engineering , School of Intelligence Science and Technology , University of Science and Technology Beijing , Beijing 100083 , China; Key Laboratory of Intelligent Bionic Unmanned Systems , Ministry of Education , University of Science and Technology Beijing , Beijing 100083 , China; Institute of Materials Intelligent Technology , Liaoning Academy of Materials , Shenyang 110004 , China"}],"abstract":"Image harmonization, aiming to seamlessly blend extraneous foreground objects with background images, is a promising and challenging task.Ensuring a synthetic image appears realistic requires maintaining consistency in visual characteristics, such as texture and style, across global and semantic regions.In this paper, We approach image harmonization as a semantic routed style transfer problem, and propose an imageharmonization model by routing semantic similarity explicitly to enhance the consistency of appearance characteristics.To refine calculate the similarity between the composed foreground and background instance, we propose an InstanceSimilarity Evaluation Module(ISEM).To harness analogous semantic information effectively, we further introduceStyle Transfer Block(STB) to establish fine-grained foreground-background semantic correlation.Our method has achieved excellent experimental results on existing datasets and our model outperforms the state-of-the-art by a margin of 0.45 dB on iHarmony4 dataset.","doi":"10.24963/ijcai.2024/145","url":"https://ijcai.org/proceedings/2024/145","pdf":"https://ijcai.org/proceedings/2024/0145.pdf","markdown":"https://ijcai.org/proceedings/2024/145.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/145","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":145,"pages":{"first":1308,"last":1316},"keywords":["Computer Vision: CV: Image and video synthesis and generation","Computer Vision: CV: Computational photography"],"videos":[],"references":[{"position":1,"raw":"[Chen et al., 2022] Haoxing Chen, Zhangxuan Gu, Yaohui Li, Jun Lan, Changhua Meng, Weiqiang Wang, and Huax- iong Li. Hierarchical dynamic image harmonization. arXiv preprint arXiv:2211.08639, 2022.","title":"Hierarchical dynamic image harmonization","authors":"Chen","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cong et al., 2020] Wenyan Cong, Jianfu Zhang, Li Niu, Liu Liu, Zhixin Ling, Weiyuan Li, and Liqing Zhang. Dovenet: Deep image harmonization via domain verifi- cation. In 2020 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 8391-8400. IEEE, 2020.","title":"Dovenet: Deep image harmonization via domain verification","authors":"Wenyan Cong, Jianfu Zhang, Li Niu, Liu Liu, Zhixin Ling, Weiyuan Li, Liqing Zhang","year":"2020","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cong et al., 2021] Wenyan Cong, Li Niu, Jianfu Zhang, Jing Liang, and Liqing Zhang. Bargainnet: Background- guided domain translation for image harmonization. In 2021 IEEE International Conference on Multimedia and Expo (ICME), pages 1-6. IEEE, 2021.","title":"Bargainnet: Backgroundguided domain translation for image harmonization","authors":"Wenyan Cong, Li Niu, Jianfu Zhang, Jing Liang, Liqing Zhang","year":"2021","venue":"2021 IEEE International Conference on Multimedia and Expo (ICME)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cong et al., 2022] Wenyan Cong, Xinhao Tao, Li Niu, Jing Liang, Xuesong Gao, Qihao Sun, and Liqing Zhang. High- resolution image harmonization via collaborative dual transformations. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 18470-18479, 2022.","title":"Highresolution image harmonization via collaborative dual transformations","authors":"Wenyan Cong, Xinhao Tao, Li Niu, Jing Liang, Xuesong Gao, Qihao Sun, Liqing Zhang","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cun and Pun, 2020] Xiaodong Cun and Chi-Man Pun. Im- proving the harmony of the composite image by spatial- separated attention module. IEEE Transactions on Image Processing, 29:4759-4771, 2020.","title":"Improving the harmony of the composite image by spatialseparated attention module","authors":"Xiaodong Cun, Chi-Man Pun","year":"2020","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Guo et al., 2021a] Zonghui Guo, Dongsheng Guo, Haiyong Zheng, Zhaorui Gu, Bing Zheng, and Junyu Dong. Im- age harmonization with transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 14870-14879, 2021.","title":"Image harmonization with transformer","authors":"Guo","year":"2021","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Guo et al., 2021b] Zonghui Guo, Haiyong Zheng, Yufeng Jiang, Zhaorui Gu, and Bing Zheng. Intrinsic image har- monization. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 16367-16376, 2021.","title":"Intrinsic image harmonization","authors":"Guo","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hang et al., 2022] Yucheng Hang, Bin Xia, Wenming Yang, and Qingmin Liao. Scs-co: Self-consistent style con- trastive learning for image harmonization, 2022.","title":"Scs-co: Self-consistent style contrastive learning for image harmonization","authors":"Yucheng Hang, Bin Xia, Wenming Yang, Qingmin Liao","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hao et al., 2020] Guoqing Hao, Satoshi Iizuka, and Kazuhiro Fukui. Image harmonization with attention- based deep feature modulation. In The British Machine Vision Conference (BMCV), page 4, 2020.","title":"Image harmonization with attentionbased deep feature modulation","authors":"Guoqing Hao, Satoshi Iizuka, Kazuhiro Fukui","year":"2020","venue":"The British Machine Vision Conference (BMCV)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ho et al., 2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models, 2020.","title":"Denoising Diffusion Probabilistic Models","authors":"Jonathan Ho, Ajay Jain, Pieter Abbeel","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kaur et al., 2023] Gurpreet Kaur, Navdeep Singh, and Mu- nish Kumar. Image forgery techniques: a review. Artificial Intelligence Review, 56(2):1577-1625, 2023.","title":"Image forgery techniques: a review","authors":"Gurpreet Kaur, Navdeep Singh, Munish Kumar","year":"2023","venue":"Artificial Intelligence Review","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ke et al., 2022] Zhanghan Ke, Chunyi Sun, Lei Zhu, Ke Xu, and Rynson WH Lau. Harmonizer: Learning to perform white-box image and video harmonization. In European Conference on Computer Vision, pages 690-706. Springer, 2022.","title":"Harmonizer: Learning to perform white-box image and video harmonization","authors":"Ke","year":"2022","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kim et al., 2022] Kihong Kim, Yunho Kim, Seokju Cho, Junyoung Seo, Jisu Nam, Kychul Lee, Seungryong Kim, and KwangHee Lee. DiffFace: Diffusion-based Face Swapping with Facial Guidance, 2022.","title":"DiffFace: Diffusion-based Face Swapping with Facial Guidance","authors":"Kim","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kirillov et al., 2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643, 2023.","title":"Segment anything","authors":"Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2023] Ruibin Li, Jingcai Guo, Song Guo, Qihua Zhou, and Jie Zhang. Freepih: Training-free painterly im- age harmonization with diffusion model. arXiv preprint arXiv:2311.14926, 2023.","title":"Freepih: Training-free painterly image harmonization with diffusion model","authors":"Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liang and Pun, 2022] Jingtang Liang and Chi-Man Pun. Image harmonization with region-wise contrastive learn- ing, 2022.","title":"Jingtang Liang and Chi-Man Pun. Image harmonization with region-wise contrastive learning","authors":"Pun Liang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ling et al., 2021] Jun Ling, Han Xue, Li Song, Rong Xie, and Xiao Gu. Region-aware adaptive instance normal- ization for image harmonization. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9357-9366. IEEE, 2021.","title":"Region-aware adaptive instance normalization for image harmonization","authors":"Ling","year":"2021","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2023] Sheng Liu, Cong Phuoc Huynh, Cong Chen, Maxim Arap, and Raffay Hamid. Lemart: Label- efficient masked region transform for image harmoniza- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18290- 18299, 2023.","title":"Lemart: Labelefficient masked region transform for image harmonization","authors":"Sheng Liu, Cong Phuoc Huynh, Cong Chen, Maxim Arap, Raffay Hamid","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Loshchilov and Hutter, 2017] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Repre- sentations, 2017.","title":"Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts","authors":"Hutter Loshchilov","year":"2017","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lu et al., 2023] Lingxiao Lu, Jiangtong Li, Junyan Cao, Li Niu, and Liqing Zhang. Painterly image harmonization using diffusion model. In Proceedings of the 31st ACM International Conference on Multimedia, pages 233-241, 2023.","title":"Painterly image harmonization using diffusion model","authors":"Lu","year":"2023","venue":"Proceedings of the 31st ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Niu et al., 2023] Li Niu, Linfeng Tan, Xinhao Tao, Junyan Cao, Fengjun Guo, Teng Long, and Liqing Zhang. Deep image harmonization with globally guided feature trans- formation and relation distillation, 2023.","title":"Deep image harmonization with globally guided feature transformation and relation distillation","authors":"Li Niu, Linfeng Tan, Xinhao Tao, Junyan Cao, Fengjun Guo, Teng Long, Liqing Zhang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Pérez et al., 2003] Patrick Pérez, Michel Gangnet, and An- drew Blake. Poisson image editing. In ACM SIGGRAPH, pages 313-318. 2003.","title":"Poisson image editing","authors":"Patrick Pérez, Michel Gangnet","year":"2003","venue":"ACM SIGGRAPH","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ren and Liu, 2022] Xuqian Ren and Yifan Liu. Semantic- guided multi-mask image harmonization. In European Conference on Computer Vision, pages 564-579. Springer, 2022.","title":"Xuqian Ren and Yifan Liu. Semanticguided multi-mask image harmonization","authors":"Liu Ren","year":"2022","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Riedmiller and Lernen, 2014] Martin Riedmiller and A Ler- nen. Multi layer perceptron. Machine Learning Lab Spe- cial Lecture, University of Freiburg, 24, 2014.","title":"Multi layer perceptron","authors":"Martin Riedmiller, Lernen","year":"2014","venue":"Machine Learning Lab Special Lecture","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Shen et al., 2023] Xintian Shen, Jiangning Zhang, Jun Chen, Shipeng Bai, Yue Han, Yabiao Wang, Chengjie Wang, and Yong Liu. Learning global-aware kernel for image harmonization. arXiv preprint arXiv:2305.11676, 2023.","title":"Learning global-aware kernel for image harmonization","authors":"Xintian Shen, Jiangning Zhang, Jun Chen, Shipeng Bai, Yue Han, Yabiao Wang, Chengjie Wang, Yong Liu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sofiiuk et al., 2021] Konstantin Sofiiuk, Polina Popenova, and Anton Konushin. Foreground-aware semantic rep- resentations for image harmonization. In Foreground- Aware Semantic Representations for Image Harmoniza- tion, pages 1619-1628. IEEE, 2021.","title":"Foreground-aware semantic representations for image harmonization","authors":"Konstantin Sofiiuk, Polina Popenova, Anton Konushin","year":"2021","venue":"Foreground-Aware Semantic Representations for Image Harmonization","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Song et al., 2023] Seokbeom Song, Suhyeon Lee, Hongje Seong, Kyoungwon Min, and Euntai Kim. Shunit: style harmonization for unpaired image-to-image translation. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 2292-2302, 2023.","title":"Shunit: style harmonization for unpaired image-to-image translation","authors":"Song","year":"2023","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Sunkavalli et al., 2010] Kalyan Sunkavalli, Micah K John- son, Wojciech Matusik, and Hanspeter Pfister. Multi-scale image harmonization. ACM Transactions on Graphics (TOG), 29(4):1-10, 2010.","title":"Multi-scale image harmonization","authors":"Kalyan Sunkavalli, Micah K Johnson, Wojciech Matusik, Hanspeter Pfister","year":"2010","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tsai et al., 2017] Yi-Hsuan Tsai, Xiaohui Shen, Zhe Lin, Kalyan Sunkavalli, Xin Lu, and Ming-Hsuan Yang. Deep image harmonization. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 3789-3797, 2017.","title":"Deep image harmonization","authors":"Yi-Hsuan Tsai, Xiaohui Shen, Zhe Lin, Kalyan Sunkavalli, Xin Lu, Ming-Hsuan Yang","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Xue et al., 2012] Su Xue, Aseem Agarwala, Julie Dorsey, and Holly Rushmeier. Understanding and improving the realism of image composites. ACM Transactions on Graphics, 31(4):1-10, 2012.","title":"Understanding and improving the realism of image composites","authors":"Su Xue, Aseem Agarwala, Julie Dorsey, Holly Rushmeier","year":"2012","venue":"ACM Transactions on Graphics","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zamir et al., 2022] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration, 2022.","title":"Restormer: Efficient transformer for high-resolution image restoration","authors":"Aditya Syed Waqas Zamir, Salman Arora, Munawar Khan, Fahad Hayat, Ming-Hsuan Shahbaz Khan, Yang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhu et al., 2022] Ziyue Zhu, Zhao Zhang, Zheng Lin, Ruiqi Wu, Zhi Chai, and Chun-Le Guo. Image harmonization by matching regional references, 2022.","title":"Image harmonization by matching regional references","authors":"Ziyue Zhu, Zhao Zhang, Zheng Lin, Ruiqi Wu, Zhi Chai, Chun-Le Guo","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}