{"type":"ScholarlyArticle","title":"Eliminating the Cross-Domain Misalignment in Text-guided Image Inpainting","authors":[{"name":"Muqi Huang","slug":"muqi-huang","given":"Muqi","family":"Huang","affiliation":"Institute of Artificial Intelligence , School of Computer Science , Wuhan University"},{"name":"Chaoyue Wang","slug":"chaoyue-wang","given":"Chaoyue","family":"Wang","affiliation":"JD Explore Academy"},{"name":"Yong Luo","slug":"yong-luo","given":"Yong","family":"Luo","affiliation":"Institute of Artificial Intelligence , School of Computer Science , Wuhan University"},{"name":"Lefei Zhang","slug":"lefei-zhang","given":"Lefei","family":"Zhang","affiliation":"Institute of Artificial Intelligence , School of Computer Science , Wuhan University; Hubei Luojia Laboratory"}],"abstract":"Text-guided image inpainting has rapidly garnered prominence as a task in user-directed image synthesis, aiming to complete the occluded image regions following the textual prompt provided. However, current methods usually grapple with issues arising from the disparity between low-level pixel data and high-level semantic descriptions, which results in inpainted sections not harmonizing with the original image (either structurally or texturally). In this study, we introduce a Structure-Aware Inpainting Learning scheme and an Asymmetric Cross Domain Attention to address these cross-domain misalignment challenges. The proposed structure-aware learning scheme employs features of an intermediate modality as structure guidance to bridge the gap between text information and low-level pixels. Meanwhile, asymmetric cross-domain attention enhances the texture consistency between inpainted and unmasked regions. Our experiments show exceptional performance on leading datasets such as MS-COCO and Open Images, surpassing state-of-the-art text-guided image inpainting methods. Code is released at: https://github.com/MucciH/ECDM-inpainting","doi":"10.24963/ijcai.2024/97","url":"https://ijcai.org/proceedings/2024/97","pdf":"https://ijcai.org/proceedings/2024/0097.pdf","markdown":"https://ijcai.org/proceedings/2024/97.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/97","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":97,"pages":{"first":875,"last":883},"keywords":["Computer Vision: CV: Vision, language and reasoning","Computer Vision: CV: Image and video synthesis and generation"],"videos":[],"references":[{"position":1,"raw":"[Avrahami et al., 2022] Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven edit- ing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18187-18197, 2022.","title":"Blended diffusion for text-driven editing of natural images","authors":"Omri Avrahami, Dani Lischinski, Ohad Fried","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Barnes et al., 2009] Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B. Goldman. Patchmatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph., 28(3):24, 2009.","title":"Patchmatch: a randomized correspondence algorithm for structural image editing","authors":"Connelly Barnes, Eli Shechtman, Adam Finkelstein, Dan B Goldman","year":"2009","venue":"ACM Trans. Graph","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cao and Fu, 2021] Chenjie Cao and Yanwei Fu. Learn- ing a sketch tensor space for image inpainting of man- made scenes. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 14489- 14498, 2021.","title":"Learning a sketch tensor space for image inpainting of manmade scenes","authors":"Fu ; ] Chenjie Cao, Yanwei Cao, Fu","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Criminisi et al., 2003] Antonio Criminisi, Patrick Pérez, and Kentaro Toyama. Object removal by exemplar-based inpainting. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 721-728, 2003.","title":"Object removal by exemplar-based inpainting","authors":"Criminisi","year":"2003","venue":"Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Goodfellow et al., 2020] Ian J. Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Ben- gio. Generative adversarial networks. Commun. ACM, 63(11):139-144, 2020.","title":"Generative adversarial networks","authors":"Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C Courville, Yoshua Bengio","year":"2020","venue":"Commun. ACM","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ho et al., 2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems 33, 2020.","title":"Denoising diffusion probabilistic models","authors":"Jonathan Ho, Ajay Jain, Pieter Abbeel","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Huang and Zhang, 2022] Muqi Huang and Lefei Zhang. Atrous pyramid transformer with spectral convolution for image inpainting. In Proceedings of the 30th ACM In- ternational Conference on Multimedia, pages 4674-4683, 2022.","title":"Atrous pyramid transformer with spectral convolution for image inpainting","authors":"Zhang Huang, Muqi Huang, Lefei Zhang","year":"2022","venue":"Proceedings of the 30th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kuznetsova et al., 2020] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont- Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International Journal of Computer Vision, 128(7):1956-1981, 2020.","title":"The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale","authors":"Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov","year":"2020","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2020] Jingyuan Li, Ning Wang, Lefei Zhang, Bo Du, and Dacheng Tao. Recurrent feature reasoning for image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7757-7765, 2020.","title":"Recurrent feature reasoning for image inpainting","authors":"Jingyuan Li, Ning Wang, Lefei Zhang, Bo Du, Dacheng Tao","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2022a] Wenbo Li, Zhe Lin, Kun Zhou, Lu Qi, Yi Wang, and Jiaya Jia. MAT: mask-aware transformer for large hole image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10748-10758, 2022.","title":"MAT: mask-aware transformer for large hole image inpainting","authors":"Li","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2022b] Wenbo Li, Xin Yu, Kun Zhou, Yibing Song, Zhe Lin, and Jiaya Jia. SDM: spatial diffusion model for large hole image inpainting. arXiv preprint arXiv:2212.02963, 2022.","title":"SDM: spatial diffusion model for large hole image inpainting","authors":"Li","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge J. Be- longie, James Hays, Pietro Perona, Deva Ramanan, Pi- otr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In Proceedings of European Conference on Computer Vision, volume 8693, pages 740- 755, 2014.","title":"Microsoft COCO: common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge J Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence Zitnick","year":"2014","venue":"Proceedings of European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lin et al., 2020] Qing Lin, Bo Yan, Jichun Li, and Weimin Tan. MMFL: multimodal fusion learning for text-guided image inpainting. In Proceedings of the 28th ACM In- ternational Conference on Multimedia, pages 1094-1102, 2020.","title":"MMFL: multimodal fusion learning for text-guided image inpainting","authors":"Qing Lin, Bo Yan, Jichun Li, Weimin Tan","year":"2020","venue":"Proceedings of the 28th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2018] Guilin Liu, Fitsum A Reda, Kevin J Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. Im- age inpainting for irregular holes using partial convolu- tions. In Proceedings of the European conference on com- puter vision, pages 85-100, 2018.","title":"Image inpainting for irregular holes using partial convolutions","authors":"Liu","year":"2018","venue":"Proceedings of the European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lugmayr et al., 2022] Andreas Lugmayr, Martin Danelljan, Andrés Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. RePaint: inpainting using denoising diffusion prob- abilistic models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 11451-11461, 2022.","title":"RePaint: inpainting using denoising diffusion probabilistic models","authors":"Andreas Lugmayr, Martin Danelljan, Andrés Romero, Fisher Yu, Radu Timofte, Luc Van Gool","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nichol et al., 2022] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. In Proceedings of the International Conference on Machine Learning, volume 162, pages 16784-16804, 2022.","title":"GLIDE: towards photorealistic image generation and editing with text-guided diffusion models","authors":"Alexander Quinn, Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, Mark Chen","year":"2022","venue":"Proceedings of the International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rombach et al., 2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10674-10685, 2022.","title":"High-resolution image synthesis with latent diffusion models","authors":"Rombach","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sun et al., 2021] Liujie Sun, Qinghan Zhang, Wenju Wang, and Mingxi Zhang. Image inpainting with learnable edge- attention maps. IEEE Access, 9:3816-3827, 2021.","title":"Image inpainting with learnable edgeattention maps","authors":"Sun","year":"2021","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Suvorov et al., 2022] Roman Suvorov, Elizaveta Lo- gacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution- robust large mask inpainting with fourier convolutions. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3172-3182. IEEE, 2022.","title":"Resolutionrobust large mask inpainting with fourier convolutions","authors":"Suvorov","year":"2022","venue":"Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Sys- tems 30, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in Neural Information Processing Systems 30","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2023] Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset, Shai Noy, Stefano Pelle- grini, Yasumasa Onoe, Sarah Laszlo, David J Fleet, Radu Soricut, et al. Imagen editor and editbench: Advancing and evaluating text-guided image inpainting. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18359-18369, 2023.","title":"Imagen editor and editbench: Advancing and evaluating text-guided image inpainting","authors":"Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J Fleet, Radu Soricut","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wu et al., 2021] Xingcai Wu, Yucheng Xie, Jiaqi Zeng, Zhenguo Yang, Yi Yu, Qing Li, and Wenyin Liu. Ad- versarial learning with mask reconstruction for text-guided image inpainting. In Proceedings of the 29th ACM Multi- media Conference, pages 3464-3472, 2021.","title":"Adversarial learning with mask reconstruction for text-guided image inpainting","authors":"Xingcai Wu, Yucheng Xie, Jiaqi Zeng, Zhenguo Yang, Yi Yu, Qing Li, Wenyin Liu","year":"2021","venue":"Proceedings of the 29th ACM Multimedia Conference","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xie et al., 2022] Yucheng Xie, Zehang Lin, Zhenguo Yang, Huan Deng, Xingcai Wu, Xudong Mao, Qing Li, and Wenyin Liu. Learning semantic alignment from image for text-guided image inpainting. Vis. Comput., 38(9):3149- 3161, 2022.","title":"Learning semantic alignment from image for text-guided image inpainting","authors":"Yucheng Xie, Zehang Lin, Zhenguo Yang, Huan Deng, Xingcai Wu, Xudong Mao, Qing Li, Wenyin Liu","year":"2022","venue":"Vis. Comput","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xie et al., 2023] Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang. Smartbrush: Text and shape guided object inpainting with diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 22428-22437, 2023.","title":"Smartbrush: Text and shape guided object inpainting with diffusion model","authors":"Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, Kun Zhang","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yang et al., 2023] Shiyuan Yang, Xiaodong Chen, and Jing Liao. Uni-Paint: a unified framework for multimodal im- age inpainting with pretrained diffusion model. In Pro- ceedings of the 31st ACM International Conference on Multimedia, page 3190-3199, 2023.","title":"Uni-Paint: a unified framework for multimodal image inpainting with pretrained diffusion model","authors":"Shiyuan Yang, Xiaodong Chen, Jing Liao","year":"2023","venue":"Proceedings of the 31st ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yu et al., 2019] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S. Huang. Free-form image inpainting with gated convolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4470-4479, 2019.","title":"Free-form image inpainting with gated convolution","authors":"Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, Thomas S Huang","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2020a] Lisai Zhang, Qingcai Chen, Baotian Hu, and Shuoran Jiang. Text-guided neural image inpaint- ing. In Proceedings of the 28th ACM International Con- ference on Multimedia, pages 1302-1310, 2020.","title":"Text-guided neural image inpainting","authors":"Zhang","year":"2020","venue":"Proceedings of the 28th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2020b] Zijian Zhang, Zhou Zhao, Zhu Zhang, Baoxing Huai, and Jing Yuan. Text-guided image inpaint- ing. In Proceedings of the 28th ACM International Con- ference on Multimedia, pages 4079-4087, 2020.","title":"Text-guided image inpainting","authors":"Zhang","year":"2020","venue":"Proceedings of the 28th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2023a] Guanhua Zhang, Jiabao Ji, Yang Zhang, Mo Yu, Tommi Jaakkola, and Shiyu Chang. To- wards coherent image inpainting using denoising diffusion implicit models. In Proceedings of the International Con- ference on Machine Learning, pages 41164-41193, 2023.","title":"Towards coherent image inpainting using denoising diffusion implicit models","authors":"Zhang","year":"2023","venue":"Proceedings of the International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2023b] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image dif- fusion models. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 3836-3847, 2023.","title":"Adding conditional control to text-to-image diffusion models","authors":"Zhang","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}