{"type":"ScholarlyArticle","title":"Coarse-to-Fine Image Inpainting via Region-wise Convolutions and Non-Local Correlation","authors":[{"name":"Yuqing Ma","slug":"yuqing-ma","given":"Yuqing","family":"Ma","affiliation":"State Key Lab of Software Development Environment , Beihang University , China"},{"name":"Xianglong Liu","slug":"xianglong-liu","given":"Xianglong","family":"Liu","affiliation":"State Key Lab of Software Development Environment , Beihang University , China; Beijing Advanced Innovation Center for Big Data-Based Precision Medicine , Beihang University , Beijing , China"},{"name":"Shihao Bai","slug":"shihao-bai","given":"Shihao","family":"Bai","affiliation":"State Key Lab of Software Development Environment , Beihang University , China"},{"name":"Lei Wang","slug":"lei-wang","given":"Lei","family":"Wang","affiliation":"State Key Lab of Software Development Environment , Beihang University , China"},{"name":"Dailan He","slug":"dailan-he","given":"Dailan","family":"He","affiliation":"State Key Lab of Software Development Environment , Beihang University , China"},{"name":"Aishan Liu","slug":"aishan-liu","given":"Aishan","family":"Liu","affiliation":"State Key Lab of Software Development Environment , Beihang University , China"}],"abstract":"Recently deep neural networks have achieved promising performance for filling large missing regions in image inpainting tasks. They usually adopted the standard convolutional architecture over the corrupted image, where the same convolution filters try to restore the diverse information on both existing and missing regions, and meanwhile ignores the long-distance correlation among the regions. Only relying on the surrounding areas inevitably leads to meaningless contents and artifacts, such as color discrepancy and blur. To address these problems, we first propose region-wise convolutions to locally deal with the different types of regions, which can help exactly reconstruct existing regions and roughly infer the missing ones from existing regions at the same time. Then, a non-local operation is introduced to globally model the correlation among different regions, promising visual consistency between missing and existing regions. Finally, we integrate the region-wise convolutions and non-local correlation in a coarse-to-fine framework to restore semantically reasonable and visually realistic images. Extensive experiments on three widely-used datasets for image inpainting tasks have been conducted, and both qualitative and quantitative experimental results demonstrate that the proposed model significantly outperforms the state-of-the-art approaches, especially for the large irregular missing regions.","doi":"10.24963/ijcai.2019/433","url":"https://ijcai.org/proceedings/2019/433","pdf":"https://ijcai.org/proceedings/2019/0433.pdf","markdown":"https://ijcai.org/proceedings/2019/433.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/433","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":433,"pages":{"first":3123,"last":3129},"keywords":["Computer Vision: Computer Vision","Machine Learning: Deep Learning","Machine Learning Applications: Other Applications"],"videos":[],"references":[{"position":1,"raw":"[Ballester et al., 2000] Coloma Ballester, Marcelo Bertalmio, Vicent Caselles, Guillermo Sapiro, and Joan Verdera. Filling-in by joint interpolation of vector fields and gray levels. 2000.","title":"Filling-in by joint interpolation of vector fields and gray levels","authors":"Coloma Ballester, Marcelo Bertalmio, Vicent Caselles, Guillermo Sapiro, Joan Verdera","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Barnes et al., 2009a] Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. PatchMatch: A randomized correspondence algorithm for structural im- age editing. ACM Transactions on Graphics (Proc. SIG- GRAPH), 28(3), August 2009.","title":"PatchMatch: A randomized correspondence algorithm for structural image editing","authors":"Barnes","year":"2009","venue":"ACM Transactions on Graphics (Proc. SIG-GRAPH)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Barnes et al., 2009b] Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. Patchmatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics (ToG), 28(3):24, 2009.","title":"Patchmatch: A randomized correspondence algorithm for structural image editing","authors":"Barnes","year":"2009","venue":"ACM Transactions on Graphics (ToG)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Doersch et al., 2012] Carl Doersch, Saurabh Singh, Abhi- nav Gupta, Josef Sivic, and Alexei Efros. What makes paris look like paris? ACM Transactions on Graphics, 31(4), 2012.","title":"What makes paris look like paris?","authors":"Carl Doersch, Saurabh Singh, Abhinav Gupta, Josef Sivic, Alexei Efros","year":"2012","venue":"ACM Transactions on Graphics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Esedoglu and Shen, 2002] Selim Esedoglu and Jianhong Shen. Digital inpainting based on the mumford-shah- euler image model. European Journal of Applied Math- ematics, 13(4):353-370, 2002.","title":"Digital inpainting based on the mumford-shaheuler image model","authors":"Shen Esedoglu, Selim Esedoglu, Jianhong Shen","year":"2002","venue":"European Journal of Applied Mathematics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Iizuka et al., 2017] Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Globally and locally consistent im- age completion. ACM Transactions on Graphics (TOG), 36(4):107, 2017.","title":"Globally and locally consistent image completion","authors":"Satoshi Iizuka, Edgar Simo-Serra, Hiroshi Ishikawa","year":"2017","venue":"ACM Transactions on Graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Johnson et al., 2016] Justin Johnson, Alexandre Alahi, and Fei Fei Li. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision, 2016.","title":"Perceptual losses for real-time style transfer and super-resolution","authors":"Justin Johnson, Alexandre Alahi, Fei Fei, Li","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Karras et al., 2017] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint arX- iv:1710.10196, 2017.","title":"Progressive growing of gans for improved quality, stability, and variation","authors":"Tero Karras, Timo Aila, Samuli Laine, Jaakko Lehtinen","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kwatra et al., 2005] Vivek Kwatra, Irfan Essa, Aaron Bo- bick, and Nipun Kwatra. Texture optimization for example-based synthesis. In ACM Transactions on Graph- ics (ToG), volume 24, pages 795-802. ACM, 2005.","title":"Texture optimization for example-based synthesis","authors":"Vivek Kwatra, Irfan Essa, Aaron Bobick, Nipun Kwatra","year":"2005","venue":"ACM Transactions on Graphics (ToG)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liu et al., 2018] Guilin Liu, Fitsum A Reda, Kevin J Shi- h, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. Image inpainting for irregular holes using partial convolu- tions. arXiv preprint arXiv:1804.07723, 2018.","title":"Image inpainting for irregular holes using partial convolutions","authors":"Liu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Nazeri et al., 2019] Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Qureshi, and Mehran Ebrahimi. Edgeconnect: Gen- erative image inpainting with adversarial edge learning. arXiv preprint arXiv:1901.00212, 2019.","title":"Edgeconnect: Generative image inpainting with adversarial edge learning","authors":"Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Qureshi, Mehran Ebrahimi","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Newson et al., 2014] Alasdair Newson, Andrés Almansa, Matthieu Fradet, Yann Gousseau, and Patrick Pérez. Video inpainting of complex scenes. SIAM Journal on Imaging Sciences, 7(4):1993-2019, 2014.","title":"Video inpainting of complex scenes","authors":"Alasdair Newson, Andrés Almansa, Matthieu Fradet, Yann Gousseau, Patrick Pérez","year":"2014","venue":"SIAM Journal on Imaging Sciences","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Park et al., 2017] Eunbyung Park, Jimei Yang, Ersin Yumer, Duygu Ceylan, and Alexander C Berg. Transformation- grounded image generation network for novel 3d view synthesis. In Proceedings of the ieee conference on com- puter vision and pattern recognition, pages 3500-3509, 2017.","title":"Transformationgrounded image generation network for novel 3d view synthesis","authors":"Park","year":"2017","venue":"Proceedings of the ieee conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pathak et al., 2016] Deepak Pathak, Philipp Krahenbuhl, J- eff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2536-2544, 2016.","title":"Context encoders: Feature learning by inpainting","authors":"Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, Alexei A Efros","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sajjadi et al., 2017] Mehdi S. M. Sajjadi, Bernhard Scholkopf, and Michael Hirsch. Enhancenet: Single image super-resolution through automated texture synthe- sis. In The IEEE International Conference on Computer Vision (ICCV), Oct 2017.","title":"Enhancenet: Single image super-resolution through automated texture synthesis","authors":"S M Mehdi, Bernhard Sajjadi, Michael Scholkopf, Hirsch","year":"2017","venue":"The IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Simakov et al., 2008] Denis Simakov, Yaron Caspi, Eli Shechtman, and Michal Irani. Summarizing visual data using bidirectional similarity. In Computer Vision and Pat- tern Recognition, 2008. CVPR 2008. IEEE Conference on, pages 1-8. IEEE, 2008.","title":"Summarizing visual data using bidirectional similarity","authors":"Denis Simakov, Yaron Caspi, Eli Shechtman, Michal Irani","year":"2008","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Song et al., 2017] Yuhang Song, Chao Yang, Zhe L. Lin, Hao Li, Qin Huang, and C.-C. Jay Kuo. Image inpaint- ing using multi-scale feature image translation. CoRR, ab- s/1711.08590, 2017.","title":"Image inpainting using multi-scale feature image translation","authors":"Yuhang Song, Chao Yang, Zhe L Lin, Hao Li, Qin Huang, C.-C. Jay Kuo","year":"2017","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2018a] Xiaolong Wang, Ross Girshick, Abhi- nav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vi- sion and Pattern Recognition, pages 7794-7803, 2018.","title":"Non-local neural networks","authors":"Wang","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2018b] Yi Wang, Xin Tao, Xiaojuan Qi, Xiaoy- ong Shen, and Jiaya Jia. Image inpainting via generative multi-column convolutional neural networks, 2018.","title":"Image inpainting via generative multi-column convolutional neural networks","authors":"Wang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xu et al., 2018] Qiantong Xu, Huang Gao, Yuan Yang, Chuan Guo, and Kilian Weinberger. An empirical study on evaluation metrics of generative adversarial networks. 2018.","title":"An empirical study on evaluation metrics of generative adversarial networks","authors":"Qiantong Xu, Huang Gao, Yuan Yang, Chuan Guo, Kilian Weinberger","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yang et al., 2017] Chao Yang, Xin Lu, Zhe Lin, Eli Shecht- man, Oliver Wang, and Hao Li. High-resolution image inpainting using multi-scale neural patch synthesis. In The IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), volume 1, page 3, 2017.","title":"High-resolution image inpainting using multi-scale neural patch synthesis","authors":"Chao Yang, Xin Lu, Zhe Lin, Eli Shechtman, Oliver Wang, Hao Li","year":"2017","venue":"The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yu et al., 2018a] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpainting with gated convolution. arXiv preprint arX- iv:1806.03589, 2018.","title":"Free-form image inpainting with gated convolution","authors":"Yu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yu et al., 2018b] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. arXiv preprint, 2018.","title":"Generative image inpainting with contextual attention","authors":"Yu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhou et al., 2018] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6):1452-1464, 2018.","title":"Places: A 10 million image database for scene recognition","authors":"Zhou","year":"2018","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}