{"type":"ScholarlyArticle","title":"Generative Image Inpainting with Submanifold Alignment","authors":[{"name":"Ang Li","slug":"ang-li","given":"Ang","family":"Li","affiliation":"The University of Melbourne"},{"name":"Jianzhong Qi","slug":"jianzhong-qi","given":"Jianzhong","family":"Qi","affiliation":"The University of Melbourne"},{"name":"Rui Zhang","slug":"rui-zhang","given":"Rui","family":"Zhang","affiliation":"The University of Melbourne"},{"name":"Xingjun Ma","slug":"xingjun-ma","given":"Xingjun","family":"Ma","affiliation":"The University of Melbourne"},{"name":"Kotagiri Ramamohanarao","slug":"kotagiri-ramamohanarao","given":"Kotagiri","family":"Ramamohanarao","affiliation":"The University of Melbourne"}],"abstract":"Image inpainting aims at restoring missing regions of corrupted images, which has many applications such as image restoration and object removal. However, current GAN-based generative inpainting models do not explicitly exploit the structural or textural consistency between restored contents and their surrounding contexts. To address this limitation, we propose to enforce the alignment (or closeness) between the local data submanifolds (subspaces) around restored images and those around the original (uncorrupted) images during the learning process of GAN-based inpainting models. We exploit Local Intrinsic Dimensionality (LID) to measure, in deep feature space, the alignment between data submanifolds learned by a GAN model and those of the original data, from a perspective of both images (denoted as iLID) and local patches (denoted as pLID) of images. We then apply iLID and pLID as regularizations for GAN-based inpainting models to encourage two different levels of submanifold alignments: 1) an image-level alignment to improve structural consistency, and 2) a patch-level alignment to improve textural details. Experimental results on four benchmark datasets show that our proposed model can generate more accurate results than state-of-the-art models.","doi":"10.24963/ijcai.2019/114","url":"https://ijcai.org/proceedings/2019/114","pdf":"https://ijcai.org/proceedings/2019/0114.pdf","markdown":"https://ijcai.org/proceedings/2019/114.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/114","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Computer Vision","track":"Main track","number":114,"pages":{"first":811,"last":817},"keywords":["Computer Vision: Computer Vision","Machine Learning Applications: Applications of Supervised Learning"],"videos":[],"references":[{"position":1,"raw":"[Amsaleg et al., 2015] Laurent Amsaleg, Oussama Chelly, Teddy Furon, Stéphane Girard, Michael E. Houle, Ken- ichi Kawarabayashi, and Michael Nett. Estimating local intrinsic dimensionality. In SIGKDD, 2015.","title":"Estimating local intrinsic dimensionality","authors":"Laurent Amsaleg, Oussama Chelly, Teddy Furon, Stéphane Girard, Michael E Houle, Kenichi Kawarabayashi, Michael Nett","year":"2015","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Amsaleg et al., 2018] Laurent Amsaleg, Oussama Chelly, Teddy Furon, Stéphane Girard, Michael E. Houle, Ken Ichi Kawarabayashi, and Michael Nett. Extreme- value-theoretic estimation of local intrinsic dimensional- ity. Data Mining & Knowledge Discovery, 2018.","title":"Extremevalue-theoretic estimation of local intrinsic dimensionality","authors":"Laurent Amsaleg, Oussama Chelly, Teddy Furon, Stéphane Girard, Michael E Houle, Ken Ichi Kawarabayashi, Michael Nett","year":"2018","venue":"Data Mining & Knowledge Discovery","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Barnes et al., 2009] Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. Patchmatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics, 2009.","title":"Patchmatch: A randomized correspondence algorithm for structural image editing","authors":"Connelly Barnes, Eli Shechtman, Adam Finkelstein, Dan B Goldman","year":"2009","venue":"ACM Transactions on Graphics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Barua et al., 2019] Sukarna Barua, Xingjun Ma, Sarah Monazam Erfani, Michael E Houle, and James Bai- ley. Quality evaluation of gans using cross local intrinsic dimensionality. arXiv preprint arXiv:1905.00643, 2019.","title":"Quality evaluation of gans using cross local intrinsic dimensionality","authors":"Barua","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bertalmio et al., 2003] Marcelo Bertalmio, Luminita Vese, Guillermo Sapiro, and Stanley Osher. Simultaneous struc- ture and texture image inpainting. IEEE Transactions on Image Processing, 2003.","title":"Simultaneous structure and texture image inpainting","authors":"Marcelo Bertalmio, Luminita Vese, Guillermo Sapiro, Stanley Osher","year":"2003","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Doersch et al., 2012] Carl Doersch, Saurabh Singh, Abhi- nav Gupta, Josef Sivic, and Alexei Efros. What makes paris look like paris? ACM Transactions on Graphics, 2012.","title":"What makes paris look like paris?","authors":"Carl Doersch, Saurabh Singh, Abhinav Gupta, Josef Sivic, Alexei Efros","year":"2012","venue":"ACM Transactions on Graphics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Efros and Freeman, 2001] Alexei A Efros and William T Freeman. Image quilting for texture synthesis and transfer. In SIGGRAPH, 2001.","title":"Image quilting for texture synthesis and transfer","authors":"Alexei A Efros, William T Freeman","year":"2001","venue":"SIGGRAPH","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Efros and Leung, 1999] Alexei A Efros and Thomas K Le- ung. Texture synthesis by non-parametric sampling. In ICCV, 1999.","title":"Texture synthesis by non-parametric sampling","authors":"Efros, Leung ; Efros, Thomas K Leung","year":"1999","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Goodfellow et al., 2014] Ian J. Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In NIPS, 2014.","title":"Generative adversarial nets","authors":"Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gulrajani et al., 2017] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of wasserstein gans. In NIPS, 2017.","title":"Improved training of wasserstein gans","authors":"Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Aaron Vincent Dumoulin, Courville","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Houle, 2017] Michael E. Houle. Local intrinsic dimension- ality i: an extreme-value-theoretic foundation for similar- ity applications. In SISAP, 2017.","title":"Local intrinsic dimensionality i: an extreme-value-theoretic foundation for similarity applications","authors":"E Michael, Houle","year":"2017","venue":"SISAP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Iizuka et al., 2017] Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Globally and locally consistent image completion. ACM Transactions on Graphics (TOG), 2017.","title":"Globally and locally consistent image completion","authors":"Satoshi Iizuka, Edgar Simo-Serra, Hiroshi Ishikawa","year":"2017","venue":"ACM Transactions on Graphics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Levina and Bickel, 2005] Elizaveta Levina and Peter J. Bickel. Maximum likelihood estimation of intrinsic di- mension. In NIPS, 2005.","title":"Maximum likelihood estimation of intrinsic dimension","authors":"Elizaveta Levina, Peter J Bickel","year":"2005","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2015] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, 2015.","title":"Deep learning face attributes in the wild","authors":"Ziwei Liu, Ping Luo, Xiaogang Wang, Xiaoou Tang","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2018] Guilin Liu, Fitsum A Reda, Kevin J Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. Im- age inpainting for irregular holes using partial convolu- tions. In ECCV, 2018.","title":"Image inpainting for irregular holes using partial convolutions","authors":"Liu","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ma et al., 2018a] Xingjun Ma, Li Bo, Yisen Wang, Sarah M. Erfani, and James Bailey. Characterizing ad- versarial subspaces using local intrinsic dimensionality. In ICLR, 2018.","title":"Characterizing adversarial subspaces using local intrinsic dimensionality","authors":"Ma","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ma et al., 2018b] Xingjun Ma, Yisen Wang, Michael E. Houle, Shuo Zhou, Sarah M. Erfani, Shu-Tao Xia, Su- danthi Wijewickrema, and James Bailey. Dimensionality- driven learning with noisy labels. ICML, 2018.","title":"Dimensionalitydriven learning with noisy labels","authors":"Ma","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pathak et al., 2016] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Con- text encoders: Feature learning by inpainting. In CVPR, 2016.","title":"Context encoders: Feature learning by inpainting","authors":"Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, Alexei A Efros","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Radford et al., 2016] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. ICLR, 2016.","title":"Unsupervised representation learning with deep convolutional generative adversarial networks","authors":"Radford","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recogni- tion challenge. International Journal of Computer Vision, 2015.","title":"Imagenet large scale visual recognition challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein","year":"2015","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. Computer Science, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":"Computer Science","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Song et al., 2018] Yuhang Song, Chao Yang, Zhe Lin, Xi- aofeng Liu, Qin Huang, Hao Li, and C-C Jay Kuo. Contextual-based image inpainting: Infer, match, and translate. In ECCV, 2018.","title":"Contextual-based image inpainting: Infer, match, and translate","authors":"Yuhang Song, Chao Yang, Zhe Lin, Xiaofeng Liu, Qin Huang, Hao Li, C-C Jay Kuo","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2018] Yi Wang, Xin Tao, Xiaojuan Qi, Xiaoy- ong Shen, and Jiaya Jia. Image inpainting via generative multi-column convolutional neural networks. In NeurIPS, 2018.","title":"Image inpainting via generative multi-column convolutional neural networks","authors":"Yi Wang, Xin Tao, Xiaojuan Qi, Xiaoyong Shen, Jiaya Jia","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yang et al., 2017] Chao Yang, Xin Lu, Zhe Lin, Eli Shecht- man, Oliver Wang, and Hao Li. High-resolution image inpainting using multi-scale neural patch synthesis. In CVPR, 2017.","title":"High-resolution image inpainting using multi-scale neural patch synthesis","authors":"Chao Yang, Xin Lu, Zhe Lin, Eli Shechtman, Oliver Wang, Hao Li","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yeh et al., 2017] Raymond A Yeh, Chen Chen, Teck Yian Lim, Alexander G Schwing, Mark Hasegawa-Johnson, and Minh N Do. Semantic image inpainting with deep generative models. In CVPR, 2017.","title":"Semantic image inpainting with deep generative models","authors":"Raymond A Yeh, Chen Chen, Yian Teck, Alexander G Lim, Mark Schwing, Minh N Hasegawa-Johnson, Do","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yu et al., 2018] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In CVPR, 2018.","title":"Generative image inpainting with contextual attention","authors":"Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, Thomas S Huang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2018] Haoran Zhang, Zhenzhen Hu, Changzhi Luo, Wangmeng Zuo, and Meng Wang. Semantic im- age inpainting with progressive generative networks. In ACMMM, 2018.","title":"Semantic image inpainting with progressive generative networks","authors":"Haoran Zhang, Zhenzhen Hu, Changzhi Luo, Wangmeng Zuo, Meng Wang","year":"2018","venue":"ACMMM","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhou et al., 2017] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 2017.","title":"Places: A 10 million image database for scene recognition","authors":"Zhou","year":"2017","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}