{"type":"ScholarlyArticle","title":"MUSICAL: Multi-Scale Image Contextual Attention Learning for Inpainting","authors":[{"name":"Ning Wang","slug":"ning-wang","given":"Ning","family":"Wang","affiliation":"School of Computer Science , Wuhan University"},{"name":"Jingyuan Li","slug":"jingyuan-li","given":"Jingyuan","family":"Li","affiliation":"School of Computer Science , Wuhan University"},{"name":"Lefei Zhang","slug":"lefei-zhang","given":"Lefei","family":"Zhang","affiliation":"School of Computer Science , Wuhan University"},{"name":"Bo Du","slug":"bo-du","given":"Bo","family":"Du","affiliation":"School of Computer Science , Wuhan University"}],"abstract":"We study the task of image inpainting, where an image with missing region is recovered with plausible context. Recent approaches based on deep neural networks have exhibited potential for producing elegant detail and are able to take advantage of background information, which gives texture information about missing region in the image. These methods often perform pixel/patch level replacement on the deep feature maps of missing region and therefore enable the generated content to have similar texture as background region. However, this kind of replacement is a local strategy and often performs poorly when the background information is misleading. To this end, in this study, we propose to use a multi-scale image contextual attention learning (MUSICAL) strategy that helps to flexibly handle richer background information while avoid to misuse of it. However, such strategy may not promising in generating context of reasonable style. To address this issue, both of the style loss and the perceptual loss are introduced into the proposed method to achieve the style consistency of the generated image. Furthermore, we have also noticed that replacing some of the down sampling layers in the baseline network with the stride 1 dilated convolution layers is beneficial for producing sharper and fine-detailed results. Experiments on the Paris Street View, Places, and CelebA datasets indicate the superior performance of our approach compares to the state-of-the-arts.","doi":"10.24963/ijcai.2019/520","url":"https://ijcai.org/proceedings/2019/520","pdf":"https://ijcai.org/proceedings/2019/0520.pdf","markdown":"https://ijcai.org/proceedings/2019/520.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/520","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":520,"pages":{"first":3748,"last":3754},"keywords":["Machine Learning Applications: Applications of Supervised Learning","Machine Learning: Deep Learning","Computer Vision: Computer Vision"],"videos":[],"references":[{"position":1,"raw":"[Akeret et al., 2017] Joel Akeret, Chihway Chang, Aurelien Lucchi, and Alexandre Refregier. Radio frequency in- terference mitigation using deep convolutional neural net- works. Astronomy and Comput., 18:35-39, 2017.","title":"Radio frequency interference mitigation using deep convolutional neural networks","authors":"Akeret","year":"2017","venue":"Astronomy and Comput","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Barnes et al., 2009] Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. Patchmatch: A randomized correspondence algorithm for structural image editing. ACM TOG, 28(3):24, 2009.","title":"Patchmatch: A randomized correspondence algorithm for structural image editing","authors":"Connelly Barnes, Eli Shechtman, Adam Finkelstein, Dan B Goldman","year":"2009","venue":"ACM TOG","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Criminisi et al., 2004] Antonio Criminisi, Patrick Pérez, and Kentaro Toyama. Region filling and object removal by exemplar-based image inpainting. IEEE TIP, 13(9):1200- 1212, 2004.","title":"Region filling and object removal by exemplar-based image inpainting","authors":"Criminisi","year":"2004","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ding et al., 2019] Ding Ding, Sundaresh Ram, and Jef- frey J Rodríguez. Image inpainting using nonlocal texture matching and nonlinear filtering. IEEE TIP, 28(4):1705- 1719, 2019.","title":"Image inpainting using nonlocal texture matching and nonlinear filtering","authors":"Ding Ding, Sundaresh Ram, Jeffrey J Rodríguez","year":"2019","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Doersch et al., 2012] Carl Doersch, Saurabh Singh, Abhi- nav Gupta, Josef Sivic, and Alexei Efros. What makes paris look like paris? ACM TOG, 31(4):101, 2012.","title":"What makes paris look like paris?","authors":"Carl Doersch, Saurabh Singh, Abhinav Gupta, Josef Sivic, Alexei Efros","year":"2012","venue":"ACM TOG","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gao and Grauman, 2017] Ruohan Gao and Kristen Grau- man. On-demand learning for deep image restoration. In Proc. CVPR, pages 1086-1095, 2017.","title":"On-demand learning for deep image restoration","authors":"Grauman ; Gao, Kristen Gao, Grauman","year":"2017","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Guillemot and Le Meur, 2014] Christine Guillemot and Olivier Le Meur. Image inpainting: Overview and recent advances. IEEE SPM, 31(1):127-144, 2014.","title":"Image inpainting: Overview and recent advances","authors":"Le Guillemot, Christine Meur, Olivier Guillemot, Meur Le","year":"2014","venue":"IEEE SPM","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu et al., 2018] Jie Hu, Li Shen, and Gang Sun. Squeeze- and-excitation networks. In Proc. CVPR, pages 7132- 7141, 2018.","title":"Squeezeand-excitation networks","authors":"Jie Hu, Li Shen, Gang Sun","year":"2018","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang et al., 2017] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proc. CVPR, pages 4700- 4708, 2017.","title":"Densely connected convolutional networks","authors":"Gao Huang, Zhuang Liu, Laurens Van Der Maaten, Kilian Q Weinberger","year":"2017","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Iizuka et al., 2017] Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Globally and locally consistent image completion. ACM TOG, 36(4):107, 2017.","title":"Globally and locally consistent image completion","authors":"Satoshi Iizuka, Edgar Simo-Serra, Hiroshi Ishikawa","year":"2017","venue":"ACM TOG","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jaderberg et al., 2015] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. In Proc. NIPS, pages 2017-2025, 2015.","title":"Spatial transformer networks","authors":"Jaderberg","year":"2015","venue":"Proc. NIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Levin et al., 2003] Anat Levin, Assaf Zomet, and Yair Weiss. Learning how to inpaint from global image statis- tics. In Proc. ICCV, pages 305-312, 2003.","title":"Learning how to inpaint from global image statistics","authors":"Levin","year":"2003","venue":"Proc. ICCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liao et al., 2018] Liang Liao, Ruimin Hu, Jing Xiao, and Zhongyuan Wang. Edge-aware context encoder for image inpainting. In Proc. ICASSP, pages 3156-3160, 2018.","title":"Edge-aware context encoder for image inpainting","authors":"Liang Liao, Ruimin Hu, Jing Xiao, Zhongyuan Wang","year":"2018","venue":"Proc. ICASSP","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2015] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proc. ICCV, pages 3730-3738, 2015.","title":"Deep learning face attributes in the wild","authors":"Ziwei Liu, Ping Luo, Xiaogang Wang, Xiaoou Tang","year":"2015","venue":"Proc. ICCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2017] Weiwei Liu, Ivor W. Tsang, and Klaus- Robert Müller. An easy-to-hard learning paradigm for multiple classes and multiple labels. JMLR, 18:94:1- 94:38, 2017.","title":"An easy-to-hard learning paradigm for multiple classes and multiple labels","authors":"Weiwei Liu, Ivor W Tsang, Klaus-Robert Müller","year":"2017","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2019] Weiwei Liu, Donna Xu, Ivor W. Tsang, and Wenjie Zhang. Metric learning for multi-output tasks. IEEE TPAMI, 41(2):408-422, 2019.","title":"Metric learning for multi-output tasks","authors":"Liu","year":"2019","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pathak et al., 2016] Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Con- text encoders: Feature learning by inpainting. In Proc. CVPR, pages 2536-2544, 2016.","title":"Context encoders: Feature learning by inpainting","authors":"Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, Alexei A Efros","year":"2016","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ronneberger et al., 2015] Olaf Ronneberger, Philipp Fis- cher, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Proc. MICCAI, pages 234-241, 2015.","title":"Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation","authors":"Ronneberger","year":"2015","venue":"Proc. MICCAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rudin et al., 1992] Leonid I. Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise re- moval algorithms. Physica D, 60(1-4):259-268, 1992.","title":"Nonlinear total variation based noise removal algorithms","authors":"Leonid I Rudin, Stanley Osher, Emad Fatemi","year":"1992","venue":"Physica D","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Snelgrove, 2017] Xavier Snelgrove. High-resolution multi- scale neural texture synthesis. In Proc. SIGGRAPH Asia Technical Briefs, page 13, 2017.","title":"High-resolution multiscale neural texture synthesis","authors":"Xavier Snelgrove","year":"2017","venue":"Proc. SIGGRAPH Asia Technical Briefs","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sun et al., 2018] Qianru Sun, Liqian Ma, Seong Joon Oh, Luc Van Gool, Bernt Schiele, and Mario Fritz. Natural and effective obfuscation by head inpainting. In Proc. CVPR, pages 5050-5059, 2018.","title":"Natural and effective obfuscation by head inpainting","authors":"Sun","year":"2018","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xiao et al., 2019] Jing Xiao, Liang Liao, Qiegen Liu, and Ruimin Hu. Cisi-net: Explicit latent content inference and imitated style rendering for image inpainting. In Proc. AAAI, pages 1-7, 2019.","title":"Cisi-net: Explicit latent content inference and imitated style rendering for image inpainting","authors":"Jing Xiao, Liang Liao, Qiegen Liu, Ruimin Hu","year":"2019","venue":"Proc. AAAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yan et al., 2018] Zhaoyi Yan, Xiaoming Li, Mu Li, Wang- meng Zuo, and Shiguang Shan. Shift-net: Image inpaint- ing via deep feature rearrangement. In Proc. ECCV, pages 3-19, 2018.","title":"Shift-net: Image inpainting via deep feature rearrangement","authors":"Zhaoyi Yan, Xiaoming Li, Mu Li, Wangmeng Zuo, Shiguang Shan","year":"2018","venue":"Proc. ECCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yang et al., 2017] Chao Yang, Xin Lu, Zhe Lin, Eli Shecht- man, Oliver Wang, and Hao Li. High-resolution image in- painting using multi-scale neural patch synthesis. In Proc. CVPR, pages 6721-6729, 2017.","title":"High-resolution image inpainting using multi-scale neural patch synthesis","authors":"Chao Yang, Xin Lu, Zhe Lin, Eli Shechtman, Oliver Wang, Hao Li","year":"2017","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yeh et al., 2017] Raymond A Yeh, Chen Chen, Teck Yian Lim, Alexander G Schwing, Mark Hasegawa- Johnson, and Minh N Do. Semantic image inpainting with deep generative models. In Proc. CVPR, pages 5485- 5493, 2017.","title":"Semantic image inpainting with deep generative models","authors":"Raymond A Yeh, Chen Chen, Yian Teck, Alexander G Lim, Mark Schwing, Minh N Hasegawa-Johnson, Do","year":"2017","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yu and Koltun, 2015] Fisher Yu and Vladlen Koltun. Multi- scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122, 2015.","title":"Multiscale context aggregation by dilated convolutions","authors":"Koltun ; Fisher Yu, Vladlen Yu, Koltun","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yu et al., 2018] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image in- painting with contextual attention. In Proc. CVPR, pages 5505-5514, 2018.","title":"Generative image inpainting with contextual attention","authors":"Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, Thomas S Huang","year":"2018","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhou et al., 2016] Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei A. Efros. View synthesis by appearance flow. In Proc. ECCV, pages 286-301, 2016.","title":"View synthesis by appearance flow","authors":"Zhou","year":"2016","venue":"Proc. ECCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhou et al., 2018] Bolei Zhou, Àgata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE TPAMI, 40(6):1452-1464, 2018.","title":"Places: A 10 million image database for scene recognition","authors":"Zhou","year":"2018","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}