{"type":"ScholarlyArticle","title":"Super-Resolution and Inpainting with Degraded and Upgraded Generative Adversarial Networks","authors":[{"name":"Yawen Huang","slug":"yawen-huang","given":"Yawen","family":"Huang","affiliation":"Shenzhen Malong Artificial Intelligence Research Center"},{"name":"Feng Zheng","slug":"feng-zheng","given":"Feng","family":"Zheng","affiliation":"Depatment of Computer Science and Technology , Southern University of Science and Technology; Research Institute of Trustworthy Autonomous Systems"},{"name":"Danyang Wang","slug":"danyang-wang","given":"Danyang","family":"Wang","affiliation":"Shenzhen Malong Artificial Intelligence Research Center"},{"name":"Junyu Jiang","slug":"junyu-jiang","given":"Junyu","family":"Jiang","affiliation":"Shenzhen Malong Artificial Intelligence Research Center"},{"name":"Xiaoqian Wang","slug":"xiaoqian-wang","given":"Xiaoqian","family":"Wang","affiliation":"Shenzhen Malong Artificial Intelligence Research Center"},{"name":"Ling Shao","slug":"ling-shao","given":"Ling","family":"Shao","affiliation":"Inception Institute of Artificial Intelligence"}],"abstract":"Image super-resolution (SR) and image inpainting are two topical problems in medical image processing. Existing methods for solving the problems are either tailored to recovering a high-resolution version of the low-resolution image or focus on filling missing values, thus inevitably giving rise to poor performance when the acquisitions suffer from multiple degradations. In this paper, we explore the possibility of super-resolving and inpainting images to handle multiple degradations and therefore improve their usability. We construct a unified and scalable framework to overcome the drawbacks of propagated errors caused by independent learning. We additionally provide improvements over previously proposed super-resolution approaches by modeling image degradation directly from data observations rather than bicubic downsampling. To this end, we propose HLH-GAN, which includes a high-to-low (H-L) GAN together with a low-to-high (L-H) GAN in a cyclic pipeline for solving the medical image degradation problem. Our comparative evaluation demonstrates that the effectiveness of the proposed method on different brain MRI datasets. In addition, our method outperforms many existing super-resolution and inpainting approaches.","doi":"10.24963/ijcai.2020/90","url":"https://ijcai.org/proceedings/2020/90","pdf":"https://ijcai.org/proceedings/2020/0090.pdf","markdown":"https://ijcai.org/proceedings/2020/90.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/90","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":90,"pages":{"first":645,"last":651},"keywords":["Computer Vision: Biomedical Image Understanding","Machine Learning: Deep Learning","Computer Vision: Other"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24295/file_24295.m3u8","length_sec":258,"provider":"hls"}],"references":[{"position":1,"raw":"[Criminisi et al., 2004] Antonio Criminisi, Patrick Pérez, and Kentaro Toyama. Region filling and object removal by exemplar-based image inpainting. IEEE TIP, 13(9):1200- 1212, 2004.","title":"Region filling and object removal by exemplar-based image inpainting","authors":"Criminisi","year":"2004","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dong et al., 2015] Chao Dong, Chen Change Loy, Kaim- ing He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. IEEE TPAMI, 38(2):295- 307, 2015.","title":"Image super-resolution using deep convolutional networks","authors":"Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang","year":"2015","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In NeurIPS, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[He et al., 2016] Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. Dual learn- ing for machine translation. In NeurIPS, pages 820-828, 2016.","title":"Dual learning for machine translation","authors":"Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, Wei-Ying Ma","year":"2016","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Huang et al., 2015] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-exemplars. In IEEE CVPR, pages 5197- 5206, 2015.","title":"Single image super-resolution from transformed self-exemplars","authors":"Jia-Bin Huang, Abhishek Singh, Narendra Ahuja","year":"2015","venue":"IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Huang et al., 2017a] Yawen Huang, Ling Shao, and Alejan- dro F Frangi. Dote: Dual convolutional filter learning for super-resolution and cross-modality synthesis in mri. In MICCAI, pages 89-98. Springer, 2017.","title":"Dote: Dual convolutional filter learning for super-resolution and cross-modality synthesis in mri","authors":"Huang","year":"2017","venue":"MICCAI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Huang et al., 2017b] Yawen Huang, Ling Shao, and Alejan- dro F Frangi. Simultaneous super-resolution and cross- modality synthesis of 3d medical images using weakly- supervised joint convolutional sparse coding. In IEEE CVPR, pages 6070-6079, 2017.","title":"Simultaneous super-resolution and crossmodality synthesis of 3d medical images using weaklysupervised joint convolutional sparse coding","authors":"Huang","year":"2017","venue":"IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Iizuka et al., 2017] Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Globally and locally consistent image completion. ACM ToG, 36(4):107, 2017.","title":"Globally and locally consistent image completion","authors":"Satoshi Iizuka, Edgar Simo-Serra, Hiroshi Ishikawa","year":"2017","venue":"ACM ToG","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Johnson et al., 2016] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, pages 694-711, 2016.","title":"Perceptual losses for real-time style transfer and super-resolution","authors":"Justin Johnson, Alexandre Alahi, Li Fei-Fei","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kim et al., 2016a] Jiwon Kim, Jung Kwon Lee, and Ky- oung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In IEEE CVPR, pages 1646- 1654, 2016.","title":"Accurate image super-resolution using very deep convolutional networks","authors":"Kim","year":"2016","venue":"IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kim et al., 2016b] Jiwon Kim, Jung Kwon Lee, and Ky- oung Mu Lee. Deeply-recursive convolutional network for image super-resolution. In IEEE CVPR, pages 1637-1645, 2016.","title":"Deeply-recursive convolutional network for image super-resolution","authors":"Kim","year":"2016","venue":"IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ledig et al., 2017] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super- resolution using a generative adversarial network. In IEEE CVPR, pages 4681-4690, 2017.","title":"Photo-realistic single image superresolution using a generative adversarial network","authors":"Ledig","year":"2017","venue":"IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li and Wand, 2016] Chuan Li and Michael Wand. Combin- ing markov random fields and convolutional neural net- works for image synthesis. In IEEE CVPR, pages 2479- 2486, 2016.","title":"Combining markov random fields and convolutional neural networks for image synthesis","authors":"Wand ; Li, Chuan Li, Michael Wand","year":"2016","venue":"IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2017] Ming-Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation net- works. In NeurIPS, pages 700-708, 2017.","title":"Unsupervised image-to-image translation networks","authors":"Liu","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Manjón et al., 2010] José V Manjón, Pierrick Coupé, An- tonio Buades, Vladimir Fonov, D Louis Collins, and Montserrat Robles. Non-local mri upsampling. MIA, 14(6):784-792, 2010.","title":"Non-local mri upsampling","authors":"Pierrick José V Manjón, Antonio Coupé, Vladimir Buades, Louis Fonov, Montserrat Collins, Robles","year":"2010","venue":"MIA","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Parker et al., 1983] J Anthony Parker, Robert V Kenyon, and Donald E Troxel. Comparison of interpolating meth- ods for image resampling. IEEE TMI, 2(1):31-39, 1983.","title":"Comparison of interpolating methods for image resampling","authors":"Parker","year":"1983","venue":"IEEE TMI","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pathak et al., 2016] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Con- text encoders: Feature learning by inpainting. In IEEE CVPR, pages 2536-2544, 2016.","title":"Context encoders: Feature learning by inpainting","authors":"Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, Alexei A Efros","year":"2016","venue":"IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Schulter et al., 2015] Samuel Schulter, Christian Leistner, and Horst Bischof. Fast and accurate image upscaling with super-resolution forests. In IEEE CVPR, pages 3791- 3799, 2015.","title":"Fast and accurate image upscaling with super-resolution forests","authors":"Schulter","year":"2015","venue":"IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tai et al., 2010] Yu-Wing Tai, Shuaicheng Liu, Michael S Brown, and Stephen Lin. Super resolution using edge prior and single image detail synthesis. In IEEE CVPR, pages 2400-2407. IEEE, 2010.","title":"Super resolution using edge prior and single image detail synthesis","authors":"Yu-Wing Tai, Shuaicheng Liu, Michael S Brown, Stephen Lin","year":"2010","venue":"IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ulyanov et al., 2016] Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, and Victor S Lempitsky. Texture net- works: Feed-forward synthesis of textures and stylized im- ages. In ICML, volume 1, page 4, 2016.","title":"Texture networks: Feed-forward synthesis of textures and stylized images","authors":"Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, Victor S Lempitsky","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wu et al., 2018] Lin Wu, Yang Wang, and Ling Shao. Cycle-consistent deep generative hashing for cross-modal retrieval. IEEE TIP, 28(4):1602-1612, 2018.","title":"Cycle-consistent deep generative hashing for cross-modal retrieval","authors":"Lin Wu, Yang Wang, Ling Shao","year":"2018","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yan et al., 2018] Zhaoyi Yan, Xiaoming Li, Mu Li, Wang- meng Zuo, and Shiguang Shan. Shift-net: Image inpaint- ing via deep feature rearrangement. In ECCV, pages 1-17, 2018.","title":"Shift-net: Image inpainting via deep feature rearrangement","authors":"Zhaoyi Yan, Xiaoming Li, Mu Li, Wangmeng Zuo, Shiguang Shan","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yang et al., 2010] Jianchao Yang, John Wright, Thomas S Huang, and Yi Ma. Image super-resolution via sparse rep- resentation. IEEE TIP, 19(11):2861-2873, 2010.","title":"Image super-resolution via sparse representation","authors":"Jianchao Yang, John Wright, Thomas S Huang, Yi Ma","year":"2010","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yi et al., 2017] Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image- to-image translation. In IEEE ICCV, pages 2849-2857, 2017.","title":"Dualgan: Unsupervised dual learning for imageto-image translation","authors":"Zili Yi, Hao Zhang, Ping Tan, Minglun Gong","year":"2017","venue":"IEEE ICCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yu and Koltun, 2015] Fisher Yu and Vladlen Koltun. Multi- scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122, 2015.","title":"Multiscale context aggregation by dilated convolutions","authors":"Koltun ; Fisher Yu, Vladlen Yu, Koltun","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zeyde et al., 2010] Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse- representations. In International conference on curves and surfaces, pages 711-730. Springer, 2010.","title":"On single image scale-up using sparserepresentations","authors":"Zeyde","year":"2010","venue":"International conference on curves and surfaces","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2012] Kaibing Zhang, Xinbo Gao, Dacheng Tao, and Xuelong Li. Multi-scale dictionary for single im- age super-resolution. In IEEE CVPR, pages 1114-1121. IEEE, 2012.","title":"Multi-scale dictionary for single image super-resolution","authors":"Zhang","year":"2012","venue":"IEEE CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhu et al., 2017] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image transla- tion using cycle-consistent adversarial networks. In IEEE ICCV, pages 2223-2232, 2017.","title":"Unpaired image-to-image translation using cycle-consistent adversarial networks","authors":"Zhu","year":"2017","venue":"IEEE ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}