Super-Resolution and Inpainting with Degraded and Upgraded Generative Adversarial Networks

1 Introduction

High-Resolution (HR) Magnetic Resonance Imaging (MRI), such as 7-Tesla MRI, has shown benefits in brain imaging due to its high signal-noise-to-ratio, resolution and sensitivity to capturing disease patterns. However, HR MRI is limited to long acquisition times, high costs and frequent unavailability. In addition, the variations in anatomical observations, triggered by a patient's motion, artifacts and corrupted or incomplete scans, deteriorate clinical diagnosis and other postprocessing steps. Furthermore, the scarcity of HR imaging limits the ability to model accurate healthy and disease patterns (e.g. to create of anatomical atlases of health status from clinical images presenting diverse conditions), which call for advanced approaches to handle the aforementioned problems.

∗Corresponding author

Existing approaches [Huang et al., 2017a] seek to solve the problems either by super-resolving an input low-resolution (LR) image or inpainting missing values from the surrounding pixels. Image Super-Resolution (SR) aims at reconstructing an equivalent HR image from its LR counterpart, which is an ill-posed problem caused by a multiplicity of solutions. Image inpainting (also known as image completion) aims at filling missing values of an image, which remains a challenge due to its inherent ambiguity and the complexity of content representation. Despite many remarkable achievements shown in these respective tasks, the existing methods lack versatility in handling multiple degradations, while the practical acquisitions may suffer from several serious damages, e.g., motion blur, compression artifacts, scanner noise, intensity inhomogeneity, region lost, and degradation stemming from the physical imaging (e.g. scattering in CT, Eddy current distortions in fMRI, etc.). In addition, separating the processes of SR and image inpainting requires two learning steps, with the consequence being that the reconstructed error from the first stage will be propagated to the second one, thus leading to an overall larger error.

Early works of SR and image inpainting mostly attempted to interpolate the lost information, relying on an image prior [Huang et al., 2015], e.g., nearest-neighbor interpolator [Parker et al., 1983], non-local self-similarity [Manj´on et al., 2010] or sparse representation [Yang et al., 2010]. By exploiting powerful image priors, Zeyde et al. [Zeyde et al., 2010] introduced a dictionary learning-based method by incorporating a sparse-land local model for the single image scale-up problem. To avoid using external data but leveraging internal feature redundancy for SR purposes, a selfsimilarity driven SR algorithm [Huang et al., 2015] was proposed. Rather than learning the relationship between LR and HR images, internal statistic [Huang et al., 2015] was employed to perform a decomposition of the geometric patch transformation. Schulter et al. [Schulter et al., 2015] presented a fast and accurate image upscaling approach by constructing a locally linear multivariate regression using random forests. Huang et al. [Huang et al., 2017b] proposed a hetero-domain image alignment approach by learning from both paired and unpaired data and achieved remarkable improvements in image SR. Although these methods do not require large-scale training sets to learn an activate expression, such simple and efficient techniques usually result in blurry reconstructions with very limited performance.

More recently, the field has witnessed a variety of improvements, with deep learning [Goodfellow et al., 2014] being employed to gain more ideal performance. Initial efforts involved training convolutional neural networks (CNNs) to predict texture details or inpaint small regions by leveraging learned feature maps. To make the solution space closer to its true manifold, various losses, such as the perceptual loss, have been proposed over feature maps to replace the pixelwise mean square error loss [Ulyanov et al., 2016]. More recently, GANs were introduced by inferring more reliable structures and textures to encourage the nature fidelity between generated and real images [Ledig et al., 2017]. Specifically, SRCNN [Dong et al., 2015] applied a convolutional network architecture to learn an end-to-end mapping relationship between LR and HR images for achieving fast and superior SR performance. Kim et al. [Kim et al., 2016a] introduced a very deep convolutional network (i.e. increasing the depth of a CNN to 20 layers) by employing residual learning and gradient clipping to solve exploding gradients, while obtaining more contextual information from LR images. In [Johnson et al., 2016], a perceptual-driven approach was proposed to enhance the visual quality of SR results. Based on the idea of being closer to reality, a gradient profile prior-guided detail representation [Tai et al., 2010], multi-scale redundant dictionary [Zhang et al., 2012], Markov random field regularized deep CNN [Li and Wand, 2016] were investigated to reconstruct realistic texture details.

SRGAN [Ledig et al., 2017] was then developed to generate images with photo-realistic textures, which gaining huge improvements in perceptual quality.

In parallel, image inpainting is treated as another research branch which focuses on filling missing regions in a damaged image with semantically reasonable content. The early developed methods, e.g., Region Filling [Criminisi et al., 2004] applied exemplar-based texture synthesis to propagate texture and structure information, simultaneously, for filling in the gaps. Borrowing data from surrounding pixels is a simple and flexible way to recover relative missing values; however, it lacks a deeper understanding of images and usually leads to context-unreasonable results. Recently, benefiting from the rapid development of deep learning, visual performance and contextual coherence of image inpainting have been further improved. Context Encoders [Pathak et al., 2016] leverage image analogy to give semantically hole-filled results. Yu et al. [Yu and Koltun, 2015] constructed a detailed convolution to model multiscale contextual information and increase the size of receptive fields. Iizuka et al. [Iizuka et al., 2017] utilized a fully-convolutional neural network with their proposed global and local context discriminators to approximate visually plausible content in the holes. Inspired by the U-Net model, Shift-Net [Yan et al., 2018] was proposed to optimize texture details and make sharper structures by introducing a guidance loss.

Throughout the literature, these advanced methods intuitively focus on the upgrading procedure (i.e., low-to-high or incomplete-to-complete processing) by artificial downsampling to treat the low-quality data. In practice, image degradation can be very complex and even feature multiple issues associated to degradation factors. This raises the challenge of handling unknown nuisance factors. Inevitably, lacking versatility to handle naturally degraded images leads to poor performance during testing. To alleviate this problem and achieve high-quality reconstruction with visual realism, we propose to learn the degradation procedure from real high/low-quality acquisitions first, and then super-resolve and inpaint degraded data in practical scenarios. Our method is shown in Fig. 1.

Our contributions are summarized as follows:

12

• To the best of our knowledge, we are the first to propose a unified framework (termed as HLH-GAN) to perform super-resolution and image inpainting simultaneously on medical images.
• A degradation GAN (H-L GAN) is presented for modeling the practical low-quality images using our H-L loss functions rather than bicubic downsampling.
• An upgrading GAN (L-H GAN) is introduced with three branches: adversarial loss, content loss, and texture loss. L-H GAN is calculated on feature maps of the VGG network for content-coherent and texture-realistic reconstruction.
• We confirm using several quantitative metrics on images from three datasets that HLH-GAN is the state-of-theart for the recovery of high-resolution, texture-clear and content-complete images.

2 Method

The proposed work aims at handling multiple degradations such as LR, noisy, blurry, corrupted and region-lost images. We collectively refer to all these degraded images as lowquality (LQ) data, in contrast to all superior images as HighQuality (HQ) data, and image super-resolution and inpainting tasks as SRI.

2.1 Preliminaries

Typically, GANs involve a generator G and a discriminator D that compete with each other in a zero-sum game to optimize the learning parameters. GANs produce sharp images, albeit mostly in a supervised setting and with somewhat unstable performance. CycleGAN [Zhu et al., 2017] builds upon GANs by modeling both forward and backward mapping functions in a closed loop, and is constrained on a cycle-consistency term to encourage more robust image style transformation.

2.2 ProblemFormulation

The goal of SRI is to estimate two maps F : L → H and G : H → L from a LQ image domain L = {ILi }Mi=1 , ∀i = 1, ..., M to a HQ image domain H = {IHj }Nj=1 , ∀j = 1, ..., N and vice versa, such that the distributions of mapped instances ˆ pl and ˆ ph can match their ground truths pl and ph. Following a cyclic strategy [Wu et al., 2018; Zhu et al., 2017], the cycle-consistency loss is enforced between G and F. The discriminators DG and DF are then defined to distinguish the corresponding generations. In this work, solving SRI can be decomposed into two stages:

1. High→Low (H-L) GAN is constructed to contaminate the HQ data with downsampling, noise and artifacts;
2. Low→High (L-H) GAN is used for recovering the HQ image from the LQ input.

2.3 H-LGAN

Given two sets of unpaired images {ILi }Mi=1 ∈ L and {IHj }Nj=1 ∈ H, we first model the degradation procedure using our H-L GAN, which focuses on simulating the multiple degradations that occur in practical image formation. For each given HQ input image IHj , we feed IHj into the generator by first concatenating it with potential downsampling d, blurring b, noise n, and a rectangular missing region r sampled randomly. The architecture is similar to [Ledig et al., 2017], but the first layer is replaced with the result of the concatenation. Since IH , b, n may have different dimensions, this results in multiple LQ counterparts for each HQ image. Dimension matching is therefore established by vectorizing and projecting the blur kernel onto an n-dimensional space via principal component analysis (PCA) and then stretching it into a real-valued tensor of the same size as the image channel with the noise level. By doing so, the generator G can be trained with d and formulated as M = {b, n}. G is also parameterized by the layer parameters θHL, including weights wHL and biases γHL, denoted as θHL = {wHL, γHL} in a deep network. As with [Ledig et al., 2017], θHL can be solved by a specific loss function LHL to optimize the mapping function G:

where the LQ image ˆIL can be produced by learning G(IH ). Instead of using a typical lp(p = 1, 2) norm to identify the difference, we generalize the task of assessing the quality of LQ data with our H-L GAN. That is, by defining a corresponding discriminator DG to distinguish the generated LQ images from real ones. This leads to LAHL =

Similar to [Liu et al., 2017; Yi et al., 2017; Zhu et al., 2017], LAHL is regarded as a unidirectional adversarial loss and optimized alternatively along with G by feeding the unpaired training data. In addition to being able to identify the generated images, another important property which was explored in the traditional reconstruction task is the preservation of common components between LQ and HQ images. To ensure that the resulting LQ data retains the same content as the HQ versions, we add a high-level content loss:

where φ(·) represents the high-level feature maps obtained from a VGG network. Different from the most related works [Ledig et al., 2017], we adopt an l1 distance to measure the cross-quality content loss using VGG feature maps between the HQ ground truth and the generated LQ counterparts. This is due to the fact that missing regions in LQ data show very different characteristics from the completed HQ data and, thus, the l1-norm is able to handle such changes much better than others. LAHL is then combined with LCHL to formulate the loss function of H-L GAN.

2.4 L-HGAN

The L-H GAN is built upon results obtained from H-L GAN to recover HQ images ˆIH . Specifically, LQ images ˆIL generated by H-L GAN are fed to the generator F of the L-H GAN. The generative network is a deep residual CNN similar to the ones used in [Johnson et al., 2016]. The parameters θLH = {wLH, γLH} are used for estimating ˆIH by learning a mapping function F as ˆIH = F(ˆIL). Specifically, ˆIH can be solved by optimizing F using a multi-upgraded loss function LLH on the training samples:

Different from previous GAN-based SR/inpainting works which only rely on local content matching for adversarial supervision, we propose using a dedicated L-H GAN together with an adversarial loss LALH, a content loss LCLH, and a texture loss LTLH, to enforce the realism and the consistency.

Generally, L-H GAN is based on adversarial learning, LALH is therefore provided to affect the upgrade process in F . The calculation of LALH is in accordance with Eq. (2) by simply changing the direction of the LQ/HQ data transformation, i.e.

where DF is the discriminator corresponding to F. As presented in Eq. (3), the content loss LCLH is also constrained for L-H GAN to capture the global structure of the underlying common components between the reconstructed HQ images and real ones. It is worth noting that the l1-norm is utilized in Eq. (3), which is suitable for images with different characteristics, e.g. for measuring the distance between IH and ˆIL instead of IH and ˆIH . The l2-loss, on the other hand, is widely used to penalize the discrepancy in image appearance between the ground truth and its pseudo-product. We then define

Eq. (6) is applied in the feature maps φ(·) of the VGG19 network. In order to generate highly accurate textures, the texture loss LTLH is used here to guide the generation of fine-scale details. Following [Ulyanov et al., 2016], we adopt a texture descriptor, involving several Gram matrices Tl, in the descriptor VGG network to induce

2.5 Final Objective

Our framework is based on a cyclic learning system, inspired by [He et al., 2016; Huang et al., 2017a; Liu et al., 2017], where the cycle-consistency constraint is added to enforce forward-backward transformation consistency. As with in [Zhu et al., 2017], the cycle-consistency loss Lcyc is defined as

With H-L GAN loss LHL, L-H GAN loss LLH, and cycleconsistency loss Lcyc, the model objective of our HLH-GAN is defined as

where LHL and LLH can be extended as LHL = LAHL + αLCHL,LLH = LALH + δLCLH + βLTLH.

3 Experiments

3.1 Network Architecture

H-L GAN: We designed the generator as an encoder-decoder network with the stretched real-valued tensor as the first layer, and 12 identical residual blocks as the middle layers, followed by a convolutional layer with stride 2, 4 and scale factor of 2, 4, respectively. The stride of last layer is 1, and all other convolutional layers follow the above setting using 3×3×3 kernels with 32 filters. For φ, we utilize the feature maps obtained by the 4-th convolution before the 4-th maxpooling layer within the VGG19 network. We use Adam with 105 iterations and a learning rate of 10-4 , which is decayed by a factor of 2 every 2 × 105 minibatch updates. To train the discriminative network, we follow [Ledig et al., 2017] using six residual blocks without batch normalization, followed by a fully connected layer.

L-H GAN: The generator uses 17 residual blocks distributed as 12-3-2 with skip connections between them. The architecture is generally the same as in [Ledig et al., 2017], but we choose different hidden layers depending on the loss. The content loss is calculated based on the feature maps from the relu5 1 layer, and the texture loss is computed by combining the relu3 1 and relu4 1 layers of VGG19. The discriminator is that of a general GAN.

Weights For the parameters, we set α = 10, δ = 10, β = 0.1, λ = 1.

3.2 Datasets

We evaluate the proposed method on two publicly available datasets: IXI 1 , and HCP 2 , which include real acquired LQ/HQ data. Specifically, the IXI dataset contains 578 healthy subjects acquired by a Philips 3T/1.5T system and a GE 1.5T system. One branch of the HCP dataset has a total of 200 subjects acquired via a Siemens 3T scanner. The HQ images were observed by a IXI-Philips 3T and HCP-Siemens 3T with complete content and no added blur or noise. The LQ images were acquired by IXI-Philips 1.5T/GE 1.5T and HCP-Siemens 3T scanners 3 . In addition, we collected 12 clinical LQ images from the local hospital, which consist of various LR images with unknown degradation and noise. We split the datasets into 500 (IXI) and 120 (HCP) for training, 78 (IXI) and 80 (HCP) for testing.

3.3 Comparison and Results

We compare HLH-GAN against several state-of-the-art approaches, including Bicubic, ScSR [Yang et al., 2010], SelfEx [Huang et al., 2015], Zeyde [Zeyde et al., 2010], DRCN [Kim et al., 2016b], SRCNN [Dong et al., 2015], and SRGAN [Ledig et al., 2017]. We use the default settings of the compared methods to obtain their best superresolved effects. To fully evaluate the effectiveness of our method in different scenarios, we conduct a comprehensive evaluation in three ways: (1) image super-resolution without deblurring and denoising (termed as SR-B-D); (2) image Figure 2: Example results for SR-B-D by HLH-GAN and compared methods.

1 http://brain-development.org/ixi-dataset

2 https://www.humanconnectome.org

3 HCP data were downsampled with destructed content, blur and noise

super-resolution without deblurring (termed as SR-B); (3) simultaneous super-resolution and inpainting (termed as SRI). To demonstrate the effectiveness of HLH-GAN, the quantitative results are provided in Tables 1-3 and visual results are given in Figs. 2-5.

Specifically, Fig. 2 presents a set of results for SR-B-D. As can be seen, all compared methods are able to generate HQ Ground Truth clear results but some tend to have blurry details. DRCN, SRCNN, SRGAN and HLH-GAN can recover clearer and more textural details. Another point worth noting is that, among the four superior methods (i.e., DRCN, SRCNN, SRGAN and HLH-GAN), HLH-GAN obtains the best quality performance and displays the most obvious visual fidelity. We then show the overall PSNR and SSIM indices of all the competing models for the three scenarios, experimented on the IXI dataset. The quantitative results are listed in Table 1. As can be seen, HLH-GAN consistently produces much better results than other approaches, with PSNR and SSIM gains of 1.28±0.6(dB) and 0.04±0.03, respectively. Note that the two special cases constructed i.e., SR-B and SRI, are more challenging than SR-B-D. Visual comparisons are provided in Figs. 3-5. As shown in Fig. 3, the typical Bicubic, ScSR, SelfEx and Zeyde produce HR data but with overly smooth surfaces, while deep generative methods are effective in reconstructing denoised HR images with precise smoothness. We also show the zoomed-in results of different methods in Fig. 4, from which we can see that most compared approaches can reconstruct content with higher resolution but suffer from blurry artifacts. Instead, our method (HLH-GAN) achieves sharper and more realistic details. To further prove the effectiveness of HLH-GAN, we also evaluate the above three cases on the HCP dataset and report all the quantitative results in Table 2. Compared with other methods, HLH-GAN also performs the best on various modality (i.e., T1 and T2) data throughout the experiments.

To validate our framework, we compare the results of simultaneous SR and inpainting (denoted as SRI) with (a) first SR and then inpainting (termed as SR+I) and (b) first inpainting and then SR (called as I+SR). We show their visual results in Fig. 5 and averaged PSNR and SSIM in Table 3. The first column of Fig. 5 demonstrates results generated by HLHGAN for SR-N-B, the second column displays our results for SR-B, while the last yellow block shows the compared performance of SRI, SR+I and I+SR. As can be seen from the figures, HLH-GAN can produce visually pleasant results for various cases, but the performance is still affected by intricate artifacts. Particularly when recovering the holed region, our unified framework produces better inpainting results than independent processing. However, it also generates obvious errors around the borders of completely absent areas. We also analyze how our HLH-GAN contributes to the final performance of SRI, conducting experiments on the T2-w MRI of both the IXI and HCP datasets. We compare our HLH-GAN to the model with separate processes, as shown in Table 3. For both datasets, our method consistently overcomes SR+I and I+SR, which indicates that the unified model improves the effectiveness of both image SR and inpainting, and that the integration is crucial to the success of HLH-GAN.

In summary, our method yields the best results against the compared approaches proving our claim of being able to simultaneously conduct SR and inpainting for better results (with our model performing particularly well on SRI task). Since the HCP dataset includes HQ data, the advantage of HLH-GAN is shown in the high-level details rather than overall appearance (referring to Fig. 2). Our method evaluated on the IXI dataset outperforms the existing state-of-the-art approaches by a large margin.

4 Conclusion

We proposed an HLH-GAN, including two submodules (H-L GAN and L-H GAN), for integrated image SR and inpainting. We showed that our method improves LQ images by first learning a natural image degradation process and then upgrading them. We also demonstrated good results for simultaneous SR and image inpainting using brain images, especially when leveraging the LQ ground truth during training. Our model significantly outperforms various state-ofthe-art models both quantitatively and qualitatively. As a future work, we plan to apply our method to augment HQ data in longitudinal imaging studies.

Acknowledgment

This work was supported in part by the National Natural Science Foundation of China (Grant No. 61972188), Guangdong Provincial Key Laboratory (Grant No. 2020B121201001), the Program for University Key Laboratory of Guangdong Province (Grant No. 2017KSYS008).