Context-Aware Image Inpainting with Learned Semantic Priors

1 Introduction

Image inpainting aims at generating visually plausible image structures and local details for missing regions. This task has been researched for a long time and can be applied to many applications such as image restoration, object removal, and image manipulation [Criminisi et al., 2004]. In summary, the main challenge for restoring realistic and reasonable contents lies in how to resolve the context ambiguities in missing regions. Although some approaches exploit stochastic models to learn the multi-modal distributions of the missing contents [Zheng et al., 2019; Zhao et al., 2020], in this paper, we focus on learning deterministic models that can leverage global context cues to reduce the context ambiguities.

∗Corresponding Authors

† Code available at https://github.com/WendongZh/SPL

In the realm of deterministic image inpainting, previous methods generally focus more on local texture consistency. The early work in this field usually uses the local texture similarities to perform pixel interpolations, such as texture diffusion or patch-based methods [Barnes et al., 2009; Levin et al., 2003]. However, these methods may fail to restore non-repetitive structures. Recently, many learning-based approaches are proposed in form of deep neural networks and directly reformulate the image inpainting as a conditional image generation problem [Yu et al., 2020; Li et al., 2020; Xie et al., 2019; Nazeri et al., 2019; Liu et al., 2020]. Among these methods, some approaches further use particular structural supervisions to enhance the structure completion in final results, such as edge image [Nazeri et al., 2019] and smooth image [Liu et al., 2020]. Although these learning-based methods produce more realistic local details compared with early work, restoring reasonable image contents for more complex scenes is still hard as shown in Figure 1. The main reason lies in that these methods fail to understand the global context but mainly focus on local texture consistency, which results in the distorted global structure.

In contrast to the above literature, we suggest that the highlevel semantics can be considered as common knowledge between specific pretext tasks and image inpainting and that neural networks pre-learned for the pretext tasks can provide strong global relational priors that can be used to infer lowlevel missing structures. To properly model the global relations between the missing contents and the remainder, we take the high-level feature maps of a pretrained multi-label classification (detection) model as the semantic supervisions*. These feature maps contain discriminative pre-learned semantic information of multiple objects which can help the global context understanding. Besides, as the pretext tasks mainly focus on the extractions of high-level features, they improve the robustness of our model to local textures and structural defects. The learned contextual features are referred to as semantic priors, and we visualize an example via the k-means algorithm in Figure 1. Following this line, a novel contextaware image inpainting model is proposed which considers both local texture consistency and global semantic consistency. The whole pipeline of our method can be summarized into two phases. In the first phase, we exploit two different encoders to separately extract the low-level image features and learn highlevel semantic priors. By performing knowledge distillation on networks from pretext tasks † , we obtain the global context representations. In the second phase, the learned semantic priors are spatially injected into image features to help image completion. By learning and leveraging semantic priors, our approach (SPL) not only strengthens the local consistency of the generated image but also facilitates the structural inference of the missing contents from the remainder.

Our contributions to image inpainting can be summarized as follows:

• We show that distilling high-level knowledge from specific pretext tasks can benefit the understanding of global contextual relations and is thus semantically meaningful to low-level image inpainting. It requires no additional human annotations on the image inpainting datasets.
• We propose SPL, a new model for context-aware image inpainting, which adaptively incorporates the learned semantic priors and the local image features in a unified generative network. It is trained with the semantic prior learner in an end-to-end fashion.
• SPL achieves state-of-the-art performance on three standard image inpainting datasets.

2 RelatedWork

Rule-based image inpainting. Early work for image inpainting usually designs particular rules on low-level image features to establish the pixel-level mappings. More specifically, exemplar-based approaches [Criminisi et al., 2004;

* Various pretext tasks can be used to provide semantic supervisions. In the rest of the paper, we mainly use multi-label classification as a typical pretext task without loss of generality.

† We do not train or fine-tune these models directly on the image inpainting datasets, but only use existing models that were pretrained on the Open Images dataset [Kuznetsova et al., 2018].

Barnes et al., 2009] exploit the patch or pixel-level similarities to directly copy textures in visible regions into missing regions. On the other hand, fusion-based methods [Ballester et al., 2001; Levin et al., 2003] try to smoothly propagate proper image contents into missing regions based on neighboring visible pixels. Although these methods produce vivid textures for simple background completion, they usually generate nonrealistic images containing repetitive patterns due to the lack of high-level semantics.

Learning-based image inpainting. Recent learning-based methods are mostly built upon generative adversarial networks [Goodfellow et al., 2014], and formulate the image inpainting problem as conditional image generation. The work of context encoder [Pathak et al., 2016] first utilizes an encoder-decoder architecture and proposes a channel-wise fully connected layer to estimate the missing contents. Subsequently, the global and local discriminators are introduced by [Iizuka et al., 2017] to further enhance the constraints of local consistency. After that, a two-stage coarse-to-fine generation framework is proposed by [Yu et al., 2018] with a context attention module. To exclude the influence of invalid pixels during the feature extraction process, the gated convolution layer [Yu et al., 2019] and the region normalization block [Yu et al., 2020] are proposed successively. Besides, the attention mechanism [Xie et al., 2019] and multi-scale learning framework [Wang et al., 2019; Li et al., 2020] are also introduced. Although these approaches use different network architectures for learning image features, they usually fail to restore reasonable structures in complex scenes due to the unresolved context ambiguities.

Structure-enhanced image inpainting. To resolve the structural distortion of the missing contents, some two-stage models [Nazeri et al., 2019; Ren et al., 2019] first generate edges or smooth images and then use them to enhance the structural details. Other approaches [Liu et al., 2020; Liao et al., 2020] also use edge-preserving images or semantic maps as extra supervisions and require dense human annotations on the image inpainting dataset. In comparison, our model is trained on the image inpainting dataset in a fully unsupervised manner, and only exploits pre-learned features from the pretext tasks, where data annotations are more accessible. Another difference is that, in our model, the semantic priors are learned and spatially injected into the image features in an end-to-end manner, which provides a more adaptive way for the restoration of the global and local structures.

3 Approach

The overall architecture of the proposed approach is shown in Figure 2. We name it SPL to highlight its ability to learn and leverage semantic priors. It contains three main components: an image encoder, a semantic priors learner, and an image decoder. In summary, the whole pipeline can be divided into two phases. In the encoding phase, we learn image features and semantic priors from the remainder of corrupted images. In the generation phase, the learned semantic priors are spatially incorporated into the image features to help the restoration of the global and local structural contents.

3.1 Image Encoding

Given corrupted image Im and corresponding mask M ∈ {0, 1} where 1 represents the locations for invalid pixels and 0 represents those for valid pixels, we first utilize two different encoders EI and ES to learn image feature Fm and semantic priors Sm. As these two features contain different information, we exploit different strategies to design the architectures of these two encoders, and they do not share any parameters.

For image encoder EI shown in Figure 2(a), we aim to extract rich low-level features from visible contents to form the conditional input. These features play important roles in restoring vivid local textures. Therefore we exploit two downsample layers to embed corrupted input into image feature:

where Im∈R3×H×W ,M∈R1×H×W , Fm∈Rc× H4 × W4 . The image feature Fm will be further fed to the generation phase to restore plausible visual details.

3.2 Semantic Priors Learning

For the semantic learner ES shown in Figure 2(b), we aim to learn complete semantic priors for corrupted visual elements with supervisions from pretrained deep neural networks. To understand the global context which usually contains multiple different objects, as shown in Figure 2(d), we exploit the multi-label classification model H pretrained on the Open Image dataset [Kuznetsova et al., 2018] with asymmetric loss (ASL) [Ben-Baruch et al., 2020] to provide supervisions, and we do not fine-tune this model on our image inpainting datasets. For this encoder, the texture details are less important and the encoder needs to pay more attention to understanding the global context in the remainder of the corrupted images. To this end, we make two modifications.

First, as we aim to learn semantic information from higher layers of deep neural networks, we up-sample the input images to preserve more local structures. More specifically, we upsample the full image I ∈ R3×H×W into a large one I ′ ∈ R3×2H×2W , and the corrupted image Im and corresponding mask M are also up-sampled in the same manner to form the input I ′m and M′ for learning semantic priors. After that, we utilize I ′ as the input to the pretrained multi-label classification model H to extract feature maps S∈Rd× H4 × W4 as the supervisions of the semantic priors:

Second, to establish the mappings between visible contents and semantic priors, we exploit three down-sampling layers followed by five residual blocks to form the semantic learner ES. The semantic learner ES utilizes enlarged corrupted image and mask as input to generate the semantic priors Sm:

where I ′m ∈R3×2H×2W , M′ ∈R1×2H×2W , Sm∈Rc× H4 × W4 . We then exploit a 1×1 convolutional layer to adapt Sm to the representations from the pretext task:

where S′m ∈Rd× H4 × W4 . Finally, an ℓ1 reconstruction loss with extra constraints on missing regions is proposed to supervise the semantic learner:

where ⊙ represents Hadamard product operator, α represents the extra weight for missing regions andMs represents resized mask which has the same spatial size as S. With Eq. 4 and Eq. 5, we can distill the knowledge that are useful for image inpainting and filter out the task-agnostic components in S.

3.3 Context-Aware Image Inpainting

In the generation phase, we exploit the image decoder shown in Figure 2(c) to tackle another key problem that how to utilize the learned semantic priors Sm to help the image restoration process. As the image feature Fm and learned semantic priors Sm focus on different aspects of image contents, directly concatenating these features for feature fusion will not only disturbs local textures in visible regions but also affects the learning processes of the corresponding encoders. To this end, we exploit 8 ResBlocks with the spatially-adaptive normalization module (SPADE) [Park et al., 2019] to spatially inject the semantic priors into the decoding phase. In this case, our model provides adaptive modifications on image features and decreases the unnecessary influences. Instead of utilizing images or semantic maps as conditional inputs, in our model, the affine transformation parameters in SPADE are learned from semantic priors and we provide the architecture of this module in Figure 2. In more details, the SPADE modules in our model first normalize the input image feature Fm with non-parametric instance normalization. Then, two different sets of parameters are learned from semantic priors Sm to perform the spatially pixel-wise affine transformations on image feature Fm:

where γ, β,Fm,F ′m ∈ Rc× H4 × W4 and IN represents nonparametric instance normalization. Finally, the output image is generated from the modified image feature F ′m with generator G which contains two upsampling layers:

Novelty of the use of SPADE. Although the SPADE ResBlocks are the core modules of SPL, they are used for different purposes and in different ways from the original SPADE model. Unlike those used for image-to-image translation, here they are jointly trained with two separate encoders and integrate the learned low-level features (instead of random variables) with the distilled high-level priors (instead of the raw layout data). The contribution of SPL is to show that extracting high-level knowledge from specific pretext tasks can be helpful to low-level image inpainting, which has not been thoroughly studied by previous work.

3.4 Loss Function

We exploit the reconstruction and adversarial losses to train the whole model of SPL. For the reconstruction loss, we use ℓ1 loss for the reconstructed image and focus more on the content of missing regions:

where ⊙ represents the Hadamard product operator and δ represents the weight for missing regions. To generate more plausible local details, we also utilize the adversarial loss to train our model, along with a discriminator in an iterative training scheme:Ladv = -Eˆ I [log(1 - D( ˆ I))], (10)

The full objective function of SPL can be written as

We tune the hyper-parameters with grid search, and finally obtain λ1 = 10, λ2 = λ3 = 1, α = 3 and δ = 5.

4 Experiments

4.1 Experimental Setups

Datasets. We evaluate our approach on three datasets, including Places2 [Zhou et al., 2017], CelebA [Liu et al., 2015], and Paris StreetView [Doersch et al., 2015]. We use the standard training set of Places2 with over 4 million images and evaluate all models on its validation set with 36,500 images. The CelebA human face dataset contains over 160,000 training images and about 19,900 testing images. The StreetView dataset contains 15,900 training samples and 100 testing samples. We use the mask set of [Liu et al., 2018] with 12,000 irregular masks that are pre-grouped into six intervals according to the size of the masks (0%-10%, 10%-20%, . . ., 50%-60%). All images in Places2 and StreetView are resized to 256 × 256. As for CelebA, we first apply center cropping and then resize the images to 256 × 256, which is a common practice of previous work [Li et al., 2020; Yu et al., 2020].

Compared methods. We compare our method (SPL) with state-of-the-art methods including the region normalization model (RN) [Yu et al., 2020], the recurrent feature reasoning model (RFR) [Li et al., 2020] and the mutual feature equalization mode (MFE) [Liu et al., 2020]. Among these methods, RN and RFR mainly focus on the consistency of local textures, and MFE exploits smooth images as extra supervisions to enhance structural completion. For methods that provide pretrained models, we directly evaluate them in our settings. Otherwise, we retrain the models based on the provided code.

Implementation and training details. To provide supervisions for semantic priors, we use multi-label classification as the pretext task, in which the model was pretrained on the Open Image dataset [Kuznetsova et al., 2018] with the asymmetric loss (ASL) [Ben-Baruch et al., 2020]. We do not finetune the model on any image inpainting datasets. We adopt the patch-based discriminator from previous work [Yu et al., 2020; Nazeri et al., 2019]. We apply the Adam solver with β1 = 0.0 and β2 = 0.9 for model optimization. The initial learning rate is 1e-4 for all experiments and decayed to 1e-5 at different epochs for different datasets. For Places2 and CelebA, we decay the learning rate at 30 epochs and further fine-tune the model for another 10 epochs. For Paris StreetView, we decay the learning rate at 50 epochs and fine-tune the model for 20 epochs. The batch size is set to 8 for all datasets and we train our model with two V100 GPUs.

4.2 Qualitative Comparisons

We provide qualitative results on three different datasets in Figure 3. The Places2 dataset is the most challenging dataset as it contains different complex scenes and various objects.

For this dataset, we can observe from the first four columns that results from our methods contain more complete global and local structures. On the contrary, results from other methods contain obvious distorted structures and suffer from blur effects. These results suggest that our method successfully reduces the context ambiguities in missing regions and obtains more complete visual concepts for foreground objects.

For CelebA dataset, results are shown in the fifth column. In this case, all methods produce complete human faces as this dataset only contains various aligned human faces. However, as shown in the fifth column, previous approaches generate meaningless textures in local details. In other words, these methods fail to globally understand the context information and pay more attention to fill the missing regions with textures based on local distributions. Compared with them, our results contain more clear local details, which suggests that the learned semantic priors can also help local texture restoration. Similar observations can also be found in Paris dataset.

We further provide detection results of the output images in the last column in Figure 3. From these results, we can observe that our result contains more complete objects with higher detection confidences than other methods.

4.3 Quantitative Comparisons

We further provide the quantitative comparison results in Table 1. To quantitatively evaluate our performance, we select peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and mean ℓ1 error (MAE) as our evaluation metrics which are commonly used in previous work. The evaluation process is as follows. We first randomly sample three different mask sets from the whole irregular mask dataset. These mask sets are assigned to different image datasets to form the mask-image pairs. For each image dataset, the mask-image mappings are held for different methods to obtain fair comparisons. The results of the whole testing sets are shown in columns with mask ratio being "All", and we gather those images with masks in corresponding mask ratios to calculate other results in Table 1. We can see that our method obtain obvious improvements on all three metrics, which further demonstrates the effectiveness of our method.

4.4 Ablation Studies

Alternatives of the semantic supervisions. We then conduct two different experiments to evaluate the effects of the proposed semantic supervisions. First, we directly remove Table 1: Quantitative comparisons of the generated images by our approach (SPL) and the state of the art on three image inpainting datasets.

0.872

0.87

w/o S

Concat

0.0158

0.0156

0.0154

Ours w/ Yolov5 Ours

Figure 4: Comparisons on the StreetView dataset of (1) different

ways to integrate the learned semantic priors into the image decoder,

and (2) using different semantic supervisions (vs. w/ YoloV5).

the semantic supervisions from our model, named as "w/o S". Then, we replace the pretrained multi-label classification network with a pretrained detection network YoloV5 [Jocher and others, 2020] to provide semantic supervisions. In this case, we aim to explore whether other pretext models can also provide sufficient information for global context understanding. From Figure 4, we can obtain two observations: 1) The performance of our method will suffer from significant degeneration without explicit supervision from pretrained model. 2) Supervisions from the detection model can also provide helpful semantic information for global context understanding.

Alternatives of SPADE in SPL. Instead of using the SPADE modules, in this experiment, we directly concatenate the learned semantic priors and the features from the image

28.8

28.7

encoder to form the input of ResBlocks, which is commonly used by previous tow-stages generation frameworks [Nazeri et al., 2019; Ren et al., 2019]. Results in Figure 4 show that the straightforward operation of concatenation leads to decreases in all three evaluation metrics.

4.5 Visualization of Learned Semantic Priors

We further visualize the learned semantic priors and compare the generation results with the edge pre-generation method (EC) [Nazeri et al., 2019] in Figure 5. Specifically, we directly conduct the k-means algorithm on the learned semantic priors and use different colors to represent different clusters. From this figure, we can see that the EC model cannot restore reasonable structures for complex scenes. Besides, the distorted structures in edge images will further mislead the final results. Instead, the semantic priors learned by our method reduce the context ambiguities and provide explicit guidance for both global and local structure restoration.

5 Conclusion

In this paper, we propose a new solution to image inpainting named SPL. Instead of focusing on the consistency of local textures, we show that the high-level features from deep neural networks, which were well-trained under specific pretext tasks (e.g., multi-label classification and object detection), contain rich semantic information and can benefit global context understanding. We transfer the high-level knowledge from the pretext models to low-level feature space and use the learned priors as the structural guidance of image inpainting. Finally, we propose the context-aware image inpainting model that adaptively incorporates global semantics with local textures in a unified image decoding module. SPL significantly outperforms the state-of-the-art approaches on three datasets.

Acknowledgments

This work was supported by NSFC (U19B2035, U20B2072, 61976137). It was also supported by the Shanghai Municipal Science & Technology Major Project (2021SHZDZX0102).