JPEG Artifacts Removal via Compression Quality Ranker-Guided Networks

1 Introduction

Lossy image compression methods, e.g. JPEG and WebP, have been widely used in Internet information transmission to save bandwidth, reduce storage requirements, and increase upload and download speeds. However, due to the information loss, complex artifacts are inevitably introduced into compressed images, such as blocking artifacts and ringing effects [Dong et al., 2015]. These artifacts not only decrease the visual quality, but also affect the performance of subsequent computer vision systems. Therefore, exploring methods to effectively reduce compression artifacts is urgently needed. Among various compression methods, JPEG compression based on the discrete cosine transform (DCT) is the most commonly used algorithm. The image firstly is divided into blocks with size of 8 × 8, and then DCT is applied on each block. Finally, the DCT coefficients of each block are quantized and rounded. Some high-frequency components that contain low energy are discarded to reduce the bit rate. Since JPEG compression applies DCT on each block, the correlation between adjacent blocks is ignored, which introduces blocking artifacts. Meanwhile, blurring and ringing effects occur due to the quantization and rounding operations.

∗Co-first authors contributed equally

†Corresponding author

To cope with JPEG compression artifacts, many methods have been proposed. In general, the existing methods can be divided into model-based methods and learning-based methods. Previous model-based methods are mostly based on filter design and can only solve certain types of artifacts. For example, Pointwise SA-DCT [Foi et al., 2007] removes blocking artifacts at the cost of introducing blurring effects. In recent years, due to the powerful representation ability, deep learning-based methods have been explored to directly learn a nonlinear mapping from JPEG images to its clean version. For example, AR-CNN [Dong et al., 2015] proposes a relatively shallow network to learn the mapping function. Inspired by deep residual learning [He et al., 2016], DnCNN [Zhang et al., 2017] proposes very deep networks for general image restorations.

Despite the impressive progress of deep learning, there are still some problems in this area. Firstly, most learningbased methods use only pixel-level loss functions, e.g., mean squared error (MSE), to optimize the model. The JPEG compression factor, which reflects the degree of degradation, is ignored. Secondly, existing methods directly use pre-trained VGG [Johnson et al., 2016] or construct adversarial learning [Goodfellow et al., 2014] to improve perceptual quality. However, these strategies are not designed for the specific JPEG artifacts reduction, which leads to a poor trade-off between distortion and perception. Third, most deep learning methods require different models to solve different compression qualities, which limits their practical values.

To address the above problems, we propose compression quality ranker-guided networks for JPEG artifacts reduction. The motivation of our method is that the JPEG compression factor implies the compressed image quality, which should be fully utilized for de-blocking. To address the nondifferentiability of directly using compression quality factor, we build a deep ranker to reflect different JPEG compression qualities. Based on this differentiable ranker, we design one quality-related loss and one feature matching loss to guide the optimization of both de-blocking and perceptual quality. By combining the above two loss functions with mean absolute error, the training process can be guided from both the semantic level and pixel level. To enable our single model to handle multiple JPEG qualities, we utilize the dilated convolutions to extract multi-scale local features. These task-specific designs allow our method to generate better de-blocking results.

In summary, the contributions of this paper are four-fold:

• We propose a general framework for JPEG artifacts removal by taking compression factors into consideration. The different compression degrees, which are ignored in existing methods, are fully utilized to help design algorithms for this specific de-blocking problem.
• We design a deep ranker to address the nondifferentiability of compression factors. Based on this ranker, we propose two loss functions to guide the deblocking network training from both pixel level and sematic level, which leads to better de-blocking results.
• We incorporate the dilated convolutions into the residual attention block [Zhang et al., 2019b]. In this way, multi-scale local spatial features can be fully explored to generate rich representations, which makes it possible to handle multiple JPEG qualities using a single model.
• Experiments demonstrate that our proposed model reaches comparable or superior performance compared with state-of-the-art approaches both in qualitative and quantitative measurements.

2 RelatedWork

JPEG artifacts removal. Traditional JPEG artifacts removal methods pay attention to filter design, e.g., Pointwise SA-DCT [Foi et al., 2007] proposes the shape-adaptive DCT-based filter for image denoising and de-blocking. The method [Chang et al., 2013] utilizes sparse coding to restore compressed images. Others treat JPEG artifacts removal as an ill-posed inverse problem and solve it by using regression trees [Jancsary et al., 2012] and non-local self-similarity property [Li et al., 2017]. Due to the powerful representation ability, deep convolutional neural networks (CNNs) are utilized and explored to solve image regression problems [Liu et al., 2018]. For example, the first deep CNNsbased method [Dong et al., 2015] designs a 4-layer fully connected CNNs to learn the mapping function. DeepSD [Vandal et al., 2018] obtains local details information by stacking CNN structures. HRFR [Zhang et al., 2018] restores images at the feature level. Inspired by the excellent performance of deep residual learning [He et al., 2016], very deep network architectures are introduced for image restoration [Zhang et al., 2017]. Inspired by the notable progress of generative adversarial networks (GANs) [Goodfellow et al., 2014], some methods [Guo and Chao, 2017; Zhu et al., 2018; Wang et al., 2019] are proposed to improve the visual quality by photo-realistic details generation. D3 [Wang et al., 2016] builds a cascaded network in dual-domain to obtain speed and de-blocking performance gains. In [Chen and Pock, 2017], the authors propose a trainable nonlinear reaction diffusion model for image restoration. Recently, the method [Fan et al., 2018] proposes a decoupled learning framework to combine different parameterized operators. DCSC [Fu et al., 2019] introduces a more compact and explainable deep sparse coding architecture to generate high-quality de-blocking results. RNAN [Zhang et al., 2019b] designs residual non-local attention networks for general image restoration, such as image super-resolution and JPEG artifacts removal.

Ranker for image processing. Image quality ranker has proven useful for image processing. RankCGAN [Saquil et al., 2018] designs a Siamese-network-based ranker to handle continuous attribute values for the task of image generation with semantic attributes. RankSRGAN [Zhang et al., 2019a] proposes a rank approach to generate the relative natural image quality evaluator (NIQE) value for image superresolution. However, these methods design image quality ranker to generate subjective visually pleasing results. While our method aims to design a differentiable ranker related to various JPEG compression factors. This ranker, which fits this specific JPEG artifacts removal problem, can better guide the subsequent de-blocking network training.

3 Methodology

Our proposed framework consists of two components: compression quality ranker (CQR) and de-blocking network, whose architectures are depicted in Figures 1 and 2, respectively. Our model is trained in two steps: first, to utilize non-differentiable compression quality information, we train a differentiable compression-quality-based ranker network to predict scores related to compression quality factors. The CQR is a Siamese network with pair-wise inputs, which have same content but different compression degrees, shown in Figure 1. Second, based on the previous CQR, we design the specific quality-related loss and feature matching loss to train the de-blocking network for JPEG artifacts removal. The deblocking network training strategy is shown in Figure 2. Below we will detail the architecture, training strategy and loss functions.

3.1 Compression Quality Ranker

Network architecture. The architecture of the CQR is depicted in Figure 1. Inspired by the methods [Saquil et al., 2018; Zhang et al., 2019a], we use a Siamese network to predict the relative scores of compression quality factors. Specifically, this network contains two same branches that share weights and each branch is composed of convolutional layers, batch normalization, Leaky ReLU, global average pooling and fully connected layers. To update parameters, we send the output scores to the margin loss module to learn the compression-quality ranking order of the pair-wise input images, whose compression quality factors are different. The preparation of the pair-wise input images and the optimization loss function of CQR are described below.

Compression quality paired images. First, the clean image is compressed by setting different JPEG-compression quality factors (QF), which range from a to b. Note that a lower QF indicates a higher compression degree, which results in poorer image quality and more obvious artifacts.

Then, we divide the range of QF into n levels, and their corresponding labels are set to 1, 2, · · · , n. For example, the image with [a, a + (b - a)/n] range of QF is labeled as "1". Obviously, a higher label reflects a better image quality. During the training procedure, we randomly select two compressed images which have same content but different labels as the pair-wise input images to optimize the ranker.

Margin ranking loss. After obtaining the paired input images, the CQR predicts their two scores respectively:

where si and sj represent the output scores of image xi and image xj , respectively. To train the ranker CQR(·), we adopt margin ranking loss as the optimization loss function, which is commonly used in ranking problems [Zhang et al., 2019a] and can be expressed as:

where bi and bj represent the labels of image xi and image xj, respectively. Obviously, if the relationship between si and sj is consistent with the relationship between bi and bj, the loss equals 0. Based on the well-trained CQR, a higher output score indicates a higher image quality.

3.2 De-blocking Network

Network architecture. The architecture of the de-blocking network is shown in the purple box in Figure 2. As can be seen, the network consists of two basic blocks, i.e., dilated non-local block and dilated local block. Inspired by RNAN [Zhang et al., 2019b], we directly quote its non-local mask module to extract global context information. The difference between these two blocks is that the non-local operation is utilized in dilated non-local blocks to generate the mask map. Therefore, we mainly introduce the dilated nonlocal block. It consists of dilated convolutions and residual blocks with the skip connection operation. The residual block is composed of convolutional layer and ReLU. Then we describe the multi-scale dilated convolutions. RNAN [Zhang et al., 2019b] and other methods [Zhang et al., 2017; Fan et al., 2018] mostly need to train individual models for different compression quality factors, which limits their practical values. To handle this obstacle, we introduce dilated convolutions to extract multi-scale features. As shown in Figure 2, different dilated factors (DF) can expand the filter to different scales and increase receptive fields. By using concatenation operation, we can obtain multi-scale representations, which allows a single model to achieve the ability to solve the multiple compression quality issue. Additionally, the dilated convolutions bring no extra parameters.

Training strategy. The training strategy of the de-blocking network is depicted in Figure 2.As mentioned above, the output score of well-trained CQR can effectively represent the compression quality. Therefore, we use the scores to guide the training of de-blocking network. Firstly, the compressed image is processed by de-blocking network to generate the de-blocked image. This de-blocked image and the corresponding ground truth are used to formulate mean absolute error, which represents pixel-level information. Secondly, the de-blocked image and ground truth are sent into well-trained CQR. The CQR output score of de-blocked image formulates quality-related loss, which measures the quality degree of deblocked image. Additionally, we introduce feature matching loss to measure the feature-level similarity between deblocked image and ground truth. Due to the three loss functions and dilated convolutions, the network can be efficiently optimized both in quantitative and qualitative measurements, and the single model can handle multiple compression qualities. The specific loss functions are described below.

3.3 Loss Functions

MAE loss. In this paper, we usemean absolute error to constrain the pixel-level similarity. MAE loss is expressed as:

where x and y represent the compressed image and ground truth, respectively. M (·) represents the mapping function of de-blocking network.

Quality-related loss. As mentioned above, the larger score output by the well-trained CQR indicates a better image quality. Therefore, we can utilize CQR to measure the quality of the de-blocked image. Since the proposed ranker can be seen as a classification process, which is the most fundamental semantic-related vision task [Liu et al., 2019], it is able to provide semantic-level information to guide the de-blocking network training. This loss can be presented as:

where CQR(M(x)) is the score of the de-blocked image M(x). To formulate a loss function, we take the negative value of the score, and perform a sigmoid operation to make the loss range from 0 to 1.

Featurematching loss. Perceptual loss is widely applied in various image restoration tasks using a pre-trained VGG network [Simonyan and Zisserman, 2014]. However, the VGG model is not trained to remove JPEG artifacts. Therefore, it cannot provide the JPEG artifacts-related features well. Unlike the traditional perceptual loss, our feature matching loss is formulated by several feature maps extracted from CQR, which can more accurately reflect JPEG artifacts information. Feature matching loss can be presented as:

where CQR(i) (·) represents the feature map extracted from the ith layer of CQR, and Ni is the element number.

Overall loss. The above three loss functions constitute the overall loss to train the de-blocking network:

where α and β are used to balance different loss functions.

4 Experiments

4.1 Experimental Datasets and Settings

Datasets. In our experiments, we use four datasets: DIV2K [Agustsson and Timofte, 2017], BSDS500 [Arbelaez et al., 2010], LIVE1 [Sheikh, 2005] and Classic5 [Zeyde et al., 2010]. The DIV2K dataset (800 images) is used for training, while the other three datasets are used for testing. As mentioned above, our model is trained in a stage-wise fashion: first CQR is trained and frozen, and then the de-blocking network is optimized. Note that the whole procedure is only implemented on the Y channel image of YCrCb space. The specific settings of training and testing are described below.

Training settings of CQR. To obtain pair-wise input images for CQR, we use the standard JPEG compression scheme to obtain compressed images with different JPEG quality factors. The factors range from 1 to 30 and we divide the range into 3 levels to label compressed images. Specifically, the images within range [1, 10], [11, 20], and [21, 30] are labeled as 1, 2 and 3, respectively. Then, we randomly select two compressed images with the same content but different labels to form a pair-wise training data.

Training settings of de-blocking network. We generate the compressed images by using three JPEG quality factors, which are 10, 20 and 30. Then, for each ground truth image, there are 3 corresponding compressed images with different quality factors. Other methods mostly use 3 quality factors to train 3 models, and each model can only handle specific quality factor. As mentioned above, our single model is able to handle multiple JPEG qualities. Thus, we use all the 3 quality factors to train our de-blocking network. The parameters α and β are empirically set as 0.01 and 0.9, respectively.

Testing settings of de-blocking network. We use the same compression operation on BSDS500 (500 images), LIVE1 (29 images) and Classic5 (5 images). They are compressed with 3 quality factors: 10, 20 and 30. We use these compressed images to evaluate the performance of our model both in different datasets and in different quality factors.

4.2 Experimental Details

Evaluation metrics. For quantitative assessment, we use PSNR, structural similarity (SSIM) and PSNR-B [Yim and Bovik, 2010] to evaluate de-blocking performance. Note that the PSNR-B is specifically designed and more sensitive to blocking artifacts than SSIM.

Implementation details. As for the network architecture, the compression quality ranker (CQR) utilizes the VGG network [Simonyan and Zisserman, 2014], which includes 10 convolutional layers. The specific dilated factors are set to 1, 2 and 4. The ε in margin ranking loss is set to 0.5. For optimizations, we use Adam optimizer [Kingma and Ba, 2014] to train our model. The learning rate is initialized to 10-4 and decreased to half every 1×105 iterations. We implement the proposed model with two Titan Xp GPUs by using PyTorch.

4.3 Ablation Study

Our model contains 4 important components: Compression Quality Ranker (CQR), dilated convolutions (DC), qualityrelated loss (LQ) and feature matching loss (LF ), which allow our model to achieve promising performance both in qualitative and quantitative evaluation. We perform ablation experiments on the above 4 components to analyze their effects, which are shown in the Table 1 and Figure 3.

Effect of Compression Quality Ranker. Obviously, when removing the CQR module, LQ and LF no longer exist. As shown in the Table 1, the PSNR-B of "w/o CQR" decreases 0.20 at q = 10, but only decreases 0.08 at q = 30, which indicates that, by introducing compression quality information, CQR can guide the network to effectively remove severe compression artifacts.

Effect of dilated convolutions. As shown in Table 1, when removing DC, the network recovers better at q = 10, but fails to achieve good performance at q = 20 and 30. Without DC, the multi-scale context information cannot be extracted, thus the details of the images are difficult to be recovered. The "w/o DC" can only extract single-scale features, which cannot be used to handle multiple compression qualities.

Effect of quality-related loss. InTable 1, the PSNR-Bmetric of "w/o LQ" is 0.03, 0.06, 0.08 lower than ours at all the three qualities, which indicates that LQ can improve network performance. To prove that our ranker is able to discriminate different quality factors, we visualize the learned feature maps in Figure 4. Obviously, our ranker has a low response to the clean image and high responses to JPEG compressed images, especially in the area where artifacts are noticeable. Additionally, for different quality factors, the ranker generates different feature maps, which can provide discriminative information to guide the subsequent de-blocking network training.

Effect of feature matching loss. When LF is removed, the visual quality metric SSIM drops heavily, due to the featurelevel constraint disappears. The network cannot be effectively optimized at the visual level. In addition, the PSNR and PSNR-B also decreased, which indicates that LF reflects more valuable information compared with LQ.

4.4 Comparison with the State-of-the-Art Methods We compare our method with state-of-the-art LD [Li et al., 2014], ARCNN [Dong et al., 2015], TNRD [Chen and Pock,

2017], DnCNN [Zhang et al., 2017], LPIO [Fan et al., 2018] and RNAN [Zhang et al., 2019b]. As shown in Table 2, Figures 5 and 6, our model achieves comparable or superior performance both in qualitative and quantitative measurements.

Quantitative evaluation. As shown in Table 2, we show PSNR, SSIM and PSNR-B values to evaluation our proposed model with other methods. As we can see, compared with other methods, our method achieves the best overall performance in all the three metrics. Since the PSNR-B is specifically designed for de-blocking problems, the higher PSNB-B of our method indicates our model is more suitable for this JPEG artifacts removal task.

Qualitative evaluation. As shown in Figures 5 and 6, our model is able to simultaneously achieve compression artifacts removal and details restoration. Specifically, in smooth areas, blocking artifacts are well removed. Moreover, in the areas with complex textures, such as the edge in Figure 5 and the lines in Figure 6, our de-blocked images contain more fluent edges and no additional blurring or ringing effects are introduced. Especially, in Figure 6, the lines are very close to the ground truth.

5 Conclusion

In this paper, we propose compression quality ranker-guided networks for JPEG artifacts removal. Specifically, a compression quality ranker is introduced to measure different compression degrees. Based on the differentiable ranker, we then design quality-related loss and feature matching loss to guide de-blocking network optimization. Additionally, dilated convolutions are introduced to extract multi-scale features, which allows our single model to handle multiple JPEG qualities. Experiments show that our method achieves satisfactory performance in both qualitative and quantitative quality.

Acknowledgments

This work was supported by the National Key R&D Program of China under Grant 2017YFB1300201, the National Natural Science Foundation of China (NSFC) under Grants 61901433, U19B2038 and 61620106009 as well as the Fundamental Research Funds for the Central Universities under Grant WK2100100030.