Connecting Low-Level Image Processing and High-Level Vision via Deep Learning

1 Overview

The latest development of computer vision has made exciting progress and tremendous impact in our daily lives. In this exciting era of technological advances, deep learning has gained huge popularity as a powerful tool for solving a lot of computer vision problems, and has added a great boost to this already rapidly developing field. Conventionally, the connection between different vision tasks is fragile. For example, low-level image processing and high-level vision tasks are usually coped with separately. However, the inherent relation of feature representations among various tasks should be effectively utilized rather than omitted. My research focuses on connecting low-level image processing and high-level vision via deep learning. Specifically, my goal is to design deep learning mechanisms that can efficiently and effectively learn features from low-level image processing and use them to improve the performance of high-level vision tasks.

Low-level image processing tasks, such as image restoration and image enhancement, play a key part in computer vision, as they enable the extraction of fundamental image primitives for further processing. In the past few years, we have witnessed an increasing interest in these fundamental topics from not only research communities but also industry, and substantial progress has been achieved. While the highlevel vision research has made significant progress with deep learning in recent years, such as image classification and face recognition, most models are trained, applied, and evaluated on high-quality (HQ) visual data, such as the LFW and ImageNet benchmarks. However, in many real-world scenarios, the performances of visual sensing and analytics are largely jeopardized by low-quality (LQ) visual data acquired from complex unconstrained environments, suffering from various types of degradations such as low resolution, noise, occlusion and motion blur. While some mild degradations may be compromised by sophisticated visual recognition models, their impacts turn much notable as the level of degradations passes some empirical threshold. To tackle such problems, building a model directly on LQ data is usually not robust due to the severe information loss caused by adverse conditions. On the other hand, the model trained on HQ data does not also perform well when tested on LQ data due to the domain mismatch.

I propose to use such low-level image processing techniques as an important frontend to help solve the challenging the resulting features are fused together, so that the kernels have a larger receptive field after all the feature contraction. It achieves the state-of-the-art methods denoising performance, demonstrating the strong capability of neural network to capture both local and global contextual information for denoising.

3 Ongoing Research and Future Directions

It becomes the latest trend to jointly study low-level image processing to high-level vision problems, as there is practical need to adopt high-level vision applications, such as surveillance system, intelligent human-computer interaction and autonomous driving, in low-quality imagery data. I expect to apply cutting-edge deep learning techniques to systematically investigate the mutual influence between low-level image processing and high-level vision problems.

I propose a unified deep learning framework to deal with image denoising and high-level vision tasks jointly in [Liu et al., 2017c], which is the first work investigating the benefit of exploiting image semantics simultaneously for image denoising and high-level vision tasks via deep learning. First I quantitatively show that image denoising can enhance the accuracy of image classification and semantic segmentation. Then I show that high-level semantics can be used for image denoising to generate visually appealing results in a deep learning fashion. In [Wang et al., 2016a; Liu* et al., 2018], a transfer learning method is developed to deal with deep learning based visual recognition in very low resolution conditions by exploiting pre-trained features for super-resolution. Extensive experiment results show that pre-training part of neural network models from the superresolution task is able to maintain a good balance between the feature extraction and the discrimination ability, and achieve impressive performance in the task of object recognition and face identification under adverse conditions. Moreover, I demonstrate that my video super-resolution approach is superior than other recent competing methods in terms of the accuracy of video face recognition under a low resolution setting in [Liu et al., 2017b]. These works reveal the fact that the feature representations learned from low-level image processing tasks can not only be used to generate visually pleasant results, but also provide more semantically faithful information, which benefits high-level vision tasks [Liu et al., 2017a].

For the future work, my goal is to systematically investigate the potential links between low-level image processing problems and high-level vision tasks via deep learning, with the focus on two questions, namely (1) how low-level image processing can help solving high-level vision problems, and (2) how the semantic information from high-level vision tasks can be used to guide low-level image processing. I expect to extend my current transfer learning approach to cope with more types of degradations, such as noise, occlusion and motion blur, and develop a degradation-robust recognition framework, which can be widely applied in modern surveillance system, intelligent robot and autonomous driving.

Network architecture has been shown to play an important role in deep learning models and various types of architecture are designed in a task-specific manner. Another direction of mine is to explore new universal neural network architecture with the help of multi-task learning, in order to handle visual data under various adverse conditions and extract semantically faithful features more effectively and efficiently. I believe my previous research experience and the skills I have accumulated before will put me in the special position to solve this challenging problem and make a unique contribution to the field of computer vision.