{"type":"ScholarlyArticle","title":"When Image Denoising Meets High-Level Vision Tasks: A Deep Learning Approach","authors":[{"name":"Ding Liu","slug":"ding-liu","given":"Ding","family":"Liu","affiliation":"University of Illinois at Urbana-Champaign , USA"},{"name":"Bihan Wen","slug":"bihan-wen","given":"Bihan","family":"Wen","affiliation":"University of Illinois at Urbana-Champaign , USA"},{"name":"Xianming Liu","slug":"xianming-liu","given":"Xianming","family":"Liu","affiliation":"Facebook Inc."},{"name":"Zhangyang Wang","slug":"zhangyang-wang","given":"Zhangyang","family":"Wang","affiliation":"Texas A&M University , USA"},{"name":"Thomas Huang","slug":"thomas-huang","given":"Thomas","family":"Huang","affiliation":"University of Illinois at Urbana-Champaign , USA"}],"abstract":"Conventionally, image denoising and high-level vision tasks are handled separately in computer vision. In this paper, we cope with the two jointly and explore the mutual influence between them. First we propose a convolutional neural network for image denoising which achieves the state-of-the-art performance. Second we propose a deep neural network solution that cascades two modules for image denoising and various high-level tasks, respectively, and use the joint loss for updating only the denoising network via back-propagation. We demonstrate that on one hand, the proposed denoiser has the generality to overcome the performance degradation of different high-level vision tasks. On the other hand, with the guidance of high-level vision information, the denoising network can generate more visually appealing results. To the best of our knowledge, this is the first work investigating the benefit of exploiting image semantics simultaneously for image denoising and high-level vision tasks via deep learning.","doi":"10.24963/ijcai.2018/117","url":"https://ijcai.org/proceedings/2018/117","pdf":"https://ijcai.org/proceedings/2018/0117.pdf","markdown":"https://ijcai.org/proceedings/2018/117.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/117","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":117,"pages":{"first":842,"last":848},"keywords":["Machine Learning: Neural Networks","Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Computer Vision: Computer Vision"],"videos":[],"references":[{"position":1,"raw":"Image CBM3D TNRD MCWNNM DnCNN Proposed CBM3D MCWNNM DnCNN Proposed CBM3D TNRD MCWNNM DnCNN Proposed 01 29.13 27.21 28.66 29.75 29.76 27.31 26.93 28.10 28.11 25.86 24.46 25.28 26.52 26.55 02 32.44 31.44 31.92 32.97 33.00 31.07 30.62 31.65 31.75 29.84 29.12 29.27 30.44 30.54 03 34.54 32.73 34.05 34.97 35.12 32.62 32.27 33.37 33.58 31.34 29.95 30.52 31.76 31.99 04 32.67 31.16 32.42 32.94 33.01 31.02 30.92 31.51 31.59 29.92 28.65 29.37 30.12 30.22 05 29.73 27.81 29.37 30.53 30.55 27.61 27.53 28.66 28.72 25.92 24.37 25.60 26.77 26.87 06 30.59 28.52 30.18 31.05 31.08 28.78 28.44 29.37 29.45 27.34 25.62 26.70 27.74 27.85 07 33.66 31.90 33.36 34.42 34.47 31.64 31.53 32.60 32.70 29.99 28.24 29.51 30.67 30.82 08 29.88 27.38 29.39 30.30 30.37 27.82 27.67 28.53 28.64 26.23 23.93 25.86 26.65 26.84 09 34.06 32.21 33.42 34.59 34.63 32.28 31.76 33.06 33.11 30.86 28.78 30.00 31.42 31.53 10 33.82 31.91 33.23 34.33 34.38 31.97 31.51 32.74 32.83 30.48 28.78 29.63 31.03 31.17 11 31.25 29.51 30.62 31.82 31.84 29.53 29.04 30.23 30.29 28.00 26.75 27.41 28.67 28.76 12 33.76 32.17 33.02 34.12 34.18 32.24 31.52 32.73 32.83 30.98 29.70 30.00 31.32 31.47 13 27.64 25.52 27.19 28.26 28.24 25.70 25.40 26.46 26.47 24.03 22.54 23.70 24.73 24.76 14 30.03 28.50 29.67 30.79 30.80 28.24 28.05 29.17 29.20 26.74 25.67 26.43 27.57 27.63 15 33.08 31.62 32.69 33.32 33.35 31.47 31.15 31.89 31.96 30.32 29.07 29.59 30.50 30.59 16 32.33 30.36 31.79 32.69 32.74 30.64 30.15 31.16 31.23 29.36 27.82 28.53 29.68 29.78 17 32.93 31.20 32.39 33.53 33.50 30.64 30.75 31.96 31.98 29.36 28.07 28.98 30.33 30.40 18 29.83 28.00 29.46 30.40 30.46 28.00 27.70 28.72 28.79 26.41 25.06 25.94 27.03 27.14 19 31.78 30.01 31.29 32.23 32.30 30.19 29.86 30.80 30.88 29.06 27.30 28.44 29.34 29.49 20 33.45 32.00 32.78 34.15 34.29 31.84 31.32 32.73 32.91 30.51 29.24 29.79 31.28 31.53 21 30.99 29.09 30.55 31.61 31.63 29.17 28.86 29.94 29.98 27.61 26.09 27.13 28.27 28.34 22 30.93 29.60 30.48 31.41 31.38 29.36 28.93 29.94 29.95 28.09 27.14 27.47 28.54 28.58 23 34.79 33.68 34.45 35.36 35.40 33.09 32.79 33.86 33.89 31.75 30.53 30.96 32.18 32.30 24 30.09 28.17 29.93 30.79 30.77 28.19 28.17 28.98 29.03 26.62 24.92 26.37 27.18 27.30 Average 31.81 30.08 31.35 32.35 32.39 30.04 29.70 30.76 30.83 28.62 27.17 28.02 29.16 29.27 Aharon et al., 2006] Michal Aharon, Michael Elad, and Al- fred Bruckstein. K-SVD : An algorithm for designing overcomplete dictionaries for sparse representation. TSP, 54(11):4311-4322, 2006.","title":"Image CBM3D TNRD MCWNNM DnCNN Proposed CBM3D MCWNNM DnCNN Proposed CBM3D TNRD","authors":null,"year":null,"venue":"MCWNNM DnCNN Proposed","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Burger et al., 2012] Harold C Burger, Christian J Schuler, and Stefan Harmeling. Image denoising: Can plain neural networks compete with bm3d? In CVPR, pages 2392- 2399. IEEE, 2012.","title":"Image denoising: Can plain neural networks compete with bm3d?","authors":"C Harold, Christian J Burger, Stefan Schuler, Harmeling","year":"2012","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen and Pock, 2017] Yunjin Chen and Thomas Pock. Trainable nonlinear reaction diffusion: A flexible frame- work for fast and effective image restoration. TPAMI, 39(6):1256-1272, 2017.","title":"Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration","authors":"Chen, Pock ; ] Yunjin Chen, Thomas Pock","year":"2017","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2014] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Se- mantic image segmentation with deep convolutional nets and fully connected crfs. arXiv preprint arXiv:1412.7062, 2014.","title":"Semantic image segmentation with deep convolutional nets and fully connected crfs","authors":"Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, Alan L Yuille","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dabov et al., 2007a] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Color image denoising via sparse 3d collaborative filtering with group- ing constraint in luminance-chrominance space. In ICIP, volume 1, pages I-313. IEEE, 2007.","title":"Color image denoising via sparse 3d collaborative filtering with grouping constraint in luminance-chrominance space","authors":"Dabov","year":"2007","venue":"ICIP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dabov et al., 2007b] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denois- ing by sparse 3-d transform-domain collaborative filtering. TIP, 16(8):2080-2095, 2007.","title":"Image denoising by sparse 3-d transform-domain collaborative filtering","authors":"Dabov","year":"2007","venue":"TIP","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dong et al., 2013] Weisheng Dong, Lei Zhang, Guangming Shi, and Xin Li. Nonlocally centralized sparse representa- tion for image restoration. TIP, 22(4):1620-1630, 2013.","title":"Nonlocally centralized sparse representation for image restoration","authors":"Weisheng Dong, Lei Zhang, Guangming Shi, Xin Li","year":"2013","venue":"TIP","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gu et al., 2014] Shuhang Gu, Lei Zhang, Wangmeng Zuo, and Xiangchu Feng. Weighted nuclear norm minimiza- tion with application to image denoising. In CVPR, pages 2862-2869, 2014.","title":"Weighted nuclear norm minimization with application to image denoising","authors":"Gu","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"He","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huynh-Thu and Ghanbari, 2008] Quan Huynh-Thu and Mohammed Ghanbari. Scope of validity of psnr in image/video quality assessment. Electronics letters, 44(13):800-801, 2008.","title":"Scope of validity of psnr in image/video quality assessment","authors":"Ghanbari Huynh-Thu","year":"2008","venue":"Electronics letters","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Johnson et al., 2016] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, pages 694-711. Springer, 2016.","title":"Perceptual losses for real-time style transfer and super-resolution","authors":"Justin Johnson, Alexandre Alahi, Li Fei-Fei","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2017] Boyi Li, Xiulian Peng, Zhangyang Wang, Jizheng Xu, and Dan Feng. Aod-net: All-in-one dehaz- ing network. In ICCV, 2017.","title":"Aod-net: All-in-one dehazing network","authors":"Boyi Li, Xiulian Peng, Zhangyang Wang, Jizheng Xu, Dan Feng","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2016] Ding Liu, Zhaowen Wang, Bihan Wen, Jianchao Yang, Wei Han, and Thomas S Huang. Ro- bust single image super-resolution via deep networks with sparse prior. IEEE TIP, 25(7):3194-3207, 2016.","title":"Robust single image super-resolution via deep networks with sparse prior","authors":"Ding Liu, Zhaowen Wang, Bihan Wen, Jianchao Yang, Wei Han, Thomas S Huang","year":"2016","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2017] Ding Liu, Bowen Cheng, Zhangyang Wang, Haichao Zhang, and Thomas S Huang. Enhance visual recognition under adverse conditions via deep net- works. arXiv preprint arXiv:1712.07732, 2017.","title":"Enhance visual recognition under adverse conditions via deep networks","authors":"Ding Liu, Bowen Cheng, Zhangyang Wang, Haichao Zhang, Thomas S Huang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mairal et al., 2009] J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman. Non-local sparse models for image restoration. In ICCV, 2009.","title":"Non-local sparse models for image restoration","authors":"J Mairal, F Bach, J Ponce, G Sapiro, A Zisserman","year":"2009","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mao et al., 2016] Xiaojiao Mao, Chunhua Shen, and Yu- Bin Yang. Image restoration using very deep convolu- tional encoder-decoder networks with symmetric skip con- nections. In NIPS, pages 2802-2810. 2016.","title":"Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections","authors":"Mao","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nguyen et al., 2015] Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High con- fidence predictions for unrecognizable images. In CVPR, pages 427-436, 2015.","title":"Deep neural networks are easily fooled: High confidence predictions for unrecognizable images","authors":"Anh Nguyen, Jason Yosinski, Jeff Clune","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ronneberger et al., 2015] Olaf Ronneberger, Philipp Fis- cher, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, pages 234-241. Springer, 2015.","title":"Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation","authors":"Ronneberger","year":"2015","venue":"MICCAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition","authors":"Zisserman Simonyan","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Szegedy et al., 2013] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.","title":"Intriguing properties of neural networks","authors":"Szegedy","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vincent et al., 2008] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extract- ing and composing robust features with denoising autoen- coders. In ICML, pages 1096-1103. ACM, 2008.","title":"Extracting and composing robust features with denoising autoencoders","authors":"Pascal Vincent, Hugo Larochelle, Yoshua Bengio, Pierre-Antoine Manzagol","year":"2008","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2016] Zhangyang Wang, Shiyu Chang, Yingzhen Yang, Ding Liu, and Thomas S Huang. Study- ing very low resolution recognition using deep networks. In CVPR, pages 4792-4800, 2016.","title":"Studying very low resolution recognition using deep networks","authors":"Wang","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu et al., 2017] Jiqing Wu, Radu Timofte, Zhiwu Huang, and Luc Van Gool. On the relation between color image denoising and classification. arXiv preprint arXiv:1704.01372, 2017.","title":"On the relation between color image denoising and classification","authors":"Jiqing Wu, Radu Timofte, Zhiwu Huang, Luc Van Gool","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xu et al., 2015] Jun Xu, Lei Zhang, Wangmeng Zuo, David Zhang, and Xiangchu Feng. Patch group based nonlo- cal self-similarity prior learning for image denoising. In CVPR, pages 244-252, 2015.","title":"Patch group based nonlocal self-similarity prior learning for image denoising","authors":"Xu","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu et al., 2017] Jun Xu, Lei Zhang, David Zhang, and Xi- angchu Feng. Multi-channel weighted nuclear norm min- imization for real color image denoising. 2017.","title":"Multi-channel weighted nuclear norm minimization for real color image denoising","authors":"Xu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2017a] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian de- noiser: Residual learning of deep cnn for image denoising. TIP, 2017.","title":"Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising","authors":"Zhang","year":"2017","venue":"TIP","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2017b] Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang. Learning deep cnn denoiser prior for image restoration. In CVPR, July 2017.","title":"Learning deep cnn denoiser prior for image restoration","authors":"Zhang","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zoran and Weiss, 2011] Daniel Zoran and Yair Weiss. From learning models of natural image patches to whole image restoration. In ICCV, pages 479-486. IEEE, 2011.","title":"From learning models of natural image patches to whole image restoration","authors":"Weiss ; ] Zoran, Yair Daniel Zoran, Weiss","year":"2011","venue":"ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}