{"type":"ScholarlyArticle","title":"Deep CNN Denoiser and Multi-layer Neighbor Component Embedding for Face Hallucination","authors":[{"name":"Junjun Jiang","slug":"junjun-jiang","given":"Junjun","family":"Jiang","affiliation":"Harbin Institute of Technology , Harbin , China; National Institute of Informatics , Tokyo , Japan"},{"name":"Yi Yu","slug":"yi-yu","given":"Yi","family":"Yu","affiliation":"National Institute of Informatics , Tokyo , Japan"},{"name":"Jinhui Hu","slug":"jinhui-hu","given":"Jinhui","family":"Hu","affiliation":"The Smart City Research Institute of CETC , Shenzhen , China"},{"name":"Suhua Tang","slug":"suhua-tang","given":"Suhua","family":"Tang","affiliation":"The University of Electro-Communications , Tokyo , Japan"},{"name":"Jiayi Ma","slug":"jiayi-ma","given":"Jiayi","family":"Ma","affiliation":"Wuhan University , Wuhan , China"}],"abstract":"Most of the current face hallucination methods, whether they are shallow learning-based or deep learning-based, all try to learn a relationship model between Low-Resolution (LR) and High-Resolution (HR) spaces with the help of a training set. They mainly focus on modeling image prior through either model-based optimization or discriminative inference learning. However, when the input LR face is tiny, the learned prior knowledge is no longer effective and their performance will drop sharply. To solve this problem, in this paper we propose a general face hallucination method that can integrate model-based optimization and discriminative inference. In particular, to exploit the model based prior, the Deep Convolutional Neural Networks (CNN) denoiser prior is plugged into the super-resolution optimization model with the aid of image-adaptive Laplacian regularization. Additionally, we further develop a high-frequency details compensation method by dividing the face image to facial components and performing face hallucination in a multi-layer neighbor embedding manner. Experiments demonstrate that the proposed method can achieve promising super-resolution results for tiny input LR faces.","doi":"10.24963/ijcai.2018/107","url":"https://ijcai.org/proceedings/2018/107","pdf":"https://ijcai.org/proceedings/2018/0107.pdf","markdown":"https://ijcai.org/proceedings/2018/107.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/107","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":107,"pages":{"first":771,"last":778},"keywords":["Computer Vision: Computer Vision","Machine Learning Applications: Other Applications","Computer Vision: Biometrics, Face and Gesture Recognition","Computer Vision: Statistical Methods and Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Afonso et al., 2010] Manya V Afonso, José M Bioucas-Dias, and Mário AT Figueiredo. Fast image recovery using variable split- ting and constrained optimization. IEEE Trans. Image Process., 19(9):2345-2356, 2010.","title":"Fast image recovery using variable splitting and constrained optimization","authors":"Manya V Afonso, M José, Bioucas-Dias, Mário, Figueiredo","year":"2010","venue":"IEEE Trans. Image Process","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baker and Kanade, 2000] Simon Baker and Takeo Kanade. Hallu- cinating faces. In FG, pages 83-88, 2000.","title":"Hallucinating faces","authors":"Kanade Baker, Simon Baker, Takeo Kanade","year":"2000","venue":"FG","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Boyd et al., 2011] Stephen Boyd, Neal Parikh, Eric Chu, Borja Pe- leato, and Jonathan Eckstein. Distributed optimization and statis- tical learning via the alternating direction method of multiplier- s. Foundations and Trends R in Machine Learning, 3(1):1-122, 2011.","title":"Distributed optimization and statistical learning via the alternating direction method of multipliers","authors":"Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein","year":"2011","venue":"Foundations and Trends R in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chang et al., 2004] Hong Chang, Dit-Yan Yeung, and Yimin X- iong. Super-resolution through neighbor embedding. In CVPR, volume 1, pages 275-282, 2004.","title":"Super-resolution through neighbor embedding","authors":"Chang","year":"2004","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2018] Yu Chen, Ying Tai, Xiaoming Liu, Chunhua Shen, and Jian Yang. FSRNet: End-to-end learning face super- resolution with facial priors. In CVPR, 2018.","title":"FSRNet: End-to-end learning face superresolution with facial priors","authors":"Yu Chen, Ying Tai, Xiaoming Liu, Chunhua Shen, Jian Yang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dabov et al., 2007] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Trans. Image Process., 16(8):2080-2095, 2007.","title":"Image denoising by sparse 3-d transform-domain collaborative filtering","authors":"Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, Karen Egiazarian","year":"2007","venue":"IEEE Trans. Image Process","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Danielyan et al., 2012] Aram Danielyan, Vladimir Katkovnik, and Karen Egiazarian. Bm3d frames and variational image deblur- ring. IEEE Trans. Image Process., 21(4):1715-1728, 2012.","title":"Aram Danielyan, Vladimir Katkovnik, and Karen Egiazarian. Bm3d frames and variational image deblurring","authors":"Danielyan","year":"2012","venue":"IEEE Trans. Image Process","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dong et al., 2013] Weisheng Dong, Lei Zhang, Guangming Shi, and Xin Li. Nonlocally centralized sparse representation for im- age restoration. IEEE Trans. Image Process., 22(4):1620-1630, 2013.","title":"Nonlocally centralized sparse representation for image restoration","authors":"Weisheng Dong, Lei Zhang, Guangming Shi, Xin Li","year":"2013","venue":"IEEE Trans. Image Process","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dong et al., 2016] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convo- lutional networks. IEEE Trans. Pattern Anal. Mach. Intell., 38(2):295-307, 2016.","title":"Image super-resolution using deep convolutional networks","authors":"Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang","year":"2016","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Grgic et al., 2011] Mislav Grgic, Kresimir Delac, and Sonja Grgic. Scface-surveillance cameras face database. Multimedia Tools and Applications, 51(3):863-879, 2011.","title":"Scface-surveillance cameras face database","authors":"Mislav Grgic, Kresimir Delac, Sonja Grgic","year":"2011","venue":"Multimedia Tools and Applications","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Huang et al., 2010] Hua Huang, Huiting He, Xin Fan, and Junping Zhang. Super-resolution of human face image using canonical correlation analysis. Pattern Recogn., 43(7):2532-2543, 2010.","title":"Super-resolution of human face image using canonical correlation analysis","authors":"Hua Huang, Huiting He, Xin Fan, Junping Zhang","year":"2010","venue":"Pattern Recogn","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jiang et al., 2014a] Junjun Jiang, Ruimin Hu, Zhongyuan Wang, and Zhen Han. Face super-resolution via multilayer locality- constrained iterative neighbor embedding and intermediate dic- tionary learning. IEEE Trans. Image Process., 23(10):4220- 4231, Oct 2014.","title":"Face super-resolution via multilayer localityconstrained iterative neighbor embedding and intermediate dictionary learning","authors":"Jiang","year":"2014","venue":"IEEE Trans. Image Process","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jiang et al., 2014b] Junjun Jiang, Ruimin Hu, Zhongyuan Wang, and Zhen Han. Noise robust face hallucination via locality- constrained representation. IEEE Trans. on Multimedia, 16(5):1268-1281, Aug 2014.","title":"Noise robust face hallucination via localityconstrained representation","authors":"Jiang","year":"2014","venue":"IEEE Trans. on Multimedia","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kim et al., 2016] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolu- tional networks. In CVPR, pages 1646-1654, 2016.","title":"Accurate image super-resolution using very deep convolutional networks","authors":"Jiwon Kim, Jung Kwon Lee, Kyoung Mu, Lee","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2001] Ce Liu, Heung-Yeung Shum, and Chang-Shui Zhang. A two-step approach to hallucinating faces: global para- metric model and local nonparametric model. In CVPR, vol- ume 1, pages 192-198, 2001.","title":"A two-step approach to hallucinating faces: global parametric model and local nonparametric model","authors":"Ce Liu, Heung-Yeung Shum, Chang-Shui Zhang","year":"2001","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2015a] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xi- aoou Tang. Deep learning face attributes in the wild. In ICCV, pages 3730-3738, 2015.","title":"Deep learning face attributes in the wild","authors":"Liu","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2015b] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xi- aoou Tang. Deep learning face attributes in the wild. In ICCV, pages 3730-3738, 2015.","title":"Deep learning face attributes in the wild","authors":"Liu","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Milanfar, 2013] Peyman Milanfar. A tour of modern image filter- ing. IEEE Singal Proc. Mag., 30(1):106-128, 2013.","title":"A tour of modern image filtering","authors":"Peyman Milanfar","year":"2013","venue":"IEEE Singal Proc. Mag","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Romano et al., 2017] Yaniv Romano, Michael Elad, and Peyman Milanfar. The little engine that could: Regularization by denois- ing (red). Siam Journal on Imaging Sciences, 10(4), 2017.","title":"The little engine that could: Regularization by denoising (red)","authors":"Yaniv Romano, Michael Elad, Peyman Milanfar","year":"2017","venue":"Siam Journal on Imaging Sciences","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Roweis and Saul, 2010] Sam T. Roweis and Lawrence K. Saul. Nonlinear dimensionality reduction by locally linear embedding. Science, 290(5500):2323-2326, 2010.","title":"Nonlinear dimensionality reduction by locally linear embedding","authors":"Sam T Roweis, Lawrence K Saul","year":"2010","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shi et al., 2015] Feng Shi, Jian Cheng, Li Wang, Pew-Thian Yap, and Dinggang Shen. Lrtv: Mr image super-resolution with low- rank and total variation regularizations. IEEE Trans. Med. Imag., 34(12):2459-2466, 2015.","title":"Lrtv: Mr image super-resolution with lowrank and total variation regularizations","authors":"Feng Shi, Jian Cheng, Li Wang, Pew-Thian Yap, Dinggang Shen","year":"2015","venue":"IEEE Trans. Med. Imag","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Song et al., 2017] Yibing Song, Jiawei Zhang, Shengfeng He, Lin- chao Bao, and Qingxiong Yang. Learning to hallucinate face images via component generation and enhancement. In IJCAI, pages 4537-4543, 2017.","title":"Learning to hallucinate face images via component generation and enhancement","authors":"Yibing Song, Jiawei Zhang, Shengfeng He, Linchao Bao, Qingxiong Yang","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/633"},{"position":23,"raw":"[Timofte et al., 2013] Radu Timofte, Vivek De, and Luc Van Gool. Anchored neighborhood regression for fast example-based super- resolution. In ICCV, pages 1920-1927, 2013.","title":"Anchored neighborhood regression for fast example-based superresolution","authors":"Radu Timofte, Vivek De, Luc Van Gool","year":"2013","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process., 13(4):600-612, 2004.","title":"Image quality assessment: from error visibility to structural similarity","authors":"Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli","year":"2004","venue":"IEEE Trans. Image Process","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yang et al., 2010] Jianchao Yang, John Wright, Thomas S Huang, and Yi Ma. Image super-resolution via sparse representation. IEEE Trans. Image Process., 19(11):2861-2873, 2010.","title":"Image super-resolution via sparse representation","authors":"Jianchao Yang, John Wright, Thomas S Huang, Yi Ma","year":"2010","venue":"IEEE Trans. Image Process","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang et al., 2013] Chih-Yuan Yang, Sifei Liu, and Ming-Hsuan Yang. Structured face hallucination. In CVPR, pages 1099-1106, 2013.","title":"Structured face hallucination","authors":"Chih-Yuan Yang, Sifei Liu, Ming-Hsuan Yang","year":"2013","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yu and Porikli, 2016] Xin Yu and Fatih Porikli. Ultra-resolving face images by discriminative generative networks. In ECCV, pages 318-333. Springer, 2016.","title":"Ultra-resolving face images by discriminative generative networks","authors":"Xin Yu, Fatih Porikli","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yu and Porikli, 2017] Xin Yu and Fatih Porikli. Face hallucina- tion with tiny unaligned images by transformative discriminative neural networks. In AAAI, pages 4327-4333, 2017.","title":"Face hallucination with tiny unaligned images by transformative discriminative neural networks","authors":"Xin Yu, Fatih Porikli","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yu et al., 2018] Xin Yu, Basura Fernando, Richard Hartley, and Fatih Porikli. Super-resolving very low-resolution face images with supplementary attributes. In CVPR, 2018.","title":"Super-resolving very low-resolution face images with supplementary attributes","authors":"Xin Yu, Basura Fernando, Richard Hartley, Fatih Porikli","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2011] Lin Zhang, Lei Zhang, Xuanqin Mou, and David Zhang. Fsim: A feature similarity index for image qual- ity assessment. IEEE Trans. Image Process., 20(8):2378-2386, 2011.","title":"Fsim: A feature similarity index for image quality assessment","authors":"Lin Zhang, Lei Zhang, Xuanqin Mou, David Zhang","year":"2011","venue":"IEEE Trans. Image Process","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2014] Jian Zhang, Debin Zhao, and Wen Gao. Group-based sparse representation for image restoration. IEEE Trans. Image Process., 23(8):3336-3351, 2014.","title":"Group-based sparse representation for image restoration","authors":"Jian Zhang, Debin Zhao, Wen Gao","year":"2014","venue":"IEEE Trans. Image Process","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2017] Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang. Learning deep CNN denoiser prior for image restora- tion. In CVPR, pages 2808-2817, 2017.","title":"Learning deep CNN denoiser prior for image restoration","authors":"Kai Zhang, Wangmeng Zuo, Shuhang Gu, Lei Zhang","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhu et al., 2016] Shizhan Zhu, Sifei Liu, Chen Change Loy, and Xiaoou Tang. Deep cascaded bi-network for face hallucination. In ECCV, pages 614-630. Springer, 2016.","title":"Deep cascaded bi-network for face hallucination","authors":"Zhu","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}