{"type":"ScholarlyArticle","title":"Pose-preserving Cross Spectral Face Hallucination","authors":[{"name":"Junchi Yu","slug":"junchi-yu","given":"Junchi","family":"Yu","affiliation":"NLPR&CRIPAC Institute of Automation , Chinese Academy of Sciences , China"},{"name":"Jie Cao","slug":"jie-cao","given":"Jie","family":"Cao","affiliation":"NLPR&CRIPAC Institute of Automation , Chinese Academy of Sciences , China; University of Chinese Academy of Sciences , China"},{"name":"Yi Li","slug":"yi-li","given":"Yi","family":"Li","affiliation":"NLPR&CRIPAC Institute of Automation , Chinese Academy of Sciences , China; University of Chinese Academy of Sciences , China"},{"name":"Xiaofei Jia","slug":"xiaofei-jia","given":"Xiaofei","family":"Jia","affiliation":"Central Media Technology Institute , Huawei Technology Co. , Ltd. , China"},{"name":"Ran He","slug":"ran-he","given":"Ran","family":"He","affiliation":"NLPR&CRIPAC Institute of Automation , Chinese Academy of Sciences , China; University of Chinese Academy of Sciences , China; Center for Excellence in Brain Science and Intelligence Technology , CAS , China"}],"abstract":"To narrow the inherent sensing gap in heterogeneous face recognition (HFR), recent methods have resorted to generative models and explored the ?recognition via generation? framework. Even though, it remains a very challenging task to synthesize photo-realistic visible faces (VIS) from near-infrared (NIR) images especially when paired training data are unavailable. We present an approach to avert the data misalignment problem and faithfully preserve pose, expression and identity information during cross-spectral face hallucination. At the pixel level, we introduce an unsupervised attention mechanism to warping that is jointly learned with the generator to derive pixel-wise correspondence from unaligned data. At the image level, an auxiliary generator is employed to facilitate the learning of mapping from NIR to VIS domain. At the domain level, we first apply the mutual information constraint to explicitly measure the correlation between domains and thus benefit synthesis. Extensive experiments on three heterogeneous face datasets demonstrate that our approach not only outperforms current state-of-the-art HFR methods but also produce visually appealing results at a high resolution.","doi":"10.24963/ijcai.2019/143","url":"https://ijcai.org/proceedings/2019/143","pdf":"https://ijcai.org/proceedings/2019/0143.pdf","markdown":"https://ijcai.org/proceedings/2019/143.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/143","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Computer Vision","track":"Main track","number":143,"pages":{"first":1018,"last":1024},"keywords":["Computer Vision: Computer Vision","Computer Vision: Biometrics, Face and Gesture Recognition"],"videos":[],"references":[{"position":1,"raw":"[Belghazi et al., 2018] Mohamed Ishmael Belghazi, Aristide Baratin, Saj Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: Mutual in- formation neural estimation. In NeurIPS, 2018.","title":"Mine: Mutual information neural estimation","authors":"Belghazi","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brakel and Bengio, 2017] Philemon Brakel and Yoshua Bengio. Learning independent features with adversarial nets for non-linear ica. In ICML, 2017.","title":"Learning independent features with adversarial nets for non-linear ica","authors":"Bengio Brakel, Philemon Brakel, Yoshua Bengio","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bulat and Tzimiropoulos, 2017] Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2d and 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks). In ICCV, 2017.","title":"Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2d and 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks)","authors":"Tzimiropoulos Bulat","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2009] Jie Chen, Dong Yi, Jimei Yang, Guoying Zhao, Stan Z. Li, and Matti Pietikainen. Learning map- pings for face synthesis from near infrared to visual light images. In CVPR, 2009.","title":"Learning mappings for face synthesis from near infrared to visual light images","authors":"Jie Chen, Dong Yi, Jimei Yang, Guoying Zhao, Stan Z Li, Matti Pietikainen","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In NeurIPS, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2017] Ran He, Xiang Wu, Zhenan Sun, and Tieniu Tan. Learning invariant deep representation for nir-vis face recognition. In AAAI, 2017.","title":"Learning invariant deep representation for nir-vis face recognition","authors":"Ran He, Xiang Wu, Zhenan Sun, Tieniu Tan","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2018] Ran He, Xiang Wu, Zhenan Sun, and Tieniu Tan. Wasserstein cnn: Learning invariant features for nir- vis face recognition. IEEE Trans on PAMI, 2018.","title":"Wasserstein cnn: Learning invariant features for nirvis face recognition","authors":"Ran He, Xiang Wu, Zhenan Sun, Tieniu Tan","year":"2018","venue":"IEEE Trans on PAMI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hjelm et al., 2019] R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representa- tions by mutual information estimation and maximization. In ICLR, 2019.","title":"Learning deep representations by mutual information estimation and maximization","authors":"Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, Yoshua Bengio","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang et al., 2012] D Huang, J Sun, and Y Wang. The buaa-visnir face database instructions. Tech. Rep. IRIP- TR-12-FR-001, July 2012.","title":"The buaa-visnir face database instructions","authors":"J Huang, Y Sun, Wang","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2013] X Huang, Z Lei, M Fan, X Wang, and Li S.Z. Regularized discriminative spectral regression method for heterogeneous face matching. IEEE Trans on IP, 2013.","title":"Regularized discriminative spectral regression method for heterogeneous face matching","authors":"Z Huang, M Lei, X Fan, Li S Z Wang","year":"2013","venue":"IEEE Trans on IP","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Huang et al., 2017] Rui Huang, Shu Zhang, Tianyu Li, and Ran He. Beyond face rotation: Global and local perception gan for photorealistic and identity preserving frontal view synthesis. In ICCV, 2017.","title":"Beyond face rotation: Global and local perception gan for photorealistic and identity preserving frontal view synthesis","authors":"Rui Huang, Shu Zhang, Tianyu Li, Ran He","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Isola et al., 2017] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with con- ditional adversarial networks. In CVPR, 2017.","title":"Image-to-image translation with conditional adversarial networks","authors":"Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A Efros","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Johnson et al., 2016] Justin Johnson, Alahi Alexandre, and Fei-Fei Li. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016.","title":"Perceptual losses for real-time style transfer and super-resolution","authors":"Justin Johnson, Alahi Alexandre, Fei-Fei Li","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kristan et al., 2014] P. Gurton Kristan, J. Yuffa Alex, and W. Videen Gorden. Enhanced facial recognition for ther- mal imagery using polarimetric imaging. Optics Letters, 2014.","title":"Enhanced facial recognition for thermal imagery using polarimetric imaging","authors":"P Gurton Kristan, J Yuffa Alex, W Videen Gorden","year":"2014","venue":"Optics Letters","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lezama et al., 2017] Jose Lezama, Qiu Qiang, and Sapiro Guillermo. Not afraid of the dark: Nir-vis face recognition via cross-spectral hallucination and low-rank embedding. In CVPR, 2017.","title":"Not afraid of the dark: Nir-vis face recognition via cross-spectral hallucination and low-rank embedding","authors":"Jose Lezama, Qiu Qiang, Sapiro Guillermo","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2013] Stan Z. Li, Dong Yi, Zhen Lei, and Sheng- cai Liao. The casia nir-vis 2.0 face database. In CVPR Workshops, 2013.","title":"The casia nir-vis 2.0 face database","authors":"Stan Z Li, Dong Yi, Zhen Lei, Shengcai Liao","year":"2013","venue":"CVPR Workshops","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2016] Xiaoxiang Liu, Lingxiao Song, Xiang Wu, and Tieniu Tan. Transferring deep repre-sentation for nir- vis heterogeneous face recognition. In ICB, 2016.","title":"Transferring deep repre-sentation for nirvis heterogeneous face recognition","authors":"Xiaoxiang Liu, Lingxiao Song, Xiang Wu, Tieniu Tan","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Parkhi et al., 2015] Omkar M. Parkhi, Andrea Vedaldi, and Andrew Zisserman. Deep face recognition. In BMVC, 2015.","title":"Deep face recognition","authors":"M Omkar, Andrea Parkhi, Andrew Vedaldi, Zisserman","year":"2015","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Peng et al., 2019] Chunlei Peng, Gao Xinbo, Wang Nannan, and Li Jie. Sparse graphical representation based discrim- inant analysis for heterogeneous face recognition. Signal Processing, 2019.","title":"Sparse graphical representation based discriminant analysis for heterogeneous face recognition","authors":"Chunlei Peng, Gao Xinbo, Wang Nannan, Li Jie","year":"2019","venue":"Signal Processing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Riggan et al., 2016] Benjamin S. Riggan, Nathanial J. Short, Shuowen Hu, and Heesung Kwon. Estimation of visible spectrum faces from polarimetric thermal faces. ICBTAS, 2016.","title":"Estimation of visible spectrum faces from polarimetric thermal faces","authors":"Benjamin S Riggan, Nathanial J Short, Shuowen Hu, Heesung Kwon","year":"2016","venue":"ICBTAS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ruprecht and Muller, 1995] D. Ruprecht and H. Muller. Im- age warping with scattered data interpolation. IEEE CGA, 1995.","title":"Image warping with scattered data interpolation","authors":"D Ruprecht, H Muller","year":"1995","venue":"IEEE CGA","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Song et al., 2018] Lingxiao Song, Zhang Man, Wu Xiang, and He Ran. Adversarial discriminative heterogeneous face recognition. AAAI, 2018.","title":"Adversarial discriminative heterogeneous face recognition","authors":"Song","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tang and Wang, 2003] Xiaoou Tang and Xiaogang Wang. Face sketch synthesis and recognition. In ICCV, 2003.","title":"Face sketch synthesis and recognition","authors":"Wang ; Tang, Xiaoou Tang, Xiaogang Wang","year":"2003","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2009] Rui Wang, Jimei Yang, Dong Yi, and Stan Z. Li. An analysis-by-synthesis method for hetero- geneous face biometrics. In ICB, 2009.","title":"An analysis-by-synthesis method for heterogeneous face biometrics","authors":"Rui Wang, Jimei Yang, Dong Yi, Stan Z Li","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wu et al., 2018] Xiang Wu, Ran He, Zhenan Sun, and Tie- niu Tan. A light cnn for deep face representation with noisy labels. IEEE Trans on IFS, 2018.","title":"A light cnn for deep face representation with noisy labels","authors":"Xiang Wu, Ran He, Zhenan Sun, Tieniu Tan","year":"2018","venue":"IEEE Trans on IFS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yin et al., 2017] Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, and Manmohan Chandraker. Towards large-pose face frontalization in the wild. In ICCV, 2017.","title":"Towards large-pose face frontalization in the wild","authors":"Xiang Xi Yin, Kihyuk Yu, Xiaoming Sohn, Manmohan Liu, Chandraker","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Youssef et al., 2018] Mejjati Youssef, Alami, Richardt Christian, Tompkin James, Cosker Darren, and Kim Kwang, In. Unsupervised attention-guided image-to- image translation. In NeurIPS, 2018.","title":"Unsupervised attention-guided image-toimage translation","authors":"Youssef","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2017] He Zhang, Vishal M. Patel, Benjamin S. Riggan, and Shuowen Hu. Generative adversarial network-based synthesis of visible faces from polarimetrie thermal faces. IJCB, 2017.","title":"Generative adversarial network-based synthesis of visible faces from polarimetrie thermal faces","authors":"Zhang","year":"2017","venue":"IJCB","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2018] Yizhe Zhang, Galley Michel, Gao Jian- feng, Gan Zhe, Li Xiujun, Brockett Chris, and Dolan Bill. Generating informative and diverse conversational responses via adversarial information maximization. In NeurIPS, 2018.","title":"Generating informative and diverse conversational responses via adversarial information maximization","authors":"Zhang","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhu et al., 2017] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image transla- tion using cycle-consistent adversarial networks. In ICCV, 2017.","title":"Unpaired image-to-image translation using cycle-consistent adversarial networks","authors":"Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A Efros","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}