{"type":"ScholarlyArticle","title":"3D-Aided Deep Pose-Invariant Face Recognition","authors":[{"name":"Jian Zhao","slug":"jian-zhao","given":"Jian","family":"Zhao","affiliation":"National University of Singapore; National University of Defense Technology"},{"name":"Lin Xiong","slug":"lin-xiong","given":"Lin","family":"Xiong","affiliation":"Panasonic R&D Center Singapore"},{"name":"Yu Cheng","slug":"yu-cheng","given":"Yu","family":"Cheng","affiliation":"National University of Singapore"},{"name":"Yi Cheng","slug":"yi-cheng","given":"Yi","family":"Cheng","affiliation":"Panasonic R&D Center Singapore"},{"name":"Jianshu Li","slug":"jianshu-li","given":"Jianshu","family":"Li","affiliation":"National University of Singapore"},{"name":"Li Zhou","slug":"li-zhou","given":"Li","family":"Zhou","affiliation":"National University of Singapore"},{"name":"Yan Xu","slug":"yan-xu","given":"Yan","family":"Xu","affiliation":"Panasonic R&D Center Singapore"},{"name":"Jayashree Karlekar","slug":"jayashree-karlekar","given":"Jayashree","family":"Karlekar","affiliation":"Panasonic R&D Center Singapore"},{"name":"Sugiri Pranata","slug":"sugiri-pranata","given":"Sugiri","family":"Pranata","affiliation":"Panasonic R&D Center Singapore"},{"name":"Shengmei Shen","slug":"shengmei-shen","given":"Shengmei","family":"Shen","affiliation":"Panasonic R&D Center Singapore"},{"name":"Junliang Xing","slug":"junliang-xing","given":"Junliang","family":"Xing","affiliation":"National Laboratory of Pattern Recognition , Institute of Automation , Chinese Academy of Sciences; Panasonic R&D Center Singapore"},{"name":"Shuicheng Yan","slug":"shuicheng-yan","given":"Shuicheng","family":"Yan","affiliation":"National University of Singapore; Qihoo 360 AI Institute"},{"name":"Jiashi Feng","slug":"jiashi-feng","given":"Jiashi","family":"Feng","affiliation":"National University of Singapore"}],"abstract":"Learning from synthetic faces, though perhaps appealing for high data efficiency, may not bring satisfactory performance due to the distribution discrepancy of the synthetic and real face images. To mitigate this gap, we propose a 3D-Aided Deep Pose-Invariant Face Recognition Model (3D-PIM), which automatically recovers realistic frontal faces from arbitrary poses through a 3D face model in a novel way. Specifically, 3D-PIM incorporates a simulator with the aid of a 3D Morphable Model (3D MM) to obtain shape and appearance prior for accelerating face normalization learning, requiring less training data. It further leverages a global-local Generative Adversarial Network (GAN) with multiple critical improvements as a refiner to enhance the realism of both global structures and local details of the face simulator’s output using unlabelled real data only, while preserving the identity information. Qualitative and quantitative experiments on both controlled and in-the-wild benchmarks clearly demonstrate superiority of the proposed model over state-of-the-arts.","doi":"10.24963/ijcai.2018/165","url":"https://ijcai.org/proceedings/2018/165","pdf":"https://ijcai.org/proceedings/2018/0165.pdf","markdown":"https://ijcai.org/proceedings/2018/165.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/165","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":165,"pages":{"first":1184,"last":1190},"keywords":["Machine Learning: Transfer, Adaptation, Multi-task Learning","Machine Learning: Semi-Supervised Learning","Machine Learning: Learning Generative Models","Computer Vision: Biometrics, Face and Gesture Recognition","Machine Learning: Neural Networks","Machine Learning: Deep Learning","Computer Vision: Computer Vision"],"videos":[],"references":[{"position":1,"raw":"Abadi et al., ] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghe- mawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A sys- tem for large-scale machine learning.","title":"Tensorflow: A system for large-scale machine learning","authors":"Abadi","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Amberg et al., 2007] Brian Amberg, Sami Romdhani, and Thomas Vetter. Optimal step nonrigid icp algorithms for surface registra- tion. In CVPR, pages 1-8, 2007.","title":"Optimal step nonrigid icp algorithms for surface registration","authors":"Brian Amberg, Sami Romdhani, Thomas Vetter","year":"2007","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Blanz and Vetter, 1999] Volker Blanz and Thomas Vetter. A mor- phable model for the synthesis of 3d faces. In SIGGRAPH, pages 187-194, 1999.","title":"A morphable model for the synthesis of 3d faces","authors":"Vetter Blanz, Volker Blanz, Thomas Vetter","year":"1999","venue":"SIGGRAPH","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cao et al., 2014] Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou. Facewarehouse: A 3d facial expression database for visual computing. TVCG, 20(3):413-425, 2014.","title":"Facewarehouse: A 3d facial expression database for visual computing","authors":"Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, Kun Zhou","year":"2014","venue":"TVCG","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cheng et al., 2017] Yu Cheng, Jian Zhao, Zhecan Wang, Yan Xu, Karlekar Jayashree, Shengmei Shen, and Jiashi Feng. Know you at one glance: A compact vector representation for low-shot learning. In ICCVW, pages 1924-1932, 2017.","title":"Know you at one glance: A compact vector representation for low-shot learning","authors":"Yu Cheng, Jian Zhao, Zhecan Wang, Yan Xu, Karlekar Jayashree, Shengmei Shen, Jiashi Feng","year":"2017","venue":"ICCVW","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Crosswhite et al., 2017] Nate Crosswhite, Jeffrey Byrne, Chris Stauffer, Omkar Parkhi, Qiong Cao, and Andrew Zisserman. Template adaptation for face verification and identification. In FG, pages 1-8, 2017.","title":"Template adaptation for face verification and identification","authors":"Nate Crosswhite, Jeffrey Byrne, Chris Stauffer, Omkar Parkhi, Qiong Cao, Andrew Zisserman","year":"2017","venue":"FG","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gross et al., 2010] Ralph Gross, Iain Matthews, Jeffrey Cohn, Takeo Kanade, and Simon Baker. Multi-pie. JIVC, 28(5):807- 813, 2010.","title":"Multi-pie","authors":"Ralph Gross, Iain Matthews, Jeffrey Cohn, Takeo Kanade, Simon Baker","year":"2010","venue":"JIVC","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hassner et al., 2016] Tal Hassner, Iacopo Masi, Jungyeon Kim, Jongmoo Choi, Shai Harel, Prem Natarajan, and Gerard Medioni. Pooling faces: template based face recognition with pooled face images. In CVPRW, pages 59-67, 2016.","title":"Pooling faces: template based face recognition with pooled face images","authors":"Tal Hassner, Iacopo Masi, Jungyeon Kim, Jongmoo Choi, Shai Harel, Prem Natarajan, Gerard Medioni","year":"2016","venue":"CVPRW","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2017] Rui Huang, Shu Zhang, Tianyu Li, and Ran He. Beyond face rotation: Global and local perception gan for photorealistic and identity preserving frontal view synthesis. arXiv preprint arXiv:1704.04086, 2017.","title":"Beyond face rotation: Global and local perception gan for photorealistic and identity preserving frontal view synthesis","authors":"Rui Huang, Shu Zhang, Tianyu Li, Ran He","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kan et al., 2014] Meina Kan, Shiguang Shan, Hong Chang, and Xilin Chen. Stacked progressive auto-encoders (spae) for face recognition across poses. In CVPR, pages 1883-1890, 2014.","title":"Stacked progressive auto-encoders (spae) for face recognition across poses","authors":"Meina Kan, Shiguang Shan, Hong Chang, Xilin Chen","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Klare et al., 2015] Brendan F Klare, Ben Klein, Emma Taborsky, Austin Blanton, Jordan Cheney, Kristen Allen, Patrick Grother, Alan Mah, and Anil K Jain. Pushing the frontiers of uncon- strained face detection and recognition: Iarpa janus benchmark a. In CVPR, pages 1931-1939, 2015.","title":"Pushing the frontiers of unconstrained face detection and recognition: Iarpa janus benchmark a","authors":"Ben Brendan F Klare, Emma Klein, Austin Taborsky, Jordan Blanton, Kristen Cheney, Patrick Allen, Alan Grother, Anil K Mah, Jain","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2016] Jianshu Li, Jian Zhao, Fang Zhao, Hao Liu, Jing Li, Shengmei Shen, Jiashi Feng, and Terence Sim. Robust face recognition with deep multi-view representation learning. In ACM MM, pages 1068-1072, 2016.","title":"Robust face recognition with deep multi-view representation learning","authors":"Jianshu Li, Jian Zhao, Fang Zhao, Hao Liu, Jing Li, Shengmei Shen, Jiashi Feng, Terence Sim","year":"2016","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Masi et al., 2016] Iacopo Masi, Stephen Rawls, Gérard Medioni, and Prem Natarajan. Pose-aware face recognition in the wild. In CVPR, pages 4838-4846, 2016.","title":"Pose-aware face recognition in the wild","authors":"Iacopo Masi, Stephen Rawls, Gérard Medioni, Prem Natarajan","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Parkhi et al., 2015] O. M. Parkhi, A. Vedaldi, and A. Zisserman. Deep face recognition. In BMVC, 2015.","title":"Deep face recognition","authors":"O M Parkhi, A Vedaldi, A Zisserman","year":"2015","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Paysan et al., 2009] Pascal Paysan, Reinhard Knothe, Brian Am- berg, Sami Romdhani, and Thomas Vetter. A 3d face model for pose and illumination invariant face recognition. In AVSS, pages 296-301, 2009.","title":"A 3d face model for pose and illumination invariant face recognition","authors":"Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, Thomas Vetter","year":"2009","venue":"AVSS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ranjan et al., 2017] Rajeev Ranjan, Carlos D Castillo, and Rama Chellappa. L2-constrained softmax loss for discriminative face verification. arXiv preprint arXiv:1703.09507, 2017.","title":"L2-constrained softmax loss for discriminative face verification","authors":"Rajeev Ranjan, Carlos D Castillo, Rama Chellappa","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sagonas et al., 2015] Christos Sagonas, Yannis Panagakis, Ste- fanos Zafeiriou, and Maja Pantic. Robust statistical face frontal- ization. In ICCV, pages 3871-3879, 2015.","title":"Robust statistical face frontalization","authors":"Christos Sagonas, Yannis Panagakis, Stefanos Zafeiriou, Maja Pantic","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sankaranarayanan et al., 2016] Swami Sankaranarayanan, Azadeh Alavi, Carlos D Castillo, and Rama Chellappa. Triplet probabilis- tic embedding for face verification and clustering. In BTAS, pages 1-8, 2016.","title":"Triplet probabilistic embedding for face verification and clustering","authors":"Azadeh Swami Sankaranarayanan, Carlos D Alavi, Rama Castillo, Chellappa","year":"2016","venue":"BTAS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Schroff et al., 2015] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recog- nition and clustering. In CVPR, pages 815-823, 2015.","title":"Facenet: A unified embedding for face recognition and clustering","authors":"Schroff","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shrivastava et al., 2016] Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda Wang, and Russ Webb. Learn- ing from simulated and unsupervised images through adversarial training. arXiv preprint arXiv:1612.07828, 2016.","title":"Learning from simulated and unsupervised images through adversarial training","authors":"Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda Wang, Russ Webb","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale im- age recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sun et al., 2015a] Yi Sun, Ding Liang, Xiaogang Wang, and Xi- aoou Tang. Deepid3: Face recognition with very deep neural networks. arXiv preprint arXiv:1502.00873, 2015.","title":"Deepid3: Face recognition with very deep neural networks","authors":"Sun","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sun et al., 2015b] Yi Sun, Xiaogang Wang, and Xiaoou Tang. Deeply learned face representations are sparse, selective, and ro- bust. In CVPR, pages 2892-2900, 2015.","title":"Deeply learned face representations are sparse, selective, and robust","authors":"Sun","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Taigman et al., 2014] Yaniv Taigman, Ming Yang, Marc'Aurelio Ranzato, and Lior Wolf. Deepface: Closing the gap to human- level performance in face verification. In CVPR, pages 1701- 1708, 2014.","title":"Deepface: Closing the gap to humanlevel performance in face verification","authors":"Yaniv Taigman, Ming Yang, Marc'aurelio Ranzato, Lior Wolf","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tran et al., 2017] Luan Tran, Xi Yin, and Xiaoming Liu. Disentan- gled representation learning gan for pose-invariant face recogni- tion. In CVPR, 2017.","title":"Disentangled representation learning gan for pose-invariant face recognition","authors":"Luan Tran, Xi Yin, Xiaoming Liu","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wu et al., 2015] Xiang Wu, Ran He, Zhenan Sun, and Tieniu Tan. A light cnn for deep face representation with noisy labels. arXiv preprint arXiv:1511.02683, 2015.","title":"A light cnn for deep face representation with noisy labels","authors":"Xiang Wu, Ran He, Zhenan Sun, Tieniu Tan","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xiao et al., 2016] Shengtao Xiao, Jiashi Feng, Junliang Xing, Hanjiang Lai, Shuicheng Yan, and Ashraf Kassim. Robust facial landmark detection via recurrent attentive-refinement networks. In ECCV, pages 57-72, 2016.","title":"Robust facial landmark detection via recurrent attentive-refinement networks","authors":"Xiao","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yang et al., 2016] Jiaolong Yang, Peiran Ren, Dong Chen, Fang Wen, Hongdong Li, and Gang Hua. Neural aggregation network for video face recognition. arXiv preprint arXiv:1603.05474, 2016.","title":"Neural aggregation network for video face recognition","authors":"Jiaolong Yang, Peiran Ren, Dong Chen, Fang Wen, Hongdong Li, Gang Hua","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yim et al., 2015] Junho Yim, Heechul Jung, ByungIn Yoo, Changkyu Choi, Dusik Park, and Junmo Kim. Rotating your face using multi-task deep neural network. In CVPR, pages 676-684, 2015.","title":"Rotating your face using multi-task deep neural network","authors":"Yim","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhao et al., 2017a] Jian Zhao, Jianshu Li, Fang Zhao, Shuicheng Yan, and Jiashi Feng. Marginalized cnn: Learning deep invariant representations. In BMVC, 2017.","title":"Marginalized cnn: Learning deep invariant representations","authors":"Zhao","year":"2017","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhao et al., 2017b] Jian Zhao, Lin Xiong, Panasonic Karlekar Jayashree, Jianshu Li, Fang Zhao, Zhecan Wang, Panasonic Su- giri Pranata, Panasonic Shengmei Shen, Shuicheng Yan, and Ji- ashi Feng. Dual-agent gans for photorealistic and identity pre- serving profile face synthesis. In NIPS, pages 65-75, 2017.","title":"Dual-agent gans for photorealistic and identity preserving profile face synthesis","authors":"Zhao","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhu et al., 2015] Xiangyu Zhu, Zhen Lei, Junjie Yan, Dong Yi, and Stan Z Li. High-fidelity pose and expression normalization for face recognition in the wild. In CVPR, pages 787-796, 2015.","title":"High-fidelity pose and expression normalization for face recognition in the wild","authors":"Xiangyu Zhu, Zhen Lei, Junjie Yan, Dong Yi, Stan Z Li","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}