{"type":"ScholarlyArticle","title":"Multi-Prototype Networks for Unconstrained Set-based Face Recognition","authors":[{"name":"Jian Zhao","slug":"jian-zhao","given":"Jian","family":"Zhao","affiliation":"National University of Singapore; National University of Defense Technology; Tencent FiT DeepSea AI Lab"},{"name":"Jianshu Li","slug":"jianshu-li","given":"Jianshu","family":"Li","affiliation":"National University of Singapore"},{"name":"Xiaoguang Tu","slug":"xiaoguang-tu","given":"Xiaoguang","family":"Tu","affiliation":"University of Electronic Science and Technology of China"},{"name":"Fang Zhao","slug":"fang-zhao","given":"Fang","family":"Zhao","affiliation":"National University of Singapore; National University of Defense Technology; Tencent FiT DeepSea AI Lab"},{"name":"Yuan Xin","slug":"yuan-xin","given":"Yuan","family":"Xin","affiliation":"Tencent FiT DeepSea AI Lab; Inception Institute of Artificial Intelligence"},{"name":"Junliang Xing","slug":"junliang-xing","given":"Junliang","family":"Xing","affiliation":"National Laboratory of Pattern Recognition , Institute of Automation , Chinese Academy of Sciences; scientist at Tencent FiT DeepSea AI Lab"},{"name":"Hengzhu Liu","slug":"hengzhu-liu","given":"Hengzhu","family":"Liu","affiliation":"National University of Defense Technology"},{"name":"Shuicheng Yan","slug":"shuicheng-yan","given":"Shuicheng","family":"Yan","affiliation":"National University of Singapore; Qihoo 360 AI Institute"},{"name":"Jiashi Feng","slug":"jiashi-feng","given":"Jiashi","family":"Feng","affiliation":"National University of Singapore"}],"abstract":"In this paper, we address the challenging unconstrained set-based face recognition problem where each subject face is instantiated by a set of media (images and videos) instead of a single image. Naively aggregating information from all the media within a set would suffer from the large intra-set variance caused by heterogeneous factors (e.g., varying media modalities, poses and illumination) and fail to learn discriminative face representations. A novel Multi-Prototype Network (MP- Net) model is thus proposed to learn multiple prototype face representations adaptively from the media sets. Each learned prototype is representative for the subject face under certain condition in terms of pose, illumination and media modality. Instead of handcrafting the set partition for prototype learn- ing, MPNet introduces a Dense SubGraph (DSG) learning sub-net that implicitly untangles inconsistent media and learns a number of representative prototypes. Qualitative and quantitative experiments clearly demonstrate the superiority of the proposed model over state-of-the-arts.","doi":"10.24963/ijcai.2019/611","url":"https://ijcai.org/proceedings/2019/611","pdf":"https://ijcai.org/proceedings/2019/0611.pdf","markdown":"https://ijcai.org/proceedings/2019/611.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/611","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":611,"pages":{"first":4397,"last":4403},"keywords":["Computer Vision: Computer Vision","Computer Vision: Biometrics, Face and Gesture Recognition","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[AbdAlmageed et al., 2016] Wael AbdAlmageed, Yue Wu, Stephen Rawls, Shai Harel, Tal Hassner, Iacopo Masi, Jongmoo Choi, Jatuporn Lekust, Jungyeon Kim, Prem Natarajan, et al. Face recognition using deep multi-pose representations. In WACV, pages 1-9, 2016.","title":"Face recognition using deep multi-pose representations","authors":"Wael Abdalmageed, Yue Wu, Stephen Rawls, Shai Harel, Tal Hassner, Iacopo Masi, Jongmoo Choi, Jatuporn Lekust, Jungyeon Kim, Prem Natarajan","year":"2016","venue":"WACV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cao et al., 2018] Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman. VggFace2: a dataset for recog- nising faces across pose and age. In FG, pages 67-74, 2018.","title":"VggFace2: a dataset for recognising faces across pose and age","authors":"Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, Andrew Zisserman","year":"2018","venue":"FG","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2015] Jun-Cheng Chen, Rajeev Ranjan, Amit Kumar, Ching-Hui Chen, Vishal M Patel, and Rama Chellappa. An end- to-end system for unconstrained face verification with deep con- volutional neural networks. In ICCVW, pages 118-126, 2015.","title":"An endto-end system for unconstrained face verification with deep convolutional neural networks","authors":"Jun-Cheng Chen, Rajeev Ranjan, Amit Kumar, Ching-Hui Chen, M Vishal, Rama Patel, Chellappa","year":"2015","venue":"ICCVW","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2016] Jun-Cheng Chen, Vishal M Patel, and Rama Chellappa. Unconstrained face verification using deep cnn fea- tures. In WACV, pages 1-9, 2016.","title":"Unconstrained face verification using deep cnn features","authors":"Jun-Cheng Chen, M Vishal, Rama Patel, Chellappa","year":"2016","venue":"WACV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chowdhury et al., 2016] Aruni Roy Chowdhury, Tsung-Yu Lin, Subhransu Maji, and Erik Learned-Miller. One-to-many face recognition with bilinear CNNs. In WACV, pages 1-9, 2016.","title":"One-to-many face recognition with bilinear CNNs","authors":"Aruni Roy Chowdhury, Tsung-Yu Lin, Subhransu Maji, Erik Learned-Miller","year":"2016","venue":"WACV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Crosswhite et al., 2017] Nate Crosswhite, Jeffrey Byrne, Chris Stauffer, Omkar Parkhi, Qiong Cao, and Andrew Zisserman. Template adaptation for face verification and identification. In FG, pages 1-8, 2017.","title":"Template adaptation for face verification and identification","authors":"Nate Crosswhite, Jeffrey Byrne, Chris Stauffer, Omkar Parkhi, Qiong Cao, Andrew Zisserman","year":"2017","venue":"FG","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Das and Lee, 2018a] Debasmit Das and CS George Lee. Sample- to-sample correspondence for unsupervised domain adaptation. EAAI, 73:80-91, 2018.","title":"Debasmit Das and CS George Lee. Sampleto-sample correspondence for unsupervised domain adaptation","authors":"Lee Das","year":"2018","venue":"EAAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Das and Lee, 2018b] Debasmit Das and CS George Lee. Unsuper- vised domain adaptation using regularized hyper-graph match- ing. In ICIP, pages 3758-3762, 2018.","title":"Debasmit Das and CS George Lee. Unsupervised domain adaptation using regularized hyper-graph matching","authors":"Lee Das","year":"2018","venue":"ICIP","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Deng et al., 2018] Jiankang Deng, Jia Guo, Niannan Xue, and Ste- fanos Zafeiriou. ArcFace: additive angular margin loss for deep face recognition. arXiv preprint arXiv:1801.07698, 2018.","title":"ArcFace: additive angular margin loss for deep face recognition","authors":"Deng","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dong et al., 2013] Jian Dong, Wei Xia, Qiang Chen, Jianshi Feng, Zhongyang Huang, and Shuicheng Yan. Subcategory-aware ob- ject classification. In CVPR, pages 827-834, 2013.","title":"Subcategory-aware object classification","authors":"Jian Dong, Wei Xia, Qiang Chen, Jianshi Feng, Zhongyang Huang, Shuicheng Yan","year":"2013","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hassner et al., 2016] Tal Hassner, Iacopo Masi, Jungyeon Kim, Jongmoo Choi, Shai Harel, Prem Natarajan, and Gérard Medioni. Pooling faces: template based face recognition with pooled face images. In CVPRW, pages 59-67, 2016.","title":"Pooling faces: template based face recognition with pooled face images","authors":"Tal Hassner, Iacopo Masi, Jungyeon Kim, Jongmoo Choi, Shai Harel, Prem Natarajan, Gérard Medioni","year":"2016","venue":"CVPRW","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Huang et al., 2007] Gary B Huang, Manu Ramesh, Tamara Berg, and Erik Learned-Miller. Labeled faces in the wild: a database for studying face recognition in unconstrained environments. Technical report, Technical Report 07-49, University of Mas- sachusetts, Amherst, 2007.","title":"Labeled faces in the wild: a database for studying face recognition in unconstrained environments","authors":"Manu Gary B Huang, Tamara Ramesh, Erik Berg, Learned-Miller","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Klare et al., 2015] Brendan F Klare, Ben Klein, Emma Taborsky, Austin Blanton, Jordan Cheney, Kristen Allen, Patrick Grother, Alan Mah, Mark Burge, and Anil K Jain. Pushing the frontiers of unconstrained face detection and recognition: IARPA Janus Benchmark A. In CVPR, pages 1931-1939, 2015.","title":"Pushing the frontiers of unconstrained face detection and recognition: IARPA Janus Benchmark A","authors":"Ben Brendan F Klare, Emma Klein, Austin Taborsky, Jordan Blanton, Kristen Cheney, Patrick Allen, Alan Grother, Mark Mah, Anil K Burge, Jain","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2016] Jianshu Li, Jian Zhao, Fang Zhao, Hao Liu, Jing Li, Shengmei Shen, Jiashi Feng, and Terence Sim. Robust face recognition with deep multi-view representation learning. In ACM MM, pages 1068-1072, 2016.","title":"Robust face recognition with deep multi-view representation learning","authors":"Jianshu Li, Jian Zhao, Fang Zhao, Hao Liu, Jing Li, Shengmei Shen, Jiashi Feng, Terence Sim","year":"2016","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2017] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. SphereFace: Deep hypersphere embedding for face recognition. In CVPR, volume 1, page 1, 2017.","title":"SphereFace: Deep hypersphere embedding for face recognition","authors":"Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, Le Song","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Maaten and Hinton, 2008] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. JMLR, 9(Nov):2579- 2605, 2008.","title":"Visualizing data using t-SNE","authors":"Hinton ; Laurens Maaten, Geoffrey Van Der Maaten, Hinton","year":"2008","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Masi et al., 2016] Iacopo Masi, Stephen Rawls, Gerard Medioni, and Prem Natarajan. Pose-aware face recognition in the wild. In CVPR, pages 4838-4846, 2016.","title":"Pose-aware face recognition in the wild","authors":"Masi","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Parkhi et al., 2015] O. M. Parkhi, A. Vedaldi, and A. Zisserman. Deep face recognition. In BMVC, 2015.","title":"Deep face recognition","authors":"O M Parkhi, A Vedaldi, A Zisserman","year":"2015","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ranjan et al., 2016] Rajeev Ranjan, Swami Sankaranarayanan, Carlos D Castillo, and Rama Chellappa. An all-in-one con- volutional neural network for face analysis. arXiv preprint arXiv:1611.00851, 2016.","title":"An all-in-one convolutional neural network for face analysis","authors":"Rajeev Ranjan, Swami Sankaranarayanan, Carlos D Castillo, Rama Chellappa","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ranjan et al., 2017] Rajeev Ranjan, Carlos D Castillo, and Rama Chellappa. L2-constrained Softmax loss for discriminative face verification. arXiv preprint arXiv:1703.09507, 2017.","title":"L2-constrained Softmax loss for discriminative face verification","authors":"Rajeev Ranjan, Carlos D Castillo, Rama Chellappa","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sankaranarayanan et al., 2016] Swami Sankaranarayanan, Azadeh Alavi, Carlos D Castillo, and Rama Chellappa. Triplet probabilis- tic embedding for face verification and clustering. In BTAS, pages 1-8, 2016.","title":"Triplet probabilistic embedding for face verification and clustering","authors":"Azadeh Swami Sankaranarayanan, Carlos D Alavi, Rama Castillo, Chellappa","year":"2016","venue":"BTAS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Schroff et al., 2015] Florian Schroff, Dmitry Kalenichenko, and James Philbin. FaceNet: a unified embedding for face recog- nition and clustering. In CVPR, pages 815-823, 2015.","title":"FaceNet: a unified embedding for face recognition and clustering","authors":"Schroff","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale im- age recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sun et al., 2015a] Yi Sun, Ding Liang, Xiaogang Wang, and Xi- aoou Tang. DeepID3: Face recognition with very deep neural networks. arXiv preprint arXiv:1502.00873, 2015.","title":"DeepID3: Face recognition with very deep neural networks","authors":"Sun","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sun et al., 2015b] Yi Sun, Xiaogang Wang, and Xiaoou Tang. Deeply learned face representations are sparse, selective, and ro- bust. In CVPR, pages 2892-2900, 2015.","title":"Deeply learned face representations are sparse, selective, and robust","authors":"Sun","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Taigman et al., 2014] Yaniv Taigman, Ming Yang, Marc'Aurelio Ranzato, and Lior Wolf. DeepFace: closing the gap to human- level performance in face verification. In CVPR, pages 1701- 1708, 2014.","title":"DeepFace: closing the gap to humanlevel performance in face verification","authors":"Yaniv Taigman, Ming Yang, Marc'aurelio Ranzato, Lior Wolf","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2018a] Feng Wang, Jian Cheng, Weiyang Liu, and Haijun Liu. Additive margin softmax for face verification. IEEE Signal Processing Letters, 25(7):926-930, 2018.","title":"Additive margin softmax for face verification","authors":"Wang","year":"2018","venue":"IEEE Signal Processing Letters","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2018b] Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, and Wei Liu. CosFace: large margin cosine loss for deep face recognition. In CVPR, pages 5265-5274, 2018.","title":"CosFace: large margin cosine loss for deep face recognition","authors":"Wang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wolf et al., 2011] Lior Wolf, Tal Hassner, and Itay Maoz. Face recognition in unconstrained videos with matched background similarity. In CVPR, pages 529-534, 2011.","title":"Face recognition in unconstrained videos with matched background similarity","authors":"Lior Wolf, Tal Hassner, Itay Maoz","year":"2011","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yang et al., 2017] Jiaolong Yang, Peiran Ren, Dongqing Zhang, Dong Chen, Fang Wen, Hongdong Li, and Gang Hua. Neural aggregation network for video face recognition. In CVPR, pages 5216-5225, 2017.","title":"Neural aggregation network for video face recognition","authors":"Jiaolong Yang, Peiran Ren, Dongqing Zhang, Dong Chen, Fang Wen, Hongdong Li, Gang Hua","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhao et al., 2017] Jian Zhao, Lin Xiong, Panasonic Karlekar Jayashree, Jianshu Li, Fang Zhao, Zhecan Wang, Panasonic Su- giri Pranata, Panasonic Shengmei Shen, Shuicheng Yan, and Ji- ashi Feng. Dual-agent gans for photorealistic and identity pre- serving profile face synthesis. In NIPS, pages 66-76, 2017.","title":"Dual-agent gans for photorealistic and identity preserving profile face synthesis","authors":"Jian Zhao, Lin Xiong, Panasonic Karlekar Jayashree, Jianshu Li, Fang Zhao, Zhecan Wang, Panasonic Sugiri Pranata, Panasonic Shengmei Shen, Shuicheng Yan, Jiashi Feng","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhao et al., 2018a] Jian Zhao, Yu Cheng, Yan Xu, Lin Xiong, Jian- shu Li, Fang Zhao, Karlekar Jayashree, Sugiri Pranata, Shengmei Shen, Junliang Xing, et al. Towards pose invariant face recogni- tion in the wild. In CVPR, pages 2207-2216, 2018.","title":"Towards pose invariant face recognition in the wild","authors":"Zhao","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhao et al., 2018b] Jian Zhao, Lin Xiong, Yu Cheng, Yi Cheng, Jianshu Li, Li Zhou, Yan Xu, Jayashree Karlekar, Sugiri Pranata, Shengmei Shen, Junliang Xing, Shuicheng Yan, and Jiashi Feng. 3d-aided deep pose-invariant face recognition. In IJCAI, pages 1184-1190, 2018.","title":"3d-aided deep pose-invariant face recognition","authors":"Zhao","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/165"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}