{"type":"ScholarlyArticle","title":"Face Photo-Sketch Synthesis via Knowledge Transfer","authors":[{"name":"Mingrui Zhu","slug":"mingrui-zhu","given":"Mingrui","family":"Zhu","affiliation":"State Key Laboratory of Integrated Services Networks , Xidian University , Xi'an , China; School of Electronic Engineering , Xidian University , Xi'an , China"},{"name":"Nannan Wang","slug":"nannan-wang","given":"Nannan","family":"Wang","affiliation":"State Key Laboratory of Integrated Services Networks , Xidian University , Xi'an , China; School of Telecommunications Engineering , Xidian University , Xi'an , China"},{"name":"Xinbo Gao","slug":"xinbo-gao","given":"Xinbo","family":"Gao","affiliation":"State Key Laboratory of Integrated Services Networks , Xidian University , Xi'an , China; School of Electronic Engineering , Xidian University , Xi'an , China"},{"name":"Jie Li","slug":"jie-li","given":"Jie","family":"Li","affiliation":"State Key Laboratory of Integrated Services Networks , Xidian University , Xi'an , China; School of Electronic Engineering , Xidian University , Xi'an , China"},{"name":"Zhifeng Li","slug":"zhifeng-li","given":"Zhifeng","family":"Li","affiliation":"Tencent AI Lab , Shenzhen , China"}],"abstract":"Despite deep neural networks have\r\ndemonstrated strong power in face photo-sketch synthesis task, their\r\nperformance, however, are still limited by the lack of training data\r\n(photo-sketch pairs). Knowledge Transfer (KT), which aims at training a smaller\r\nand fast student network with the information learned from a larger and\r\naccurate teacher network, has attracted much attention recently due to its\r\nsuperior performance in the acceleration and compression of deep neural\r\nnetworks. This work has brought us great inspiration that we can train a\r\nrelatively small student network on very few training data by transferring\r\nknowledge from a larger teacher model trained on enough training data for other\r\ntasks. Therefore, we propose a novel knowledge transfer framework to synthesize\r\nface photos from face sketches or synthesize face sketches from face photos.\r\nParticularly, we utilize two teacher networks trained on large amount of data\r\nin related task to learn the knowledge of face photos and face sketches\r\nseparately and transfer them to two student networks simultaneously. In\r\naddition, the two student networks, one for photo ? sketch task and the other for sketch ? photo task, can transfer their knowledge mutually. With the\r\nproposed method, we can train our model which has superior performance using a\r\nsmall set of photo-sketch pairs. We validate the effectiveness of our method\r\nacross several datasets. Quantitative and qualitative evaluations illustrate\r\nthat our model outperforms other state-of-the-art methods in generating face\r\nsketches (or photos) with high visual quality and recognition ability.","doi":"10.24963/ijcai.2019/147","url":"https://ijcai.org/proceedings/2019/147","pdf":"https://ijcai.org/proceedings/2019/0147.pdf","markdown":"https://ijcai.org/proceedings/2019/147.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/147","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Computer Vision","track":"Main track","number":147,"pages":{"first":1048,"last":1054},"keywords":["Computer Vision: Biometrics, Face and Gesture Recognition","Computer Vision: Computer Vision"],"videos":[],"references":[{"position":1,"raw":"[Chen et al., 2000] Li-Fen Chen, Hong-Yuan Mark Liao, Ming-Tat Ko, Ja-Chen Lin, and Gwo-Jong Yu. A new lda- based face recognition system which can solve the small sample size problem. Pattern Recognition, 33(10):1713- 1726, 2000.","title":"A new ldabased face recognition system which can solve the small sample size problem","authors":"Li-Fen Chen, Hong-Yuan Mark Liao, Ming-Tat Ko, Ja-Chen Lin, Gwo-Jong Yu","year":"2000","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2018] Chaofeng Chen, Wei Liu, Xiao Tan, and Kwan-Yee K. Wong. Semi-supervised learning for face sketch synthesis in the wild. In ACCV, 2018.","title":"Semi-supervised learning for face sketch synthesis in the wild","authors":"Chaofeng Chen, Wei Liu, Xiao Tan, K Kwan-Yee, Wong","year":"2018","venue":"ACCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Goodfellow et al., 2014] Ian J. Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In NIPS, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hinton et al., 2014] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In NIPS Workshop, 2014.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2014","venue":"NIPS Workshop","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2018] Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Enhua Wu. Squeeze-and-excitation networks. In CVPR, 2018.","title":"Squeeze-and-excitation networks","authors":"Jie Hu, Li Shen, Samuel Albanie, Gang Sun, Enhua Wu","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang and Wang, 2017] Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selec- tivity transfer. arXiv:1707.01219, 2017.","title":"Like what you like: Knowledge distill via neuron selectivity transfer","authors":"Wang ; Huang, Zehao Huang, Naiyan Wang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Isola et al., 2017] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with con- ditional adversarial networks. In CVPR, pages 1125-1134, 2017.","title":"Image-to-image translation with conditional adversarial networks","authors":"Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A Efros","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In ICLR, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Lei Kingma, Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liang Chang et al., 2010] Mingquan Zhou Liang Chang, Yanjun Han, and Xiaoming Deng. Face sketch synthe- sis via sparse representation. In ICPR, pages 2146-2149, 2010.","title":"Face sketch synthesis via sparse representation","authors":"Liang Chang","year":"2010","venue":"ICPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2005] Qingshan Liu, Xiaoou Tang, Hongliang Jin, Hanqing Lu, and Songde Ma. A nonlinear approach for face sketch synthesis and recognition. In CVPR, pages 1005-1010, 2005.","title":"A nonlinear approach for face sketch synthesis and recognition","authors":"Qingshan Liu, Xiaoou Tang, Hongliang Jin, Hanqing Lu, Songde Ma","year":"2005","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Martinez and Benavente, 1998] A. M. Martinez and Robert Benavente. The ar face database. Technical report, CVC Technical Report #24, 1998.","title":"The ar face database","authors":"A M Martinez, Robert Benavente","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Messer et al., 1999] Kieron Messer, Jiri Matas, J. Kittler, J. Luettin, and G. Maitre. Xm2vtsdb: The extended m2vts database. In AVBPA, pages 72-77, 1999.","title":"Xm2vtsdb: The extended m2vts database","authors":"Kieron Messer, Jiri Matas, J Kittler, J Luettin, G Maitre","year":"1999","venue":"AVBPA","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Phillips et al., 2000] P. Jonathon Phillips, Hyeonjoon Moon, Syed A. Rizvi, and Patrick J. Rauss. The feret evaluation methodology for face recognition algorithms. TPAMI, 22(10):1090-1104, 2000.","title":"The feret evaluation methodology for face recognition algorithms","authors":"P, Jonathon Phillips, Hyeonjoon Moon, A Syed, Patrick J Rizvi, Rauss","year":"2000","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556, 2014.","title":"Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition","authors":"Zisserman Simonyan","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tang and Wang, 2003] Xiaoou Tang and Xiaogang Wang. Face sketch synthesis and recognition. In ICCV, pages 687-694, 2003.","title":"Face sketch synthesis and recognition","authors":"Wang ; Tang, Xiaoou Tang, Xiaogang Wang","year":"2003","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tang and Wang, 2009] Xiaoou Tang and Xiaogang Wang. Face photo-sketch synthesis and recognition. TPAMI, 31(11):1955-1967, November 2009.","title":"Face photo-sketch synthesis and recognition","authors":"Wang ; Tang, Xiaoou Tang, Xiaogang Wang","year":"2009","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2004] Zhou Wang, Alan Conrad Bovik, Hamid Rahim Sheikh, and Eero P. Simoncelli. Image quality assessment: From error visibility to structural similarity. TIP, 13(4):600-612, 2004.","title":"Image quality assessment: From error visibility to structural similarity","authors":"Zhou Wang, Alan Conrad Bovik, Hamid Rahim Sheikh, Eero P Simoncelli","year":"2004","venue":"TIP","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2014] Nannan Wang, Dacheng Tao, Xinbo Gao, Xuelong Li, and Jie Li. A comprehensive survey to face hallucination. IJCV, 106(1):9-30, January 2014.","title":"A comprehensive survey to face hallucination","authors":"Nannan Wang, Dacheng Tao, Xinbo Gao, Xuelong Li, Jie Li","year":"2014","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2018] Lidan Wang, Vishwanath A. Sindagi, and Vishal M. Patel. High-quality facial photo-sketch synthesis using multi-adversarial networks. In FG 2018, pages 83-90, 2018.","title":"High-quality facial photo-sketch synthesis using multi-adversarial networks","authors":"Lidan Wang, A Vishwanath, Vishal M Sindagi, Patel","year":"2018","venue":"FG 2018","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xu et al., 2018] Zheng Xu, Yen-Chang Hsu, and Jiawei Huang. Training shallow and thin networks for accelera- tion via knowledge distillation with conditional adversarial networks. arXiv:1709.00513v2, 2018.","title":"Training shallow and thin networks for acceleration via knowledge distillation with conditional adversarial networks","authors":"Zheng Xu, Yen-Chang Hsu, Jiawei Huang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yu et al., 2018] Jun Yu, Shengjie Shi, Fei Gao, Dacheng Tao, and Qingming Huang. Composition-aided face photo-sketch synthesis. arXiv:1712.00899v3, 2018.","title":"Composition-aided face photo-sketch synthesis","authors":"Jun Yu, Shengjie Shi, Fei Gao, Dacheng Tao, Qingming Huang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zagoruyko and Komodakis, 2017] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Im- proving the performance of convolutional neural networks via attention transfer. In ICLR, 2017.","title":"Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer","authors":"Komodakis Zagoruyko, Sergey Zagoruyko, Nikos Komodakis","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2011a] Lin Zhang, Lei Zhang, Xuanqin Mou, and David Zhang. Fsim: A feature similarity index for image quality assessment. TIP, 20(8):2378-2386, 2011.","title":"Fsim: A feature similarity index for image quality assessment","authors":"Zhang","year":"2011","venue":"TIP","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2011b] Wei Zhang, Xiaogang Wang, and Xi- aoou Tang. Coupled information-theoretic encoding for face photo-sketch recognition. In CVPR, pages 513-520, 2011.","title":"Coupled information-theoretic encoding for face photo-sketch recognition","authors":"Zhang","year":"2011","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2015] Liliang Zhang, Liang Lin, Xian Wu, Shengyong Ding, and Lei Zhang. End-to-end photo-sketch generation via fully convolutional representation learning. In ACM ICMR, pages 627-634, 2015.","title":"End-to-end photo-sketch generation via fully convolutional representation learning","authors":"Liliang Zhang, Liang Lin, Xian Wu, Shengyong Ding, Lei Zhang","year":"2015","venue":"ACM ICMR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhou et al., 2012] Hao Zhou, Zhanghui Kuang, and Kwan- Yee K. Wong. Markov weight fields for face sketch syn- thesis. In CVPR, pages 1091-1097, 2012.","title":"Markov weight fields for face sketch synthesis","authors":"Hao Zhou, Zhanghui Kuang, K Kwan-Yee, Wong","year":"2012","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhu et al., 2017a] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image transla- tion using cycle-consistent adversarial networks. In CVPR, pages 2223-2232, 2017.","title":"Unpaired image-to-image translation using cycle-consistent adversarial networks","authors":"Zhu","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhu et al., 2017b] Mingrui Zhu, Nannan Wang, Xinbo Gao, and Jie Li. Deep graphical feature learning for face sketch synthesis. In IJCAI, pages 3574-3580, 2017.","title":"Deep graphical feature learning for face sketch synthesis","authors":"Zhu","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/500"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}