{"type":"ScholarlyArticle","title":"Real-World Automatic Makeup via Identity Preservation Makeup Net","authors":[{"name":"Zhikun Huang","slug":"zhikun-huang","given":"Zhikun","family":"Huang","affiliation":"Hangzhou Dianzi University"},{"name":"Zhedong Zheng","slug":"zhedong-zheng","given":"Zhedong","family":"Zheng","affiliation":"University of Technology Sydney; Baidu Research"},{"name":"Chenggang Yan","slug":"chenggang-yan","given":"Chenggang","family":"Yan","affiliation":"Hangzhou Dianzi University"},{"name":"Hongtao Xie","slug":"hongtao-xie","given":"Hongtao","family":"Xie","affiliation":"University of Science and Technology of China"},{"name":"Yaoqi Sun","slug":"yaoqi-sun","given":"Yaoqi","family":"Sun","affiliation":"Hangzhou Dianzi University; University of Technology Sydney ."},{"name":"Jianzhong Wang","slug":"jianzhong-wang","given":"Jianzhong","family":"Wang","affiliation":"Hangzhou Dianzi University"},{"name":"Jiyong Zhang","slug":"jiyong-zhang","given":"Jiyong","family":"Zhang","affiliation":"Hangzhou Dianzi University"}],"abstract":"This paper focuses on the real-world automatic makeup problem. Given one non-makeup target image and one reference image, the automatic makeup is to generate one face image, which maintains the original identity with the makeup style in the reference image. In the real-world scenario, face makeup task demands a robust system against the environmental variants.  The two main challenges in real-world face makeup could be summarized as follow: first, the background in real-world images is complicated. The previous methods are prone to change the style of background as well;  second, the foreground faces are also easy to be affected. For instance, the ``heavy'' makeup may lose the discriminative information of the original identity. To address these two challenges, we introduce a new makeup model, called Identity Preservation Makeup Net (IPM-Net), which preserves not only the background but the critical patterns of the original identity. Specifically, we disentangle the face images to two different information codes, i.e., identity content code and makeup style code. When inference, we only need to change the makeup style code to generate various makeup images of the target person. In the experiment, we show the proposed method achieves not only better accuracy in both realism (FID) and diversity (LPIPS) in the test set, but also works well on the real-world images collected from the Internet.","doi":"10.24963/ijcai.2020/91","url":"https://ijcai.org/proceedings/2020/91","pdf":"https://ijcai.org/proceedings/2020/0091.pdf","markdown":"https://ijcai.org/proceedings/2020/91.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/91","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":91,"pages":{"first":652,"last":658},"keywords":["Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25066/file_25066.m3u8","length_sec":296,"provider":"hls"}],"references":[{"position":1,"raw":"[Cao et al., 2018] Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman. Vggface2: A dataset for recog- nising faces across pose and age. In International Conference on Automatic Face & Gesture Recognition (FG 2018), 2018.","title":"Vggface2: A dataset for recognising faces across pose and age","authors":"Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, Andrew Zisserman","year":"2018","venue":"International Conference on Automatic Face & Gesture Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chang et al., 2018] Huiwen Chang, Jingwan Lu, Fisher Yu, and Adam Finkelstein. Pairedcyclegan: Asymmetric style transfer for applying and removing makeup. In CVPR, 2018.","title":"Pairedcyclegan: Asymmetric style transfer for applying and removing makeup","authors":"Huiwen Chang, Jingwan Lu, Fisher Yu, Adam Finkelstein","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2019] Hung-Jen Chen, Ka-Ming Hui, Szu-Yu Wang, Li-Wu Tsao, Hong-Han Shuai, and Wen-Huang Cheng. Beau- tyglow: On-demand makeup transfer framework with reversible generative network. In CVPR, 2019.","title":"Beautyglow: On-demand makeup transfer framework with reversible generative network","authors":"Hung-Jen Chen, Ka-Ming Hui, Szu-Yu Wang, Li-Wu Tsao, Hong-Han Shuai, Wen-Huang Cheng","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical im- age database. In CVPR, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"NeurlPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Guo and Sim, 2009] Dong Guo and Terence Sim. Digital face makeup by example. In CVPR, 2009.","title":"Digital face makeup by example","authors":"Dong Guo, Terence Sim","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Heusel et al., 2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurlPS, 2017.","title":"Gans trained by a two time-scale update rule converge to a local nash equilibrium","authors":"Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Sepp Hochreiter","year":"2017","venue":"NeurlPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang and Belongie, 2017] Xun Huang and Serge Belongie. Ar- bitrary style transfer in real-time with adaptive instance normal- ization. In ICCV, 2017.","title":"Arbitrary style transfer in real-time with adaptive instance normalization","authors":"Belongie Huang, Xun Huang, Serge Belongie","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2018] Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to-image trans- lation. In ECCV, 2018.","title":"Multimodal unsupervised image-to-image translation","authors":"Xun Huang, Ming-Yu Liu, Serge Belongie, Jan Kautz","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by re- ducing internal covariate shift. arXiv:1502.03167, 2015.","title":"Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jiang et al., 2019] Wentao Jiang, Si Liu, Chen Gao, Jie Cao, Ran He, Jiashi Feng, and Shuicheng Yan. Psgan: Pose-robust spatial- aware gan for customizable makeup transfer. arXiv:1909.06956, 2019.","title":"Psgan: Pose-robust spatialaware gan for customizable makeup transfer","authors":"Wentao Jiang, Si Liu, Chen Gao, Jie Cao, Ran He, Jiashi Feng, Shuicheng Yan","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jin et al., 2019] Xin Jin, Rui Han, Ning Ning, Xiaodong Li, and Xiaokun Zhang. Facial makeup transfer combining illumination transfer. IEEE Access, 2019.","title":"Facial makeup transfer combining illumination transfer","authors":"Xin Jin, Rui Han, Ning Ning, Xiaodong Li, Xiaokun Zhang","year":"2019","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lai et al., 2018] Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. Fast and accurate image super-resolution with deep laplacian pyramid networks. TPAMI, 2018.","title":"Fast and accurate image super-resolution with deep laplacian pyramid networks","authors":"Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, Ming-Hsuan Yang","year":"2018","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Leyvand et al., 2006] Tommer Leyvand, Daniel Cohen-Or, Gideon Dror, and Dani Lischinski. Digital face beautification. In ACM Siggraph, 2006.","title":"Digital face beautification","authors":"Tommer Leyvand, Daniel Cohen-Or, Gideon Dror, Dani Lischinski","year":"2006","venue":"ACM Siggraph","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2015] Chen Li, Kun Zhou, and Stephen Lin. Simulating makeup through physics-based manipulation of intrinsic image layers. In CVPR, 2015.","title":"Simulating makeup through physics-based manipulation of intrinsic image layers","authors":"Chen Li, Kun Zhou, Stephen Lin","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2018] Tingting Li, Ruihe Qian, Chao Dong, Si Liu, Qiong Yan, Wenwu Zhu, and Liang Lin. Beautygan: Instance- level facial makeup transfer with deep generative adversarial net- work. In ACM MM, 2018.","title":"Beautygan: Instancelevel facial makeup transfer with deep generative adversarial network","authors":"Tingting Li, Ruihe Qian, Chao Dong, Si Liu, Qiong Yan, Wenwu Zhu, Liang Lin","year":"2018","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2016] Si Liu, Xinyu Ou, Ruihe Qian, Wei Wang, and Xiaochun Cao. Makeup like a superstar: Deep localized makeup transfer network. arXiv:1604.07102, 2016.","title":"Makeup like a superstar: Deep localized makeup transfer network","authors":"Si Liu, Xinyu Ou, Ruihe Qian, Wei Wang, Xiaochun Cao","year":"2016","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2016/365"},{"position":20,"raw":"[Luo et al., 2005] Jiebo Luo, Andreas E Savakis, and Amit Sing- hal. A bayesian network-based framework for semantic image understanding. Pattern recognition, 38(6):919-934, 2005.","title":"and Amit Singhal. A bayesian network-based framework for semantic image understanding","authors":"Luo","year":"2005","venue":"Pattern recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Paszke et al., 2017] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.","title":"Automatic differentiation in pytorch","authors":"Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Devito, Zeming Lin, Alban Desmaison, Luca Antiga, Adam Lerer","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Pu et al., 2008] Yifei Pu, Weixing Wang, Jiliu Zhou, Yiyang Wang, and Huading Jia. Fractional differential approach to de- tecting textural features of digital image and its fractional differ- ential filter implementation. Science in China Series F: Informa- tion Sciences, 51(9):1319-1339, 2008.","title":"Fractional differential approach to detecting textural features of digital image and its fractional differential filter implementation","authors":"Yifei Pu, Weixing Wang, Jiliu Zhou, Yiyang Wang, Huading Jia","year":"2008","venue":"Science in China Series F: Information Sciences","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Senft and Baym, 2015] Theresa M Senft and Nancy K Baym. What does the selfie say? investigating a global phenomenon. International journal of communication, 2015.","title":"What does the selfie say? investigating a global phenomenon","authors":"M Theresa, Nancy K Senft, Baym","year":"2015","venue":"International journal of communication","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shashua and Riklin-Raviv, 2001] Amnon Shashua and Tammy Riklin-Raviv. The quotient image: Class-based re-rendering and recognition with varying illuminations. TPAMI, 2001.","title":"Amnon Shashua and Tammy Riklin-Raviv. The quotient image: Class-based re-rendering and recognition with varying illuminations","authors":"Riklin-Raviv Shashua","year":"2001","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sorokowski et al., 2015] Piotr Sorokowski, Agnieszka Sorokowska, Anna Oleszkiewicz, Tomasz Frackowiak, Anna Huk, and Katarzyna Pisanski. Selfie posting behaviors are asso- ciated with narcissism among men. Personality and Individual Differences, 2015.","title":"Selfie posting behaviors are associated with narcissism among men","authors":"Piotr Sorokowski, Agnieszka Sorokowska, Anna Oleszkiewicz, Tomasz Frackowiak, Anna Huk, Katarzyna Pisanski","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Szegedy et al., 2016] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.","title":"Rethinking the inception architecture for computer vision","authors":"Szegedy","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tang et al., 2018] Wei Tang, Teng Li, Fudong Nian, and Meng Wang. Mscgan: Multi-scale conditional generative adversarial networks for person image generation. arXiv:1810.08534, 2018.","title":"Mscgan: Multi-scale conditional generative adversarial networks for person image generation","authors":"Wei Tang, Teng Li, Fudong Nian, Meng Wang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tong et al., 2007] Wai-Shun Tong, Chi-Keung Tang, Michael S Brown, and Ying-Qing Xu. Example-based cosmetic transfer. In 15th Pacific Conference on Computer Graphics and Applications (PG'07), 2007.","title":"Example-based cosmetic transfer","authors":"Wai-Shun Tong, Chi-Keung Tang, Michael S Brown, Ying-Qing Xu","year":"2007","venue":"15th Pacific Conference on Computer Graphics and Applications (PG'07)","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yan et al., 2019] Chenggang Yan, Liang Li, Chunjie Zhang, Bing- tao Liu, Yongdong Zhang, and Qionghai Dai. Cross-modality bridging and knowledge transferring for image understanding. IEEE Transactions on Multimedia, 21(10):2675-2685, 2019.","title":"Cross-modality bridging and knowledge transferring for image understanding","authors":"Chenggang Yan, Liang Li, Chunjie Zhang, Bingtao Liu, Yongdong Zhang, Qionghai Dai","year":"2019","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yu et al., 2018] Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. Bisenet: Bilateral segmentation network for real-time semantic segmentation. In ECCV, 2018.","title":"Bisenet: Bilateral segmentation network for real-time semantic segmentation","authors":"Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, Nong Sang","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.","title":"The unreasonable effectiveness of deep features as a perceptual metric","authors":"Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, Oliver Wang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2019] Honglun Zhang, Wenqing Chen, Hao He, and Yaohui Jin. Disentangled makeup transfer with generative adver- sarial network. arXiv:1907.01144, 2019.","title":"Disentangled makeup transfer with generative adversarial network","authors":"Zhang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zheng et al., 2019] Zhedong Zheng, Xiaodong Yang, Zhiding Yu, Liang Zheng, Yi Yang, and Jan Kautz. Joint discriminative and generative learning for person re-identification. In CVPR, 2019.","title":"Joint discriminative and generative learning for person re-identification","authors":"Zheng","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhu et al., 2017] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, 2017.","title":"Unpaired image-to-image translation using cycle-consistent adversarial networks","authors":"Zhu","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}