{"type":"ScholarlyArticle","title":"Generating Person Images with Appearance-aware Pose Stylizer","authors":[{"name":"Siyu Huang","slug":"siyu-huang","given":"Siyu","family":"Huang","affiliation":"Baidu Research"},{"name":"Haoyi Xiong","slug":"haoyi-xiong","given":"Haoyi","family":"Xiong","affiliation":"Baidu Research"},{"name":"Zhi-Qi Cheng","slug":"zhi-qi-cheng","given":"Zhi-Qi","family":"Cheng","affiliation":"Carnegie Mellon University"},{"name":"Qingzhong Wang","slug":"qingzhong-wang","given":"Qingzhong","family":"Wang","affiliation":"City University of Hong Kong"},{"name":"Xingran Zhou","slug":"xingran-zhou","given":"Xingran","family":"Zhou","affiliation":"Zhejiang University"},{"name":"Bihan Wen","slug":"bihan-wen","given":"Bihan","family":"Wen","affiliation":"Nanyang Technological University"},{"name":"Jun Huan","slug":"jun-huan","given":"Jun","family":"Huan","affiliation":"Styling AI"},{"name":"Dejing Dou","slug":"dejing-dou","given":"Dejing","family":"Dou","affiliation":"Baidu Research"}],"abstract":"Generation of high-quality person images is challenging, due to the sophisticated entanglements among image factors, e.g., appearance, pose, foreground, background, local details, global structures, etc. In this paper, we present a novel end-to-end framework to generate realistic person images based on given person poses and appearances. The core of our framework is a novel generator called Appearance-aware Pose Stylizer (APS) which generates human images by coupling the target pose with the conditioned person appearance progressively. The framework is highly flexible and controllable by effectively decoupling various complex person image factors in the encoding phase, followed by re-coupling them in the decoding phase. In addition, we present a new normalization method named adaptive patch normalization, which enables region-specific normalization and shows a good performance when adopted in person image generation model. Experiments on two benchmark datasets show that our method is capable of generating visually appealing and realistic-looking results using arbitrary image and pose inputs.","doi":"10.24963/ijcai.2020/87","url":"https://ijcai.org/proceedings/2020/87","pdf":"https://ijcai.org/proceedings/2020/0087.pdf","markdown":"https://ijcai.org/proceedings/2020/87.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/87","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":87,"pages":{"first":623,"last":629},"keywords":["Computer Vision: Other","Machine Learning: Deep Generative Models"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24014/file_24014.m3u8","length_sec":788,"provider":"hls"}],"references":[{"position":1,"raw":"[Andriluka et al., 2014] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estimation: New benchmark and state of the art analysis. In CVPR, 2014.","title":"2d human pose estimation: New benchmark and state of the art analysis","authors":"Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, Bernt Schiele","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cao et al., 2017] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In CVPR, 2017.","title":"Realtime multi-person 2d pose estimation using part affinity fields","authors":"Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chan et al., 2019] Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros. Everybody dance now. In ICCV, 2019.","title":"Everybody dance now","authors":"Caroline Chan, Shiry Ginosar, Tinghui Zhou, Alexei A Efros","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dong et al., 2019] Haoye Dong, Xiaodan Liang, Xiaohui Shen, Bochao Wang, Hanjiang Lai, Jia Zhu, Zhiting Hu, and Jian Yin. Towards multi-pose guided virtual try-on network. In ICCV, 2019.","title":"Towards multi-pose guided virtual try-on network","authors":"Haoye Dong, Xiaodan Liang, Xiaohui Shen, Bochao Wang, Hanjiang Lai, Jia Zhu, Zhiting Hu, Jian Yin","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Esser et al., 2018] Patrick Esser, Ekaterina Sutter, and Björn Om- mer. A variational u-net for conditional appearance and shape generation. In CVPR, 2018.","title":"A variational u-net for conditional appearance and shape generation","authors":"Patrick Esser, Ekaterina Sutter, Björn Ommer","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Han et al., 2018] Xintong Han, Zuxuan Wu, Zhe Wu, Ruichi Yu, and Larry S Davis. Viton: An image-based virtual try-on net- work. In CVPR, 2018.","title":"Viton: An image-based virtual try-on network","authors":"Han","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hinton et al., 2012] Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. Im- proving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012.","title":"Improving neural networks by preventing co-adaptation of feature detectors","authors":"Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, Ruslan R Salakhutdinov","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang and Belongie, 2017] Xun Huang and Serge Belongie. Ar- bitrary style transfer in real-time with adaptive instance normal- ization. In ICCV, 2017.","title":"Arbitrary style transfer in real-time with adaptive instance normalization","authors":"Belongie Huang, Xun Huang, Serge Belongie","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Johnson et al., 2016] Justin Johnson, Alexandre Alahi, and Li Fei- Fei. Perceptual losses for real-time style transfer and super- resolution. In ECCV, pages 694-711, 2016.","title":"Perceptual losses for real-time style transfer and superresolution","authors":"Justin Johnson, Alexandre Alahi, Li Fei-Fei","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Karras et al., 2018] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved qual- ity, stability, and variation. In ICLR, 2018.","title":"Progressive growing of gans for improved quality, stability, and variation","authors":"Tero Karras, Timo Aila, Samuli Laine, Jaakko Lehtinen","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Karras et al., 2019] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial net- works. In CVPR, 2019.","title":"A style-based generator architecture for generative adversarial networks","authors":"Tero Karras, Samuli Laine, Timo Aila","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lassner et al., 2017] Christoph Lassner, Gerard Pons-Moll, and Peter V Gehler. A generative model of people in clothing. In ICCV, 2017.","title":"A generative model of people in clothing","authors":"Christoph Lassner, Gerard Pons-Moll, Peter V Gehler","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2016] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recog- nition and retrieval with rich annotations. In CVPR, 2016.","title":"Deepfashion: Powering robust clothes recognition and retrieval with rich annotations","authors":"Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, Xiaoou Tang","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ma et al., 2017] Liqian Ma, Xu Jia, Qianru Sun, Bernt Schiele, Tinne Tuytelaars, and Luc Van Gool. Pose guided person image generation. In NIPS, 2017.","title":"Pose guided person image generation","authors":"Liqian Ma, Xu Jia, Qianru Sun, Bernt Schiele, Tinne Tuytelaars, Luc Van Gool","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ma et al., 2018] Liqian Ma, Qianru Sun, Stamatios Georgoulis, Luc Van Gool, Bernt Schiele, and Mario Fritz. Disentangled per- son image generation. In CVPR, 2018.","title":"Disentangled person image generation","authors":"Ma","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mirza and Osindero, 2014] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014.","title":"Conditional generative adversarial nets","authors":"Mehdi Mirza, Simon Osindero","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Neverova et al., 2018] Natalia Neverova, Riza Alp Guler, and Ia- sonas Kokkinos. Dense pose transfer. In ECCV, 2018.","title":"Dense pose transfer","authors":"Natalia Neverova, Riza Alp Guler, Iasonas Kokkinos","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Park et al., 2019] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially- adaptive normalization. In CVPR, 2019.","title":"Semantic image synthesis with spatiallyadaptive normalization","authors":"Park","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Paszke et al., 2017] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.","title":"Automatic differentiation in PyTorch","authors":"Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Devito, Zeming Lin, Alban Desmaison, Luca Antiga, Adam Lerer","year":"2017","venue":"NIPS Autodiff Workshop","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Salimans et al., 2016] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NIPS, 2016.","title":"Improved techniques for training gans","authors":"Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Siarohin et al., 2018] Aliaksandr Siarohin, Enver Sangineto, Stéphane Lathuilière, and Nicu Sebe. Deformable gans for pose-based human image generation. In CVPR, 2018.","title":"Deformable gans for pose-based human image generation","authors":"Aliaksandr Siarohin, Enver Sangineto, Stéphane Lathuilière, Nicu Sebe","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Song et al., 2019] Sijie Song, Wei Zhang, Jiaying Liu, and Tao Mei. Unsupervised person image generation with semantic pars- ing transformation. In CVPR, 2019.","title":"Unsupervised person image generation with semantic parsing transformation","authors":"Sijie Song, Wei Zhang, Jiaying Liu, Tao Mei","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Walker et al., 2017] Jacob Walker, Kenneth Marino, Abhinav Gupta, and Martial Hebert. The pose knows: Video forecasting by generating pose futures. In ICCV, 2017.","title":"The pose knows: Video forecasting by generating pose futures","authors":"Jacob Walker, Kenneth Marino, Abhinav Gupta, Martial Hebert","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli, et al. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 13(4):600-612, 2004.","title":"Image quality assessment: from error visibility to structural similarity","authors":"Zhou Wang, Alan C Bovik, Eero P Hamid R Sheikh, Simoncelli","year":"2004","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2018a] Bochao Wang, Huabin Zheng, Xiaodan Liang, Yimin Chen, Liang Lin, and Meng Yang. Toward characteristic- preserving image-based virtual try-on network. In ECCV, 2018.","title":"Toward characteristicpreserving image-based virtual try-on network","authors":"Wang","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2018b] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video- to-video synthesis. arXiv preprint arXiv:1808.06601, 2018.","title":"Videoto-video synthesis","authors":"Wang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wei et al., 2018] Longhui Wei, Shiliang Zhang, Wen Gao, and Qi Tian. Person transfer gan to bridge domain gap for person re-identification. In CVPR, 2018.","title":"Person transfer gan to bridge domain gap for person re-identification","authors":"Longhui Wei, Shiliang Zhang, Wen Gao, Qi Tian","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yildirim et al., 2019] Gokhan Yildirim, Nikolay Jetchev, Roland Vollgraf, and Urs Bergmann. Generating high-resolution fashion model images wearing custom outfits. In ICCV Workshops, 2019.","title":"Generating high-resolution fashion model images wearing custom outfits","authors":"Gokhan Yildirim, Nikolay Jetchev, Roland Vollgraf, Urs Bergmann","year":"2019","venue":"ICCV Workshops","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zanfir et al., 2018] Mihai Zanfir, Alin-Ionut Popa, Andrei Zanfir, and Cristian Sminchisescu. Human appearance transfer. In CVPR, 2018.","title":"Human appearance transfer","authors":"Mihai Zanfir, Alin-Ionut Popa, Andrei Zanfir, Cristian Sminchisescu","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhao et al., 2018] Bo Zhao, Xiao Wu, Zhi-Qi Cheng, Hao Liu, Ze- qun Jie, and Jiashi Feng. Multi-view image generation from a single-view. In ACM MM, 2018.","title":"Multi-view image generation from a single-view","authors":"Bo Zhao, Xiao Wu, Zhi-Qi Cheng, Hao Liu, Zequn Jie, Jiashi Feng","year":"2018","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zheng et al., 2015] Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. Scalable person re- identification: A benchmark. In ICCV, 2015.","title":"Scalable person reidentification: A benchmark","authors":"Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, Qi Tian","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zheng et al., 2017] Zhedong Zheng, Liang Zheng, and Yi Yang. Unlabeled samples generated by gan improve the person re- identification baseline in vitro. In ICCV, 2017.","title":"Unlabeled samples generated by gan improve the person reidentification baseline in vitro","authors":"Zheng","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhou et al., 2019] Xingran Zhou, Siyu Huang, Bin Li, Yingming Li, Jiachen Li, and Zhongfei Zhang. Text guided person image synthesis. In CVPR, 2019.","title":"Text guided person image synthesis","authors":"Xingran Zhou, Siyu Huang, Bin Li, Yingming Li, Jiachen Li, Zhongfei Zhang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhu et al., 2017] Shizhan Zhu, Raquel Urtasun, Sanja Fidler, Dahua Lin, and Chen Change Loy. Be your own prada: Fash- ion synthesis with structural coherence. In ICCV, 2017.","title":"Be your own prada: Fashion synthesis with structural coherence","authors":"Zhu","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhu et al., 2019] Zhen Zhu, Tengteng Huang, Baoguang Shi, Miao Yu, Bofei Wang, and Xiang Bai. Progressive pose attention trans- fer for person image generation. In CVPR, 2019.","title":"Progressive pose attention transfer for person image generation","authors":"Zhen Zhu, Tengteng Huang, Baoguang Shi, Miao Yu, Bofei Wang, Xiang Bai","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}