{"type":"ScholarlyArticle","title":"CR-GAN: Learning Complete Representations for Multi-view Generation","authors":[{"name":"Yu Tian","slug":"yu-tian","given":"Yu","family":"Tian","affiliation":"Rutgers University"},{"name":"Xi Peng","slug":"xi-peng","given":"Xi","family":"Peng","affiliation":"Rutgers University"},{"name":"Long Zhao","slug":"long-zhao","given":"Long","family":"Zhao","affiliation":"Rutgers University"},{"name":"Shaoting Zhang","slug":"shaoting-zhang","given":"Shaoting","family":"Zhang","affiliation":"University of North Carolina at Charlotte"},{"name":"Dimitris N. Metaxas","slug":"dimitris-n-metaxas","given":"Dimitris N.","family":"Metaxas","affiliation":"Rutgers University"}],"abstract":"Generating multi-view images from a single-view input is an important yet challenging problem. It has broad applications in vision, graphics, and robotics. Our study indicates that the widely-used generative adversarial network (GAN) may learn ?incomplete? representations due to the single-pathway framework: an encoder-decoder network followed by a discriminator network.We propose CR-GAN to address this problem. In addition to the single reconstruction path, we introduce a generation sideway to maintain the completeness of the learned embedding space. The two learning paths collaborate and compete in a parameter-sharing manner, yielding largely improved generality to ?unseen? dataset. More importantly, the two-pathway framework makes it possible to combine both labeled and unlabeled data for self-supervised learning, which further enriches the embedding space for realistic generations. We evaluate our approach on a wide range of datasets. The results prove that CR-GAN significantly outperforms state-of-the-art methods, especially when generating from ?unseen? inputs in wild conditions.","doi":"10.24963/ijcai.2018/131","url":"https://ijcai.org/proceedings/2018/131","pdf":"https://ijcai.org/proceedings/2018/0131.pdf","markdown":"https://ijcai.org/proceedings/2018/131.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/131","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":131,"pages":{"first":942,"last":948},"keywords":["Humans and AI: Human-Computer Interaction","Robotics: Robotics and Vision","Computer Vision: Biometrics, Face and Gesture Recognition","Computer Vision: 2D and 3D Computer Vision"],"videos":[],"references":[{"position":1,"raw":"[Arjovsky et al., 2017] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein Generative Adversarial Net- works. In ICML, pages 214-223, 2017.","title":"Wasserstein Generative Adversarial Networks","authors":"Arjovsky","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Blanz and Vetter, 1999] Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3D faces. In SIG- GRAPH, pages 187-194, 1999.","title":"A morphable model for the synthesis of 3D faces","authors":"Vetter Blanz, Volker Blanz, Thomas Vetter","year":"1999","venue":"SIG-GRAPH","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Donahue et al., 2017] Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. In ICLR, 2017.","title":"Adversarial feature learning","authors":"Jeff Donahue, Philipp Krähenbühl, Trevor Darrell","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dumoulin et al., 2017] Vincent Dumoulin, Ishmael Belg- hazi, Ben Poole, Alex Lamb, Martin Arjovsky, Olivier Mastropietro, and Aaron Courville. Adversarially learned inference. In ICLR, 2017.","title":"Adversarially learned inference","authors":"Ishmael Vincent Dumoulin, Ben Belghazi, Alex Poole, Martin Lamb, Olivier Arjovsky, Aaron Mastropietro, Courville","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fan et al., 2016] Miao Fan, Qiang Zhou, and Thomas Fang Zheng. Learning embedding representations for knowledge inference on imperfect and incomplete repositories. In Web Intelligence (WI), pages 42-48, 2016.","title":"Learning embedding representations for knowledge inference on imperfect and incomplete repositories","authors":"Qiang Miao Fan, Thomas Zhou, Zheng Fang","year":"2016","venue":"Web Intelligence (WI)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fischler and Bolles, 1981] Martin A Fischler and Robert C Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM, 24(6):381-395, 1981.","title":"Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography","authors":"A Martin, Robert C Fischler, Bolles","year":"1981","venue":"Commun. ACM","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sher- jil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gross et al., 2010] Ralph Gross, Iain Matthews, Jeffrey Cohn, Takeo Kanade, and Simon Baker. Multi-pie. Image Vision Computer, 28(5):807-813, 2010.","title":"Multi-pie","authors":"Ralph Gross, Iain Matthews, Jeffrey Cohn, Takeo Kanade, Simon Baker","year":"2010","venue":"Image Vision Computer","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gulrajani et al., 2017] Ishaan Gulrajani, Faruk Ahmed, Mar- tin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved Training of Wasserstein GANs. In NIPS, pages 5769-5779, 2017.","title":"Improved Training of Wasserstein GANs","authors":"Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Aaron Vincent Dumoulin, Courville","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hardoon et al., 2004] David R Hardoon, Sandor Szedmak, and John Shawe-Taylor. Canonical correlation analysis: An overview with application to learning methods. Neural computation, 16(12):2639-2664, 2004.","title":"Canonical correlation analysis: An overview with application to learning methods","authors":"Sandor David R Hardoon, John Szedmak, Shawe-Taylor","year":"2004","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hinton et al., 2011] Geoffrey E Hinton, Alex Krizhevsky, and Sida D Wang. Transforming auto-encoders. In ICANN, pages 44-51, 2011.","title":"Transforming auto-encoders","authors":"Geoffrey E Hinton, Alex Krizhevsky, Sida D Wang","year":"2011","venue":"ICANN","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Huang et al., 2017] Rui Huang, Shu Zhang, Tianyu Li, and Ran He. Beyond face rotation: Global and local perception gan for photorealistic and identity preserving frontal view synthesis. In ICCV, 2017.","title":"Beyond face rotation: Global and local perception gan for photorealistic and identity preserving frontal view synthesis","authors":"Rui Huang, Shu Zhang, Tianyu Li, Ran He","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kingma and Ba, 2015] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Klare et al., 2015] Brendan F Klare, Ben Klein, Emma Taborsky, Austin Blanton, Jordan Cheney, Kristen Allen, Patrick Grother, Alan Mah, and Anil K Jain. Pushing the frontiers of unconstrained face detection and recognition: Iarpa janus benchmark a. In CVPR, pages 1931-1939, 2015.","title":"Pushing the frontiers of unconstrained face detection and recognition: Iarpa janus benchmark a","authors":"Ben Brendan F Klare, Emma Klein, Austin Taborsky, Jordan Blanton, Kristen Cheney, Patrick Allen, Alan Grother, Anil K Mah, Jain","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2016] Yingming Li, Ming Yang, and Zhongfei Zhang. Multi-view representation learning: A survey from shallow methods to deep methods. arXiv preprint arXiv:1610.01206, 2016.","title":"Multi-view representation learning: A survey from shallow methods to deep methods","authors":"Li","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2015] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, pages 3730-3738, 2015.","title":"Deep learning face attributes in the wild","authors":"Ziwei Liu, Ping Luo, Xiaogang Wang, Xiaoou Tang","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Odena et al., 2017] Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxil- iary classifier gans. In ICML, 2017.","title":"Conditional image synthesis with auxiliary classifier gans","authors":"Odena","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Peng et al., 2015] Xi Peng, Junzhou Huang, Qiong Hu, Shaoting Zhang, Ahmed Elgammal, and Dimitris Metaxas. From circle to 3-sphere: Head pose estimation by instance parameterization. Computer Vision and Image Understand- ing, 136:92-102, 2015.","title":"From circle to 3-sphere: Head pose estimation by instance parameterization","authors":"Peng","year":"2015","venue":"Computer Vision and Image Understanding","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Peng et al., 2016] Xi Peng, Rogerio S Feris, Xiaoyu Wang, and Dimitris N Metaxas. A recurrent encoder-decoder network for sequential face alignment. In ECCV, pages 38-56, 2016.","title":"A recurrent encoder-decoder network for sequential face alignment","authors":"Xi Peng, S Rogerio, Xiaoyu Feris, Dimitris N Wang, Metaxas","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Peng et al., 2017] Xi Peng, Xiang Yu, Kihyuk Sohn, Dimitris N Metaxas, and Manmohan Chandraker. Reconstruction-based disentanglement for pose-invariant face recognition. In ICCV, 2017.","title":"Reconstruction-based disentanglement for pose-invariant face recognition","authors":"Xi Peng, Xiang Yu, Kihyuk Sohn, Dimitris N Metaxas, Manmohan Chandraker","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Radford et al., 2016] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR, 2016.","title":"Unsupervised representation learning with deep convolutional generative adversarial networks","authors":"Radford","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rezende et al., 2016] Danilo Jimenez Rezende, SM Ali Es- lami, Shakir Mohamed, Peter Battaglia, Max Jaderberg, and Nicolas Heess. Unsupervised learning of 3d structure from images. In NIPS, pages 4996-5004, 2016.","title":"Unsupervised learning of 3d structure from images","authors":"Danilo Jimenez Rezende, Ali Sm, Shakir Eslami, Peter Mohamed, Max Battaglia, Nicolas Jaderberg, Heess","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sagonas et al., 2013] Christos Sagonas, Georgios Tz- imiropoulos, Stefanos Zafeiriou, and Maja Pantic. 300 faces in-the-wild challenge: The first facial landmark localization challenge. In ICCVW, pages 397-403, 2013.","title":"300 faces in-the-wild challenge: The first facial landmark localization challenge","authors":"Christos Sagonas, Georgios Tzimiropoulos, Stefanos Zafeiriou, Maja Pantic","year":"2013","venue":"ICCVW","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Schroff et al., 2015] Florian Schroff, Dmitry Kalenichenko, and James Philbin. \"FaceNet\": A unified embedding for face recognition and clustering. In CVPR, pages 815-823, 2015.","title":"FaceNet\": A unified embedding for face recognition and clustering","authors":"Florian Schroff, Dmitry Kalenichenko, James Philbin","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tran et al., 2017] Luan Tran, Xi Yin, and Xiaoming Liu. Dis- entangled Representation Learning GAN for Pose-Invariant Face Recognition. In CVPR, 2017.","title":"Disentangled Representation Learning GAN for Pose-Invariant Face Recognition","authors":"Luan Tran, Xi Yin, Xiaoming Liu","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yan et al., 2016] Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee. Perspective transformer nets: Learning single-view 3D object reconstruction without 3D supervision. In NIPS, pages 1696-1704, 2016.","title":"Perspective transformer nets: Learning single-view 3D object reconstruction without 3D supervision","authors":"Yan","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yin et al., 2017] Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, and Manmohan Chandraker. Towards large-pose face frontalization in the wild. In ICCV, 2017.","title":"Towards large-pose face frontalization in the wild","authors":"Xiang Xi Yin, Kihyuk Yu, Xiaoming Sohn, Manmohan Liu, Chandraker","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhao et al., 2017] Bo Zhao, Xiao Wu, Zhi-Qi Cheng, Hao Liu, and Jiashi Feng. Multi-view image generation from a single-view. arXiv preprint arXiv:1704.04886, 2017.","title":"Multi-view image generation from a single-view","authors":"Bo Zhao, Xiao Wu, Zhi-Qi Cheng, Hao Liu, Jiashi Feng","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhou et al., 2016] Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei A Efros. View synthesis by appearance flow. In ECCV, pages 286-301, 2016.","title":"View synthesis by appearance flow","authors":"Zhou","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhu et al., 2013] Zhenyao Zhu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning identity-preserving face space. In ICCV, pages 113-120, 2013.","title":"Deep learning identity-preserving face space","authors":"Zhenyao Zhu, Ping Luo, Xiaogang Wang, Xiaoou Tang","year":"2013","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhu et al., 2014] Zhenyao Zhu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Multi-view perceptron: a deep model for learning face identity and view representations. In NIPS, pages 217-225, 2014.","title":"Multi-view perceptron: a deep model for learning face identity and view representations","authors":"Zhenyao Zhu, Ping Luo, Xiaogang Wang, Xiaoou Tang","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhu et al., 2016] X. Zhu, Z. Lei, X. Liu, H. Shi, and S.Z. Li. Face Alignment Across Large Poses: A 3D Solution. In CVPR, pages 146-155, 2016.","title":"Face Alignment Across Large Poses: A 3D Solution","authors":"Zhu","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}