{"type":"ScholarlyArticle","title":"Tag Disentangled Generative Adversarial Network for Object Image Re-rendering","authors":[{"name":"Chaoyue Wang","slug":"chaoyue-wang","given":"Chaoyue","family":"Wang","affiliation":"† Centre for Artificial Intelligence , School of Software , University of Technology Sydney , Australia"},{"name":"Chaohui Wang","slug":"chaohui-wang","given":"Chaohui","family":"Wang","affiliation":"‡ Université Paris-Est , LIGM (UMR 8049) , CNRS , ENPC , ESIEE Paris , UPEM , Marne-la-Vallée , France"},{"name":"Chang Xu","slug":"chang-xu","given":"Chang","family":"Xu","affiliation":"UBTech Sydney AI Institute , School of IT , FEIT , The University of Sydney , Australia"},{"name":"Dacheng Tao","slug":"dacheng-tao","given":"Dacheng","family":"Tao","affiliation":"UBTech Sydney AI Institute , School of IT , FEIT , The University of Sydney , Australia"}],"abstract":"In this paper, we propose a principled Tag Disentangled Generative Adversarial Networks (TD-GAN) for re-rendering new images for the object of interest from a single image of it by specifying multiple scene properties (such as viewpoint, illumination, expression, etc.). The whole framework consists of a disentangling network, a generative network, a tag mapping net, and a discriminative network, which are trained jointly based on a given set of images that are completely/partially tagged (i.e., supervised/semi-supervised setting). Given an input image, the disentangling network extracts disentangled and interpretable representations, which are then used to generate images by the generative network. In order to boost the quality of disentangled representations, the tag mapping net is integrated to explore the consistency between the image and its tags. Furthermore, the discriminative network is introduced to implement the adversarial training strategy for generating more realistic images. Experiments on two challenging datasets demonstrate the state-of-the-art performance of the proposed framework in the problem of interest.","doi":"10.24963/ijcai.2017/404","url":"https://ijcai.org/proceedings/2017/404","pdf":"https://ijcai.org/proceedings/2017/0404.pdf","markdown":"https://ijcai.org/proceedings/2017/404.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/404","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":404,"pages":{"first":2901,"last":2907},"keywords":["Machine Learning: Deep Learning","Machine Learning: Neural Networks"],"videos":[],"references":[{"position":1,"raw":"[Aubry et al., 2014] Mathieu Aubry, Daniel Maturana, Alexei A Efros, Bryan C Russell, and Josef Sivic. Seeing 3d chairs: exemplar part-based 2d-3d alignment using a large dataset of cad models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3762-3769, 2014.","title":"Seeing 3d chairs: exemplar part-based 2d-3d alignment using a large dataset of cad models","authors":"Aubry Mathieu, Daniel Maturana, Alexei A Efros, Bryan C Russell, Josef Sivic","year":"2014","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2013] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1828, 2013.","title":"Representation learning: A review and new perspectives","authors":"Yoshua Bengio, Aaron Courville, Pascal Vincent","year":"2013","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bengio et al., 2015] Yoshua Bengio, Ian J Goodfellow, and Aaron Courville. Deep learning. An MIT Press book in preparation. Draft chapters available at http://www. iro. umontreal. ca/ bengioy/dlbook, 2015.","title":"Deep learning. An MIT Press book in preparation","authors":"Yoshua Bengio, Ian J Goodfellow, Aaron Courville","year":null,"venue":null,"doi":null,"url":"http://www.iro.umontreal.ca/bengioy/dlbook,2015","internal_url":null},{"position":4,"raw":"[Bergstra et al., 2010] James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guil- laume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. Theano: A cpu and gpu math com- piler in python. In Proc. 9th Python in Science Conf, pages 1-7, 2010.","title":"Theano: A cpu and gpu math compiler in python","authors":"James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, Yoshua Bengio","year":"2010","venue":"Proc. 9th Python in Science Conf","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2016] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Info- gan: Interpretable representation learning by information maximizing generative adversarial nets. arXiv preprint arXiv:1606.03657, 2016.","title":"Infogan: Interpretable representation learning by information maximizing generative adversarial nets","authors":"Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, Pieter Abbeel","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cheung et al., 2014] Brian Cheung, Jesse A Livezey, Ar- jun K Bansal, and Bruno A Olshausen. Discovering hid- den factors of variation in deep networks. arXiv preprint arXiv:1412.6583, 2014.","title":"Discovering hidden factors of variation in deep networks","authors":"Brian Cheung, Jesse A Livezey, Arjun K Bansal, Bruno A Olshausen","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ding et al., 2015] Changxing Ding, Chang Xu, and Dacheng Tao. Multi-task pose-invariant face recognition. IEEE Transactions on Image Processing, 24(3):980-993, 2015.","title":"Multi-task pose-invariant face recognition","authors":"Changxing Ding, Chang Xu, Dacheng Tao","year":"2015","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dosovitskiy et al., 2015] Alexey Dosovitskiy, Jost To- bias Springenberg, and Thomas Brox. Learning to generate chairs with convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1538-1546, 2015.","title":"Learning to generate chairs with convolutional neural networks","authors":"Alexey Dosovitskiy, Jost Tobias Springenberg, Thomas Brox","year":"2015","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gauthier, 2014] Jon Gauthier. Conditional generative adver- sarial nets for convolutional face generation. Class Project for Stanford CS231N: Convolutional Neural Networks for Visual Recognition, Winter semester, 2014, 2014.","title":"Conditional generative adversarial nets for convolutional face generation","authors":"Jon Gauthier, Gauthier","year":"2014","venue":"Class Project for Stanford CS231N: Convolutional Neural Networks for Visual Recognition, Winter semester","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In Advances in Neural Informa- tion Processing Systems, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gross et al., 2010] Ralph Gross, Iain Matthews, Jeffrey Cohn, Takeo Kanade, and Simon Baker. Multi-pie. Im- age and Vision Computing, 28(5):807-813, 2010.","title":"Multi-pie","authors":"Ralph Gross, Iain Matthews, Jeffrey Cohn, Takeo Kanade, Simon Baker","year":"2010","venue":"Image and Vision Computing","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hinton et al., 2011] Geoffrey E Hinton, Alex Krizhevsky, and Sida D Wang. Transforming auto-encoders. In Inter- national Conference on Artificial Neural Networks, pages 44-51. Springer, 2011.","title":"Transforming auto-encoders","authors":"Geoffrey E Hinton, Alex Krizhevsky, Sida D Wang","year":"2011","venue":"International Conference on Artificial Neural Networks","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kar et al., 2015] Abhishek Kar, Shubham Tulsiani, Joo Car- reira, and Jitendra Malik. Category-specific object recon- struction from a single image. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1966-1974. IEEE, 2015.","title":"Category-specific object reconstruction from a single image","authors":"Kar","year":"2015","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kingma and Welling, 2013] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kulkarni et al., 2015] Tejas D Kulkarni, William F Whit- ney, Pushmeet Kohli, and Josh Tenenbaum. Deep convo- lutional inverse graphics network. In Advances in Neural Information Processing Systems, pages 2539-2547, 2015.","title":"Deep convolutional inverse graphics network","authors":"D Tejas, William F Kulkarni, Pushmeet Whitney, Josh Kohli, Tenenbaum","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2017] Tongliang Liu, Dacheng Tao, Mingli Song, and Stephen J. Maybank. Algorithm-dependent general- ization bounds for multi-task learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(2):227- 241, February 2017.","title":"Algorithm-dependent generalization bounds for multi-task learning","authors":"Liu","year":"2017","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Radford et al., 2015] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.","title":"Unsupervised representation learning with deep convolutional generative adversarial networks","authors":"Radford","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Reed et al., 2016] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In Proceed- ings of The 33rd International Conference on Machine Learning, volume 3, 2016.","title":"Generative adversarial text to image synthesis","authors":"Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, Honglak Lee","year":"2016","venue":"Proceedings of The 33rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tatarchenko et al., 2016] Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox. Multi-view 3d models from single images with a convolutional network. In Eu- ropean Conference on Computer Vision, pages 322-337. Springer, 2016.","title":"Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox. Multi-view 3d models from single images with a convolutional network","authors":"Tatarchenko","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vicente et al., 2014] Sara Vicente, João Carreira, Lourdes Agapito, and Jorge Batista. Reconstructing pascal voc. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 41-48. IEEE, 2014.","title":"Reconstructing pascal voc","authors":"Sara Vicente, João Carreira, Lourdes Agapito, Jorge Batista","year":"2014","venue":"2014 IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2011] Chaohui Wang, Yun Zeng, Loic Simon, Ioannis Kakadiaris, Dimitris Samaras, and Nikos Paragios. Viewpoint invariant 3d landmark model inference from monocular 2d images using higher-order priors. In 2011 International Conference on Computer Vision, pages 319- 326. IEEE, 2011.","title":"Viewpoint invariant 3d landmark model inference from monocular 2d images using higher-order priors","authors":"Wang","year":"2011","venue":"2011 International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xu et al., 2015] Chang Xu, Dacheng Tao, and Chao Xu. Multi-view intact space learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(12):2531- 2544, 2015.","title":"Multi-view intact space learning","authors":"Chang Xu, Dacheng Tao, Chao Xu","year":"2015","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yang et al., 2015] Jimei Yang, Scott E Reed, Ming-Hsuan Yang, and Honglak Lee. Weakly-supervised disentan- gling with recurrent transformations for 3d view synthe- sis. In Advances in Neural Information Processing Sys- tems, pages 1099-1107, 2015.","title":"Weakly-supervised disentangling with recurrent transformations for 3d view synthesis","authors":"Yang","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}