{"type":"ScholarlyArticle","title":"IRC-GAN: Introspective Recurrent Convolutional GAN for Text-to-video Generation","authors":[{"name":"Kangle Deng","slug":"kangle-deng","given":"Kangle","family":"Deng","affiliation":"Institute of Computer Science and Technology , Peking University , Beijing , China"},{"name":"Tianyi Fei","slug":"tianyi-fei","given":"Tianyi","family":"Fei","affiliation":"Institute of Computer Science and Technology , Peking University , Beijing , China"},{"name":"Xin Huang","slug":"xin-huang","given":"Xin","family":"Huang","affiliation":"Institute of Computer Science and Technology , Peking University , Beijing , China"},{"name":"Yuxin Peng","slug":"yuxin-peng","given":"Yuxin","family":"Peng","affiliation":"Institute of Computer Science and Technology , Peking University , Beijing , China"}],"abstract":"Automatically generating videos according to the given text is a highly challenging task, where visual quality and semantic consistency with captions are two critical issues. In existing methods, when generating a specific frame, the information in those frames generated before is not fully exploited. And an effective way to measure the semantic accordance between videos and captions remains to be established. To address these issues, we present a novel Introspective Recurrent Convolutional GAN (IRC-GAN) approach. First, we propose a recurrent transconvolutional generator, where LSTM cells are integrated with 2D transconvolutional layers. As 2D transconvolutional layers put more emphasis on the details of each frame than 3D ones, our generator takes both the definition of each video frame and temporal coherence across the whole video into consideration, and thus can generate videos with better visual quality. Second, we propose mutual information introspection to semantically align the generated videos to text. Unlike other methods simply judging whether the video and the text match or not, we further take mutual information to concretely measure the semantic consistency. In this way,  our model is able to introspect the semantic distance between the generated video and the corresponding text, and try to minimize it to boost the semantic consistency.We conduct experiments on 3 datasets and compare with state-of-the-art methods. Experimental results demonstrate the effectiveness of our IRC-GAN to generate plausible videos from given text.","doi":"10.24963/ijcai.2019/307","url":"https://ijcai.org/proceedings/2019/307","pdf":"https://ijcai.org/proceedings/2019/0307.pdf","markdown":"https://ijcai.org/proceedings/2019/307.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/307","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":307,"pages":{"first":2216,"last":2222},"keywords":["Computer Vision: Language and Vision","Machine Learning: Learning Generative Models"],"videos":[],"references":[{"position":1,"raw":"[Chen et al., 2016] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Info- gan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pages 2172-2180, 2016.","title":"Infogan: Interpretable representation learning by information maximizing generative adversarial nets","authors":"Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, Pieter Abbeel","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dai and Le, 2015] Andrew M Dai and Quoc V Le. Semi- supervised sequence learning. In Advances in neural in- formation processing systems, pages 3079-3087, 2015.","title":"Semisupervised sequence learning","authors":"M Andrew, Quoc V Dai, Le","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Doersch, 2016] Carl Doersch. Tutorial on variational au- toencoders. 2016.","title":"Tutorial on variational autoencoders","authors":"Carl Doersch","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hao et al., 2018] Guang-Yuan Hao, Hong-Xing Yu, and Wei-Shi Zheng. Mixgan: learning concepts from differ- ent domains for mixture generation. IJCAI, 2018.","title":"Mixgan: learning concepts from different domains for mixture generation","authors":"Guang-Yuan Hao, Hong-Xing Yu, Wei-Shi Zheng","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/306"},{"position":6,"raw":"[Im et al., 2016] Daniel Jiwoong Im, Chris Dongjoo Kim, Hui Jiang, and Roland Memisevic. Generating im- ages with recurrent adversarial networks. arXiv preprint arXiv:1602.05110, 2016.","title":"Generating images with recurrent adversarial networks","authors":"Jiwoong Daniel, Chris Dongjoo Im, Hui Kim, Roland Jiang, Memisevic","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Laptev et al., 2004] Ivan Laptev, Barbara Caputo, et al. Recognizing human actions: a local svm approach. In null, pages 32-36. IEEE, 2004.","title":"Recognizing human actions: a local svm approach","authors":"Ivan Laptev, Barbara Caputo","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Mao and Li, 2018] Xudong Mao and Qing Li. Unpaired multi-domain image generation via regularized condi- tional gans. IJCAI, 2018.","title":"Unpaired multi-domain image generation via regularized conditional gans","authors":"Xudong Mao, Qing Li","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/354"},{"position":9,"raw":"[Mittal et al., 2016] Gaurav Mittal, Tanya Marwah, and Vi- neeth N Balasubramanian. Sync-draw: Automatic video generation using deep recurrent attentive architectures. 2016.","title":"Sync-draw: Automatic video generation using deep recurrent attentive architectures","authors":"Gaurav Mittal, Tanya Marwah, Vineeth N Balasubramanian","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ni et al., 2018] Yao Ni, Dandan Song, Xi Zhang, Hao Wu, and Lejian Liao. Cagan: Consistent adversarial training enhanced gans. In IJCAI, pages 2588-2594, 2018.","title":"Cagan: Consistent adversarial training enhanced gans","authors":"Yao Ni, Dandan Song, Xi Zhang, Hao Wu, Lejian Liao","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/359"},{"position":11,"raw":"[Pan et al., 2017] Yingwei Pan, Zhaofan Qiu, Ting Yao, Houqiang Li, and Tao Mei. To create what you tell: Generating videos from captions. In Proceedings of the 25th ACM international conference on Multimedia, pages 1789-1798. ACM, 2017.","title":"To create what you tell: Generating videos from captions","authors":"Pan","year":"2017","venue":"Proceedings of the 25th ACM international conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Park et al., 2018] David Keetae Park, Seungjoo Yoo, Hyo- jin Bahng, Jaegul Choo, and Noseong Park. Megan: mix- ture of experts of generative adversarial networks for mul- timodal image generation. IJCAI, 2018.","title":"Megan: mixture of experts of generative adversarial networks for multimodal image generation","authors":"David Keetae Park, Seungjoo Yoo, Hyojin Bahng, Jaegul Choo, Noseong Park","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/122"},{"position":13,"raw":"[Saito et al., 2017] Masaki Saito, Eiichi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with singular value clipping. In Proceedings of the IEEE In- ternational Conference on Computer Vision, pages 2830- 2839, 2017.","title":"Temporal generative adversarial nets with singular value clipping","authors":"Masaki Saito, Eiichi Matsumoto, Shunta Saito","year":"2017","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Schuster and Paliwal, 1997] Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11):2673-2681, 1997.","title":"Bidirectional recurrent neural networks","authors":"Mike Schuster, Kuldip K Paliwal","year":"1997","venue":"IEEE Transactions on Signal Processing","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Song et al., 2018] Jingkuan Song, Jingqiu Zhang, Lianli Gao, Xianglong Liu, and Heng Tao Shen. Dual conditional gans for face aging and rejuvenation. In IJCAI, pages 899- 905, 2018.","title":"Dual conditional gans for face aging and rejuvenation","authors":"Song","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/125"},{"position":16,"raw":"[Tian et al., 2018] Yu Tian, Xi Peng, Long Zhao, Shaoting Zhang, and Dimitris N Metaxas. Cr-gan: learning com- plete representations for multi-view generation. IJCAI, 2018.","title":"Cr-gan: learning complete representations for multi-view generation","authors":"Yu Tian, Xi Peng, Long Zhao, Shaoting Zhang, Dimitris N Metaxas","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/131"},{"position":17,"raw":"[Tulyakov et al., 2018] Sergey Tulyakov, Ming-Yu Liu, Xi- aodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1526-1535, 2018.","title":"Mocogan: Decomposing motion and content for video generation","authors":"Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, Jan Kautz","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Vondrick et al., 2016] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dy- namics. In Advances In Neural Information Processing Systems, pages 613-621, 2016.","title":"Generating videos with scene dynamics","authors":"Carl Vondrick, Hamed Pirsiavash, Antonio Torralba","year":"2016","venue":"Advances In Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Zeiler et al., 2010] Matthew D Zeiler, Dilip Krishnan, Gra- ham W Taylor, and Rob Fergus. Deconvolutional net- works. 2010.","title":"Deconvolutional networks","authors":"Dilip Matthew D Zeiler, Graham W Krishnan, Rob Taylor, Fergus","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zhang and Peng, 2018a] Chenrui Zhang and Yuxin Peng. Stacking vae and gan for context-aware text-to-image gen- eration. In 2018 IEEE Fourth International Conference on Multimedia Big Data (BigMM), pages 1-5. IEEE, 2018.","title":"Stacking vae and gan for context-aware text-to-image generation","authors":"Peng Zhang, Chenrui Zhang, Yuxin Peng","year":"2018","venue":"IEEE Fourth International Conference on Multimedia Big Data (BigMM)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhang and Peng, 2018b] Chenrui Zhang and Yuxin Peng. Visual data synthesis via gan for zero-shot video classi- fication. IJCAI, 2018.","title":"Visual data synthesis via gan for zero-shot video classification","authors":"Peng Zhang, Chenrui Zhang, Yuxin Peng","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/157"},{"position":22,"raw":"[Zhang et al., 2017] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial net- works. In Proceedings of the IEEE International Confer- ence on Computer Vision, pages 5907-5915, 2017.","title":"Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks","authors":"Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, Dimitris N Metaxas","year":"2017","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}