{"type":"ScholarlyArticle","title":"Conditional GAN with Discriminative Filter Generation for Text-to-Video Synthesis","authors":[{"name":"Yogesh Balaji","slug":"yogesh-balaji","given":"Yogesh","family":"Balaji","affiliation":"University of Maryland , College Park"},{"name":"Martin Renqiang Min","slug":"martin-renqiang-min","given":"Martin Renqiang","family":"Min","affiliation":""},{"name":"Bing Bai","slug":"bing-bai","given":"Bing","family":"Bai","affiliation":"NEC Labs America -Princeton"},{"name":"Rama Chellappa","slug":"rama-chellappa","given":"Rama","family":"Chellappa","affiliation":"University of Maryland , College Park"},{"name":"Hans Peter Graf","slug":"hans-peter-graf","given":"Hans Peter","family":"Graf","affiliation":"NEC Labs America -Princeton"}],"abstract":"Developing conditional generative models for text-to-video synthesis is an extremely challenging yet an important topic of research in machine learning. In this work, we address this problem by introducing Text-Filter conditioning Generative Adversarial Network (TFGAN), a conditional GAN model with a novel multi-scale text-conditioning scheme that improves text-video associations. By combining the proposed conditioning scheme with a deep GAN architecture, TFGAN generates high quality videos from text on challenging real-world video datasets. In addition, we construct a synthetic dataset of text-conditioned moving shapes to systematically evaluate our conditioning scheme. Extensive experiments demonstrate that TFGAN significantly outperforms existing approaches, and can also generate videos of novel categories not seen during training.","doi":"10.24963/ijcai.2019/276","url":"https://ijcai.org/proceedings/2019/276","pdf":"https://ijcai.org/proceedings/2019/0276.pdf","markdown":"https://ijcai.org/proceedings/2019/276.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/276","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":276,"pages":{"first":1995,"last":2001},"keywords":["Machine Learning: Learning Generative Models","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Brock et al., 2018] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fi- delity natural image synthesis. CoRR, abs/1809.11096, 2018.","title":"Large scale GAN training for high fidelity natural image synthesis","authors":"Andrew Brock, Jeff Donahue, Karen Simonyan","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Damen et al., 2018] Dima Damen, Hazel Doughty, Gio- vanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling ego- centric vision: The epic-kitchens dataset. In European Conference on Computer Vision (ECCV), 2018.","title":"Scaling egocentric vision: The epic-kitchens dataset","authors":"Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, Michael Wray","year":"2018","venue":"European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In Advances in Neural Informa- tion Processing Systems 27, pages 2672-2680. 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Heusel et al., 2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural In- formation Processing Systems 30, pages 6626-6637. 2017.","title":"Gans trained by a two time-scale update rule converge to a local nash equilibrium","authors":"Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Sepp Hochreiter","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Jia et al., 2016] Xu Jia, Bert De Brabandere, Tinne Tuyte- laars, and Luc Van Gool. Dynamic filter networks. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Sys- tems 2016, December 5-10, 2016, Barcelona, Spain, pages 667-675, 2016.","title":"Dynamic filter networks","authors":"Xu Jia, Bert De Brabandere, Tinne Tuytelaars, Luc Van Gool","year":"2016","venue":"Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Karras et al., 2017] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. CoRR, abs/1710.10196, 2017.","title":"Progressive growing of gans for improved quality, stability, and variation","authors":"Tero Karras, Timo Aila, Samuli Laine, Jaakko Lehtinen","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kay et al., 2017] Will Kay, João Carreira, Karen Si- monyan, Brian Zhang, Chloe Hillier, Sudheendra Vi- jayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisser- man. The kinetics human action video dataset. CoRR, abs/1705.06950, 2017.","title":"The kinetics human action video dataset","authors":"Kay","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kingma and Welling, 2014] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In ICLR 2014, April 2014.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2014","venue":"ICLR 2014","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2018] Yitong Li, Martin Renqiang Min, Dinghan Shen, David E. Carlson, and Lawrence Carin. Video gen- eration from text. In AAAI, 2018.","title":"Video generation from text","authors":"Yitong Li, Martin Renqiang Min, Dinghan Shen, David E Carlson, Lawrence Carin","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Mescheder et al., 2018] Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. Which training meth- ods for gans do actually converge? In International Conference on Machine Learning (ICML), 2018.","title":"Which training methods for gans do actually converge","authors":"Mescheder","year":"2018","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mirza and Osindero, 2014] Mehdi Mirza and Simon Osin- dero. Conditional generative adversarial nets. ICLR Work- shop, 2014.","title":"Conditional generative adversarial nets","authors":"Mehdi Mirza, Simon Osindero","year":"2014","venue":"ICLR Workshop","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Odena et al., 2017] Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxil- iary classifier GANs. In Proceedings of the 34th Inter- national Conference on Machine Learning, pages 2642- 2651, 2017.","title":"Conditional image synthesis with auxiliary classifier GANs","authors":"Odena","year":"2017","venue":"Proceedings of the 34th Inter-national Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Pan et al., 2017] Yingwei Pan, Zhaofan Qiu, Ting Yao, Houqiang Li, and Tao Mei. To create what you tell: Gen- erating videos from captions. 2017.","title":"To create what you tell: Generating videos from captions","authors":"Pan","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Reed et al., 2016] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In Proceed- ings of The 33rd International Conference on Machine Learning, pages 1060-1069, 2016.","title":"Generative adversarial text to image synthesis","authors":"Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, Honglak Lee","year":"2016","venue":"Proceedings of The 33rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Salimans et al., 2016] Tim Salimans, Ian Goodfellow, Wo- jciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems 29, pages 2234-2242. 2016.","title":"Improved techniques for training gans","authors":"Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, Xi Chen","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Tulyakov et al., 2018] Sergey Tulyakov, Ming-Yu Liu, Xi- aodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.","title":"Mocogan: Decomposing motion and content for video generation","authors":"Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, Jan Kautz","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Vondrick et al., 2016] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dy- namics. In Advances in Neural Information Processing Systems 29, pages 613-621. 2016.","title":"Generating videos with scene dynamics","authors":"Carl Vondrick, Hamed Pirsiavash, Antonio Torralba","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2018a] Heng Wang, Zengchang Qin, and Tao Wan. Text generation based on generative adversarial nets with latent variables. In Advances in Knowledge Discovery and Data Mining -22nd Pacific-Asia Conference, PAKDD 2018, Melbourne, VIC, Australia, June 3-6, 2018, Pro- ceedings, Part II, 2018.","title":"Text generation based on generative adversarial nets with latent variables","authors":"Wang","year":"2018","venue":"Advances in Knowledge Discovery and Data Mining -22nd Pacific-Asia Conference, PAKDD 2018","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2018b] Hongwei Wang, Jia Wang, Jialin Wang, Miao Zhao, Weinan Zhang, Fuzheng Zhang, Xing Xie, and Minyi Guo. Graphgan: Graph representation learning with generative adversarial nets. In AAAI, 2018.","title":"Graphgan: Graph representation learning with generative adversarial nets","authors":"Wang","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xu et al., 2018] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xi- aodong He. Attngan: Fine-grained text to image gen- eration with attentional generative adversarial networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.","title":"Attngan: Fine-grained text to image generation with attentional generative adversarial networks","authors":"Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, Xiaodong He","year":"2018","venue":"The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yan et al., 2016] Xinchen Yan, Jimei Yang, Kihyuk Sohn, and Honglak Lee. Attribute2image: Conditional image generation from visual attributes. In Computer Vision - ECCV 2016 -14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, 2016.","title":"Attribute2image: Conditional image generation from visual attributes","authors":"Yan","year":"2016","venue":"Computer Vision -ECCV 2016 -14th European Conference","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang et al., 2017] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In ICCV, 2017.","title":"Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks","authors":"Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, Dimitris Metaxas","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhu et al., 2017] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image transla- tion using cycle-consistent adversarial networks. In ICCV, 2017.","title":"Unpaired image-to-image translation using cycle-consistent adversarial networks","authors":"Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A Efros","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}