{"type":"ScholarlyArticle","title":"Visual Data Synthesis via GAN for Zero-Shot Video Classification","authors":[{"name":"Chenrui Zhang","slug":"chenrui-zhang","given":"Chenrui","family":"Zhang","affiliation":"Institute of Computer Science and Technology , Peking University , Beijing 100871 , China"},{"name":"Yuxin Peng","slug":"yuxin-peng","given":"Yuxin","family":"Peng","affiliation":"Institute of Computer Science and Technology , Peking University , Beijing 100871 , China"}],"abstract":"Zero-Shot Learning (ZSL) in video classification\r\nis a promising research direction, which aims to\r\ntackle the challenge from explosive growth of video\r\ncategories. Most existing methods exploit seento-\r\nunseen correlation via learning a projection between\r\nvisual and semantic spaces. However, such\r\nprojection-based paradigms cannot fully utilize the\r\ndiscriminative information implied in data distribution,\r\nand commonly suffer from the information\r\ndegradation issue caused by \"heterogeneity gap\".\r\nIn this paper, we propose a visual data synthesis\r\nframework via GAN to address these problems.\r\nSpecifically, both semantic knowledge and visual\r\ndistribution are leveraged to synthesize video feature\r\nof unseen categories, and ZSL can be turned\r\ninto typical supervised problem with the synthetic\r\nfeatures. First, we propose multi-level semantic\r\ninference to boost video feature synthesis, which\r\ncaptures the discriminative information implied in\r\njoint visual-semantic distribution via feature-level\r\nand label-level semantic inference. Second, we\r\npropose Matching-aware Mutual Information Correlation\r\nto overcome information degradation issue,\r\nwhich captures seen-to-unseen correlation in\r\nmatched and mismatched visual-semantic pairs by\r\nmutual information, providing the zero-shot synthesis\r\nprocedure with robust guidance signals. Experimental\r\nresults on four video datasets demonstrate\r\nthat our approach can improve the zero-shot\r\nvideo classification performance significantly.","doi":"10.24963/ijcai.2018/157","url":"https://ijcai.org/proceedings/2018/157","pdf":"https://ijcai.org/proceedings/2018/0157.pdf","markdown":"https://ijcai.org/proceedings/2018/157.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/157","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":157,"pages":{"first":1128,"last":1134},"keywords":["Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Computer Vision: Computer Vision"],"videos":[],"references":[{"position":1,"raw":"[Akata et al., 2015] Zeynep Akata, Scott Reed, Daniel Wal- ter, Honglak Lee, and Bernt Schiele. Evaluation of out- put embeddings for fine-grained image classification. In CVPR, pages 2927-2936, 2015.","title":"Evaluation of output embeddings for fine-grained image classification","authors":"Zeynep Akata, Scott Reed, Daniel Walter, Honglak Lee, Bernt Schiele","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Akata et al., 2016] Zeynep Akata, Mateusz Malinowski, Mario Fritz, and Bernt Schiele. Multi-cue zero-shot learn- ing with strong supervision. In CVPR, pages 59-68, 2016.","title":"Multi-cue zero-shot learning with strong supervision","authors":"Zeynep Akata, Mateusz Malinowski, Mario Fritz, Bernt Schiele","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Arjovsky et al., 2017] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. arXiv preprint arXiv:1701.07875, 2017.","title":"Wasserstein gan","authors":"Martin Arjovsky, Soumith Chintala, Léon Bottou","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Changpinyo et al., 2016] Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, and Fei Sha. Synthesized classifiers for zero-shot learning. In CVPR, pages 5327-5336, 2016.","title":"Synthesized classifiers for zero-shot learning","authors":"Changpinyo","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2016] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Info- gan: Interpretable representation learning by information maximizing generative adversarial nets. In NIPS, pages 2172-2180, 2016.","title":"Infogan: Interpretable representation learning by information maximizing generative adversarial nets","authors":"Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, Pieter Abbeel","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In NIPS, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jayaraman et al., 2014] Dinesh Jayaraman, Fei Sha, and Kristen Grauman. Decorrelating semantic visual attributes by resisting the urge to share. In CVPR, pages 1629-1636, 2014.","title":"Decorrelating semantic visual attributes by resisting the urge to share","authors":"Dinesh Jayaraman, Fei Sha, Kristen Grauman","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jiang et al., 2011] Yu-Gang Jiang, Guangnan Ye, Shih-Fu Chang, Daniel Ellis, and Alexander C Loui. Consumer video understanding: A benchmark database and an eval- uation of human and machine performance. In ICMR, page 29, 2011.","title":"Consumer video understanding: A benchmark database and an evaluation of human and machine performance","authors":"Yu-Gang Jiang, Guangnan Ye, Shih-Fu Chang, Daniel Ellis, Alexander C Loui","year":"2011","venue":"ICMR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kuehne et al., 2013] Hilde Kuehne, Hueihan Jhuang, Rainer Stiefelhagen, and Thomas Serre. Hmdb51: A large video database for human motion recognition. In HPCSE, pages 571-582. 2013.","title":"Hmdb51: A large video database for human motion recognition","authors":"Kuehne","year":"2013","venue":"HPCSE","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lampert et al., 2009] Christoph H Lampert, Hannes Nick- isch, and Stefan Harmeling. Learning to detect unseen ob- ject classes by between-class attribute transfer. In CVPR, pages 951-958, 2009.","title":"Learning to detect unseen object classes by between-class attribute transfer","authors":"Hannes Christoph H Lampert, Stefan Nickisch, Harmeling","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lampert et al., 2014] Christoph H Lampert, Hannes Nick- isch, and Stefan Harmeling. Attribute-based classifica- tion for zero-shot visual object categorization. PAMI, 36(3):453-465, 2014.","title":"Attribute-based classification for zero-shot visual object categorization","authors":"Hannes Christoph H Lampert, Stefan Nickisch, Harmeling","year":"2014","venue":"PAMI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"Long et al., ] Yang Long, Li Liu, Ling Shao, Fumin Shen, Guiguang Ding, and Jungong Han. From zero-shot learn- ing to conventional supervised classification: Unseen vi- sual data synthesis. In CVPR, pages 1627-1636.","title":"From zero-shot learning to conventional supervised classification: Unseen visual data synthesis","authors":"Long","year":null,"venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Maaten and Hinton, 2008] Laurens van der Maaten and Ge- offrey Hinton. Visualizing data using t-sne. JMLR, 9(Nov):2579-2605, 2008.","title":"Laurens van der Maaten and Geoffrey Hinton","authors":"Hinton Maaten","year":"2008","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mirza and Osindero, 2014] Mehdi Mirza and Simon Osin- dero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014.","title":"Conditional generative adversarial nets","authors":"Mehdi Mirza, Simon Osindero","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Niebles et al., 2010] Juan Carlos Niebles, Chih-Wei Chen, and Li Fei-Fei. Modeling temporal structure of decompos- able motion segments for activity classification. In ECCV, pages 392-405, 2010.","title":"Modeling temporal structure of decomposable motion segments for activity classification","authors":"Juan Carlos Niebles, Chih-Wei Chen, Li Fei-Fei","year":"2010","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Norouzi et al., 2014] Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea Frome, Greg S Corrado, and Jeffrey Dean. Zero-shot learning by convex combination of semantic embeddings. In ICLR, 2014.","title":"Zero-shot learning by convex combination of semantic embeddings","authors":"Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea Frome, Greg S Corrado, Jeffrey Dean","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Palatucci et al., 2009] Mark Palatucci, Dean Pomerleau, Geoffrey E Hinton, and Tom M Mitchell. Zero-shot learn- ing with semantic output codes. In NIPS, pages 1410- 1418, 2009.","title":"Zero-shot learning with semantic output codes","authors":"Mark Palatucci, Dean Pomerleau, Geoffrey E Hinton, Tom M Mitchell","year":"2009","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In EMNLP, pages 1532-1543, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher Manning","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Radovanović et al., 2010] Miloš Radovanović, Alexandros Nanopoulos, and Mirjana Ivanović. Hubs in space: Pop- ular nearest neighbors in high-dimensional data. JMLR, 11(Sep):2487-2531, 2010.","title":"Hubs in space: Popular nearest neighbors in high-dimensional data","authors":"Radovanović","year":"2010","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Reed et al., 2016] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In ICML, 2016.","title":"Generative adversarial text to image synthesis","authors":"Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, Honglak Lee","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Two-stream convolutional networks for action recognition in videos. In NIPS, pages 568-576, 2014.","title":"Two-stream convolutional networks for action recognition in videos","authors":"Zisserman Simonyan","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Soomro et al., 2012] Khurram Soomro, Amir Roshan Za- mir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012.","title":"Ucf101: A dataset of 101 human actions classes from videos in the wild","authors":"Khurram Soomro, Mubarak Amir Roshan Zamir, Shah","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xian et al., 2017] Yongqin Xian, Bernt Schiele, and Zeynep Akata. Zero-shot learning-the good, the bad and the ugly. In CVPR, pages 1980-1990, 2017.","title":"Zero-shot learning-the good, the bad and the ugly","authors":"Xian","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xu et al., 2017] Xun Xu, Timothy Hospedales, and Shao- gang Gong. Transductive zero-shot action recognition by word-vector embedding. IJCV, 123(3):309-333, 2017.","title":"Transductive zero-shot action recognition by word-vector embedding","authors":"Xun Xu, Timothy Hospedales, Shaogang Gong","year":"2017","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang and Saligrama, 2015] Ziming Zhang and Venkatesh Saligrama. Zero-shot learning via semantic similarity em- bedding. In ICCV, pages 4166-4174, 2015.","title":"Zero-shot learning via semantic similarity embedding","authors":"Ziming Zhang, Venkatesh Saligrama","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang and Saligrama, 2016] Ziming Zhang and Venkatesh Saligrama. Zero-shot learning via joint semantic similarity embedding. In CVPR, pages 4166-4174, 2016.","title":"Zero-shot learning via joint semantic similarity embedding","authors":"Ziming Zhang, Venkatesh Saligrama","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2017] Li Zhang, Tao Xiang, and Shaogang Gong. Learning a deep embedding model for zero-shot learning. CVPR, pages 2021-2030, 2017.","title":"Learning a deep embedding model for zero-shot learning","authors":"Li Zhang, Tao Xiang, Shaogang Gong","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}