{"type":"ScholarlyArticle","title":"Exploiting Images for Video Recognition with Hierarchical Generative Adversarial Networks","authors":[{"name":"Feiwu Yu","slug":"feiwu-yu","given":"Feiwu","family":"Yu","affiliation":"Beijing Laboratory of Intelligent Information Technology , School of Computer Science , Beijing Institute of Technology"},{"name":"Xinxiao Wu","slug":"xinxiao-wu","given":"Xinxiao","family":"Wu","affiliation":"Beijing Laboratory of Intelligent Information Technology , School of Computer Science , Beijing Institute of Technology"},{"name":"Yuchao Sun","slug":"yuchao-sun","given":"Yuchao","family":"Sun","affiliation":"Beijing Laboratory of Intelligent Information Technology , School of Computer Science , Beijing Institute of Technology"},{"name":"Lixin Duan","slug":"lixin-duan","given":"Lixin","family":"Duan","affiliation":"Big Data Research Center , University of Electronic Science and Technology of China"}],"abstract":"Existing deep learning methods of video recognition usually require a large number of labeled videos for training.  But for a new task, videos are often unlabeled and it is also time-consuming and labor-intensive to annotate them. Instead of human annotation, we try to make use of existing fully labeled images to help recognize those videos. However, due to the problem of domain shifts and heterogeneous feature representations, the performance of classifiers trained on images may be dramatically degraded for video recognition tasks. In this paper, we propose a novel method, called Hierarchical Generative Adversarial Networks (HiGAN), to enhance recognition in videos (i.e., target domain) by transferring knowledge from images (i.e., source domain). The HiGAN model consists of a \\emph{low-level} conditional GAN and a \\emph{high-level} conditional GAN. By taking advantage of these two-level adversarial learning, our method is capable of learning a domain-invariant feature representation of source images and target videos. Comprehensive experiments on two challenging video recognition datasets (i.e. UCF101 and HMDB51) demonstrate the effectiveness of the proposed method when compared with the existing state-of-the-art domain adaptation methods.","doi":"10.24963/ijcai.2018/154","url":"https://ijcai.org/proceedings/2018/154","pdf":"https://ijcai.org/proceedings/2018/0154.pdf","markdown":"https://ijcai.org/proceedings/2018/154.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/154","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":154,"pages":{"first":1107,"last":1113},"keywords":["Computer Vision: Action Recognition","Machine Learning: Transfer, Adaptation, Multi-task Learning","Machine Learning: Unsupervised Learning"],"videos":[],"references":[{"position":1,"raw":"Akaho, 2006","title":null,"authors":"Akaho","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"S Akaho. A kernel method for canonical cor- relation analysis. arXiv preprint cs/0609071, 2006.","title":"A kernel method for canonical correlation analysis","authors":"Akaho","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bousmalis et al., 2017] K Bousmalis, N Silberman, D Do- han, D Erhan, and D Krishnan. Unsupervised pixel-level domain adaptation with generative adversarial networks. In CVPR, 2017.","title":"Unsupervised pixel-level domain adaptation with generative adversarial networks","authors":"N Bousmalis, D Silberman, D Dohan, Erhan, Krishnan","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Busto and Gall, 2017] PP Busto and J Gall. Open set do- main adaptation. In ICCV, 2017.","title":"Open set domain adaptation","authors":"Gall ; Pp Busto, J Busto, Gall","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Carlucci et al., 2017] FM Carlucci, L Porzi, B Caputo, E Ricci, and SR Bulo. Autodial: Automatic domain align- ment layers. In ICCV, 2017.","title":"Autodial: Automatic domain alignment layers","authors":"Carlucci","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2016] WY Chen, TM Hsu, YH Tsai, YC Wang, and MS Chen. Transfer neural trees for heterogeneous do- main adaptation. In ECCV, pages 399-414, 2016.","title":"Transfer neural trees for heterogeneous domain adaptation","authors":"Chen","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Duan et al., 2012a] L Duan, D Xu, and SF Chang. Exploit- ing web images for event recognition in consumer videos: A multiple source domain adaptation approach. In CVPR, pages 1338-1345, 2012.","title":"Exploiting web images for event recognition in consumer videos: A multiple source domain adaptation approach","authors":"Duan","year":"2012","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Duan et al., 2012b] L Duan, D Xu, and I Tsang. Learning with augmented features for heterogeneous domain adap- tation. arXiv preprint arXiv:1206.4660, 2012.","title":"Learning with augmented features for heterogeneous domain adaptation","authors":"Duan","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gan et al., 2016] C Gan, C Sun, L Duan, and B Gong. Webly-supervised video recognition by mutually voting for relevant web images and web video frames. In ECCV, pages 849-866, 2016.","title":"Webly-supervised video recognition by mutually voting for relevant web images and web video frames","authors":"C Gan, L Sun, Duan, Gong","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Goodfellow et al., 2014] I Goodfellow, J Pouget-Abadie, M Mirza, B Xu, D Warde-Farley, S Ozair, A Courville, and Y Bengio. Generative adversarial nets. In NIPS, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Goodfellow","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2016] K He, X Zhang, S Ren, and J Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"He, S Zhang, Ren, Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jia et al., 2014] Y Jia, E Shelhamer, J Donahue, S Karayev, J Long, R Girshick, S Guadarrama, and T Darrell. Caffe: Convolutional architecture for fast feature embedding. In ACM MM, pages 675-678, 2014.","title":"Caffe: Convolutional architecture for fast feature embedding","authors":"Jia, J Shelhamer, S Donahue, J Karayev, R Long, S Girshick, Guadarrama, Darrell","year":"2014","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kingma and Ba, 2014] D Kingma and J Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"D Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Long et al., 2014] M Long, J Wang, G Ding, J Sun, and PS Yu. Transfer joint matching for unsupervised domain adaptation. In CVPR, pages 1410-1417, 2014.","title":"Transfer joint matching for unsupervised domain adaptation","authors":"Long, G Wang, J Ding, Sun, Yu","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Long et al., 2015a] M Long, Y Cao, J Wang, and M Jor- dan. Learning transferable features with deep adaptation networks. In ICML, pages 97-105, 2015.","title":"Learning transferable features with deep adaptation networks","authors":"Long","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Long et al., 2015b] M Long, J Wang, J Sun, and S Yu Philip. Domain invariant transfer kernel learning. TKDE, 27(6):1519-1532, 2015.","title":"Domain invariant transfer kernel learning","authors":"Long","year":"2015","venue":"TKDE","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Long et al., 2016] M Long, H Zhu, J Wang, and MI Jor- dan. Unsupervised domain adaptation with residual trans- fer networks. In NIPS, pages 136-144, 2016.","title":"Unsupervised domain adaptation with residual transfer networks","authors":"H Long, J Zhu, M I Wang, Jordan","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Long et al., 2017] M Long, H Zhu, J Wang, and MI. Jordan. Deep transfer learning with joint adaptation networks. In ICML, pages 2208-2217, 2017.","title":"Deep transfer learning with joint adaptation networks","authors":"H Long, J Zhu, M I Wang, Jordan","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mao et al., 2016] X Mao, Q Li, H Xie, R YK Lau, and Z Wang. Multi-class generative adversarial networks with the l2 loss function. arXiv preprint arXiv:1611.04076, 2016.","title":"Multi-class generative adversarial networks with the l2 loss function","authors":"Q Mao, H Li, R Yk Xie, Lau, Wang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sankaranarayanan et al., 2017] S Sankaranarayanan, Y Balaji, CD Castillo, and R Chellappa. Generate to adapt: Aligning domains using generative adversarial networks. arXiv preprint arXiv:1704.01705, 2017.","title":"Generate to adapt: Aligning domains using generative adversarial networks","authors":"Sankaranarayanan, Balaji, Castillo, Chellappa","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shi et al., 2010] X Shi, Q Liu, W Fan, SY Philip, and R Zhu. Transfer learning on heterogenous feature spaces via spectral transformation. In ICDM, pages 1049-1054. IEEE, 2010.","title":"Transfer learning on heterogenous feature spaces via spectral transformation","authors":"Shi","year":"2010","venue":"ICDM","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Soomro et al., 2012] K Soomro, A Roshan Zamir, and M Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012.","title":"Ucf101: A dataset of 101 human actions classes from videos in the wild","authors":"Soomro, M Roshan Zamir, Shah","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sun and Saenko, 2016] B Sun and K Saenko. Deep coral: Correlation alignment for deep domain adaptation. In ECCV 2016 Workshops, 2016.","title":"Deep coral: Correlation alignment for deep domain adaptation","authors":"B Sun, Saenko","year":"2016","venue":"ECCV 2016 Workshops","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sun et al., 2016] B Sun, J Feng, and K Saenko. Return of frustratingly easy domain adaptation. In AAAI, volume 6, page 8, 2016.","title":"Return of frustratingly easy domain adaptation","authors":"Sun, Feng, Saenko","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tanisik et al., 2016] G Tanisik, C Zalluhoglu, and N Ikizler- Cinbis. Facial descriptors for human interaction recogni- tion in still images. PR, 73:44-51, 2016.","title":"Facial descriptors for human interaction recognition in still images","authors":"Tanisik, N Zalluhoglu, Ikizler-Cinbis","year":"2016","venue":"PR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tran et al., 2015] D Tran, L Bourdev, R Fergus, L Torre- sani, and M Paluri. Learning spatiotemporal features with 3d convolutional networks. In ICCV, pages 4489-4497, 2015.","title":"Learning spatiotemporal features with 3d convolutional networks","authors":"L Tran, R Bourdev, L Fergus, M Torresani, Paluri","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tsai et al., 2016] YH Tsai, YR Yeh, and YC Wang. Learn- ing cross-domain landmarks for heterogeneous domain adaptation. In CVPR, pages 5081-5090, 2016.","title":"Learning cross-domain landmarks for heterogeneous domain adaptation","authors":"Tsai, Yeh, Wang","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang and Mahadevan, 2011] C Wang and S Mahadevan. Heterogeneous domain adaptation using manifold align- ment. In IJCAI, volume 22, page 1541, 2011.","title":"Heterogeneous domain adaptation using manifold alignment","authors":"C Wang, S Mahadevan","year":"2011","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2011/259"},{"position":29,"raw":"[Xu et al., 2013] X Xu, IW Tsang, and D Xu. Soft margin multiple kernel learning. TNNLS, 24(5):749-761, 2013.","title":"Soft margin multiple kernel learning","authors":"Xu","year":"2013","venue":"TNNLS","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Xu et al., 2016] Y Xu, X Fang, J Wu, X Li, and D Zhang. Discriminative transfer subspace learning via low-rank and sparse representation. TIP, 25(2):850-863, 2016.","title":"Discriminative transfer subspace learning via low-rank and sparse representation","authors":"X Xu, J Fang, X Wu, Li, Zhang","year":"2016","venue":"TIP","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yao et al., 2011] B Yao, X Jiang, A Khosla, AL Lin, L Guibas, and Li FF. Human action recognition by learn- ing bases of action attributes and parts. In ICCV, pages 1331-1338, 2011.","title":"Human action recognition by learning bases of action attributes and parts","authors":"X Yao, Jiang, A L Khosla, L Lin, Li Guibas, Ff","year":"2011","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2017] J Zhang, Y Han, J Tang, Q Hu, and J Jiang. Semi-supervised image-to-video adaptation for video action recognition. TCYB, 47(4):960-973, 2017.","title":"Semi-supervised image-to-video adaptation for video action recognition","authors":"Y Zhang, J Han, Q Tang, Hu, Jiang","year":"2017","venue":"TCYB","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhu et al., 2017] JY Zhu, T Park, P Isola, and AA. Efros. Unpaired image-to-image translation using cycle- consistent adversarial networks. In ICCV, 2017.","title":"Unpaired image-to-image translation using cycleconsistent adversarial networks","authors":"Jy Zhu, P Park, A A Isola, Efros","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}