{"type":"ScholarlyArticle","title":"Dual Adversarial Networks for Zero-shot Cross-media Retrieval","authors":[{"name":"Jingze Chi","slug":"jingze-chi","given":"Jingze","family":"Chi","affiliation":"Institute of Computer Science and Technology , Peking University , Beijing , China"},{"name":"Yuxin Peng","slug":"yuxin-peng","given":"Yuxin","family":"Peng","affiliation":"Institute of Computer Science and Technology , Peking University , Beijing , China"}],"abstract":"Existing cross-media retrieval methods usually require that testing categories remain the same with training categories, which cannot support the retrieval of increasing new categories. Inspired by zero-shot learning, this paper proposes zeroshot cross-media retrieval for addressing the above problem, which aims to retrieve data of new categories across different media types. It is challenging that zero-shot cross-media retrieval has to handle not only the inconsistent semantics across new and known categories, but also the heterogeneous distributions across different media types. To address the above challenges, this paper proposes Dual Adversarial Networks for Zero-shot Crossmedia Retrieval (DANZCR), which is the first approach to address zero-shot cross-media retrieval to the best of our knowledge. Our DANZCR approach consists of two GANs in a dual structure for common representation generation and original representation reconstruction respectively, which capture the underlying data structures as well as strengthen relations between input data and semantic space to generalize across seen and unseen categories. Our DANZCR approach exploits word embeddings to learn common representations in semantic space via an adversarial learning method, which preserves the inherent cross-media correlation and enhances the knowledge transfer to new categories. Experiments on three widely-used cross-media retrieval datasets show the effectiveness of our approach.","doi":"10.24963/ijcai.2018/92","url":"https://ijcai.org/proceedings/2018/92","pdf":"https://ijcai.org/proceedings/2018/0092.pdf","markdown":"https://ijcai.org/proceedings/2018/92.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/92","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":92,"pages":{"first":663,"last":669},"keywords":["Multidisciplinary Topics and Applications: Information Retrieval","Computer Vision: Language and Vision"],"videos":[],"references":[{"position":1,"raw":"[Ba et al., 2016] Jimmy Lei Ba, Kevin Swersky, Sanja Fi- dler, and Ruslan Salakhutdinov. Predicting deep zero-shot convolutional neural networks using textual descriptions. In ICCV, pages 4247-4255, 2016.","title":"Predicting deep zero-shot convolutional neural networks using textual descriptions","authors":"Jimmy Lei Ba, Kevin Swersky, Sanja Fidler, Ruslan Salakhutdinov","year":"2016","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chua et al., 2009] Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng. NUS- WIDE: a real-world web image database from national university of singapore. In CIVR, pages 1-9, 2009.","title":"NUS-WIDE: a real-world web image database from national university of singapore","authors":"Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, Yantao Zheng","year":"2009","venue":"CIVR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Elhoseiny et al., 2014] Mohamed Elhoseiny, Babak Saleh, and Ahmed Elgammal. Write a classifier: Zero-shot learn- ing using purely textual descriptions. In ICCV, pages 2584-2591, 2014.","title":"Write a classifier: Zero-shot learning using purely textual descriptions","authors":"Mohamed Elhoseiny, Babak Saleh, Ahmed Elgammal","year":"2014","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Farhadi et al., 2009] Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. Describing objects by their attributes. In CVPR, pages 1778-1785, 2009.","title":"Describing objects by their attributes","authors":"Ali Farhadi, Ian Endres, Derek Hoiem, David Forsyth","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Farhadi et al., 2010] Ali Farhadi, Mohsen Hejrati, Moham- mad Sadeghi, Peter Young, Cyrus Rashtchian, Julia Hock- enmaier, and David Forsyth. Every picture tells a story: Generating sentences from images. In ECCV, pages 15- 29, 2010.","title":"Every picture tells a story: Generating sentences from images","authors":"Ali Farhadi, Mohsen Hejrati, Mohammad Sadeghi, Peter Young, Cyrus Rashtchian, Julia Hockenmaier, David Forsyth","year":"2010","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In NIPS, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kang et al., 2015] Cuicui Kang, Shiming Xiang, Shengcai Liao, Changsheng Xu, and Chunhong Pan. Learning consistent feature representation for cross-modal multime- dia retrieval. IEEE Transactions on Multimedia (TMM), 17(3):370-381, 2015.","title":"Learning consistent feature representation for cross-modal multimedia retrieval","authors":"Kang","year":"2015","venue":"IEEE Transactions on Multimedia (TMM)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Larochelle et al., 2008] Hugo Larochelle, Dumitru Erhan, and Yoshua Bengio. Zero-data learning of new tasks. In AAAI, pages 646-651, 2008.","title":"Zero-data learning of new tasks","authors":"Larochelle","year":"2008","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Le and Mikolov, 2014] Quoc Le and Tomas Mikolov. Dis- tributed representations of sentences and documents. In ICML, pages 1188-1196, 2014.","title":"Quoc Le and Tomas Mikolov. Distributed representations of sentences and documents","authors":"Mikolov Le","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ledig et al., 2017] Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P. Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo-realistic single im- age super-resolution using a generative adversarial net- work. In CVPR, pages 105-114, 2017.","title":"Photo-realistic single image super-resolution using a generative adversarial network","authors":"Ledig","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2018] Linghui Li, Sheng Tang, Yongdong Zhang, Lixi Deng, and Qi Tian. Gla: Global-local attention for image description. IEEE Transactions on Multimedia, 20(3):726-737, 2018.","title":"Gla: Global-local attention for image description","authors":"Linghui Li, Sheng Tang, Yongdong Zhang, Lixi Deng, Qi Tian","year":"2018","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2017] Ruoyu Liu, Yao Zhao, Liang Zheng, Shikui Wei, and Yi Yang. A new evaluation protocol and benchmarking results for extendable cross-media retrieval. arXiv preprint arXiv:1703.03567, 2017.","title":"A new evaluation protocol and benchmarking results for extendable cross-media retrieval","authors":"Ruoyu Liu, Yao Zhao, Liang Zheng, Shikui Wei, Yi Yang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013.","title":"Efficient estimation of word representations in vector space","authors":"Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mirza and Osindero, 2014] Mehdi Mirza and Simon Osin- dero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014.","title":"Conditional generative adversarial nets","authors":"Osindero Mirza","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Peng et al., 2017] Yuxin Peng, Xin Huang, and Yunzhen Zhao. An overview of cross-media retrieval: Concepts, methodologies, benchmarks and challenges. IEEE Trans- actions on Circuits and Systems for Video Technology (TCSVT), 2017.","title":"An overview of cross-media retrieval: Concepts, methodologies, benchmarks and challenges","authors":"Yuxin Peng, Xin Huang, Yunzhen Zhao","year":"2017","venue":"IEEE Transactions on Circuits and Systems for Video Technology","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Radford et al., 2015] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.","title":"Unsupervised representation learning with deep convolutional generative adversarial networks","authors":"Radford","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rasiwasia et al., 2010] Nikhil Rasiwasia, Jose Costa Pereira, Emanuele Coviello, Gabriel Doyle, Gert RG Lanckriet, Roger Levy, and Nuno Vasconcelos. A new approach to cross-modal multimedia retrieval. In ACM MM, pages 251-260, 2010.","title":"A new approach to cross-modal multimedia retrieval","authors":"Nikhil Rasiwasia, Jose Costa Pereira, Emanuele Coviello, Gabriel Doyle, Gert Rg Lanckriet, Roger Levy, Nuno Vasconcelos","year":"2010","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2017] Bokun Wang, Yang Yang, Xing Xu, Alan Hanjalic, and Heng Tao Shen. Adversarial cross- modal retrieval. In ACM MM, pages 154-162, 2017.","title":"Adversarial crossmodal retrieval","authors":"Bokun Wang, Yang Yang, Xing Xu, Alan Hanjalic, Heng Tao Shen","year":"2017","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wei et al., 2017] Yunchao Wei, Yao Zhao, Canyi Lu, Shikui Wei, Luoqi Liu, Zhenfeng Zhu, and Shuicheng Yan. Cross-modal retrieval with cnn visual features: A new baseline. IEEE Transactions on Cybernetics (TCYB), 47(2):449-460, 2017.","title":"Cross-modal retrieval with cnn visual features: A new baseline","authors":"Yunchao Wei, Yao Zhao, Canyi Lu, Shikui Wei, Luoqi Liu, Zhenfeng Zhu, Shuicheng Yan","year":"2017","venue":"IEEE Transactions on Cybernetics (TCYB)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wu et al., 2014] Shuang Wu, Sravanthi Bondugula, Florian Luisier, Xiaodan Zhuang, and Pradeep Natarajan. Zero- shot event detection using multi-modal fusion of weakly supervised concepts. In CVPR, pages 2665-2672, 2014.","title":"Zeroshot event detection using multi-modal fusion of weakly supervised concepts","authors":"Shuang Wu, Sravanthi Bondugula, Florian Luisier, Xiaodan Zhuang, Pradeep Natarajan","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xian et al., 2017] Yongqin Xian, Bernt Schiele, and Zeynep Akata. Zero-shot learning-the good, the bad and the ugly. In CVPR, pages 3077-3086, 2017.","title":"Zero-shot learning-the good, the bad and the ugly","authors":"Xian","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yi et al., 2017] Zili Yi, Hao (Richard) Zhang, Ping Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image-to-image translation. In ICCV, pages 2868-2876, 2017.","title":"Dualgan: Unsupervised dual learning for image-to-image translation","authors":"Zili Yi, ( Hao, ) Richard, Ping Zhang, Minglun Tan, Gong","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhai et al., 2014] Xiaohua Zhai, Yuxin Peng, and Jianguo Xiao. Learning cross-media joint representation with sparse and semisupervised regularization. IEEE Trans- actions on Circuits and Systems for Video Technology (TCSVT), 24(6):965-978, 2014.","title":"Learning cross-media joint representation with sparse and semisupervised regularization","authors":"Xiaohua Zhai, Yuxin Peng, Jianguo Xiao","year":"2014","venue":"IEEE Transactions on Circuits and Systems for Video Technology (TCSVT)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhu et al., 2017] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image transla- tion using cycle-consistent adversarial networks. In ICCV, pages 2242-2251, 2017.","title":"Unpaired image-to-image translation using cycle-consistent adversarial networks","authors":"Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A Efros","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhuang et al., 2013] Yueting Zhuang, Yanfei Wang, Fei Wu, Yin Zhang, and Weiming Lu. Supervised coupled dictionary learning with group structures for multi-modal retrieval. In AAAI, pages 1070-1076, 2013.","title":"Supervised coupled dictionary learning with group structures for multi-modal retrieval","authors":"Yueting Zhuang, Yanfei Wang, Fei Wu, Yin Zhang, Weiming Lu","year":"2013","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}