{"type":"ScholarlyArticle","title":"Lifelong Zero-Shot Learning","authors":[{"name":"Kun Wei","slug":"kun-wei","given":"Kun","family":"Wei","affiliation":"School of Electronic Engineering , Xidian University , Xi'an 710071 , China"},{"name":"Cheng Deng","slug":"cheng-deng","given":"Cheng","family":"Deng","affiliation":"School of Electronic Engineering , Xidian University , Xi'an 710071 , China"},{"name":"Xu Yang","slug":"xu-yang","given":"Xu","family":"Yang","affiliation":"School of Electronic Engineering , Xidian University , Xi'an 710071 , China"}],"abstract":"Zero-Shot Learning (ZSL) handles the problem that some testing classes never appear in training set. Existing ZSL methods are designed for learning from a fixed training set, which do not have the ability to capture and accumulate the knowledge of multiple training sets, causing them infeasible to many real-world applications. In this paper, we propose a new ZSL setting, named as Lifelong Zero-Shot Learning (LZSL), which aims to accumulate the knowledge during the learning from multiple datasets and recognize unseen classes of all trained datasets. Besides, a novel method is conducted to realize LZSL, which effectively alleviates the Catastrophic Forgetting in the continuous training process. Specifically, considering those datasets containing different semantic embeddings, we utilize Variational Auto-Encoder to obtain unified semantic representations. Then, we leverage selective retraining strategy to preserve the\r\ntrained weights of previous tasks and avoid negative transfer when fine-tuning the entire model. Finally, knowledge distillation is employed to transfer knowledge from previous training stages to current stage. We also design the LZSL evaluation protocol and the challenging benchmarks. Extensive experiments on these benchmarks indicate that our method tackles LZSL problem effectively, while existing ZSL methods fail.","doi":"10.24963/ijcai.2020/77","url":"https://ijcai.org/proceedings/2020/77","pdf":"https://ijcai.org/proceedings/2020/0077.pdf","markdown":"https://ijcai.org/proceedings/2020/77.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/77","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":77,"pages":{"first":551,"last":557},"keywords":["Machine Learning: Transfer, Adaptation, Multi-task Learning","Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Machine Learning: Deep Generative Models"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24182/file_24182.m3u8","length_sec":228,"provider":"hls"}],"references":[{"position":1,"raw":"[Chang et al., 2020] Dongliang Chang, Yifeng Ding, Jiyang Xie, Ayan Kumar Bhunia, Xiaoxu Li, Zhanyu Ma, Ming Wu, Jun Guo, and Yi-Zhe Song. The devil is in the chan- nels: Mutual-channel loss for fine-grained image classifi- cation. IEEE Transactions on Image Processing, 29:4683- 4695, 2020.","title":"The devil is in the channels: Mutual-channel loss for fine-grained image classification","authors":"Dongliang Chang, Yifeng Ding, Jiyang Xie, Ayan Kumar Bhunia, Xiaoxu Li, Zhanyu Ma, Ming Wu, Jun Guo, Yi-Zhe Song","year":"2020","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2018] Long Chen, Hanwang Zhang, Jun Xiao, Wei Liu, and Shih-Fu Chang. Zero-shot visual recog- nition using semantics-preserving adversarial embedding networks. In CVPR, pages 1043-1052, 2018.","title":"Zero-shot visual recognition using semantics-preserving adversarial embedding networks","authors":"Long Chen, Hanwang Zhang, Jun Xiao, Wei Liu, Shih-Fu Chang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2019] Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Data-free learning of student net- works. In ICCV, pages 3514-3522, 2019.","title":"Data-free learning of student networks","authors":"Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, Qi Tian","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Farhadi et al., 2009] Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. Describing objects by their attributes. In CVPR, pages 1778-1785. IEEE, 2009.","title":"Describing objects by their attributes","authors":"Ali Farhadi, Ian Endres, Derek Hoiem, David Forsyth","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Felix et al., 2018] Rafael Felix, Vijay BG Kumar, Ian Reid, and Gustavo Carneiro. Multi-modal cycle-consistent gen- eralized zero-shot learning. In ECCV, pages 21-37, 2018.","title":"Multi-modal cycle-consistent generalized zero-shot learning","authors":"Rafael Felix, B G Vijay, Ian Kumar, Gustavo Reid, Carneiro","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In Nips, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"Nips","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kingma and Welling, 2013] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv, 2013.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li and Hoiem, 2017] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell, 40(12):2935-2947, 2017.","title":"Learning without forgetting","authors":"Zhizhong Li, Derek Hoiem","year":"2017","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liu et al., 2018] Xialei Liu, Marc Masana, Luis Herranz, Joost Van de Weijer, Antonio M Lopez, and Andrew D Bagdanov. Rotate your networks: Better weight consol- idation and less catastrophic forgetting. In ICPR, pages 2262-2268. IEEE, 2018.","title":"Rotate your networks: Better weight consolidation and less catastrophic forgetting","authors":"Liu","year":"2018","venue":"ICPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[McCloskey and Cohen, 1989] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165. Elsevier, 1989.","title":"Catastrophic interference in connectionist networks: The sequential learning problem","authors":"Cohen Mccloskey, Michael Mccloskey, Neal J Cohen","year":"1989","venue":"Psychology of learning and motivation","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Patterson and Hays, 2012] Genevieve Patterson and James Hays. Sun attribute database: Discovering, annotating, and recognizing scene attributes. In CVPR, pages 2751-2758. IEEE, 2012.","title":"Sun attribute database: Discovering, annotating, and recognizing scene attributes","authors":"Genevieve Patterson, James Hays","year":"2012","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Rebuffi et al., 2017] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In CVPR, pages 2001-2010, 2017.","title":"icarl: Incremental classifier and representation learning","authors":"Sylvestre-Alvise, Alexander Rebuffi, Georg Kolesnikov, Christoph H Sperl, Lampert","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Schonfeld et al., 2019] Edgar Schonfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, and Zeynep Akata. Gen- eralized zero-and few-shot learning via aligned variational autoencoders. In CVPR, pages 8247-8255, 2019.","title":"Generalized zero-and few-shot learning via aligned variational autoencoders","authors":"Edgar Schonfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, Zeynep Akata","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Shin et al., 2017] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep gener- ative replay. In Nips, pages 2990-2999, 2017.","title":"Continual learning with deep generative replay","authors":"Hanul Shin, Jung Kwon Lee, Jaehong Kim, Jiwon Kim","year":"2017","venue":"Nips","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Socher et al., 2013] Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng. Zero-shot learning through cross-modal transfer. In Nips, pages 935-943, 2013.","title":"Zero-shot learning through cross-modal transfer","authors":"Richard Socher, Milind Ganjoo, Christopher D Manning, Andrew Ng","year":"2013","venue":"Nips","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wah et al., 2011] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech- ucsd birds-200-2011 dataset. 2011.","title":"The caltechucsd birds-200-2011 dataset","authors":"Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, Serge Belongie","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wei et al., 2019] Kun Wei, Muli Yang, Hao Wang, Cheng Deng, and Xianglong Liu. Adversarial fine-grained com- position learning for unseen attribute-object recognition. In ICCV, pages 3741-3749, 2019.","title":"Adversarial fine-grained composition learning for unseen attribute-object recognition","authors":"Kun Wei, Muli Yang, Hao Wang, Cheng Deng, Xianglong Liu","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wu et al., 2018] Chenshen Wu, Luis Herranz, Xialei Liu, Joost van de Weijer, Bogdan Raducanu, et al. Memory replay gans: Learning to generate new categories without forgetting. In Nips, pages 5962-5972, 2018.","title":"Memory replay gans: Learning to generate new categories without forgetting","authors":"Wu","year":"2018","venue":"Nips","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xian et al., 2018a] Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly. IEEE Trans. Pattern Anal. Mach. Intell, 2018.","title":"Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly","authors":"Xian","year":"2018","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Xian et al., 2018b] Yongqin Xian, Tobias Lorenz, Bernt Schiele, and Zeynep Akata. Feature generating networks for zero-shot learning. In CVPR, pages 5542-5551, 2018.","title":"Feature generating networks for zero-shot learning","authors":"Xian","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xu et al., 2019] Xinyi Xu, Huanhuan Cao, Yanhua Yang, Erkun Yang, and Cheng Deng. Zero-shot metric learning. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 3996-4002. International Joint Conferences on Artificial Intelligence Organization, 7 2019.","title":"Zero-shot metric learning","authors":"Xinyi Xu, Huanhuan Cao, Yanhua Yang, Erkun Yang, Cheng Deng","year":"2019","venue":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yang et al., 2019] Xu Yang, Cheng Deng, Feng Zheng, Junchi Yan, and Wei Liu. Deep spectral clustering using dual autoencoder network. In CVPR, June 2019.","title":"Deep spectral clustering using dual autoencoder network","authors":"Xu Yang, Cheng Deng, Feng Zheng, Junchi Yan, Wei Liu","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yoon et al., 2017] Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynami- cally expandable networks. arXiv, 2017.","title":"Lifelong learning with dynamically expandable networks","authors":"Yoon","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2017] Li Zhang, Tao Xiang, and Shaogang Gong. Learning a deep embedding model for zero-shot learning. In CVPR, pages 2021-2030, 2017.","title":"Learning a deep embedding model for zero-shot learning","authors":"Li Zhang, Tao Xiang, Shaogang Gong","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhao et al., 2018] Bo Zhao, Xinwei Sun, Yanwei Fu, Yuan Yao, and Yizhou Wang. Msplit lbi: Realizing feature se- lection and dense estimation simultaneously in few-shot and zero-shot learning. ICML, 2018.","title":"Msplit lbi: Realizing feature selection and dense estimation simultaneously in few-shot and zero-shot learning","authors":"Bo Zhao, Xinwei Sun, Yanwei Fu, Yuan Yao, Yizhou Wang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhao et al., 2019] Bo Zhao, Xinwei Sun, Xiaopeng Hong, Yuan Yao, and Yizhou Wang. Zero-shot learning via re- current knowledge transfer. In WACV, pages 1308-1317. IEEE, 2019.","title":"Zero-shot learning via recurrent knowledge transfer","authors":"Bo Zhao, Xinwei Sun, Xiaopeng Hong, Yuan Yao, Yizhou Wang","year":"2019","venue":"WACV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhu et al., 2018] Yizhe Zhu, Mohamed Elhoseiny, Bingchen Liu, Xi Peng, and Ahmed Elgammal. A generative adversarial approach for zero-shot learning from noisy texts. In CVPR, pages 1004-1013, 2018.","title":"A generative adversarial approach for zero-shot learning from noisy texts","authors":"Zhu","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}