{"type":"ScholarlyArticle","title":"Learning Prototype via Placeholder for Zero-shot Recognition","authors":[{"name":"Zaiquan Yang","slug":"zaiquan-yang","given":"Zaiquan","family":"Yang","affiliation":"Beihang University"},{"name":"Yang Liu","slug":"yang-liu","given":"Yang","family":"Liu","affiliation":"Beihang University"},{"name":"Wenjia Xu","slug":"wenjia-xu","given":"Wenjia","family":"Xu","affiliation":"Beijing University of Posts and Telecommunications"},{"name":"Chong Huang","slug":"chong-huang","given":"Chong","family":"Huang","affiliation":"Beihang University"},{"name":"Lei Zhou","slug":"lei-zhou","given":"Lei","family":"Zhou","affiliation":"Beihang University"},{"name":"Chao Tong","slug":"chao-tong","given":"Chao","family":"Tong","affiliation":"Beihang University"}],"abstract":"Zero-shot learning (ZSL) aims to recognize unseen classes by exploiting semantic descriptions shared between seen classes and unseen classes. \r\nCurrent methods show that it is effective to learn visual-semantic alignment by projecting semantic embeddings into the visual space as class prototypes. However, such a projection function is only concerned with seen classes. When applied to unseen classes, the prototypes often perform suboptimally due to domain shift. \r\nIn this paper, we propose to learn prototypes via placeholders, termed LPL, to eliminate the domain shift between seen and unseen classes. Specifically, we combine seen classes to hallucinate new classes which play as placeholders of the unseen classes in the visual and semantic space. Placed between seen classes, the placeholders encourage prototypes of seen classes to be highly dispersed. And more space is spared for the insertion of well-separated unseen ones. Empirically, well-separated prototypes help counteract visual-semantic misalignment caused by domain shift.\r\nFurthermore, we exploit a novel semantic-oriented fine-tuning method to guarantee the semantic reliability of placeholders. \r\nExtensive experiments on five benchmark datasets demonstrate the significant performance gain of LPL over the state-of-the-art methods.","doi":"10.24963/ijcai.2022/217","url":"https://ijcai.org/proceedings/2022/217","pdf":"https://ijcai.org/proceedings/2022/0217.pdf","markdown":"https://ijcai.org/proceedings/2022/217.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/217","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":217,"pages":{"first":1559,"last":1565},"keywords":["Computer Vision: Recognition (object detection, categorization)","Computer Vision: Representation Learning","Computer Vision: Vision and language","Computer Vision: Transfer, low-shot, semi- and un- supervised learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986464","length_sec":83,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984607","length_sec":331,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bucher et al., 2016] Maxime Bucher, Stéphane Herbin, and Frédéric Jurie. Improving semantic embedding consis- tency by metric learning for zero-shot classiffication. In Proc. of ECCV, 2016.","title":"Improving semantic embedding consistency by metric learning for zero-shot classiffication","authors":"Maxime Bucher, Stéphane Herbin, Frédéric Jurie","year":"2016","venue":"Proc. of ECCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2021a] Shiming Chen, Wenjie Wang, Beihao Xia, Qinmu Peng, Xinge You, Feng Zheng, and Ling Shao. Free: Feature refinement for generalized zero-shot learn- ing. In Proc. of ICCV, pages 122-131, 2021.","title":"Free: Feature refinement for generalized zero-shot learning","authors":"Chen","year":"2021","venue":"Proc. of ICCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2021b] Shiming Chen, Guosen Xie, Yang Liu, Qinmu Peng, Baigui Sun, Hao Li, Xinge You, and Ling Shao. Hsva: Hierarchical semantic-visual adaptation for zero-shot learning. Proc. of NeurIPS, 2021.","title":"Hsva: Hierarchical semantic-visual adaptation for zero-shot learning","authors":"Chen","year":"2021","venue":"Proc. of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Farhadi et al., 2009] Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. Describing objects by their attributes. In Proc. of CVPR, 2009.","title":"Describing objects by their attributes","authors":"Ali Farhadi, Ian Endres, Derek Hoiem, David Forsyth","year":"2009","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fu et al., 2015] Yanwei Fu, Timothy M Hospedales, Tao Xiang, and Shaogang Gong. Transductive multi-view zero-shot learning. IEEE TPAMI, 2015.","title":"Transductive multi-view zero-shot learning","authors":"Yanwei Fu, Timothy M Hospedales, Tao Xiang, Shaogang Gong","year":"2015","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Han et al., 2021] Zongyan Han, Zhenyong Fu, Shuo Chen, and Jian Yang. Contrastive embedding for generalized zero-shot learning. In Proc. of CVPR, 2021.","title":"Contrastive embedding for generalized zero-shot learning","authors":"Han","year":"2021","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Huang et al., 2020] Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves cross- domain generalization. In Proc. of ECCV, 2020.","title":"Self-challenging improves crossdomain generalization","authors":"Zeyi Huang, Haohan Wang, Eric P Xing, Dong Huang","year":"2020","venue":"Proc. of ECCV","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jiang et al., 2019] Huajie Jiang, Ruiping Wang, Shiguang Shan, and Xilin Chen. Transferable contrastive network for generalized zero-shot learning. In Proc. of ICCV, 2019.","title":"Transferable contrastive network for generalized zero-shot learning","authors":"Huajie Jiang, Ruiping Wang, Shiguang Shan, Xilin Chen","year":"2019","venue":"Proc. of ICCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kodirov et al., 2017] Elyor Kodirov, Tao Xiang, and Shao- gang Gong. Semantic autoencoder for zero-shot learning. In Proc. of CVPR, 2017.","title":"Elyor Kodirov, Tao Xiang, and Shaogang Gong. Semantic autoencoder for zero-shot learning","authors":"Kodirov","year":"2017","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lampert et al., 2009] Christoph H Lampert, Hannes Nick- isch, and Stefan Harmeling. Learning to detect unseen ob- ject classes by between-class attribute transfer. In Proc. of CVPR, 2009.","title":"Learning to detect unseen object classes by between-class attribute transfer","authors":"Hannes Christoph H Lampert, Stefan Nickisch, Harmeling","year":"2009","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2019] Kai Li, Martin Renqiang Min, and Yun Fu. Rethinking zero-shot learning: A conditional visual clas- sification perspective. In Proc. of ICCV, 2019.","title":"Rethinking zero-shot learning: A conditional visual classification perspective","authors":"Kai Li, Martin Renqiang Min, Yun Fu","year":"2019","venue":"Proc. of ICCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2020] Yang Liu, Lei Zhou, Xiao Bai, Lin Gu, Tatsuya Harada, and Jun Zhou. Information bottleneck constrained latent bidirectional embedding for zero-shot learning. arXiv preprint arXiv:2009.07451, 2020.","title":"Information bottleneck constrained latent bidirectional embedding for zero-shot learning","authors":"Yang Liu, Lei Zhou, Xiao Bai, Lin Gu, Tatsuya Harada, Jun Zhou","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2021] Yang Liu, Lei Zhou, Xiao Bai, Yifei Huang, Lin Gu, Jun Zhou, and Tatsuya Harada. Goal- oriented gaze estimation for zero-shot learning. In Proc. of CVPR, 2021.","title":"Goaloriented gaze estimation for zero-shot learning","authors":"Yang Liu, Lei Zhou, Xiao Bai, Yifei Huang, Lin Gu, Jun Zhou, Tatsuya Harada","year":"2021","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Min et al., 2020] Shaobo Min, Hantao Yao, Hongtao Xie, Chaoqun Wang, Zheng-Jun Zha, and Yongdong Zhang. Domain-aware visual bias eliminating for generalized zero-shot learning. In Proc. of CVPR, 2020.","title":"Domain-aware visual bias eliminating for generalized zero-shot learning","authors":"Hantao Shaobo Min, Hongtao Yao, Chaoqun Xie, Zheng-Jun Wang, Yongdong Zha, Zhang","year":"2020","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Narayan et al., 2020] Sanath Narayan, Akshita Gupta, Fa- had Shahbaz Khan, Cees GM Snoek, and Ling Shao. La- tent embedding feedback and discriminative features for zero-shot classification. In Proc. of ECCV, 2020.","title":"Latent embedding feedback and discriminative features for zero-shot classification","authors":"Sanath Narayan, Akshita Gupta, Fahad Shahbaz Khan, G M Cees, Ling Snoek, Shao","year":"2020","venue":"Proc. of ECCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nilsback and Zisserman, 2008] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Confer- ence on Computer Vision, Graphics & Image Processing, 2008.","title":"Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes","authors":"Zisserman Nilsback","year":"2008","venue":"Sixth Indian Conference on Computer Vision, Graphics & Image Processing","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Patterson and Hays, 2012] Genevieve Patterson and James Hays. Sun attribute database: Discovering, annotating, and recognizing scene attributes. In Proc. of CVPR, 2012.","title":"Sun attribute database: Discovering, annotating, and recognizing scene attributes","authors":"Genevieve Patterson, James Hays","year":"2012","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Radovanovic et al., 2010] Milos Radovanovic, Alexandros Nanopoulos, and Mirjana Ivanovic. Hubs in space: Popu- lar nearest neighbors in high-dimensional data. Journal of Machine Learning Research, 2010.","title":"Hubs in space: Popular nearest neighbors in high-dimensional data","authors":"Radovanovic","year":"2010","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Reed et al., 2016] Scott Reed, Zeynep Akata, Honglak Lee, and Bernt Schiele. Learning deep representations of fine- grained visual descriptions. In Proc. of CVPR, 2016.","title":"Learning deep representations of finegrained visual descriptions","authors":"Scott Reed, Zeynep Akata, Honglak Lee, Bernt Schiele","year":"2016","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Schonfeld et al., 2019] Edgar Schonfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, and Zeynep Akata. Gen- eralized zero-and few-shot learning via aligned variational autoencoders. In Proc. of CVPR, 2019.","title":"Generalized zero-and few-shot learning via aligned variational autoencoders","authors":"Edgar Schonfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, Zeynep Akata","year":"2019","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Skorokhodov and Elhoseiny, 2020] Ivan Skorokhodov and Mohamed Elhoseiny. Class normalization for (continual)? generalized zero-shot learning. In International Confer- ence on Learning Representations, 2020.","title":"Ivan Skorokhodov and Mohamed Elhoseiny. Class normalization for (continual)? generalized zero-shot learning","authors":"Elhoseiny Skorokhodov","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Verma et al., 2019] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better rep- resentations by interpolating hidden states. In Proc. of ICML, 2019.","title":"Manifold mixup: Better representations by interpolating hidden states","authors":"Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, Yoshua Bengio","year":"2019","venue":"Proc. of ICML","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Welinder et al., 2010] P Welinder, S Branson, T Mita, C Wah, F Schroff, S Belongie, and P Perona. Caltech-ucsd birds 200. technical report cns-tr-2010-001. California In- stitute of Technology, 2010.","title":"California Institute of Technology","authors":"Welinder, T Branson, C Mita, F Wah, S Schroff, Belongie, Perona","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xian et al., 2017] Yongqin Xian, Bernt Schiele, and Zeynep Akata. Zero-shot learning-the good, the bad and the ugly. In Proc. of CVPR, 2017.","title":"Zero-shot learning-the good, the bad and the ugly","authors":"Xian","year":"2017","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xian et al., 2019] Yongqin Xian, Saurabh Sharma, Bernt Schiele, and Zeynep Akata. f-vaegan-d2: A feature gener- ating framework for any-shot learning. In Proc. of CVPR, 2019.","title":"f-vaegan-d2: A feature generating framework for any-shot learning","authors":"Xian","year":"2019","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xie et al., 2019] Guo-Sen Xie, Li Liu, Xiaobo Jin, Fan Zhu, Zheng Zhang, Jie Qin, Yazhou Yao, and Ling Shao. Atten- tive region embedding network for zero-shot learning. In Proc. of CVPR, 2019.","title":"Attentive region embedding network for zero-shot learning","authors":"Guo-Sen Xie, Li Liu, Xiaobo Jin, Fan Zhu, Zheng Zhang, Jie Qin, Yazhou Yao, Ling Shao","year":"2019","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xie et al., 2020] Guo-Sen Xie, Li Liu, Fan Zhu, Fang Zhao, Zheng Zhang, Yazhou Yao, Jie Qin, and Ling Shao. Re- gion graph embedding network for zero-shot learning. In Proc. of ECCV, 2020.","title":"Region graph embedding network for zero-shot learning","authors":"Guo-Sen Xie, Li Liu, Fan Zhu, Fang Zhao, Zheng Zhang, Yazhou Yao, Jie Qin, Ling Shao","year":"2020","venue":"Proc. of ECCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xu et al., 2020] Wenjia Xu, Yongqin Xian, Jiuniu Wang, Bernt Schiele, and Zeynep Akata. Attribute prototype net- work for zero-shot learning. Proc. of NeurIPS, 2020.","title":"Attribute prototype network for zero-shot learning","authors":"Xu","year":"2020","venue":"Proc. of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang and Wang, 2021] Weilin Zhang and Yu-Xiong Wang. Hallucination improves few-shot object detection. In Proc. of CVPR, 2021.","title":"Hallucination improves few-shot object detection","authors":"Weilin Zhang, Yu-Xiong Wang","year":"2021","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}