{"type":"ScholarlyArticle","title":"Continual Compositional Zero-Shot Learning","authors":[{"name":"Yang Zhang","slug":"yang-zhang","given":"Yang","family":"Zhang","affiliation":"School of Computer Science and Technology , Beijing Jiaotong University , Beijing , China"},{"name":"Songhe Feng","slug":"songhe-feng","given":"Songhe","family":"Feng","affiliation":"Tangshan Research Institute , Beijing Jiaotong University , Beijing , China"},{"name":"Jiazheng Yuan","slug":"jiazheng-yuan","given":"Jiazheng","family":"Yuan","affiliation":"College of Science and Technology , Beijing Open University , Beijing , China"}],"abstract":"Compositional Zero-Shot Learning (CZSL) aims to recognize unseen compositions with the knowledge learned from seen compositions, where each composition is composed of two primitives (attribute and object). However, existing CZSL methods are designed to learn compositions from fixed primitive set, which cannot handle the continually expanding primitive set in real-world applications. In this paper, we propose a new CZSL setting, named Continual Compositional Zero-Shot Learning (CCZSL), which requires the model to recognize unseen compositions composed of learned primitive set while continually increasing the size of learned primitive set. Contextuality and catastrophic forgetting are the main issues to be addressed in this setting. Specifically, we capture similar contextuality in compositions through several learnable Super-Primitives that can modify the invariant primitive embedding to better adapt the contextuality in the corresponding composition. Then we introduce a dual knowledge distillation loss which aims at maintaining old knowledge learned from previous sessions and avoiding overfitting of new session. We design the CCZSL evaluation protocol and conduct extensive experiments on widely used benchmarks, demonstrating the superiority of our method compared to the state-of-the-art CZSL methods.","doi":"10.24963/ijcai.2024/191","url":"https://ijcai.org/proceedings/2024/191","pdf":"https://ijcai.org/proceedings/2024/0191.pdf","markdown":"https://ijcai.org/proceedings/2024/191.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/191","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":191,"pages":{"first":1724,"last":1732},"keywords":["Machine Learning: ML: Incremental learning","Computer Vision: CV: Transfer, low-shot, semi- and un- supervised learning"],"videos":[],"references":[{"position":1,"raw":"[Atzmon et al., 2016] Yuval Atzmon, Jonathan Berant, Vahid Kezami, Amir Globerson, and Gal Chechik. Learning to generalize to new compositions in image understanding. arXiv preprint arXiv:1608.07639, 2016.","title":"Learning to generalize to new compositions in image understanding","authors":"Atzmon","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chi et al., 2022] Zhixiang Chi, Li Gu, Huan Liu, Yang Wang, Yuanhao Yu, and Jin Tang. Metafscil: A meta- learning approach for few-shot class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14166- 14175, June 2022.","title":"Metafscil: A metalearning approach for few-shot class incremental learning","authors":"Zhixiang Chi, Li Gu, Huan Liu, Yang Wang, Yuanhao Yu, Jin Tang","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dhar et al., 2019] Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa. Learning without memorizing. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 5138-5146, 2019.","title":"Learning without memorizing","authors":"Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, Rama Chellappa","year":"2019","venue":"Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gautam et al., 2020] Chandan Gautam, Sethupathy Parameswaran, Ashish Mishra, and Suresh Sundaram. Generalized continual zero-shot learning. arXiv preprint arXiv:2011.08508, 2020.","title":"Generalized continual zero-shot learning","authors":"Chandan Gautam, Sethupathy Parameswaran, Ashish Mishra, Suresh Sundaram","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hao et al., 2023] Shaozhe Hao, Kai Han, and Kwan-Yee K Wong. Learning attention as disentangler for composi- tional zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15315-15324, 2023.","title":"Learning attention as disentangler for compositional zero-shot learning","authors":"Hao","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu and Wang, 2023] Xiaoming Hu and Zilei Wang. Lever- aging sub-class discimination for compositional zero-shot learning. In Proceedings of the AAAI Conference on Arti- ficial Intelligence, volume 37, pages 890-898, 2023.","title":"Leveraging sub-class discimination for compositional zero-shot learning","authors":"Xiaoming Hu, Zilei Wang","year":"2023","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huynh and Elhamifar, 2020] Dat Huynh and Ehsan Elham- ifar. A shared multi-attention framework for multi-label zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8776-8786, 2020.","title":"Dat Huynh and Ehsan Elhamifar. A shared multi-attention framework for multi-label zero-shot learning","authors":"Elhamifar Huynh","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Karthik et al., 2021] Shyamgopal Karthik, Massimiliano Mancini, and Zeynep Akata. Revisiting visual product for compositional zero-shot learning. In NeurIPS 2021 Work- shop on Distribution Shifts: Connecting Methods and Ap- plications, 2021.","title":"Revisiting visual product for compositional zero-shot learning","authors":"Karthik","year":"2021","venue":"NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kim et al., 2023] Hanjae Kim, Jiyoung Lee, Seongheon Park, and Kwanghoon Sohn. Hierarchical visual primitive experts for compositional zero-shot learning. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 5675-5685, 2023.","title":"Hierarchical visual primitive experts for compositional zero-shot learning","authors":"Kim","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kirkpatrick et al., 2017] James Kirkpatrick, Razvan Pas- canu, Neil Rabinowitz, Joel Veness, Guillaume Des- jardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcom- ing catastrophic forgetting in neural networks. Proceed- ings of the national academy of sciences, 114(13):3521- 3526, 2017.","title":"Overcoming catastrophic forgetting in neural networks","authors":"James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska","year":"2017","venue":"Proceedings of the national academy of sciences","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lake et al., 2017] Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017.","title":"Building machines that learn and think like people","authors":"Brenden M Lake, Joshua B Tomer D Ullman, Samuel J Tenenbaum, Gershman","year":"2017","venue":"Behavioral and brain sciences","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li and Hoiem, 2017] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pat- tern Analysis and Machine Intelligence, 40(12):2935- 2947, 2017.","title":"Learning without forgetting","authors":"Zhizhong Li, Derek Hoiem","year":"2017","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2020] Yong-Lu Li, Yue Xu, Xiaohan Mao, and Cewu Lu. Symmetry and group in attribute-object com- positions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11316- 11325, 2020.","title":"Symmetry and group in attribute-object compositions","authors":"Yong-Lu Li, Yue Xu, Xiaohan Mao, Cewu Lu","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2022] Xiangyu Li, Xu Yang, Kun Wei, Cheng Deng, and Muli Yang. Siamese contrastive embedding net- work for compositional zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 9326-9335, 2022.","title":"Siamese contrastive embedding network for compositional zero-shot learning","authors":"Xiangyu Li, Xu Yang, Kun Wei, Cheng Deng, Muli Yang","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2023] Zechao Li, Hao Tang, Zhimao Peng, Guo- Jun Qi, and Jinhui Tang. Knowledge-guided semantic transfer network for few-shot image recognition. IEEE Transactions on Neural Networks and Learning Systems, 2023.","title":"Knowledge-guided semantic transfer network for few-shot image recognition","authors":"Zechao Li, Hao Tang, Zhimao Peng, Guo-Jun Qi, Jinhui Tang","year":"2023","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Masana et al., 2022] Marc Masana, Xialei Liu, Bartłomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer. Class-incremental learning: sur- vey and performance evaluation on image classification. IEEE Transactions on Pattern Analysis and Machine In- telligence, 45(5):5513-5533, 2022.","title":"Class-incremental learning: survey and performance evaluation on image classification","authors":"Marc Masana, Xialei Liu, Bartłomiej Twardowski, Mikel Menta, Andrew D Bagdanov, Joost Van De Weijer","year":"2022","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Misra et al., 2017] Ishan Misra, Abhinav Gupta, and Mar- tial Hebert. From red wine to red tomato: Composition with context. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 1792-1801, 2017.","title":"From red wine to red tomato: Composition with context","authors":"Ishan Misra, Abhinav Gupta, Martial Hebert","year":"2017","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Naeem et al., 2021] Muhammad Ferjad Naeem, Yongqin Xian, Federico Tombari, and Zeynep Akata. Learning graph embeddings for compositional zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 953-962, 2021.","title":"Learning graph embeddings for compositional zero-shot learning","authors":"Naeem","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nagarajan and Grauman, 2018] Tushar Nagarajan and Kris- ten Grauman. Attributes as operators: factorizing unseen attribute-object compositions. In Proceedings of the Euro- pean Conference on Computer Vision (ECCV), pages 169- 185, 2018.","title":"Attributes as operators: factorizing unseen attribute-object compositions","authors":"Tushar Nagarajan, Kristen Grauman","year":"2018","venue":"Proceedings of the Euro-pean Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Palatucci et al., 2009] Mark Palatucci, Dean Pomerleau, Geoffrey E Hinton, and Tom M Mitchell. Zero-shot learn- ing with semantic output codes. Advances in Neural Infor- mation Processing Systems, 22, 2009.","title":"Zero-shot learning with semantic output codes","authors":"Mark Palatucci, Dean Pomerleau, Geoffrey E Hinton, Tom M Mitchell","year":"2009","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Saini et al., 2022] Nirat Saini, Khoi Pham, and Abhinav Shrivastava. Disentangling visual embeddings for at- tributes and objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13658-13667, 2022.","title":"Disentangling visual embeddings for attributes and objects","authors":"Nirat Saini, Khoi Pham, Abhinav Shrivastava","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Skorokhodov and Elhoseiny, 2020] Ivan Skorokhodov and Mohamed Elhoseiny. Class normalization for (contin- ual)? generalized zero-shot learning. arXiv preprint arXiv:2006.11328, 2020.","title":"Ivan Skorokhodov and Mohamed Elhoseiny. Class normalization for (continual)? generalized zero-shot learning","authors":"Elhoseiny Skorokhodov","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tang et al., 2020] Hao Tang, Zechao Li, Zhimao Peng, and Jinhui Tang. Blockmix: meta regularization and self- calibrated inference for metric-based meta-learning. In Proceedings of the 28th ACM international conference on multimedia, pages 610-618, 2020.","title":"Blockmix: meta regularization and selfcalibrated inference for metric-based meta-learning","authors":"Hao Tang, Zechao Li, Zhimao Peng, Jinhui Tang","year":"2020","venue":"Proceedings of the 28th ACM international conference on multimedia","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tang et al., 2022] Hao Tang, Chengcheng Yuan, Zechao Li, and Jinhui Tang. Learning attention-guided pyramidal fea- tures for few-shot fine-grained recognition. Pattern Recog- nition, 130:108792, 2022.","title":"Learning attention-guided pyramidal features for few-shot fine-grained recognition","authors":"Hao Tang, Chengcheng Yuan, Zechao Li, Jinhui Tang","year":"2022","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Titsias et al., 2019] Michalis K Titsias, Jonathan Schwarz, Alexander G de G Matthews, Razvan Pascanu, and Yee Whye Teh. Functional regularisation for contin- ual learning with gaussian processes. arXiv preprint arXiv:1901.11356, 2019.","title":"Functional regularisation for continual learning with gaussian processes","authors":"Jonathan Michalis K Titsias, Alexander G Schwarz, Razvan De G Matthews, Yee Whye Pascanu, Teh","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2023] Qingsheng Wang, Lingqiao Liu, Chenchen Jing, Hao Chen, Guoqiang Liang, Peng Wang, and Chunhua Shen. Learning conditional attributes for compositional zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11197-11206, 2023.","title":"Learning conditional attributes for compositional zero-shot learning","authors":"Wang","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wei et al., 2020] Kun Wei, Cheng Deng, Xu Yang, et al. Lifelong zero-shot learning. In IJCAI, pages 551-557, 2020.","title":"Lifelong zero-shot learning","authors":"Kun Wei, Cheng Deng, Xu Yang","year":"2020","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2020/77"},{"position":29,"raw":"[Wu et al., 2023a] Yanan Wu, Zhixiang Chi, Yang Wang, and Songhe Feng. Metagcd: Learning to continually learn in generalized category discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1655-1665, October 2023.","title":"Metagcd: Learning to continually learn in generalized category discovery","authors":"Wu","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wu et al., 2023b] Yanan Wu, Tengfei Liang, Songhe Feng, Yi Jin, Gengyu Lyu, Haojun Fei, and Yang Wang. Metazs- cil: A meta-learning approach for generalized zero-shot class incremental learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 10408-10416, 2023.","title":"Metazscil: A meta-learning approach for generalized zero-shot class incremental learning","authors":"Wu","year":"2023","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Xian et al., 2018] Yongqin Xian, Tobias Lorenz, Bernt Schiele, and Zeynep Akata. Feature generating networks for zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5542-5551, 2018.","title":"Feature generating networks for zero-shot learning","authors":"Xian","year":"2018","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yu and Grauman, 2014] Aron Yu and Kristen Grauman. Fine-grained visual comparisons with local learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 192-199, 2014.","title":"Fine-grained visual comparisons with local learning","authors":"Aron Yu, Kristen Grauman","year":"2014","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang and Feng, 2023] Yang Zhang and Songhe Feng. En- hancing domain-invariant parts for generalized zero-shot learning. In Proceedings of the 31st ACM International Conference on Multimedia, pages 6283-6291, 2023.","title":"Enhancing domain-invariant parts for generalized zero-shot learning","authors":"Feng Zhang, Yang Zhang, Songhe Feng","year":"2023","venue":"Proceedings of the 31st ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhang et al., 2020] Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Hem- ing Zhang, and C-C Jay Kuo. Class-incremental learn- ing via deep model consolidation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Com- puter Vision, pages 1131-1140, 2020.","title":"Class-incremental learning via deep model consolidation","authors":"Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, C-C Jay Kuo","year":"2020","venue":"Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhang et al., 2022] Tian Zhang, Kongming Liang, Ruoyi Du, Xian Sun, Zhanyu Ma, and Jun Guo. Learning in- variant visual representations for compositional zero-shot learning. In Proceedings of the European Conference on Computer Vision (ECCV), pages 339-355. Springer, 2022.","title":"Learning invariant visual representations for compositional zero-shot learning","authors":"Tian Zhang, Kongming Liang, Ruoyi Du, Xian Sun, Zhanyu Ma, Jun Guo","year":"2022","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}