{"type":"ScholarlyArticle","title":"Few-shot Visual Learning with Contextual Memory and Fine-grained Calibration","authors":[{"name":"Yuqing Ma","slug":"yuqing-ma","given":"Yuqing","family":"Ma","affiliation":"State Key Lab of Software Development Environment , Beihang University , China"},{"name":"Wei Liu","slug":"wei-liu","given":"Wei","family":"Liu","affiliation":"State Key Lab of Software Development Environment , Beihang University , China"},{"name":"Shihao Bai","slug":"shihao-bai","given":"Shihao","family":"Bai","affiliation":"State Key Lab of Software Development Environment , Beihang University , China"},{"name":"Qingyu Zhang","slug":"qingyu-zhang","given":"Qingyu","family":"Zhang","affiliation":"State Key Lab of Software Development Environment , Beihang University , China"},{"name":"Aishan Liu","slug":"aishan-liu","given":"Aishan","family":"Liu","affiliation":"State Key Lab of Software Development Environment , Beihang University , China"},{"name":"Weimin Chen","slug":"weimin-chen","given":"Weimin","family":"Chen","affiliation":"NetEase Fuxi AI Lab , Hangzhou , China"},{"name":"Xianglong Liu","slug":"xianglong-liu","given":"Xianglong","family":"Liu","affiliation":"State Key Lab of Software Development Environment , Beihang University , China; Beijing Advanced Innovation Center for Big Data-Based Precision Medicine , Beihang University , China"}],"abstract":"Few-shot learning aims to learn a model that can be readily adapted to new unseen classes (concepts) by accessing one or few examples. Despite the successful progress, most of the few-shot learning approaches, concentrating on either global or local characteristics of examples, still suffer from weak generalization abilities. Inspired by the inverted pyramid theory, to address this problem, we propose an inverted pyramid network (IPN) that intimates the human's coarse-to-fine cognition paradigm. The proposed IPN consists of two consecutive stages, namely global stage and local stage. At the global stage, a class-sensitive contextual memory network (CCMNet) is introduced to learn discriminative support-query relation embeddings and predict the query-to-class similarity based on the contextual memory. Then at the local stage, a fine-grained calibration is further appended to complement the coarse relation embeddings, targeting more precise query-to-class similarity evaluation. To the best of our knowledge, IPN is the first work that simultaneously integrates both global and local characteristics in few-shot learning, approximately imitating the human cognition mechanism. Our extensive experiments on multiple benchmark datasets demonstrate the superiority of IPN, compared to a number of state-of-the-art approaches.","doi":"10.24963/ijcai.2020/113","url":"https://ijcai.org/proceedings/2020/113","pdf":"https://ijcai.org/proceedings/2020/0113.pdf","markdown":"https://ijcai.org/proceedings/2020/113.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/113","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":113,"pages":{"first":811,"last":817},"keywords":["Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Computer Vision: Structural and Model-Based Approaches, Knowledge Representation and Reasoning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24905/file_24905.m3u8","length_sec":269,"provider":"hls"}],"references":[{"position":1,"raw":"[Ahissar and Hochstein, 2004] Merav Ahissar and Shaul Hochstein. The reverse hierarchy theory of visual percep- tual learning. Trends in cognitive sciences, 8(10):457-464, 2004.","title":"The reverse hierarchy theory of visual perceptual learning","authors":"Hochstein Ahissar, Merav Ahissar, Shaul Hochstein","year":"2004","venue":"Trends in cognitive sciences","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2019] Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu- Chiang Frank Wang, and Jia-Bin Huang. A closer look at few- shot classification, 2019.","title":"A closer look at fewshot classification","authors":"Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank, Wang, Jia-Bin Huang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.","title":"Learning phrase representations using rnn encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical im- age database. In 2009 IEEE Computer Vision and Pattern Recog- nition, pages 248-255. Ieee, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"IEEE Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Finn et al., 2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep net- works. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1126-1135. JMLR. org, 2017.","title":"Model-agnostic meta-learning for fast adaptation of deep networks","authors":"Finn","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Finn et al., 2018] Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In Advances in Neu- ral Information Processing Systems, pages 9516-9527, 2018.","title":"Probabilistic model-agnostic meta-learning","authors":"Finn","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Garcia and Bruna, 2017] Victor Garcia and Joan Bruna. Few- shot learning with graph neural networks. arXiv preprint arXiv:1711.04043, 2017.","title":"Fewshot learning with graph neural networks","authors":"Bruna ; Garcia, Victor Garcia, Joan Bruna","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gidaris and Komodakis, 2018] Spyros Gidaris and Nikos Ko- modakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE Computer Vision and Pattern Recog- nition, pages 4367-4375, 2018.","title":"Dynamic few-shot visual learning without forgetting","authors":"Spyros Gidaris, Nikos Komodakis","year":"2018","venue":"Proceedings of the IEEE Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gidaris and Komodakis, 2019] Spyros Gidaris and Nikos Ko- modakis. Generating classification weights with gnn de- noising autoencoders for few-shot learning. arXiv preprint arXiv:1905.01102, 2019.","title":"Generating classification weights with gnn denoising autoencoders for few-shot learning","authors":"Spyros Gidaris, Nikos Komodakis","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gordon et al., 2018] Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard E Turner. Meta- learning probabilistic inference for prediction. arXiv preprint arXiv:1805.09921, 2018.","title":"Metalearning probabilistic inference for prediction","authors":"Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, Richard E Turner","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ji et al., 2013] Zhong Ji, Jing Wang, Yuting Su, Zhanjie Song, and Shikai Xing. Balance between object and background: Object- enhanced features for scene image classification. Neurocomput- ing, 120:15-23, 2013.","title":"Balance between object and background: Objectenhanced features for scene image classification","authors":"Zhong Ji, Jing Wang, Yuting Su, Zhanjie Song, Shikai Xing","year":"2013","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kim et al., 2019] Jongmin Kim, Taesup Kim, Sungwoong Kim, and Chang D Yoo. Edge-labeling graph neural network for few- shot learning. In Proceedings of the IEEE Computer Vision and Pattern Recognition, pages 11-20, 2019.","title":"Edge-labeling graph neural network for fewshot learning","authors":"Jongmin Kim, Taesup Kim, Sungwoong Kim, Chang D Yoo","year":"2019","venue":"Proceedings of the IEEE Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2017] Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint arXiv:1707.09835, 2017.","title":"Meta-sgd: Learning to learn quickly for few-shot learning","authors":"Zhenguo Li, Fengwei Zhou, Fei Chen, Hang Li","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2019a] Hongyang Li, David Eigen, Samuel Dodge, Matthew Zeiler, and Xiaogang Wang. Finding task-relevant fea- tures for few-shot learning by category traversal. In Proceedings of the IEEE Computer Vision and Pattern Recognition, pages 1- 10, 2019.","title":"Finding task-relevant features for few-shot learning by category traversal","authors":"Li","year":"2019","venue":"Proceedings of the IEEE Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2019b] Wenbin Li, Lei Wang, Jinglin Xu, Jing Huo, Yang Gao, and Jiebo Luo. Revisiting local descriptor based image- to-class measure for few-shot learning. In Proceedings of the IEEE Computer Vision and Pattern Recognition, pages 7260- 7268, 2019.","title":"Revisiting local descriptor based imageto-class measure for few-shot learning","authors":"Li","year":"2019","venue":"Proceedings of the IEEE Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2019c] Wenbin Li, Jinglin Xu, Jing Huo, Lei Wang, Yang Gao, and Jiebo Luo. Distribution consistency based covariance metric networks for few-shot learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8642-8649, 2019.","title":"Distribution consistency based covariance metric networks for few-shot learning","authors":"Li","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2018] Yanbin Liu, Juho Lee, Minseop Park, Saehoon Kim, Eunho Yang, Sung Ju Hwang, and Yi Yang. Learning to propagate labels: Transductive propagation network for few-shot learning. arXiv preprint arXiv:1805.10002, 2018.","title":"Learning to propagate labels: Transductive propagation network for few-shot learning","authors":"Yanbin Liu, Juho Lee, Minseop Park, Saehoon Kim, Eunho Yang, Sung Ju Hwang, Yi Yang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mishra et al., 2017] Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. arXiv preprint arXiv:1707.03141, 2017.","title":"A simple neural attentive metalearner","authors":"Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, Pieter Abbeel","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nichol et al., 2018] Alex Nichol, Joshua Achiam, and John Schul- man. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999, 2018.","title":"On first-order meta-learning algorithms","authors":"Alex Nichol, Joshua Achiam, John Schulman","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Oreshkin et al., 2018] Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems, pages 721-731, 2018.","title":"Tadam: Task dependent adaptive metric for improved few-shot learning","authors":"Boris Oreshkin, Pau Rodríguez López, Alexandre Lacoste","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Qiao et al., 2018] Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L Yuille. Few-shot image recognition by predicting parameters from activations. In Proceedings of the IEEE Computer Vision and Pattern Recognition, pages 7229-7238, 2018.","title":"Few-shot image recognition by predicting parameters from activations","authors":"Siyuan Qiao, Chenxi Liu, Wei Shen, Alan L Yuille","year":"2018","venue":"Proceedings of the IEEE Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ravi and Larochelle, 2017] Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2017.","title":"Optimization as a model for few-shot learning","authors":"Larochelle ; Ravi, Sachin Ravi, Hugo Larochelle","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ren et al., 2018] Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi-supervised few-shot classification. arXiv preprint arXiv:1803.00676, 2018.","title":"Meta-learning for semi-supervised few-shot classification","authors":"Ren","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Rusu et al., 2018] Andrei A Rusu, Dushyant Rao, Jakub Syg- nowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv preprint arXiv:1807.05960, 2018.","title":"Meta-learning with latent embedding optimization","authors":"Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, Raia Hadsell","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Snell et al., 2017] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neu- ral Information Processing Systems, pages 4077-4087, 2017.","title":"Prototypical networks for few-shot learning","authors":"Jake Snell, Kevin Swersky, Richard Zemel","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sun et al., 2019] Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. In Proceedings of the IEEE Computer Vision and Pattern Recogni- tion, pages 403-412, 2019.","title":"Meta-transfer learning for few-shot learning","authors":"Sun","year":"2019","venue":"Proceedings of the IEEE Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sung et al., 2018] Flood Sung, Yongxin Yang, Li Zhang, Tao Xi- ang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceed- ings of the IEEE Computer Vision and Pattern Recognition, pages 1199-1208, 2018.","title":"Learning to compare: Relation network for few-shot learning","authors":"Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Timothy M Philip Hs Torr, Hospedales","year":"2018","venue":"Proceedings of the IEEE Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Vinyals et al., 2016] Oriol Vinyals, Charles Blundell, Timothy Lil- licrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pages 3630-3638, 2016.","title":"Matching networks for one shot learning","authors":"Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yoon et al., 2019] Sung Whan Yoon, Jun Seo, and Jaekyun Moon. Tapnet: Neural network augmented with task-adaptive projection for few-shot learning. arXiv preprint arXiv:1905.06549, 2019.","title":"Tapnet: Neural network augmented with task-adaptive projection for few-shot learning","authors":"Sung Whan Yoon, Jun Seo, Jaekyun Moon","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zagoruyko and Komodakis, 2016] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.","title":"Wide residual networks","authors":"Sergey Zagoruyko, Nikos Komodakis","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}