{"type":"ScholarlyArticle","title":"Learning Prototype Classifiers for Long-Tailed Recognition","authors":[{"name":"Saurabh Sharma","slug":"saurabh-sharma","given":"Saurabh","family":"Sharma","affiliation":"University of California Santa Barbara , USA"},{"name":"Yongqin Xian","slug":"yongqin-xian","given":"Yongqin","family":"Xian","affiliation":"Google , Switzerland"},{"name":"Ning Yu","slug":"ning-yu","given":"Ning","family":"Yu","affiliation":"Salesforce Research , USA"},{"name":"Ambuj Singh","slug":"ambuj-singh","given":"Ambuj","family":"Singh","affiliation":"University of California Santa Barbara , USA"}],"abstract":"The problem of long-tailed recognition (LTR) has received attention in recent years due to the fundamental power-law distribution of objects in the real-world. Most recent works in LTR use softmax classifiers that are biased in that they correlate classifier norm with the amount of training data for a given class. In this work, we show that learning prototype classifiers addresses the biased softmax problem in LTR. Prototype classifiers can deliver promising results simply using Nearest-Class-Mean (NCM), a special case where prototypes are empirical centroids. We go one step further and propose to jointly learn prototypes by using distances to prototypes in representation space as the logit scores for classification. Further, we theoretically analyze the properties of Euclidean distance based prototype classifiers that lead to stable gradient-based optimization which is robust to outliers. To enable independent distance scales along each channel, we enhance Prototype classifiers by learning channel-dependent temperature parameters. Our analysis shows that prototypes learned by Prototype classifiers are better separated than empirical centroids. Results on four LTR benchmarks show that Prototype classifier outperforms or is comparable to state-of-the-art methods. Our code is made available at https://github.com/saurabhsharma1993/prototype-classifier-ltr.","doi":"10.24963/ijcai.2023/151","url":"https://ijcai.org/proceedings/2023/151","pdf":"https://ijcai.org/proceedings/2023/0151.pdf","markdown":"https://ijcai.org/proceedings/2023/151.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/151","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":151,"pages":{"first":1360,"last":1368},"keywords":["Computer Vision: CV: Transfer, low-shot, semi- and un- supervised learning","Machine Learning: ML: Few-shot learning"],"videos":[],"references":[{"position":1,"raw":"[Alshammari et al., 2022] Shaden Alshammari, Yu-Xiong Wang, Deva Ramanan, and Shu Kong. Long-tailed recognition via weight balancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6897-6907, 2022.","title":"Long-tailed recognition via weight balancing","authors":"Shaden Alshammari, Yu-Xiong Wang, Deva Ramanan, Shu Kong","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cao et al., 2019] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aware margin loss. In NeurIPS, 2019.","title":"Learning imbalanced datasets with label-distribution-aware margin loss","authors":"Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, Tengyu Ma","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chawla et al., 2002] Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: syn- thetic minority over-sampling technique. JAIR, 2002.","title":"Smote: synthetic minority over-sampling technique","authors":"Kevin W Nitesh V Chawla, Lawrence O Bowyer, Philip Hall, Kegelmeyer","year":"2002","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chawla et al., 2003] Nitesh V Chawla, Aleksandar Lazare- vic, Lawrence O Hall, and Kevin W Bowyer. Smoteboost: Improving prediction of the minority class in boosting. In European conference on principles of data mining and knowledge discovery, 2003.","title":"Smoteboost: Improving prediction of the minority class in boosting","authors":"Aleksandar Nitesh V Chawla, Lawrence O Lazarevic, Kevin W Hall, Bowyer","year":"2003","venue":"European conference on principles of data mining and knowledge discovery","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cubuk et al., 2018] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.09501, 2018.","title":"Autoaugment: Learning augmentation policies from data","authors":"Barret Ekin D Cubuk, Dandelion Zoph, Vijay Mane, Quoc V Vasudevan, Le","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cubuk et al., 2020] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical auto- mated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition workshops, pages 702-703, 2020.","title":"Randaugment: Practical automated data augmentation with a reduced search space","authors":"Barret Ekin D Cubuk, Jonathon Zoph, Quoc V Shlens, Le","year":"2020","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cui et al., 2019] Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In CVPR, 2019.","title":"Class-balanced loss based on effective number of samples","authors":"Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, Serge Belongie","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cui et al., 2021] Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 715-724, 2021.","title":"Parametric contrastive learning","authors":"Cui","year":"2021","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[DeVries and Taylor, 2017] Terrance DeVries and Gra- ham W Taylor. Improved regularization of convolu- tional neural networks with cutout. arXiv preprint arXiv:1708.04552, 2017.","title":"Improved regularization of convolutional neural networks with cutout","authors":"Terrance Devries, Graham W Taylor","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Drummond et al., 2003] Chris Drummond, Robert C Holte, et al. C4. 5, class imbalance, and cost sensitivity: why under-sampling beats over-sampling. In Workshop on learning from imbalanced datasets II, 2003.","title":"C4. 5, class imbalance, and cost sensitivity: why under-sampling beats over-sampling","authors":"Chris Drummond, Robert C Holte","year":"2003","venue":"Workshop on learning from imbalanced datasets II","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Estabrooks et al., 2004] Andrew Estabrooks, Taeho Jo, and Nathalie Japkowicz. A multiple resampling method for learning from imbalanced data sets. Computational intel- ligence, 2004.","title":"A multiple resampling method for learning from imbalanced data sets","authors":"Andrew Estabrooks, Taeho Jo, Nathalie Japkowicz","year":"2004","venue":"Computational intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Fang et al., 2021] Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer- peeled model: Minority collapse in imbalanced train- ing. Proceedings of the National Academy of Sciences, 118(43):e2103091118, 2021.","title":"Exploring deep neural networks via layerpeeled model: Minority collapse in imbalanced training","authors":"Fang","year":"2021","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Guerriero et al., 2018] Samantha Guerriero, Barbara Ca- puto, and Thomas Mensink. Deepncm: Deep nearest class mean classifiers. In International Conference on Learning Representations Workshop, 2018.","title":"Deepncm: Deep nearest class mean classifiers","authors":"Samantha Guerriero, Barbara Caputo, Thomas Mensink","year":"2018","venue":"International Conference on Learning Representations Workshop","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[He and Garcia, 2009] Haibo He and Edwardo A Garcia. Learning from imbalanced data. TKDE, 2009.","title":"Learning from imbalanced data","authors":"Haibo He, Edwardo A Garcia","year":"2009","venue":"TKDE","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[He et al., 2020] Kaiming He, Haoqi Fan, Yuxin Wu, Sain- ing Xie, and Ross Girshick. Momentum contrast for unsu- pervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9738, 2020.","title":"Momentum contrast for unsupervised visual representation learning","authors":"Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick","year":"2020","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kang et al., 2020] Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classi- fier for long-tailed recognition. In Eighth International Conference on Learning Representations (ICLR), 2020.","title":"Decoupling representation and classifier for long-tailed recognition","authors":"Kang","year":"2020","venue":"Eighth International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lin et al., 2017] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.","title":"Focal loss for dense object detection","authors":"Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollár","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2019] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long-tailed recognition in an open world. In CVPR, 2019.","title":"Large-scale long-tailed recognition in an open world","authors":"Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, Stella X Yu","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu et al., 2021] Bo Liu, Haoxiang Li, Hao Kang, Gang Hua, and Nuno Vasconcelos. Gistnet: a geometric struc- ture transfer network for long-tailed recognition. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 8209-8218, 2021.","title":"Gistnet: a geometric structure transfer network for long-tailed recognition","authors":"Bo Liu, Haoxiang Li, Hao Kang, Gang Hua, Nuno Vasconcelos","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Loshchilov and Hutter, 2017] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Repre- sentations, 2017.","title":"Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts","authors":"Hutter Loshchilov","year":"2017","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Menon et al., 2021] Aditya Krishna Menon, Sadeep Jaya- sumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit ad- justment. In International Conference on Learning Repre- sentations, 2021.","title":"Long-tail learning via logit adjustment","authors":"Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, Sanjiv Kumar","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Mensink et al., 2013] Thomas Mensink, Jakob Verbeek, Florent Perronnin, and Gabriela Csurka. Distance-based image classification: Generalizing to new classes at near- zero cost. TPAMI, 2013.","title":"Distance-based image classification: Generalizing to new classes at nearzero cost","authors":"Thomas Mensink, Jakob Verbeek, Florent Perronnin, Gabriela Csurka","year":"2013","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Nogueira and others, 2014] Fernando Nogueira et al. Bayesian optimization: Open source constrained global optimization tool for python. URL https://github. com/fmfn/BayesianOptimization, 2014.","title":"Bayesian optimization: Open source constrained global optimization tool for python","authors":"Fernando Nogueira","year":null,"venue":null,"doi":null,"url":"https://github.com/fmfn/BayesianOptimization,2014","internal_url":null},{"position":25,"raw":"[Papyan et al., 2020] Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the Na- tional Academy of Sciences, 117(40):24652-24663, 2020.","title":"Prevalence of neural collapse during the terminal phase of deep learning training","authors":"Vardan Papyan, David L Han, Donoho","year":"2020","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Rebuffi et al., 2017] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Pro- ceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2017.","title":"icarl: Incremental classifier and representation learning","authors":"Sylvestre-Alvise, Alexander Rebuffi, Georg Kolesnikov, Christoph H Sperl, Lampert","year":"2017","venue":"Proceedings of the IEEE conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Ren et al., 2020] Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for long- tailed visual recognition. Advances in Neural Information Processing Systems, 33:4175-4186, 2020.","title":"Balanced meta-softmax for longtailed visual recognition","authors":"Cunjun Jiawei Ren, Xiao Yu, Haiyu Ma, Shuai Zhao, Yi","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Samuel and Chechik, 2021] Dvir Samuel and Gal Chechik. Distributional robustness loss for long-tail learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.","title":"Distributional robustness loss for long-tail learning","authors":"Chechik ; Samuel, Dvir Samuel, Gal Chechik","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sharma et al., 2020] Saurabh Sharma, Ning Yu, Mario Fritz, and Bernt Schiele. Long-tailed recognition using class-balanced experts. In DAGM German Conference on Pattern Recognition. Springer, 2020.","title":"Long-tailed recognition using class-balanced experts","authors":"Saurabh Sharma, Ning Yu, Mario Fritz, Bernt Schiele","year":"2020","venue":"DAGM German Conference on Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Snell et al., 2017] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NeurIPS, 2017.","title":"Prototypical networks for few-shot learning","authors":"Jake Snell, Kevin Swersky, Richard Zemel","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sun et al., 2019] Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. In CVPR, 2019.","title":"Meta-transfer learning for few-shot learning","authors":"Sun","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Thrampoulidis et al., 2022] Christos Thrampoulidis, Ganesh Ramachandra Kini, Vala Vakilian, and Tina Behnia. Imbalance trouble: Revisiting neural-collapse geometry. In Advances in Neural Information Processing Systems, 2022.","title":"Imbalance trouble: Revisiting neural-collapse geometry","authors":"Christos Thrampoulidis, Ramachandra Ganesh, Vala Kini, Tina Vakilian, Behnia","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Van Horn and Perona, 2017] Grant Van Horn and Pietro Perona. The devil is in the tails: Fine-grained classification in the wild. arXiv preprint arXiv:1709.01450, 2017.","title":"The devil is in the tails: Fine-grained classification in the wild","authors":"Van Horn, Perona, Grant Van Horn, Pietro Perona","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Van Horn et al., 2018] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8769-8778, 2018.","title":"The inaturalist species classification and detection dataset","authors":"Van Horn","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Wang et al., 2020] Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella Yu. Long-tailed recognition by routing diverse distribution-aware experts. In Interna- tional Conference on Learning Representations, 2020.","title":"Long-tailed recognition by routing diverse distribution-aware experts","authors":"Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, Stella Yu","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Xian et al., 2019] Yongqin Xian, Saurabh Sharma, Bernt Schiele, and Zeynep Akata. f-vaegan-d2: A feature gener- ating framework for any-shot learning. In CVPR, 2019.","title":"f-vaegan-d2: A feature generating framework for any-shot learning","authors":"Xian","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Xiang et al., 2020] Liuyu Xiang, Guiguang Ding, and Jun- gong Han. Learning from multiple experts: Self-paced knowledge distillation for long-tailed classification. In Eu- ropean Conference on Computer Vision. Springer, 2020.","title":"Liuyu Xiang, Guiguang Ding, and Jungong Han. Learning from multiple experts: Self-paced knowledge distillation for long-tailed classification","authors":"Xiang","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhang et al., 2021] Songyang Zhang, Zeming Li, Shipeng Yan, Xuming He, and Jian Sun. Distribution alignment: A unified framework for long-tail visual recognition. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2361-2370, 2021.","title":"Distribution alignment: A unified framework for long-tail visual recognition","authors":"Songyang Zhang, Zeming Li, Shipeng Yan, Xuming He, Jian Sun","year":"2021","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zhou et al., 2020] Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 9719-9728, 2020.","title":"Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition","authors":"Boyan Zhou, Quan Cui, Xiu-Shen Wei, Zhao-Min Chen","year":"2020","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}