{"type":"ScholarlyArticle","title":"Multi-Proxy Learning from an Entropy Optimization Perspective","authors":[{"name":"Yunlong Yu","slug":"yunlong-yu","given":"Yunlong","family":"Yu","affiliation":"College of Information Science and Electronic Engineering , Zhejiang University"},{"name":"Dingyi Zhang","slug":"dingyi-zhang","given":"Dingyi","family":"Zhang","affiliation":"College of Information Science and Electronic Engineering , Zhejiang University"},{"name":"Yingming Li","slug":"yingming-li","given":"Yingming","family":"Li","affiliation":"College of Information Science and Electronic Engineering , Zhejiang University"},{"name":"Zhongfei Zhang","slug":"zhongfei-zhang","given":"Zhongfei","family":"Zhang","affiliation":"Computer Science Department , Binghamton University"}],"abstract":"Deep Metric Learning, a task that learns a feature embedding space where semantically similar samples are located closer than dissimilar samples, is a cornerstone of many computer vision applications. Most of the existing proxy-based approaches usually exploit the global context via learning a single proxy for each training class, which struggles in capturing the complex non-uniform data distribution with different patterns. In this work, we present an easy-to-implement framework to effectively capture the local neighbor relationships via learning multiple proxies for each class that collectively approximate the intra-class distribution. In the context of large intra-class visual diversity, we revisit the entropy learning under the multi-proxy learning framework and provide a training routine that both minimizes the entropy of intra-class probability distribution and maximizes the entropy of inter-class probability distribution. In this way, our model is able to better capture the intra-class variations and smooth the inter-class differences and thus facilitates to extract more semantic feature representations for the downstream tasks. Extensive experimental results demonstrate that the proposed approach achieves competitive performances. Codes and an appendix are provided.","doi":"10.24963/ijcai.2022/222","url":"https://ijcai.org/proceedings/2022/222","pdf":"https://ijcai.org/proceedings/2022/0222.pdf","markdown":"https://ijcai.org/proceedings/2022/222.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/222","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":222,"pages":{"first":1594,"last":1600},"keywords":["Computer Vision: Transfer, low-shot, semi- and un- supervised learning","Computer Vision: Image and Video retrieval","Machine Learning: Feature Extraction, Selection and Dimensionality Reduction"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987020","length_sec":77,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985163","length_sec":709,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Boudiaf et al., 2020] Malik Boudiaf, Jérôme Rony, Imtiaz Masud Ziko, Eric Granger, Marco Pedersoli, Pablo Piantanida, and Is- mail Ben Ayed. A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses. In ECCV, pages 548- 564, 2020.","title":"A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses","authors":"Malik Boudiaf, Jérôme Rony, Masud Imtiaz, Eric Ziko, Marco Granger, Pablo Pedersoli, Ismail Piantanida, Ayed Ben","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chopra et al., 2005] Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In CVPR, pages 539-546, 2005.","title":"Learning a similarity metric discriminatively, with application to face verification","authors":"Sumit Chopra, Raia Hadsell, Yann Lecun","year":"2005","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dubey et al., 2018] Abhimanyu Dubey, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Maximum-entropy fine-grained classifi- cation. In NeurIPS, 2018.","title":"Maximum-entropy fine-grained classification","authors":"Abhimanyu Dubey, Otkrist Gupta, Ramesh Raskar, Nikhil Naik","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Grandvalet and Bengio, 2005] Yves Grandvalet and Yoshua Ben- gio. Semi-supervised learning by entropy minimization. In CAP, pages 281-296, 2005.","title":"Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization","authors":"Bengio Grandvalet","year":"2005","venue":"CAP","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gu et al., 2021] Geonmo Gu, Byungsoo Ko, and Han-Gyu Kim. Proxy synthesis: Learning with synthetic classes for deep metric learning. In AAAI, 2021.","title":"Proxy synthesis: Learning with synthetic classes for deep metric learning","authors":"Gu","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hoffer and Ailon, 2015] Elad Hoffer and Nir Ailon. Deep metric learning using triplet network. In ICLR Workshop, pages 84-92, 2015.","title":"Deep metric learning using triplet network","authors":"Elad Hoffer, Nir Ailon","year":"2015","venue":"ICLR Workshop","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"Ioffe and Normalization, ] Sergey Ioffe and Christian Szegedy Batch Normalization. Accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167.","title":"Accelerating deep network training by reducing internal covariate shift","authors":"Normalization Ioffe","year":null,"venue":"Sergey Ioffe and Christian Szegedy Batch Normalization","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kim et al., 2020] Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak. Proxy anchor loss for deep metric learning. In CVPR, pages 3238-3247, 2020.","title":"Proxy anchor loss for deep metric learning","authors":"Kim","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kingma and Ba, 2015] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Krause et al., 2013] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categoriza- tion. In CVPR workshop, pages 554-561, 2013.","title":"3d object representations for fine-grained categorization","authors":"Jonathan Krause, Michael Stark, Jia Deng, Li Fei-Fei","year":"2013","venue":"CVPR workshop","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2016] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recog- nition and retrieval with rich annotations. In CVPR, pages 1096- 1104, 2016.","title":"Deepfashion: Powering robust clothes recognition and retrieval with rich annotations","authors":"Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, Xiaoou Tang","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Melacci and Gori, 2012] Stefano Melacci and Marco Gori. Un- supervised learning by minimal entropy encoding. TNNLS, 23(12):1849-1861, 2012.","title":"Unsupervised learning by minimal entropy encoding","authors":"Stefano Melacci, Marco Gori","year":"2012","venue":"TNNLS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Meng et al., 2021] Qiang Meng, Shichao Zhao, Zhida Huang, and Feng Zhou. Magface: A universal representation for face recogni- tion and quality assessment. arXiv:2103.06627, 2021.","title":"Magface: A universal representation for face recognition and quality assessment","authors":"Qiang Meng, Shichao Zhao, Zhida Huang, Feng Zhou","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Milbich et al., 2020] Timo Milbich, Karsten Roth, Homanga Bharadhwaj, Samarth Sinha, Yoshua Bengio, Björn Ommer, and Joseph Paul Cohen. Diva: Diverse visual feature aggregation for deep metric learning. In ECCV, pages 590-607, 2020.","title":"Diva: Diverse visual feature aggregation for deep metric learning","authors":"Timo Milbich, Karsten Roth, Homanga Bharadhwaj, Samarth Sinha, Yoshua Bengio, Björn Ommer, Joseph Paul Cohen","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Movshovitz-Attias et al., 2017] Yair Movshovitz-Attias, Alexander Toshev, Thomas K Leung, Sergey Ioffe, and Saurabh Singh. No fuss distance metric learning using proxies. In ICCV, pages 360- 368, 2017.","title":"No fuss distance metric learning using proxies","authors":"Yair Movshovitz-Attias, Alexander Toshev, Thomas K Leung, Sergey Ioffe, Saurabh Singh","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Musgrave et al., 2020] Kevin Musgrave, Serge Belongie, and Ser- Nam Lim. A metric learning reality check. In ECCV, pages 681-699, 2020.","title":"A metric learning reality check","authors":"Kevin Musgrave, Serge Belongie, Ser-Nam Lim","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Oh Song et al., 2016] Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In CVPR, pages 4004-4012, 2016.","title":"Deep metric learning via lifted structured feature embedding","authors":"Hyun Oh Song, Yu Xiang, Stefanie Jegelka, Silvio Savarese","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Qian et al., 2019] Qi Qian, Lei Shang, Baigui Sun, Juhua Hu, Hao Li, and Rong Jin. Softtriple loss: Deep metric learning without triplet sampling. In CVPR, pages 6450-6458, 2019.","title":"Softtriple loss: Deep metric learning without triplet sampling","authors":"Lei Qi Qian, Baigui Shang, Juhua Sun, Hao Hu, Rong Li, Jin","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Roth et al., 2020] Karsten Roth, Timo Milbich, Samarth Sinha, Pra- teek Gupta, Bjorn Ommer, and Joseph Paul Cohen. Revisiting training strategies and generalization performance in deep metric learning. In ICML, pages 8242-8252, 2020.","title":"Revisiting training strategies and generalization performance in deep metric learning","authors":"Karsten Roth, Timo Milbich, Samarth Sinha, Prateek Gupta, Bjorn Ommer, Joseph Paul Cohen","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Roth et al., 2021] Karsten Roth, Timo Milbich, Bjorn Ommer, Joseph Paul Cohen, and Marzyeh Ghassemi. Simultaneous similarity-based self-distillation for deep metric learning. In ICML, pages 9095-9106, 2021.","title":"Simultaneous similarity-based self-distillation for deep metric learning","authors":"Karsten Roth, Timo Milbich, Bjorn Ommer, Joseph Paul Cohen, Marzyeh Ghassemi","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rutquist, 2019] Per Rutquist. Unsupervised learning through tem- poral smoothing and entropy maximization. In CDC, 2019.","title":"Per Rutquist. Unsupervised learning through temporal smoothing and entropy maximization","authors":"Rutquist","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Saito et al., 2019] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Semi-supervised domain adap- tation via minimax entropy. In ICCV, pages 8050-8058, 2019.","title":"Semi-supervised domain adaptation via minimax entropy","authors":"Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, Kate Saenko","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Snell et al., 2017] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NeurIPS, 2017.","title":"Prototypical networks for few-shot learning","authors":"Jake Snell, Kevin Swersky, Richard Zemel","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sohn et al., 2020] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. In NeurIPS, 2020.","title":"Fixmatch: Simplifying semisupervised learning with consistency and confidence","authors":"Sohn","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sohn, 2016] Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In NeurIPS, pages 1857-1865, 2016.","title":"Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective","authors":"Sohn","year":"2016","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sun et al., 2020] Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. Circle loss: A unified perspective of pair similarity optimization. In CVPR, pages 6398-6407, 2020.","title":"Circle loss: A unified perspective of pair similarity optimization","authors":"Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, Yichen Wei","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Teh et al., 2020] Eu Wern Teh, Terrance DeVries, and Graham W Taylor. Proxynca++: Revisiting and revitalizing proxy neighbor- hood component analysis. In ECCV, 2020.","title":"Proxynca++: Revisiting and revitalizing proxy neighborhood component analysis","authors":"Terrance Eu Wern Teh, Graham W Devries, Taylor","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wah et al., 2011] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200- 2011 dataset. http://www.vision.caltech.edu/datasets/cub 200 2011, 2011.","title":"The caltech-ucsd birds-200-2011 dataset","authors":"Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, Serge Belongie","year":"2011","venue":null,"doi":null,"url":"http://www.vision.caltech.edu/datasets/cub2002011","internal_url":null},{"position":30,"raw":"[Wang et al., 2019] Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew R Scott. Multi-similarity loss with general pair weighting for deep metric learning. In CVPR, pages 5022-5030, 2019.","title":"Multi-similarity loss with general pair weighting for deep metric learning","authors":"Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, Matthew R Scott","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wu et al., 2017] Chao-Yuan Wu, R Manmatha, Alexander J Smola, and Philipp Krahenbuhl. Sampling matters in deep embedding learning. In ICCV, pages 2840-2848, 2017.","title":"Sampling matters in deep embedding learning","authors":"Chao-Yuan, R Wu, Alexander J Manmatha, Philipp Smola, Krahenbuhl","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhai and Wu, 2018] Andrew Zhai and Hao-Yu Wu. Classification is a strong baseline for deep metric learning. In BMVC, 2018.","title":"Classification is a strong baseline for deep metric learning","authors":"Andrew Zhai, Hao-Yu Wu","year":"2018","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zheng et al., 2021] Wenzhao Zheng, Chengkun Wang, Jiwen Lu, and Jie Zhou. Deep compositional metric learning. In CVPR, pages 9320-9329, 2021.","title":"Deep compositional metric learning","authors":"Zheng","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhu et al., 2020] Yuehua Zhu, Muli Yang, Cheng Deng, and Wei Liu. Fewer is more: A deep graph metric learning perspective using fewer proxies. In NeurIPS, pages 17792-17803, 2020.","title":"Fewer is more: A deep graph metric learning perspective using fewer proxies","authors":"Yuehua Zhu, Muli Yang, Cheng Deng, Wei Liu","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}