{"type":"ScholarlyArticle","title":"Towards Universal Backward-Compatible Representation Learning","authors":[{"name":"Binjie Zhang","slug":"binjie-zhang","given":"Binjie","family":"Zhang","affiliation":"Tsinghua University; ARC Lab , Tencent PCG"},{"name":"Yixiao Ge","slug":"yixiao-ge","given":"Yixiao","family":"Ge","affiliation":"ARC Lab , Tencent PCG"},{"name":"Yantao Shen","slug":"yantao-shen","given":"Yantao","family":"Shen","affiliation":"AWS / Amazon AI"},{"name":"Shupeng Su","slug":"shupeng-su","given":"Shupeng","family":"Su","affiliation":"ARC Lab , Tencent PCG"},{"name":"Fanzi Wu","slug":"fanzi-wu","given":"Fanzi","family":"Wu","affiliation":"AWS / Amazon AI"},{"name":"Chun Yuan","slug":"chun-yuan","given":"Chun","family":"Yuan","affiliation":"Tsinghua University"},{"name":"Xuyuan Xu","slug":"xuyuan-xu","given":"Xuyuan","family":"Xu","affiliation":"AI Technology Center of Tencent Video"},{"name":"Yexin Wang","slug":"yexin-wang","given":"Yexin","family":"Wang","affiliation":"AI Technology Center of Tencent Video"},{"name":"Ying Shan","slug":"ying-shan","given":"Ying","family":"Shan","affiliation":"ARC Lab , Tencent PCG"}],"abstract":"Conventional model upgrades for visual search systems require offline refresh of gallery features by feeding gallery images into new models (dubbed as “backfill”), which is time-consuming and expensive, especially in large-scale applications. The task of backward-compatible representation learning is therefore introduced to support backfill-free model upgrades, where the new query features are interoperable with the old gallery features. Despite the success, previous works only investigated a close-set training scenario (i.e., the new training set shares the same classes as the old one), and are limited by more realistic and challenging open-set scenarios. To this end, we first introduce a new problem of universal backward-compatible representation learning, covering all possible data split in model upgrades. We further propose a simple yet effective method, dubbed as Universal Backward-Compatible Training (UniBCT) with a novel structural prototype refinement algorithm, to learn compatible representations in all kinds of model upgrading benchmarks in a unified manner. Comprehensive experiments on the large-scale face recognition datasets MS1Mv3 and IJB-C fully demonstrate the effectiveness of our method. Source code is available at https://github.com/TencentARC/OpenCompatible.","doi":"10.24963/ijcai.2022/225","url":"https://ijcai.org/proceedings/2022/225","pdf":"https://ijcai.org/proceedings/2022/0225.pdf","markdown":"https://ijcai.org/proceedings/2022/225.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/225","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":225,"pages":{"first":1615,"last":1621},"keywords":["Computer Vision: Representation Learning","Computer Vision: Image and Video retrieval"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986984","length_sec":92,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985127","length_sec":125,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Budnik and Avrithis, 2020] Mateusz Budnik and Yannis Avrithis. Asymmetric metric learning for knowledge trans- fer. arXiv preprint arXiv:2006.16331, 2020.","title":"Asymmetric metric learning for knowledge transfer","authors":"Mateusz Budnik, Yannis Avrithis","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Deng et al., 2019] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In CVPR, 2019.","title":"Arcface: Additive angular margin loss for deep face recognition","authors":"Deng","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ge et al., 2020] Yixiao Ge, Haibo Wang, Feng Zhu, Rui Zhao, and Hongsheng Li. Self-supervising fine-grained region similarities for large-scale image localization. In ECCV, 2020.","title":"Self-supervising fine-grained region similarities for large-scale image localization","authors":"Yixiao Ge, Haibo Wang, Feng Zhu, Rui Zhao, Hongsheng Li","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ghifary et al., 2016] Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, David Balduzzi, and Wen Li. Deep reconstruction-classification networks for unsupervised do- main adaptation. In ECCV. Springer, 2016.","title":"Deep reconstruction-classification networks for unsupervised domain adaptation","authors":"Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, David Balduzzi, Wen Li","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Guo et al., 2016] Yandong Guo, Lei Zhang, Yuxiao Hu, Xi- aodong He, and Jianfeng Gao. Ms-celeb-1m: A dataset and benchmark for large-scale face recognition. In ECCV, 2016.","title":"Ms-celeb-1m: A dataset and benchmark for large-scale face recognition","authors":"Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, Jianfeng Gao","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Li and Hoiem, 2017] Zhizhong Li and Derek Hoiem. Learn- ing without forgetting. PAMI, 2017.","title":"Learning without forgetting","authors":"Zhizhong Li, Derek Hoiem","year":"2017","venue":"PAMI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Liu et al., 2017] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In CVPR, 2017.","title":"Sphereface: Deep hypersphere embedding for face recognition","authors":"Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, Le Song","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Maze et al., 2018] Brianna Maze, Jocelyn Adams, James A Duncan, Nathan Kalka, Tim Miller, Charles Otto, Anil K Jain, W Tyler Niggel, Janet Anderson, Jordan Cheney, et al. Iarpa janus benchmark-c: Face dataset and protocol. In ICB, 2018.","title":"Iarpa janus benchmark-c: Face dataset and protocol","authors":"Brianna Maze, Jocelyn Adams, James A Duncan, Nathan Kalka, Tim Miller, Charles Otto, Anil K Jain, W Tyler Niggel, Janet Anderson, Jordan Cheney","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Meng et al., 2021] Qiang Meng, Chixiang Zhang, Xiaoqiang Xu, and Feng Zhou. Learning compatible embeddings. ICCV, 2021.","title":"Learning compatible embeddings","authors":"Qiang Meng, Chixiang Zhang, Xiaoqiang Xu, Feng Zhou","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Philbin et al., 2007] James Philbin, Ondrej Chum, Michael Isard, Josef Sivic, and Andrew Zisserman. Object retrieval with large vocabularies and fast spatial matching. In 2007 IEEE conference on computer vision and pattern recogni- tion. IEEE, 2007.","title":"Object retrieval with large vocabularies and fast spatial matching","authors":"James Philbin, Ondrej Chum, Michael Isard, Josef Sivic, Andrew Zisserman","year":"2007","venue":"IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Philbin et al., 2008] James Philbin, Ondrej Chum, Michael Isard, Josef Sivic, and Andrew Zisserman. Lost in quanti- zation: Improving particular object retrieval in large scale image databases. In 2008 IEEE conference on computer vision and pattern recognition. IEEE, 2008. [Radenović et al., 2018] Filip Radenović, Ahmet Iscen, Gior- gos Tolias, Yannis Avrithis, and Ondřej Chum. Revisiting oxford and paris: Large-scale image retrieval benchmark- ing. In CVPR, 2018.","title":"Lost in quantization: Improving particular object retrieval in large scale image databases","authors":"James Philbin, Ondrej Chum, Michael Isard, Josef Sivic, Andrew Zisserman, ; Radenović","year":"2008","venue":"IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Saito et al., 2020] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, and Kate Saenko. Universal domain adaptation through self supervision. In NeurIPS, 2020.","title":"Universal domain adaptation through self supervision","authors":"Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Kate Saenko","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Shen et al., 2020] Yantao Shen, Yuanjun Xiong, Wei Xia, and Stefano Soatto. Towards backward-compatible repre- sentation learning. In CVPR, 2020.","title":"Towards backward-compatible representation learning","authors":"Shen","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Su et al., 2022] Shupeng Su, Binjie Zhang, Yixiao Ge, Xuyuan Xu, Yexin Wang, Chun Yuan, and Ying Shan. Privacy-preserving model upgrades with bidirectional compatible training in image retrieval. arXiv preprint arXiv:2204.13919, 2022.","title":"Privacy-preserving model upgrades with bidirectional compatible training in image retrieval","authors":"Su","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Wang et al., 2017] Feng Wang, Xiang Xiang, Jian Cheng, and Alan Loddon Yuille. Normface: l 2 hypersphere em- bedding for face verification. In ACMMM. ACM, 2017.","title":"Normface: l 2 hypersphere embedding for face verification","authors":"Feng Wang, Xiang Xiang, Jian Cheng, Alan Loddon, Yuille","year":"2017","venue":"ACMMM. ACM","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wang et al., 2018] Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In CVPR, 2018.","title":"Cosface: Large margin cosine loss for deep face recognition","authors":"Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, Wei Liu","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2020] Chien-Yi Wang, Ya-Liang Chang, Shang- Ta Yang, Dong Chen, and Shang-Hong Lai. Unified rep- resentation learning for cross model compatibility. arXiv preprint arXiv:2008.04821, 2020.","title":"Unified representation learning for cross model compatibility","authors":"Chien-Yi Wang, Ya-Liang Chang, Shang-Ta Yang, Dong Chen, Shang-Hong Lai","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Weyand et al., 2020] Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2-a large- scale benchmark for instance-level recognition and retrieval. In CVPR, 2020.","title":"Google landmarks dataset v2-a largescale benchmark for instance-level recognition and retrieval","authors":"Weyand","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yang et al., 2017] Jiaolong Yang, Peiran Ren, Dongqing Zhang, Dong Chen, Fang Wen, Hongdong Li, and Gang Hua. Neural aggregation network for video face recogni- tion. In CVPR, 2017.","title":"Neural aggregation network for video face recognition","authors":"Jiaolong Yang, Peiran Ren, Dongqing Zhang, Dong Chen, Fang Wen, Hongdong Li, Gang Hua","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[You et al., 2019] Kaichao You, Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. Universal do- main adaptation. In CVPR, 2019.","title":"Universal domain adaptation","authors":"You","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang et al., 2020] Xiao Zhang, Rui Zhao, Yu Qiao, and Hongsheng Li. Rbf-softmax: Learning deep representative prototypes with radial basis function softmax. pages 296- 311. Springer, 2020.","title":"Rbf-softmax: Learning deep representative prototypes with radial basis function softmax","authors":"Xiao Zhang, Rui Zhao, Yu Qiao, Hongsheng Li","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2021] Binjie Zhang, Yixiao Ge, Yantao Shen, Yu Li, Chun Yuan, XUYUAN XU, Yexin Wang, and Ying Shan. Hot-refresh model upgrades with regression-free compatible training in image retrieval. In International Conference on Learning Representations, 2021.","title":"Hot-refresh model upgrades with regression-free compatible training in image retrieval","authors":"Zhang","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}