{"type":"ScholarlyArticle","title":"Generalization Bounds for Adversarial Metric Learning","authors":[{"name":"Wen Wen","slug":"wen-wen","given":"Wen","family":"Wen","affiliation":"College of Informatics , Huazhong Agricultural University , Wuhan 430070 , China"},{"name":"Han Li","slug":"han-li","given":"Han","family":"Li","affiliation":"College of Informatics , Huazhong Agricultural University , Wuhan 430070 , China"},{"name":"Hong Chen","slug":"hong-chen","given":"Hong","family":"Chen","affiliation":"College of Informatics , Huazhong Agricultural University , Wuhan 430070 , China; Engineering Research Center of Intelligent Technology for Agriculture , Ministry of Education , Wuhan 430070 , China; Key Laboratory of Smart Farming for Agricultural Animals , Wuhan 430070 , China"},{"name":"Rui Wu","slug":"rui-wu","given":"Rui","family":"Wu","affiliation":"Horizon Robotics , Haidian District , BeiJing 100190 , China"},{"name":"Lingjuan Wu","slug":"lingjuan-wu","given":"Lingjuan","family":"Wu","affiliation":"College of Informatics , Huazhong Agricultural University , Wuhan 430070 , China"},{"name":"Liangxuan Zhu","slug":"liangxuan-zhu","given":"Liangxuan","family":"Zhu","affiliation":"College of Informatics , Huazhong Agricultural University , Wuhan 430070 , China"}],"abstract":"Recently, adversarial metric learning has been proposed to enhance the robustness of the learned distance metric against adversarial perturbations. Despite rapid progress in validating its effectiveness empirically, theoretical guarantees on adversarial robustness and generalization are far less understood. To fill this gap, this paper focuses on unveiling the generalization properties of adversarial metric learning by developing the uniform convergence analysis techniques. Based on the capacity estimation of covering numbers, we establish the first high-probability generalization bounds with order O(n^{-1/2}) for adversarial metric learning with pairwise perturbations and general losses, where n is the number of training samples. Moreover, we obtain the refined generalization bounds with order O(n^{-1}) for the smooth loss by using local Rademacher complexity, which is faster than the previous result of adversarial pairwise learning, e.g., adversarial bipartite ranking. Experimental evaluation on real-world datasets validates our theoretical findings.","doi":"10.24963/ijcai.2023/489","url":"https://ijcai.org/proceedings/2023/489","pdf":"https://ijcai.org/proceedings/2023/0489.pdf","markdown":"https://ijcai.org/proceedings/2023/489.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/489","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":489,"pages":{"first":4397,"last":4405},"keywords":["Machine Learning: ML: Learning theory","Machine Learning: ML: Adversarial machine learning"],"videos":[],"references":[{"position":1,"raw":"[Awasthi et al., 2021] Pranjal Awasthi, George Yu, Chun- Sung Ferng, Andrew Tomkins, and Da-Cheng Juan. Ad- versarial robustness across representation spaces. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7608-7616, 2021.","title":"Adversarial robustness across representation spaces","authors":"Pranjal Awasthi, George Yu, Chun-Sung Ferng, Andrew Tomkins, Da-Cheng Juan","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bartlett et al., 2005] Peter L Bartlett, Olivier Bousquet, and Shahar Mendelson. Local rademacher complexities. The Annals of Statistics, 33(4):1497-1537, 2005.","title":"Local rademacher complexities","authors":"Olivier Peter L Bartlett, Shahar Bousquet, Mendelson","year":"2005","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bartlett et al., 2017] Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. Advances in Neural Information Pro- cessing Systems, pages 6240-6249, 2017.","title":"Spectrally-normalized margin bounds for neural networks","authors":"Dylan J Peter L Bartlett, Matus J Foster, Telgarsky","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bouniot et al., 2020] Quentin Bouniot, Romaric Audigier, and Angelique Loesch. Vulnerability of person re- identification models to metric adversarial attacks. In Pro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition Workshops, pages 794-795, 2020.","title":"Vulnerability of person reidentification models to metric adversarial attacks","authors":"Quentin Bouniot, Romaric Audigier, Angelique Loesch","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bousquet et al., 2020] Olivier Bousquet, Yegor Klochkov, and Nikita Zhivotovskiy. Sharper bounds for uniformly stable algorithms. In Conference on Learning Theory, pages 610-626, 2020.","title":"Sharper bounds for uniformly stable algorithms","authors":"Bousquet","year":"2020","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cao et al., 2016] Qiong Cao, Zheng-Chu Guo, and Yiming Ying. Generalization bounds for metric and similarity learning. Machine Learning, 102(1):115-132, 2016.","title":"Generalization bounds for metric and similarity learning","authors":"Qiong Cao, Zheng-Chu Guo, Yiming Ying","year":"2016","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Carlini and Wagner, 2017] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In Proceedings of 2017 IEEE Symposium on Security and Privacy (SP), pages 39-57, 2017.","title":"Towards evaluating the robustness of neural networks","authors":"Wagner Carlini, Nicholas Carlini, David Wagner","year":"2017","venue":"Proceedings of 2017 IEEE Symposium on Security and Privacy (SP)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen and Deng, 2019] Binghui Chen and Weihong Deng. Energy confused adversarial metric learning for zero-shot image retrieval and clustering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8134-8141, 2019.","title":"Energy confused adversarial metric learning for zero-shot image retrieval and clustering","authors":"Chen, Deng ; ] Binghui Chen, Weihong Deng","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dai et al., 2018] Pingyang Dai, Rongrong Ji, Haibin Wang, Qiong Wu, and Yuyu Huang. Cross-modality person re- identification with generative adversarial training. In Pro- ceedings of International Joint Conference on Artificial In- telligence, volume 1, page 6, 2018.","title":"Cross-modality person reidentification with generative adversarial training","authors":"Pingyang Dai, Rongrong Ji, Haibin Wang, Qiong Wu, Yuyu Huang","year":"2018","venue":"Proceedings of International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Farnia and Ozdaglar, 2021] Farzan Farnia and Asuman Ozdaglar. Train simultaneously, generalize better: Sta- bility of gradient-based minimax learners. In Interna- tional Conference on Machine Learning, pages 3174- 3185. PMLR, 2021.","title":"Train simultaneously, generalize better: Stability of gradient-based minimax learners","authors":"Farzan Farnia, Asuman Ozdaglar","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hahnloser et al., 2000] Richard HR Hahnloser, Rahul Sarpeshkar, Misha A Mahowald, Rodney J Douglas, and H Sebastian Seung. Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit. Nature, 405(6789):947-951, 2000.","title":"Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit","authors":"Rahul Richard Hr Hahnloser, Misha A Sarpeshkar, Rodney J Mahowald, H Douglas, Seung Sebastian","year":"2000","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Huai et al., 2019] Mengdi Huai, Hongfei Xue, Chenglin Miao, Liuyi Yao, Lu Su, Changyou Chen, and Aidong Zhang. Deep metric learning: The generalization analysis and an adaptive algorithm. In Proceedings of the Twenty- Eighth International Joint Conference on Artificial Intelli- gence, IJCAI-19, pages 2535-2541, 2019.","title":"Deep metric learning: The generalization analysis and an adaptive algorithm","authors":"Huai","year":"2019","venue":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19","doi":null,"url":null,"internal_url":"/proceedings/2019/352"},{"position":13,"raw":"[Huai et al., 2022] Mengdi Huai, Tianhang Zheng, Chenglin Miao, Liuyi Yao, and Aidong Zhang. On the robustness of metric learning: An adversarial perspective. ACM Trans- actions on Knowledge Discovery from Data (TKDD), 16(5):1-25, 2022.","title":"On the robustness of metric learning: An adversarial perspective","authors":"Huai","year":"2022","venue":"ACM Transactions on Knowledge Discovery from Data (TKDD)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Huang et al., 2019] He Huang, Changhu Wang, Philip S Yu, and Chang-Dong Wang. Generative dual adversarial net- work for generalized zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 801-810, 2019.","title":"Generative dual adversarial network for generalized zero-shot learning","authors":"Huang","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Khim and Loh, 2018] Justin Khim and Po-Ling Loh. Ad- versarial risk bounds via function transformation. arXiv preprint arXiv:1810.09519, 2018.","title":"Adversarial risk bounds via function transformation","authors":"Loh Khim, Justin Khim, Po-Ling Loh","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Klochkov and Zhivotovskiy, 2021] Yegor Klochkov and Nikita Zhivotovskiy. Stability and deviation optimal risk bounds with convergence rate o(1/n). Advances in Neural Information Processing Systems, 34:5065-5076, 2021.","title":"Yegor Klochkov and Nikita Zhivotovskiy. Stability and deviation optimal risk bounds with convergence rate o(1/n)","authors":"Zhivotovskiy Klochkov","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kurakin et al., 2018] Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In Artificial Intelligence Safety and Security, pages 99-112. 2018.","title":"Adversarial examples in the physical world","authors":"Alexey Kurakin, Ian J Goodfellow, Samy Bengio","year":"2018","venue":"Artificial Intelligence Safety and Security","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lei et al., 2020] Yunwen Lei, Antoine Ledent, and Marius Kloft. Sharper generalization bounds for pairwise learn- ing. Advances in Neural Information Processing Systems, 33:21236-21246, 2020.","title":"Sharper generalization bounds for pairwise learning","authors":"Lei","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li and Liu, 2021] Shaojie Li and Yong Liu. High proba- bility generalization bounds with fast rates for minimax problems. In International Conference on Learning Rep- resentations, 2021.","title":"High probability generalization bounds with fast rates for minimax problems","authors":"Li, Liu, Shaojie Li, Yong Liu","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu et al., 2022] Deyin Liu, Lin Wu, Richang Hong, Zongyuan Ge, Jialie Shen, Farid Boussaid, and Mo- hammed Bennamoun. Generative metric learning for adversarially robust open-world person re-identification. ACM Transactions on Multimedia Computing, Communi- cations, and Applications (TOMM), 2022.","title":"Generative metric learning for adversarially robust open-world person re-identification","authors":"Liu","year":"2022","venue":"ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Madry et al., 2018] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In Proceedings of International Conference on Learning Representations, 2018.","title":"Towards deep learning models resistant to adversarial attacks","authors":"Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, Adrian Vladu","year":"2018","venue":"Proceedings of International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Mo et al., 2022] Yingxiang Mo, Hong Chen, Yuxiang Han, and Hao Deng. Error bounds of adversarial bipartite rank- ing. Neurocomputing, 478:81-88, 2022.","title":"Error bounds of adversarial bipartite ranking","authors":"Mo","year":"2022","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Mohri et al., 2018] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018.","title":"Foundations of machine learning","authors":"Mehryar Mohri, Afshin Rostamizadeh, Ameet Talwalkar","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Mustafa et al., 2022] Waleed Mustafa, Yunwen Lei, and Marius Kloft. On the generalization analysis of adversarial learning. In International Conference on Machine Learn- ing, pages 16174-16196, 2022.","title":"On the generalization analysis of adversarial learning","authors":"Waleed Mustafa, Yunwen Lei, Marius Kloft","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Reeve and Kaban, 2020] Henry Reeve and Ata Kaban. Op- timistic bounds for multi-output learning. In Interna- tional Conference on Machine Learning, pages 8030- 8040, 2020.","title":"Optimistic bounds for multi-output learning","authors":"Kaban ; ] Henry Reeve, Ata Reeve, Kaban","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Srebro et al., 2010] Nathan Srebro, Karthik Sridharan, and Ambuj Tewari. Smoothness, low noise and fast rates. Advances in Neural Information Processing Systems, 23, 2010.","title":"Smoothness, low noise and fast rates","authors":"Nathan Srebro, Karthik Sridharan, Ambuj Tewari","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tu et al., 2019] Zhuozhuo Tu, Jingwei Zhang, and Dacheng Tao. Theoretical analysis of adversarial learning: A min- imax approach. Advances in Neural Information Process- ing Systems, 32, 2019.","title":"Theoretical analysis of adversarial learning: A minimax approach","authors":"Tu","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Vapnik, 1999] Vladimir Vapnik. The nature of statistical learning theory. Springer Science & Business Media, 1999.","title":"The nature of statistical learning theory","authors":"Vladimir Vapnik","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2020] Zhuoyi Wang, Yigong Wang, Bo Dong, Sahoo Pracheta, Kevin Hamlen, and Latifur Khan. Adap- tive margin based deep adversarial metric learning. In 2020 IEEE 6th Intl Conference on Big Data Security on Cloud (BigDataSecurity), IEEE Intl Conference on High Performance and Smart Computing,(HPSC) and IEEE Intl Conference on Intelligent Data and Security (IDS), pages 100-108, 2020.","title":"Adaptive margin based deep adversarial metric learning","authors":"Wang","year":"2020","venue":"IEEE 6th Intl Conference on Big Data Security on Cloud (BigDataSecurity), IEEE Intl Conference on High Performance and Smart Computing","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Xiao et al., 2022] Jiancong Xiao, Yanbo Fan, Ruoyu Sun, Jue Wang, and Zhi-Quan Luo. Stability analysis and gen- eralization bounds of adversarial training. In Advances in Neural Information Processing Systems, 2022.","title":"Stability analysis and generalization bounds of adversarial training","authors":"Xiao","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Xing et al., 2021] Yue Xing, Qifan Song, and Guang Cheng. On the algorithmic stability of adversarial train- ing. Advances in Neural Information Processing Systems, 34:26523-26535, 2021.","title":"On the algorithmic stability of adversarial training","authors":"Yue Xing, Qifan Song, Guang Cheng","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Xu et al., 2019] Xing Xu, Li He, Huimin Lu, Lianli Gao, and Yanli Ji. Deep adversarial metric learning for cross- modal retrieval. World Wide Web, 22(2):657-672, 2019.","title":"Deep adversarial metric learning for crossmodal retrieval","authors":"Xing Xu, Li He, Huimin Lu, Lianli Gao, Yanli Ji","year":"2019","venue":"World Wide Web","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yang et al., 2021] Xiaochen Yang, Mingzhi Dong, Yiwen Guo, and Jing-Hao Xue. Metric learning for categorical and ambiguous features: An adversarial method. In Joint European Conference on Machine Learning and Knowl- edge Discovery in Databases, pages 223-238, 2021.","title":"Metric learning for categorical and ambiguous features: An adversarial method","authors":"Xiaochen Yang, Mingzhi Dong, Yiwen Guo, Jing-Hao Xue","year":"2021","venue":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Ye et al., 2019] Han-Jia Ye, De-Chuan Zhan, and Yuan Jiang. Fast generalization rates for distance metric learn- ing. Machine Learning, 108(2):267-295, 2019.","title":"Fast generalization rates for distance metric learning","authors":"Han-Jia Ye, Yuan De-Chuan Zhan, Jiang","year":"2019","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Yin et al., 2019] Dong Yin, Ramchandran Kannan, and Pe- ter Bartlett. Rademacher complexity for adversarially ro- bust generalization. In International Conference on Ma- chine Learning, pages 7085-7094, 2019.","title":"Rademacher complexity for adversarially robust generalization","authors":"Dong Yin, Ramchandran Kannan, Peter Bartlett","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zantedeschi et al., 2016] Valentina Zantedeschi, Rémi Emonet, and Marc Sebban. Metric learning as convex combinations of local models with generalization guaran- tees. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1478-1486, 2016.","title":"Metric learning as convex combinations of local models with generalization guarantees","authors":"Valentina Zantedeschi, Rémi Emonet, Marc Sebban","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhang, 2002] Tong Zhang. Covering number bounds of cer- tain regularized linear function classes. Journal of Ma- chine Learning Research, 2(Mar):527-550, 2002.","title":"Covering number bounds of certain regularized linear function classes","authors":"Tong Zhang","year":"2002","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}