{"type":"ScholarlyArticle","title":"Generalization Bounds for Regularized Pairwise Learning","authors":[{"name":"Yunwen Lei","slug":"yunwen-lei","given":"Yunwen","family":"Lei","affiliation":"Shenzhen Key Laboratory of Computational Intelligence , Department of Computer Science and Engineering , Southern University of Science and Technology , Shenzhen"},{"name":"Shao-Bo Lin","slug":"shao-bo-lin","given":"Shao-Bo","family":"Lin","affiliation":"Department of Mathematics , Wenzhou University , Wenzhou"},{"name":"Ke Tang","slug":"ke-tang","given":"Ke","family":"Tang","affiliation":"Shenzhen Key Laboratory of Computational Intelligence , Department of Computer Science and Engineering , Southern University of Science and Technology , Shenzhen"}],"abstract":"Pairwise learning refers to learning tasks with the associated loss functions depending on pairs of examples. Recently, pairwise learning has received increasing attention since it covers many machine learning schemes, e.g., metric learning, ranking and AUC maximization, in a unified framework. In this paper, we establish a unified generalization error bound for regularized pairwise learning without either Bernstein conditions or capacity assumptions. We apply this general result to typical learning tasks including distance metric learning and ranking, for each of which our discussion is able to improve the state-of-the-art results.","doi":"10.24963/ijcai.2018/329","url":"https://ijcai.org/proceedings/2018/329","pdf":"https://ijcai.org/proceedings/2018/0329.pdf","markdown":"https://ijcai.org/proceedings/2018/329.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/329","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":329,"pages":{"first":2376,"last":2382},"keywords":["Machine Learning: Machine Learning","Machine Learning: Learning Theory","Machine Learning: Learning Preferences or Rankings","Machine Learning: Kernel Methods"],"videos":[],"references":[{"position":1,"raw":"[Agarwal and Niyogi, 2009] Shivani Agarwal and Partha Niyogi. Generalization bounds for ranking algorithms via algorithmic stability. Journal of Machine Learning Re- search, 10:441-474, 2009.","title":"Generalization bounds for ranking algorithms via algorithmic stability","authors":"Shivani Agarwal, Partha Niyogi","year":"2009","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bartlett et al., 2005] P.L. Bartlett, O. Bousquet, and S. Mendelson. Local Rademacher complexities. Annals of Statistics, 33(4):1497-1537, 2005.","title":"Local Rademacher complexities","authors":"P L Bartlett, O Bousquet, S Mendelson","year":"2005","venue":"Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bellet and Habrard, 2015] Aurélien Bellet and Amaury Habrard. Robustness and generalization for metric learn- ing. Neurocomputing, 151:259-267, 2015.","title":"Aurélien Bellet and Amaury Habrard. Robustness and generalization for metric learning","authors":"Habrard Bellet","year":"2015","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cao et al., 2016] Qiong Cao, Zheng-Chu Guo, and Yiming Ying. Generalization bounds for metric and similarity learning. Machine Learning, 102(1):115-132, 2016.","title":"Generalization bounds for metric and similarity learning","authors":"Qiong Cao, Zheng-Chu Guo, Yiming Ying","year":"2016","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Christmann and Zhou, 2016] Andreas Christmann and Ding-Xuan Zhou. On the robustness of regularized pairwise learning methods based on kernels. Journal of Complexity, 37:1-33, 2016.","title":"On the robustness of regularized pairwise learning methods based on kernels","authors":"Christmann, Zhou, Andreas Christmann, Ding-Xuan Zhou","year":"2016","venue":"Journal of Complexity","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Clémenc ¸on et al., 2008] Stéphan Clémenc ¸on, Gabor Lu- gosi, and Nicolas Vayatis. Ranking and empirical mini- mization of U-statistics. Annals of Statistics, 36(2):844- 874, 2008.","title":"Ranking and empirical minimization of U-statistics","authors":"Stéphan Clémenc ¸on, Gabor Lugosi, Nicolas Vayatis","year":"2008","venue":"Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cortes and Mohri, 2004] Corinna Cortes and Mehryar Mohri. AUC optimization vs. error rate minimization. In NIPS, pages 313-320, 2004.","title":"AUC optimization vs. error rate minimization","authors":"Mohri Cortes, Corinna Cortes, Mehryar Mohri","year":"2004","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cucker and Zhou, 2007] Felipe Cucker and Ding-Xuan Zhou. Learning theory: an approximation theory view- point. Cambridge Univ. Press, Cambridge, 2007.","title":"Felipe Cucker and Ding-Xuan Zhou. Learning theory: an approximation theory viewpoint","authors":"Zhou Cucker","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gao and Zhou, 2015] Wei Gao and Zhi-Hua Zhou. On the consistency of AUC pairwise optimization. In IJCAI, pages 939-945, 2015.","title":"On the consistency of AUC pairwise optimization","authors":"Wei Gao, Zhi-Hua Zhou","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/137"},{"position":10,"raw":"[Gao et al., 2013] Wei Gao, Rong Jin, Shenghuo Zhu, and Zhi-Hua Zhou. One-pass AUC optimization. In ICML, pages 906-914, 2013.","title":"One-pass AUC optimization","authors":"Wei Gao, Rong Jin, Shenghuo Zhu, Zhi-Hua Zhou","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Guo et al., 2016] Zheng-Chu Guo, Yiming Ying, and Ding- Xuan Zhou. Online regularized learning with pairwise loss functions. Advances in Computational Mathematics, pages 1-24, 2016.","title":"Online regularized learning with pairwise loss functions","authors":"Zheng-Chu Guo, Yiming Ying, Ding-Xuan Zhou","year":"2016","venue":"Advances in Computational Mathematics","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hu et al., 2015] Ting Hu, Jun Fan, Qiang Wu, and Ding- Xuan Zhou. Regularization schemes for minimum error entropy principle. Analysis and Applications, 13(04):437- 455, 2015.","title":"Regularization schemes for minimum error entropy principle","authors":"Ting Hu, Jun Fan, Qiang Wu, Ding-Xuan Zhou","year":"2015","venue":"Analysis and Applications","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jin et al., 2009] Rong Jin, Shijun Wang, and Yang Zhou. Regularized distance metric learning: theory and algo- rithm. In NIPS, pages 862-870, 2009.","title":"Regularized distance metric learning: theory and algorithm","authors":"Rong Jin, Shijun Wang, Yang Zhou","year":"2009","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kakade et al., 2012] Sham M Kakade, Shai Shalev- Shwartz, and Ambuj Tewari. Regularization techniques for learning with matrices. Journal of Machine Learning Research, 13:1865-1890, 2012.","title":"Regularization techniques for learning with matrices","authors":"Shai Sham M Kakade, Ambuj Shalev-Shwartz, Tewari","year":"2012","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kar et al., 2013] Purushottam Kar, Bharath Sriperumbudur, Prateek Jain, and Harish Karnick. On the generalization ability of online learning algorithms for pairwise loss func- tions. In ICML, pages 441-449, 2013.","title":"On the generalization ability of online learning algorithms for pairwise loss functions","authors":"Purushottam Kar, Bharath Sriperumbudur, Prateek Jain, Harish Karnick","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kriukova et al., 2016] Galyna Kriukova, Oleksandra Pana- siuk, Sergei V Pereverzyev, and Pavlo Tkachenko. A linear functional strategy for regularized ranking. Neural Net- works, 73:26-35, 2016.","title":"A linear functional strategy for regularized ranking","authors":"Kriukova","year":"2016","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lei and Ying, 2016] Yunwen Lei and Yiming Ying. Gener- alization analysis of multi-modal metric learning. Analysis and Applications, 14(04):503-521, 2016.","title":"Yunwen Lei and Yiming Ying. Generalization analysis of multi-modal metric learning","authors":"Ying Lei","year":"2016","venue":"Analysis and Applications","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lin and Zhou, 2018] Shao-Bo Lin and Ding-Xuan Zhou. Distributed kernel-based gradient descent algorithms. Constructive Approximation, 47(2):249-276, 2018.","title":"Distributed kernel-based gradient descent algorithms","authors":"Bo Shao, Ding-Xuan Lin, Zhou","year":"2018","venue":"Constructive Approximation","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lin et al., 2017] Junhong Lin, Yunwen Lei, Bo Zhang, and Ding-Xuan Zhou. Online pairwise learning algorithms with convex loss functions. Information Sciences, 406:57- 70, 2017.","title":"Online pairwise learning algorithms with convex loss functions","authors":"Junhong Lin, Yunwen Lei, Bo Zhang, Ding-Xuan Zhou","year":"2017","venue":"Information Sciences","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mohri et al., 2012] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning. MIT press, 2012.","title":"Foundations of Machine Learning","authors":"Mehryar Mohri, Afshin Rostamizadeh, Ameet Talwalkar","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Mukherjee and Zhou, 2006] Sayan Mukherjee and Ding- Xuan Zhou. Learning coordinate covariances via gradi- ents. Journal of Machine Learning Research, 7:519-549, 2006.","title":"Learning coordinate covariances via gradients","authors":"Sayan Mukherjee, Ding-Xuan Zhou","year":"2006","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rejchel, 2012] Wojciech Rejchel. On ranking and general- ization bounds. Journal of Machine Learning Research, 13(1):1373-1392, 2012.","title":"On ranking and generalization bounds","authors":"Wojciech Rejchel","year":"2012","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Smale and Zhou, 2003] Steve Smale and Ding-Xuan Zhou. Estimating the approximation error in learning theory. Analysis and Applications, 1(01):17-41, 2003.","title":"Estimating the approximation error in learning theory","authors":"Zhou ; ] Steve Smale, Ding-Xuan Smale, Zhou","year":"2003","venue":"Analysis and Applications","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sridharan et al., 2009] Karthik Sridharan, Shai Shalev- Shwartz, and Nathan Srebro. Fast rates for regularized objectives. In NIPS, pages 1545-1552, 2009.","title":"Fast rates for regularized objectives","authors":"Karthik Sridharan, Shai Shalev-Shwartz, Nathan Srebro","year":"2009","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2012] Yuyang Wang, Roni Khardon, Dmitry Pechyony, and Rosie Jones. Generalization bounds for on- line learning algorithms with pairwise loss functions. In COLT, pages 1-22, 2012.","title":"Generalization bounds for online learning algorithms with pairwise loss functions","authors":"Yuyang Wang, Roni Khardon, Dmitry Pechyony, Rosie Jones","year":"2012","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xing et al., 2003] Eric P Xing, Andrew Y Ng, Michael I Jordan, and Stuart Russell. Distance metric learning with application to clustering with side-information. In NIPS, pages 505-512, 2003.","title":"Distance metric learning with application to clustering with side-information","authors":"Eric P Xing, Andrew Y Ng, Michael I Jordan, Stuart Russell","year":"2003","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Ying and Zhou, 2016] Yiming Ying and Ding-Xuan Zhou. Online pairwise learning algorithms. Neural Computation, 28(4):743-777, 2016.","title":"Online pairwise learning algorithms","authors":"Yiming Ying, Ding-Xuan Zhou","year":"2016","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Ying et al., 2009] Yiming Ying, Kaizhu Huang, and Colin Campbell. Sparse metric learning via smooth optimiza- tion. In NIPS, pages 2214-2222, 2009.","title":"Sparse metric learning via smooth optimization","authors":"Yiming Ying, Kaizhu Huang, Colin Campbell","year":"2009","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2017] Lijun Zhang, Tianbao Yang, and Rong Jin. Empirical risk minimization for stochastic convex op- timization: O(1/n)-and O(1/n 2 )-type of risk bounds. In COLT, pages 1954-1979, 2017.","title":"Empirical risk minimization for stochastic convex optimization: O(1/n)-and O(1/n 2 )-type of risk bounds","authors":"Lijun Zhang, Tianbao Yang, Rong Jin","year":"2017","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhao et al., 2011] Peilin Zhao, Rong Jin, Tianbao Yang, and Steven C Hoi. Online AUC maximization. In ICML, pages 233-240, 2011.","title":"Online AUC maximization","authors":"Peilin Zhao, Rong Jin, Tianbao Yang, Steven C Hoi","year":"2011","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhao et al., 2017] Yulong Zhao, Jun Fan, and Lei Shi. Learning rates for regularized least squares ranking algo- rithm. Analysis and Applications, 15(06):815-836, 2017.","title":"Learning rates for regularized least squares ranking algorithm","authors":"Zhao","year":"2017","venue":"Analysis and Applications","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}