{"type":"ScholarlyArticle","title":"On the Optimization of Margin Distribution","authors":[{"name":"Meng-Zhang Qian","slug":"meng-zhang-qian","given":"Meng-Zhang","family":"Qian","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , China"},{"name":"Zheng Ai","slug":"zheng-ai","given":"Zheng","family":"Ai","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , China"},{"name":"Teng Zhang","slug":"teng-zhang","given":"Teng","family":"Zhang","affiliation":"School of Computer Science and Technology , Huazhong University of Science and Technology , China"},{"name":"Wei Gao","slug":"wei-gao","given":"Wei","family":"Gao","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , China"}],"abstract":"Margin has played an important role on the design and analysis of learning algorithms during the past years, mostly working with the maximization of the minimum margin. Recent years have witnessed the increasing empirical studies on the optimization of margin distribution according to different statistics such as medium margin, average margin, margin variance, etc., whereas there is a relative paucity of theoretical understanding.\r\n\r\nIn this work, we take one step on this direction by providing a new generalization error bound, which is heavily relevant to margin distribution by incorporating ingredients such as average margin and semi-variance, a new margin statistics for the characterization of margin distribution. Inspired by the theoretical findings, we propose the MSVMAv, an efficient approach to achieve better performance by optimizing margin distribution in terms of its empirical average margin and semi-variance. We finally conduct extensive experiments to show the superiority of the proposed MSVMAv approach.","doi":"10.24963/ijcai.2022/470","url":"https://ijcai.org/proceedings/2022/470","pdf":"https://ijcai.org/proceedings/2022/0470.pdf","markdown":"https://ijcai.org/proceedings/2022/470.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/470","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":470,"pages":{"first":3387,"last":3393},"keywords":["Machine Learning: Classification","Machine Learning: Learning Theory"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987109","length_sec":90,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985252","length_sec":635,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Aiolli et al., 2008] F. Aiolli, G. Da San Martino, and A. Sperduti. A kernel method for the optimization of the margin distribution. In ICANN, pages 305-314, 2008.","title":"A kernel method for the optimization of the margin distribution","authors":"F Aiolli, G Da San, A Martino, Sperduti","year":"2008","venue":"ICANN","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bartlett and Mendelson, 2002] P. Bartlett and S. Mendel- son. Rademacher and gaussian complexities: Risk bounds and structural results. JMLR, 3:463-482, 2002.","title":"Rademacher and gaussian complexities: Risk bounds and structural results","authors":"P Bartlett, S Mendelson","year":"2002","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bartlett and Shawe-Taylor, 1999] P. Bartlett and J. Shawe- Taylor. Generalization performance of support vector machines and other pattern classifiers. Advances in Kernel Methods: Support Vector Learning, pages 43-54, 1999.","title":"Generalization performance of support vector machines and other pattern classifiers","authors":"P Bartlett, J Shawe-Taylor","year":"1999","venue":"Advances in Kernel Methods: Support Vector Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bartlett et al., 2017] P. Bartlett, D. J. Foster, and M. Tel- garsky. Spectrally-normalized margin bounds for neural networks. In NIPS 30, pages 6240-6249. 2017.","title":"Spectrally-normalized margin bounds for neural networks","authors":"P Bartlett, D J Foster, M Telgarsky","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Boser et al., 1992] B. E. Boser, I. M. Guyon, and V. Vapnik. A training algorithm for optimal margin classifiers. In COLT, pages 144-152, 1992.","title":"A training algorithm for optimal margin classifiers","authors":"B E Boser, I M Guyon, V Vapnik","year":"1992","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Breiman, 1999] L. Breiman. Prediction games and arcing algorithms. Neural Comput., 11(7):1493-1517, 1999.","title":"Prediction games and arcing algorithms","authors":"L Breiman","year":"1999","venue":"Neural Comput","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cheng et al., 2016] F.-Y. Cheng, J. Zhang, and C.-H. Wen. Cost-sensitive large margin distribution machine for clas- sification of imbalanced data. Pattern Recognit. Lett., 80:107-112, 2016.","title":"Cost-sensitive large margin distribution machine for classification of imbalanced data","authors":"F.-Y Cheng, J Zhang, C.-H Wen","year":"2016","venue":"Pattern Recognit. Lett","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cortes and Vapnik, 1995] C. Cortes and V. Vapnik. Support-vector networks. Mach. Learn., 20(3):273-297, 1995.","title":"Support-vector networks","authors":"C Cortes, V Vapnik","year":"1995","venue":"Mach. Learn","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Drucker et al., 1997] H. Drucker, J. C. Burges, L. Kaufman, A. Smola, and V. Vapnik. Support vector regression machines. In NIPS 9, pages 155-161. 1997.","title":"Support vector regression machines","authors":"H Drucker, J C Burges, L Kaufman, A Smola, V Vapnik","year":"1997","venue":"NIPS 9","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Freund and Schapire, 1996] Y. Freund and R. E. Schapire. Experiments with a new boosting algorithm. In ICML, pages 148-156, 1996.","title":"Experiments with a new boosting algorithm","authors":"Y Freund, R E Schapire","year":"1996","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gao and Zhou, 2013] W. Gao and Z.-H. Zhou. On the doubt about margin explanation of boosting. Artif. Intell., 203:1- 18, 2013.","title":"On the doubt about margin explanation of boosting","authors":"W Gao, Z.-H Zhou","year":"2013","venue":"Artif. Intell","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Garg and Roth, 2003] A. Garg and D. Roth. Margin distri- bution and learning. In ICML, pages 210-217, 2003.","title":"Margin distribution and learning","authors":"A Garg, D Roth","year":"2003","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Grønlund et al., 2020] A. Grønlund, L. Kamma, and K. G. Larsen. Near-tight margin-based generalization bounds for support vector machines. In ICML, pages 3779-3788, 2020.","title":"Near-tight margin-based generalization bounds for support vector machines","authors":"A Grønlund, L Kamma, K G Larsen","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ji et al., 2021] Z.-W. Ji, N. Srebro, and M. Telgarsky. Fast margin maximization via dual acceleration. In ICML, pages 4860-4869, 2021.","title":"Fast margin maximization via dual acceleration","authors":"Z.-W Ji, N Srebro, M Telgarsky","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jiang et al., 2019] Y. Jiang, D. Krishnan, H. Mobahi, and S. Bengio. Predicting the generalization gap in deep networks with margin distributions. In ICLR, 2019.","title":"Predicting the generalization gap in deep networks with margin distributions","authors":"Y Jiang, D Krishnan, H Mobahi, S Bengio","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kabán and Durrant, 2020] Ata Kabán and Robert J Durrant. Structure from randomness in halfspace learning with the zero-one loss. JAIR, 69:733-764, 2020.","title":"Structure from randomness in halfspace learning with the zero-one loss","authors":"Durrant ; ] Ata Kabán, Robert J Kabán, Durrant","year":"2020","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Markowitz, 1952] Harry Markowitz. Portfolio selection. Journal of Finance, 7(1):77-91, 1952.","title":"Portfolio selection","authors":"Markowitz","year":"1952","venue":"Journal of Finance","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pelckmans et al., 2007] K. Pelckmans, J. A. K. Suykens, and B. De Moor. A risk minimization principle for a class of parzen estimators. In NIPS 20, pages 1137-1144. 2007.","title":"A risk minimization principle for a class of parzen estimators","authors":"K Pelckmans, J A K Suykens, B De Moor","year":"2007","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Qian et al., 2022] M.-Z. Qian, Z. Ai, T. Zhang, and W. Gao. On the optimization of margin distribution. CoRR, abs/2204.14118, 2022.","title":"On the optimization of margin distribution","authors":"M.-Z Qian, Z Ai, T Zhang, W Gao","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rastogi et al., 2020] R. Rastogi, P. Anand, and S. Chandra. Large-margin distribution machine-based regression. Neu- ral Comput. Appl., 32(8):3633-3648, 2020.","title":"Large-margin distribution machine-based regression","authors":"R Rastogi, P Anand, S Chandra","year":"2020","venue":"Neural Comput. Appl","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rosset et al., 2003] S. Rosset, J. Zhu, and T. Hastie. Margin maximizing loss functions. In NIPS 16, pages 1237-1244. 2003.","title":"Margin maximizing loss functions","authors":"S Rosset, J Zhu, T Hastie","year":"2003","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Schapire et al., 1998] R. E. Schapire, Y. Freund, P. Bartlett, and W. S. Lee. Boosting the margin: A new explana- tion for the effectiveness of voting methods. Ann. Stat., 26(5):1651-1686, 1998.","title":"Boosting the margin: A new explanation for the effectiveness of voting methods","authors":"R E Schapire, Y Freund, P Bartlett, W S Lee","year":"1998","venue":"Ann. Stat","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Schölkopf et al., 2002] B. Schölkopf, A. J. Smola, and F. Bach. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, Cambridge, 2002.","title":"Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond","authors":"B Schölkopf, A J Smola, F Bach","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shalev-Shwartz and Ben-David, 2014] S. Shalev-Shwartz and S. Ben-David. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, Cambridge, 2014.","title":"Understanding Machine Learning: From Theory to Algorithms","authors":"Shalev-Shwartz, Ben-David ; S Shalev-Shwartz, S Ben-David","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sherman and Morrison, 1950] J. Sherman and W. J. Morri- son. Adjustment of an inverse matrix corresponding to a change in one element of a given matrix. Ann. Math. stat., 21(1):124-127, 1950.","title":"Adjustment of an inverse matrix corresponding to a change in one element of a given matrix","authors":"J Sherman, W J Morrison","year":"1950","venue":"Ann. Math. stat","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Shivaswamy and Jebara, 2010] P. K. Shivaswamy and T. Je- bara. Maximum relative margin and data-dependent regu- larization. JMLR, 11(2), 2010.","title":"Maximum relative margin and data-dependent regularization","authors":"P K Shivaswamy, T Jebara","year":"2010","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sokolić et al., 2017] J. Sokolić, R. Giryes, G. Sapiro, and M. R. Rodrigues. Robust large margin deep neural net- works. IEEE Trans. Signal Process., 65(16):4265-4280, 2017.","title":"Robust large margin deep neural networks","authors":"J Sokolić, R Giryes, G Sapiro, M R Rodrigues","year":"2017","venue":"IEEE Trans. Signal Process","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Suykens et al., 2002] J. A. K. Suykens, T. Van Gestel, J. De Brabanter, B. De Moor, and J. Vandewalle. Least Squares Support Vector Machines. World Scientific, Singapore, 2002.","title":"Least Squares Support Vector Machines","authors":"J A K Suykens, T Van Gestel, J De Brabanter, B De Moor, J Vandewalle","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Vapnik, 1982] V. Vapnik. Estimation of Dependences based on Empirical Data. Springer-Verlag, New York, 1982.","title":"Estimation of Dependences based on Empirical Data","authors":"V Vapnik","year":"1982","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wei and Ma, 2020] C. Wei and T.-Y. Ma. Improved sample complexities for deep neural networks and robust classifi- cation via an all-layer margin. In ICLR, 2020.","title":"Improved sample complexities for deep neural networks and robust classification via an all-layer margin","authors":"C Wei, T.-Y Ma","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Weinstein et al., 2020] B. Weinstein, S. Fine, and Y. Hel- Or. Margin-based regularization and selective sampling in deep neural networks. CoRR, abs/2009.06011, 2020.","title":"Margin-based regularization and selective sampling in deep neural networks","authors":"B Weinstein, S Fine, Y Hel-Or","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang and Zhou, 2014] T. Zhang and Z.-H. Zhou. Large margin distribution machine. In KDD, pages 313-322, 2014.","title":"Large margin distribution machine","authors":"T Zhang, Z.-H Zhou","year":"2014","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang and Zhou, 2019] T. Zhang and Z.-H. Zhou. Optimal margin distribution machine. IEEE Trans. Knowl. Data Eng., 32(6):1143-1156, 2019.","title":"Optimal margin distribution machine","authors":"T Zhang, Z.-H Zhou","year":"2019","venue":"IEEE Trans. Knowl. Data Eng","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}