{"type":"ScholarlyArticle","title":"Incremental and Decremental Optimal Margin Distribution Learning","authors":[{"name":"Li-Jun Chen","slug":"li-jun-chen","given":"Li-Jun","family":"Chen","affiliation":"National Engineering Research Center for Big Data Technology and System Services Computing Technology and System Lab , Cluster and Grid Computing Lab School of Computer Science and Technology , Huazhong University of Science and Technology , China"},{"name":"Teng Zhang","slug":"teng-zhang","given":"Teng","family":"Zhang","affiliation":"National Engineering Research Center for Big Data Technology and System Services Computing Technology and System Lab , Cluster and Grid Computing Lab School of Computer Science and Technology , Huazhong University of Science and Technology , China"},{"name":"Xuanhua Shi","slug":"xuanhua-shi","given":"Xuanhua","family":"Shi","affiliation":"National Engineering Research Center for Big Data Technology and System Services Computing Technology and System Lab , Cluster and Grid Computing Lab School of Computer Science and Technology , Huazhong University of Science and Technology , China"},{"name":"Hai Jin","slug":"hai-jin","given":"Hai","family":"Jin","affiliation":"National Engineering Research Center for Big Data Technology and System Services Computing Technology and System Lab , Cluster and Grid Computing Lab School of Computer Science and Technology , Huazhong University of Science and Technology , China"}],"abstract":"Incremental and decremental learning (IDL) deals with the tasks where new data arrives sequentially as a stream or old data turns unavailable continually due to the privacy protection. Existing IDL methods mainly focus on support vector machine and its variants with linear-type loss. There are few studies about the quadratic-type loss, whose Lagrange multipliers are unbounded and much more difficult to track. In this paper, we take the latest statistical learning framework optimal margin distribution machine (ODM) which involves a quadratic-type loss due to the optimization of margin variance, for example, and equip it with the ability to handle IDL tasks. Our proposed ID-ODM can avoid updating the Lagrange multipliers in an infinite range by determining their optimal values beforehand so as to enjoy much more efficiency. Moreover, ID-ODM is also applicable when multiple instances come and leave simultaneously. Extensive empirical studies show that ID-ODM can achieve 9.1x speedup on average with almost no generalization lost compared to retraining ODM on new data set from scratch.","doi":"10.24963/ijcai.2023/392","url":"https://ijcai.org/proceedings/2023/392","pdf":"https://ijcai.org/proceedings/2023/0392.pdf","markdown":"https://ijcai.org/proceedings/2023/392.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/392","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":392,"pages":{"first":3523,"last":3531},"keywords":["Machine Learning: ML: Incremental learning","Machine Learning: ML: Classification"],"videos":[],"references":[{"position":1,"raw":"[Boyd and Vandenberghe, 2004] Stephen P. Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, Cambridge, United Kingdom, 2004.","title":"Convex Optimization","authors":"Stephen P Boyd, Lieven Vandenberghe","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cauwenberghs and Poggio, 2000] Gert Cauwenberghs and Tomaso A. Poggio. Incremental and decremental support vector machine learning. In Advances in Neural Infor- mation Processing Systems, pages 409-415, Denver, CO, 2000.","title":"Incremental and decremental support vector machine learning","authors":"Poggio Cauwenberghs, Gert Cauwenberghs, Tomaso A Poggio","year":"2000","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2023] Haiyan Chen, Ying Yu, Yizhen Jia, and Bin Gu. Incremental learning for transductive support vec- tor machine. Pattern Recognition, 133:10982, 2023.","title":"Incremental learning for transductive support vector machine","authors":"Haiyan Chen, Ying Yu, Yizhen Jia, Bin Gu","year":"2023","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cherubin et al., 2021] Giovanni Cherubin, Konstantinos Chatzikokolakis, and Martin Jaggi. Exact optimization of conformal predictors via incremental and decremental learning. In Proceedings of the 38th International Con- ference on Machine Learning, pages 1836-1845, virtual, 2021.","title":"Exact optimization of conformal predictors via incremental and decremental learning","authors":"Giovanni Cherubin, Konstantinos Chatzikokolakis, Martin Jaggi","year":"2021","venue":"Proceedings of the 38th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cortes and Vapnik, 1995] Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine Learning, 20(3):273-297, 1995.","title":"Corinna Cortes and Vladimir Vapnik. Support-vector networks","authors":"Vapnik Cortes","year":"1995","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ertekin et al., 2007] Seyda Ertekin, Jian Huang, Léon Bot- tou, and C. Lee Giles. Learning on the border: active learn- ing in imbalanced data classification. In Proceedings of the 16th ACM Conference on Information and Knowledge Management, pages 127-136, Lisbon, Portugal, 2007.","title":"Learning on the border: active learning in imbalanced data classification","authors":"Seyda Ertekin, Jian Huang, Léon Bottou, C Lee Giles","year":"2007","venue":"Proceedings of the 16th ACM Conference on Information and Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gâlmeanu and Andonie, 2022] Honorius Gâlmeanu and Razvan Andonie. Weighted incremental-decremental support vector machines for concept drift with shifting window. Neural Networks, 152(5):528-541, 2022.","title":"Honorius Gâlmeanu and Razvan Andonie. Weighted incremental-decremental support vector machines for concept drift with shifting window","authors":"Andonie Gâlmeanu","year":"2022","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gao and Zhou, 2013] Wei Gao and Zhi-Hua Zhou. On the doubt about margin explanation of boosting. Artificial In- telligence, 203:1-18, 2013.","title":"On the doubt about margin explanation of boosting","authors":"Wei Gao, Zhi-Hua Zhou","year":"2013","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Golub and Van Loan, 1983] Gene H. Golub and Charles F. Van Loan. Matrix Computations. Johns Hopkins Univer- sity Press, Baltimore, MD, 1983.","title":"Matrix Computations","authors":"Gene H Golub, Charles F Van Loan","year":"1983","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gu et al., 2012] Bin Gu, Jiandong Wang, Yuecheng Yu, Guansheng Zheng, Yu-Fan Huang, and Tao Xu. Accu- rate on-line ν-support vector learning. Neural Networks, 27:51-59, 2012.","title":"Accurate on-line ν-support vector learning","authors":"Bin Gu, Jiandong Wang, Yuecheng Yu, Guansheng Zheng, Yu-Fan Huang, Tao Xu","year":"2012","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gu et al., 2015a] Bin Gu, Victor S. Sheng, Keng Yeow Tay, Walter Romano, and Shuo Li. Incremental support vector learning for ordinal regression. IEEE Transactions Neural Networks Learning System, 26(7):1403-1416, 2015.","title":"Incremental support vector learning for ordinal regression","authors":"Gu","year":"2015","venue":"IEEE Transactions Neural Networks Learning System","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gu et al., 2015b] Bin Gu, Victor S. Sheng, Zhijie Wang, Derek Ho, Said Osman, and Shuo Li. Incremental learning for ν-support vector regression. Neural Networks, 67:140- 150, 2015.","title":"Incremental learning for ν-support vector regression","authors":"Gu","year":"2015","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gu et al., 2018] Bin Gu, Xiao-Tong Yuan, Songcan Chen, and Heng Huang. New incremental learning algorithm for semi-supervised support vector machine. In Proceed- ings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1475- 1484, London, United Kingdom, 2018.","title":"New incremental learning algorithm for semi-supervised support vector machine","authors":"Gu","year":"2018","venue":"Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Huang et al., 2020] Sheng-Jun Huang, Guo-Xiang Li, Wen- Yu Huang, and Shao-Yuan Li. Incremental multi-label learning with active queries. Journal of Computer Science and Technology, 35(2):234-246, 2020.","title":"Incremental multi-label learning with active queries","authors":"Sheng-Jun Huang, Guo-Xiang Li, Wen-Yu Huang, Shao-Yuan Li","year":"2020","venue":"Journal of Computer Science and Technology","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jiang et al., 2019] Hansi Jiang, Haoyu Wang, Wenhao Hu, Deovrat Kakde, and Arin Chaudhuri. Fast incremental SVDD learning algorithm with the gaussian kernel. In Proceedings of 33rd AAAI Conference on Artificial Intelli- gence, pages 3991-3998, Honolulu, HI, 2019.","title":"Fast incremental SVDD learning algorithm with the gaussian kernel","authors":"Hansi Jiang, Haoyu Wang, Wenhao Hu, Deovrat Kakde, Arin Chaudhuri","year":"2019","venue":"Proceedings of 33rd AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Karasuyama and Takeuchi, 2009] Masayuki Karasuyama and Ichiro Takeuchi. Multiple incremental decremental learning of support vector machines. In Advances in Neural Information Processing Systems, pages 907-915, Vancouver, Canada, 2009.","title":"Multiple incremental decremental learning of support vector machines","authors":"Takeuchi Karasuyama, Masayuki Karasuyama, Ichiro Takeuchi","year":"2009","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Laskov et al., 2006] Pavel Laskov, Christian Gehl, Stefan Krüger, and Klaus-Robert Müller. Incremental support vector learning: Analysis, implementation and applica- tions. Journal of Machine Learning Research, 7(9):1909- 1936, 2006.","title":"Incremental support vector learning: Analysis, implementation and applications","authors":"Laskov","year":"2006","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Laxhammar and Falkman, 2014] Rikard Laxhammar and Göran Falkman. Online learning and sequential anomaly detection in trajectories. IEEE Transactions on Pat- tern Analysis and Machine Intelligence, 36(6):1158-1173, 2014.","title":"Online learning and sequential anomaly detection in trajectories","authors":"Falkman Laxhammar, Rikard Laxhammar, Göran Falkman","year":"2014","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liang and Li, 2009] Zhizheng Liang and Youfu Li. In- cremental support vector machine learning in the primal and applications. Neurocomputing, 72(10-12):2249-2258, 2009.","title":"Incremental support vector machine learning in the primal and applications","authors":"Li ; Liang, Zhizheng Liang, Youfu Li","year":"2009","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nguyen et al., 2019] Thanh Tam Nguyen, Matthias Wei- dlich, Bolong Zheng, Hongzhi Yin, Quoc Viet Hung Nguyen, and Bela Stantic. From anomaly detection to rumour detection using data streams of social platforms. Proceedings of the VLDB Endowment, 12(9):1016-1029, 2019.","title":"From anomaly detection to rumour detection using data streams of social platforms","authors":"Thanh Tam Nguyen, Matthias Weidlich, Bolong Zheng, Hongzhi Yin, Quoc Viet, Hung Nguyen, Bela Stantic","year":"2019","venue":"Proceedings of the VLDB Endowment","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Nguyen et al., 2020] Quoc Phong Nguyen, Bryan Kian Hsiang Low, and Patrick Jaillet. Variational bayesian unlearning. In Advances in Neural Information Processing Systems, pages 16025-16036, virtual, 2020.","title":"Variational bayesian unlearning","authors":"Phong Quoc, Bryan Kian Hsiang Nguyen, Patrick Low, Jaillet","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ouellette, 1981] Diane Valérie Ouellette. Schur comple- ments and statistics. Linear Algebra and its Applications, 36:187-295, 1981.","title":"Schur complements and statistics","authors":"Diane Valérie, Ouellette","year":"1981","venue":"Linear Algebra and its Applications","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rüping, 2001] Stefan Rüping. Incremental learning with support vector machines. In Proceedings of the 1st IEEE International Conference on Data Mining, pages 641-642, San Jose, CA, 2001.","title":"Incremental learning with support vector machines","authors":"Stefan Rüping","year":"2001","venue":"Proceedings of the 1st IEEE International Conference on Data Mining","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Schlimmer and Fisher, 1986] Jeffrey C. Schlimmer and Douglas H. Fisher. A case study of incremental concept in- duction. In Proceedings of the 5th National Conference on Artificial Intelligence, pages 496-501, Philadelphia, PA, 1986.","title":"A case study of incremental concept induction","authors":"Jeffrey C Schlimmer, Douglas H Fisher","year":"1986","venue":"Proceedings of the 5th National Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Schölkopf et al., 2000] Bernhard Schölkopf, Alexander J. Smola, Robert C. Williamson, and Peter L. Bartlett. New support vector algorithms. Neural Computation, 12(5):1207-1245, 2000.","title":"New support vector algorithms","authors":"Bernhard Schölkopf, Alexander J Smola, Robert C Williamson, Peter L Bartlett","year":"2000","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sekhari et al., 2021] Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remem- ber what you want to forget: Algorithms for machine un- learning. In Advances in Neural Information Processing Systems, pages 18075-18086, Virtual, 2021.","title":"Remember what you want to forget: Algorithms for machine unlearning","authors":"Ayush Sekhari, Jayadev Acharya, Gautam Kamath, Ananda Theertha, Suresh","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Syed et al., 1999] Nadeem Ahmed Syed, Huan Liu, and Kah Kay Sung. Handling concept drifts in incremen- tal learning with support vector machines. In Proceed- ings of the 5th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 317-321, San Diego, CA, 1999.","title":"Handling concept drifts in incremental learning with support vector machines","authors":"Ahmed Nadeem, Huan Syed, Kah Liu, Sung Kay","year":"1999","venue":"Proceedings of the 5th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tax and Duin, 2004] David M. J. Tax and Robert P. W. Duin. Support vector data description. Machine Learn- ing, 54(1):45-66, 2004.","title":"Support vector data description","authors":"M J David, Tax, P W Robert, Duin","year":"2004","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang and Vucetic, 2009] Zhuang Wang and Slobodan Vucetic. Fast online training of ramp loss support vector machines. In Proceedings of the 9th IEEE International Conference on Data Mining, pages 569-577, Miami, FL, 2009.","title":"Fast online training of ramp loss support vector machines","authors":"Wang, Vucetic ; ] Zhuang Wang, Slobodan Vucetic","year":"2009","venue":"Proceedings of the 9th IEEE International Conference on Data Mining","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang and Zhou, 2019] Teng Zhang and Zhi-Hua Zhou. Optimal margin distribution machine. IEEE Transactions on Knowledge and Data Engineering, 32(6):1143-1156, 2019.","title":"Optimal margin distribution machine","authors":"Teng Zhang, Zhi-Hua Zhou","year":"2019","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang, 2006] Fuzhen Zhang. The Schur complement and its applications. Springer Science and Business Media, Berlin, Germany, 2006.","title":"The Schur complement and its applications","authors":"Fuzhen Zhang","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}