{"type":"ScholarlyArticle","title":"Scalable Semi-Supervised SVM via Triply Stochastic Gradients","authors":[{"name":"Xiang Geng","slug":"xiang-geng","given":"Xiang","family":"Geng","affiliation":"School of Computer & Software , Nanjing University of Information Science & Technology , P.R.China"},{"name":"Bin Gu","slug":"bin-gu","given":"Bin","family":"Gu","affiliation":"School of Computer & Software , Nanjing University of Information Science & Technology , P.R.China; JD Finance America Corporation"},{"name":"Xiang Li","slug":"xiang-li","given":"Xiang","family":"Li","affiliation":"Computer Science Department , University of Western Ontario , Canada"},{"name":"Wanli Shi","slug":"wanli-shi","given":"Wanli","family":"Shi","affiliation":"School of Computer & Software , Nanjing University of Information Science & Technology , P.R.China"},{"name":"Guansheng Zheng","slug":"guansheng-zheng","given":"Guansheng","family":"Zheng","affiliation":"School of Computer & Software , Nanjing University of Information Science & Technology , P.R.China"},{"name":"Heng Huang","slug":"heng-huang","given":"Heng","family":"Huang","affiliation":"JD Finance America Corporation; Department of Electrical & Computer Engineering , University of Pittsburgh , USA"}],"abstract":"Semi-supervised learning (SSL) plays an increasingly important role in the big data era because a large number of unlabeled samples can be used effectively to improve the performance of the classifier. Semi-supervised support vector machine (S3VM) is one of the most appealing methods for SSL, but scaling up S3VM for kernel learning is still an open problem. Recently, a doubly stochastic gradient (DSG) algorithm has been proposed to achieve efficient and scalable training for kernel methods. However, the algorithm and theoretical analysis of DSG are developed based on the convexity assumption which makes them incompetent for non-convex problems such as S3VM. To address this problem, in this paper, we propose a triply stochastic gradient algorithm for S3VM, called TSGS3VM. Specifically, to handle two types of data instances involved in S3VM, TSGS3VM samples a labeled instance and  an unlabeled instance as well with the random features in each iteration to compute a triply stochastic gradient. We use the approximated gradient to update the  solution. More importantly, we establish new theoretic analysis for TSGS3VM which guarantees that TSGS3VM can converge to a stationary point. Extensive experimental results on a variety of datasets demonstrate that TSGS3VM is much more efficient and scalable than existing S3VM algorithms.","doi":"10.24963/ijcai.2019/328","url":"https://ijcai.org/proceedings/2019/328","pdf":"https://ijcai.org/proceedings/2019/0328.pdf","markdown":"https://ijcai.org/proceedings/2019/328.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/328","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":328,"pages":{"first":2364,"last":2370},"keywords":["Machine Learning: Data Mining","Machine Learning: Classification"],"videos":[],"references":[{"position":1,"raw":"[Bennett and Demiriz, 1999] Kristin P Bennett and Ayhan Demiriz. Semi-supervised support vector machines. In Advances in Neural Information processing systems, pages 368-374, 1999.","title":"Semi-supervised support vector machines","authors":"P Kristin, Ayhan Bennett, Demiriz","year":"1999","venue":"Advances in Neural Information processing systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cai and Cherkassky, 2012] Feng Cai and Vladimir Cherkassky. Generalized smo algorithm for svm- based multitask learning. IEEE transactions on neural networks and learning systems, 23(6):997-1003, 2012.","title":"Generalized smo algorithm for svmbased multitask learning","authors":"Cherkassky Cai, Feng Cai, Vladimir Cherkassky","year":"2012","venue":"IEEE transactions on neural networks and learning systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Carratino et al., 2018] Luigi Carratino, Alessandro Rudi, and Lorenzo Rosasco. Learning with sgd and random fea- tures. In Advances in Neural Information Processing Sys- tems, pages 10213-10224, 2018.","title":"Learning with sgd and random features","authors":"Carratino","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chang and Lin, 2011] Chih-Chung Chang and Chih-Jen Lin. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1-27:27, 2011.","title":"LIBSVM: A library for support vector machines","authors":"Chang, Lin ; ] Chih-Chung Chang, Chih-Jen Lin","year":"2011","venue":"ACM Transactions on Intelligent Systems and Technology","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chapelle and Zien, 2005] Olivier Chapelle and Alexander Zien. Semi-supervised classification by low density sepa- ration. In AISTATS, volume 2005, pages 57-64. Citeseer, 2005.","title":"Semi-supervised classification by low density separation","authors":"Zien Chapelle, Olivier Chapelle, Alexander Zien","year":"2005","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chapelle, 2007] Olivier Chapelle. Training a support vector machine in the primal. Neural computation, 19(5):1155- 1178, 2007.","title":"Training a support vector machine in the primal","authors":"Olivier Chapelle, Chapelle","year":"2007","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Collobert et al., 2006] Ronan Collobert, Fabian Sinz, Jason Weston, and Léon Bottou. Large scale transductive svm- s. Journal of Machine Learning Research, 7(Aug):1687- 1712, 2006.","title":"Large scale transductive svms","authors":"Ronan Collobert, Fabian Sinz, Jason Weston, Léon Bottou","year":"2006","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dai et al., 2014] Bo Dai, Bo Xie, Niao He, Yingyu Liang, Anant Raj, Maria-Florina F Balcan, and Le Song. Scalable kernel methods via doubly stochastic gradients. In Ad- vances in Neural Information Processing Systems, pages 3041-3049, 2014.","title":"Scalable kernel methods via doubly stochastic gradients","authors":"Bo Dai, Bo Xie, Niao He, Yingyu Liang, Anant Raj, Maria-Florina F Balcan, Le Song","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Drineas and Mahoney, 2005] Petros Drineas and Michael W Mahoney. On the nyström method for approximating a gram matrix for improved kernel- based learning. journal of machine learning research, 6(Dec):2153-2175, 2005.","title":"On the nyström method for approximating a gram matrix for improved kernelbased learning","authors":"Petros Drineas, Michael W Mahoney","year":"2005","venue":"journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ghadimi and Lan, 2013] Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for non- convex stochastic programming. SIAM Journal on Opti- mization, 23(4):2341-2368, 2013.","title":"Stochastic first-and zeroth-order methods for nonconvex stochastic programming","authors":"Saeed Ghadimi, Guanghui Lan","year":"2013","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gu et al., 2018a] Bin Gu, Zhouyuan Huo, Cheng Deng, and Heng Huang. Faster derivative-free stochastic algorithm for shared memory machines. In International Conference on Machine Learning, pages 1807-1816, 2018.","title":"Faster derivative-free stochastic algorithm for shared memory machines","authors":"Gu","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gu et al., 2018b] Bin Gu, Yingying Shan, Xiang Geng, and Guansheng Zheng. Accelerated asynchronous greedy co- ordinate descent algorithm for svms. In IJCAI, pages 2170-2176, 2018.","title":"Accelerated asynchronous greedy coordinate descent algorithm for svms","authors":"Gu","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/300"},{"position":13,"raw":"[Gu et al., 2018c] Bin Gu, Miao Xin, Zhouyuan Huo, and Heng Huang. Asynchronous doubly stochastic sparse k- ernel learning. In Thirty-Second AAAI Conference on Ar- tificial Intelligence, 2018.","title":"Asynchronous doubly stochastic sparse kernel learning","authors":"Gu","year":"2018","venue":"Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Gu et al., 2018d] Bin Gu, Xiao-Tong Yuan, Songcan Chen, and Heng Huang. New incremental learning algorithm for semi-supervised support vector machine. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1475-1484. ACM, 2018.","title":"New incremental learning algorithm for semi-supervised support vector machine","authors":"Gu","year":"2018","venue":"Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Huo et al., 2018] Zhouyuan Huo, Bin Gu, and Heng Huang. Training neural networks using features replay. In Ad- vances in Neural Information Processing Systems, pages 6659-6668, 2018.","title":"Training neural networks using features replay","authors":"Huo","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Joachims, 1999] Thorsten Joachims. Transductive inference for text classification using support vector machines. In Icml, volume 99, pages 200-209, 1999.","title":"Transductive inference for text classification using support vector machines","authors":"Thorsten Joachims","year":"1999","venue":"Icml","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Le et al., 2016] Trung Le, Phuong Duong, Mi Dinh, Tu D- inh Nguyen, Vu Nguyen, and Dinh Q Phung. Budgeted semi-supervised support vector machine. In UAI, 2016.","title":"Budgeted semi-supervised support vector machine","authors":"Trung Le, Phuong Duong, Mi Dinh, Tu Dinh Nguyen, Vu Nguyen, Dinh Q Phung","year":"2016","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2017] Xiang Li, Bin Gu, Shuang Ao, Huaimin Wang, and Charles X Ling. Triply stochastic gradients on multiple kernel learning. In UAI, 2017.","title":"Triply stochastic gradients on multiple kernel learning","authors":"Xiang Li, Bin Gu, Shuang Ao, Huaimin Wang, Charles X Ling","year":"2017","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lopez-Paz et al., 2014] David Lopez-Paz, Suvrit Sra, Alex Smola, Zoubin Ghahramani, and Bernhard Schölkopf. Randomized nonlinear component analysis. Computer Science, 4:1359-1367, 2014.","title":"Randomized nonlinear component analysis","authors":"David Lopez-Paz, Suvrit Sra, Alex Smola, Zoubin Ghahramani, Bernhard Schölkopf","year":"2014","venue":"Computer Science","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rahimi and Recht, 2008] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Ad- vances in neural information processing systems, pages 1177-1184, 2008.","title":"Random features for large-scale kernel machines","authors":"Recht Rahimi, Ali Rahimi, Benjamin Recht","year":"2008","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2007] Junhui Wang, Xiaotong Shen, and Wei Pan. On transductive support vector machines. Contem- porary Mathematics, 443:7-20, 2007.","title":"On transductive support vector machines","authors":"Junhui Wang, Xiaotong Shen, Wei Pan","year":"2007","venue":"Contemporary Mathematics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wendland, 2004] Holger Wendland. Scattered data approx- imation, volume 17. Cambridge university press, 2004.","title":"Scattered data approximation","authors":"Holger Wendland, Wendland","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xie et al., 2015] Bo Xie, Yingyu Liang, and Le Song. Scale up nonlinear component analysis with doubly stochastic gradients. In Advances in Neural Information Processing Systems, pages 2341-2349, 2015.","title":"Scale up nonlinear component analysis with doubly stochastic gradients","authors":"Bo Xie, Yingyu Liang, Le Song","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yuille and Rangarajan, 2002] Alan L Yuille and Anand Rangarajan. The concave-convex procedure (cccp). In Advances in neural information processing systems, pages 1033-1040, 2002.","title":"The concave-convex procedure (cccp)","authors":"Alan L Yuille, Anand Rangarajan","year":"2002","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}