{"type":"ScholarlyArticle","title":"Quadruply Stochastic Gradients for Large Scale Nonlinear Semi-Supervised AUC Optimization","authors":[{"name":"Wanli Shi","slug":"wanli-shi","given":"Wanli","family":"Shi","affiliation":"School of Computer & Software , Nanjing University of Information Science & Technology , P.R.China"},{"name":"Bin Gu","slug":"bin-gu","given":"Bin","family":"Gu","affiliation":"School of Computer & Software , Nanjing University of Information Science & Technology , P.R.China; JD Finance America Corporation"},{"name":"Xiang Li","slug":"xiang-li","given":"Xiang","family":"Li","affiliation":"Computer Science Department , University of Western Ontario , Canada"},{"name":"Xiang Geng","slug":"xiang-geng","given":"Xiang","family":"Geng","affiliation":"School of Computer & Software , Nanjing University of Information Science & Technology , P.R.China"},{"name":"Heng Huang","slug":"heng-huang","given":"Heng","family":"Huang","affiliation":"JD Finance America Corporation; Department of Electrical & Computer Engineering , University of Pittsburgh , USA"}],"abstract":"Semi-supervised learning is pervasive in real-world applications, where only a few labeled data are available and large amounts of instances remain unlabeled. Since AUC is an important model evaluation metric in classification, directly optimizing AUC in semi-supervised learning scenario has drawn much attention in the machine learning community. Recently, it has been shown that one could find an unbiased solution for the semi-supervised AUC maximization problem without knowing the class prior distribution. However, this method is hardly scalable for nonlinear classification problems with kernels. To address this problem, in this paper, we propose a novel scalable quadruply stochastic gradient algorithm (QSG-S2AUC) for nonlinear semi-supervised AUC optimization. In each iteration of the stochastic optimization process, our method randomly samples a positive instance, a negative instance, an unlabeled instance and their random features to compute the gradient and then update the model by using this quadruply stochastic gradient to approach the optimal solution. More importantly, we prove that QSG-S2AUC can converge to the optimal solution in O(1/t), where t is the iteration number. Extensive experimental results on  a variety of benchmark datasets show that QSG-S2AUC is far more efficient than the existing state-of-the-art algorithms for semi-supervised AUC maximization, while retaining the similar generalization performance.","doi":"10.24963/ijcai.2019/474","url":"https://ijcai.org/proceedings/2019/474","pdf":"https://ijcai.org/proceedings/2019/0474.pdf","markdown":"https://ijcai.org/proceedings/2019/474.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/474","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":474,"pages":{"first":3418,"last":3424},"keywords":["Machine Learning: Semi-Supervised Learning","Machine Learning: Kernel Methods","Machine Learning Applications: Big data ; Scalability"],"videos":[],"references":[{"position":1,"raw":"[Amini et al., 2008] Massih-Reza Amini, Tuong-Vinh Truong, and Cyril Goutte. A boosting algorithm for learning bipartite ranking functions with partially labeled data. 2008.","title":"A boosting algorithm for learning bipartite ranking functions with partially labeled data","authors":"Massih-Reza Amini, Tuong-Vinh Truong, Cyril Goutte","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dai et al., 2014] Bo Dai, Bo Xie, Niao He, Yingyu Liang, Anant Raj, Maria-Florina F Balcan, and Le Song. Scalable kernel methods via doubly stochastic gradients. In Ad- vances in Neural Information Processing Systems, pages 3041-3049, 2014.","title":"Scalable kernel methods via doubly stochastic gradients","authors":"Bo Dai, Bo Xie, Niao He, Yingyu Liang, Anant Raj, Maria-Florina F Balcan, Le Song","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ding et al., 2017] Yi Ding, Chenghao Liu, Peilin Zhao, and Steven CH Hoi. Large scale kernel methods for online auc maximization. In 2017 IEEE International Conference on Data Mining (ICDM), pages 91-100. IEEE, 2017.","title":"Large scale kernel methods for online auc maximization","authors":"Yi Ding, Chenghao Liu, Peilin Zhao, Steven Ch Hoi","year":"2017","venue":"IEEE International Conference on Data Mining (ICDM)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Drineas and Mahoney, 2005] Petros Drineas and Michael W Mahoney. On the nyström method for approximating a gram matrix for improved kernel- based learning. journal of machine learning research, 6(Dec):2153-2175, 2005.","title":"On the nyström method for approximating a gram matrix for improved kernelbased learning","authors":"Petros Drineas, Michael W Mahoney","year":"2005","venue":"journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[du Plessis et al., 2015] Marthinus Christoffel du Plessis, Gang Niu, and Masashi Sugiyama. Class-prior estimation for learning from positive and unlabeled data. In ACML, pages 221-236, 2015.","title":"Class-prior estimation for learning from positive and unlabeled data","authors":"Marthinus Christoffel Du Plessis, Gang Niu, Masashi Sugiyama","year":"2015","venue":"ACML","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fine and Scheinberg, 2001] Shai Fine and Katya Schein- berg. Efficient svm training using low-rank kernel rep- resentations. Journal of Machine Learning Research, 2(Dec):243-264, 2001.","title":"Efficient svm training using low-rank kernel representations","authors":"Scheinberg Fine","year":"2001","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fujino and Ueda, 2016] Akinori Fujino and Naonori Ueda. A semi-supervised auc optimization method with gener- ative models. In Data Mining (ICDM), 2016 IEEE 16th International Conference on, pages 883-888. IEEE, 2016.","title":"A semi-supervised auc optimization method with generative models","authors":"Ueda Fujino, Akinori Fujino, Naonori Ueda","year":"2016","venue":"IEEE 16th International Conference on","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gao and Zhou, 2015] Wei Gao and Zhi-Hua Zhou. On the consistency of auc pairwise optimization. In IJCAI, pages 939-945, 2015.","title":"On the consistency of auc pairwise optimization","authors":"Wei Gao, Zhi-Hua Zhou","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/137"},{"position":9,"raw":"[Gao et al., 2013] Wei Gao, Rong Jin, Shenghuo Zhu, and Zhi-Hua Zhou. One-pass auc optimization. In Interna- tional Conference on Machine Learning, pages 906-914, 2013.","title":"One-pass auc optimization","authors":"Wei Gao, Rong Jin, Shenghuo Zhu, Zhi-Hua Zhou","year":"2013","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gu and Huo, 2018] Bin Gu and Zhouyuan Huo. Asyn- chronous doubly stochastic group regularized learning. In International Conference on Artificial Intelligence and Statistics (AISTATS 2018), 2018.","title":"Asynchronous doubly stochastic group regularized learning","authors":"Huo ; ] Bin Gu, Zhouyuan Gu, Huo","year":"2018","venue":"International Conference on Artificial Intelligence and Statistics (AISTATS 2018)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gu et al., 2016] Bin Gu, Zhouyuan Huo, and Heng Huang. Asynchronous stochastic block coordinate descent with variance reduction. arXiv preprint arXiv:1610.09447, 2016.","title":"Asynchronous stochastic block coordinate descent with variance reduction","authors":"Gu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gu et al., 2018a] Bin Gu, Yingying Shan, Xiang Geng, and Guansheng Zheng. Accelerated asynchronous greedy co- ordinate descent algorithm for svms. In IJCAI, pages 2170-2176, 2018.","title":"Accelerated asynchronous greedy coordinate descent algorithm for svms","authors":"Gu","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/300"},{"position":13,"raw":"[Gu et al., 2018b] Bin Gu, Miao Xin, Zhouyuan Huo, and Heng Huang. Asynchronous doubly stochastic sparse ker- nel learning. In Thirty-Second AAAI Conference on Artifi- cial Intelligence, 2018.","title":"Asynchronous doubly stochastic sparse kernel learning","authors":"Gu","year":"2018","venue":"Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Gu et al., 2018c] Bin Gu, Xiao-Tong Yuan, Songcan Chen, and Heng Huang. New incremental learning algorithm for semi-supervised support vector machine. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1475-1484. ACM, 2018.","title":"New incremental learning algorithm for semi-supervised support vector machine","authors":"Gu","year":"2018","venue":"Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hanley and McNeil, 1982] James A Hanley and Barbara J McNeil. The meaning and use of the area under a re- ceiver operating characteristic (roc) curve. Radiology, 143(1):29-36, 1982.","title":"The meaning and use of the area under a receiver operating characteristic (roc) curve","authors":"A James, Barbara J Hanley, Mcneil","year":"1982","venue":"Radiology","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2017] Xiang Li, Bin Gu, Shuang Ao, Huaimin Wang, and Charles X Ling. Triply stochastic gradients on multiple kernel learning. UAI, 2017.","title":"Triply stochastic gradients on multiple kernel learning","authors":"Xiang Li, Bin Gu, Shuang Ao, Huaimin Wang, Charles X Ling","year":"2017","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liang, 2005] Percy Liang. Semi-supervised learning for natural language. PhD thesis, Massachusetts Institute of Technology, 2005.","title":"Semi-supervised learning for natural language","authors":"Percy Liang","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2018] Mingrui Liu, Xiaoxuan Zhang, Zaiyi Chen, Xiaoyu Wang, and Tianbao Yang. Fast stochastic auc maximization with o (1/n)-convergence rate. In Inter- national Conference on Machine Learning, pages 3195- 3203, 2018.","title":"Fast stochastic auc maximization with o (1/n)-convergence rate","authors":"Liu","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rahimi and Recht, 2008] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Ad- vances in neural information processing systems, pages 1177-1184, 2008.","title":"Random features for large-scale kernel machines","authors":"Recht Rahimi, Ali Rahimi, Benjamin Recht","year":"2008","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rudin, 2017] Walter Rudin. Fourier analysis on groups. Courier Dover Publications, 2017.","title":"Fourier analysis on groups","authors":"Walter Rudin","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sakai et al., 2017] Tomoya Sakai, Marthinus Christoffel du Plessis, Gang Niu, and Masashi Sugiyama. Semi- supervised classification based on classification from pos- itive and unlabeled data. In Proceedings of the 34th In- ternational Conference on Machine Learning-Volume 70, pages 2998-3006. JMLR. org, 2017.","title":"Semisupervised classification based on classification from positive and unlabeled data","authors":"Tomoya Sakai, Marthinus Christoffel Du Plessis, Gang Niu, Masashi Sugiyama","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sakai et al., 2018] Tomoya Sakai, Gang Niu, and Masashi Sugiyama. Semi-supervised auc optimization based on positive-unlabeled learning. Machine Learning, 107(4):767-794, 2018.","title":"Semi-supervised auc optimization based on positive-unlabeled learning","authors":"Tomoya Sakai, Gang Niu, Masashi Sugiyama","year":"2018","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sholokhov et al., 2018] Alexey Sholokhov, Md Sahidullah, and Tomi Kinnunen. Semi-supervised speech activity de- tection with an application to automatic speaker verifica- tion. Computer Speech & Language, 47:132-156, 2018.","title":"Semi-supervised speech activity detection with an application to automatic speaker verification","authors":"Alexey Sholokhov, Md Sahidullah, Tomi Kinnunen","year":"2018","venue":"Computer Speech & Language","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Smola and Schölkopf, 2000] Alex J Smola and Bernhard Schölkopf. Sparse greedy matrix approximation for ma- chine learning. 2000.","title":"Sparse greedy matrix approximation for machine learning","authors":"Alex J Smola, Bernhard Schölkopf","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2010] Jun Wang, Sanjiv Kumar, and Shih-Fu Chang. Semi-supervised hashing for scalable image re- trieval. 2010.","title":"Semi-supervised hashing for scalable image retrieval","authors":"Jun Wang, Sanjiv Kumar, Shih-Fu Chang","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xie and Li, 2018] Zheng Xie and Ming Li. Semi-supervised auc optimization without guessing labels of unlabeled data. 2018.","title":"Semi-supervised auc optimization without guessing labels of unlabeled data","authors":"Zheng Xie, Ming Li","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Ying et al., 2016] Yiming Ying, Longyin Wen, and Siwei Lyu. Stochastic online auc maximization. In Advances in neural information processing systems, pages 451-459, 2016.","title":"Stochastic online auc maximization","authors":"Yiming Ying, Longyin Wen, Siwei Lyu","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}