{"type":"ScholarlyArticle","title":"Storage Fit Learning with Unlabeled Data","authors":[{"name":"Bo-Jian Hou","slug":"bo-jian-hou","given":"Bo-Jian","family":"Hou","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"},{"name":"Lijun Zhang","slug":"lijun-zhang","given":"Lijun","family":"Zhang","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"},{"name":"Zhi-Hua Zhou","slug":"zhi-hua-zhou","given":"Zhi-Hua","family":"Zhou","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"}],"abstract":"By using abundant unlabeled data, semi-supervised learning approaches have been found very useful in various tasks. Existing approaches, however, neglect the fact that the storage available for the learning process is different under different situations, and thus, the learning approaches should be flexible subject to the storage budget limit. In this paper, we focus on graph-based semi-supervised learning and propose two storage fit learning approaches which can adjust their behaviors to different storage budgets. Specifically, we utilize techniques of low-rank matrix approximation to find a low-rank approximator of the similarity matrix so as to reduce the space complexity. The first approach is based on stochastic optimization, which is an iterative approach that converges to the optimal low-rank approximator globally. The second approach is based on Nystrom method, which can find a good low-rank approximator efficiently and is suitable for real-time applications. Experiments on classification tasks show that the proposed methods can fit dynamically different storage budgets and obtain good performances in different scenarios.","doi":"10.24963/ijcai.2017/256","url":"https://ijcai.org/proceedings/2017/256","pdf":"https://ijcai.org/proceedings/2017/0256.pdf","markdown":"https://ijcai.org/proceedings/2017/256.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/256","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":256,"pages":{"first":1844,"last":1850},"keywords":["Machine Learning: Cost-Sensitive Learning","Machine Learning: Semi-Supervised Learning"],"videos":[],"references":[{"position":1,"raw":"3: AUC on UCI data sets without storage limit. The number in parentheses shows the relative rank of the algorithm on the corresponding data set. The smaller the rank, the better the relative performance. \"size\" means the number of instances; \"dim\" means the dimensionality. The best performance on each data set is bolded. Dataset [size, dim] KNN Harmonic CMN TSVM ClusK SoCK NysCK australian[690,42] .743(7) .754(5) .754(6) .851(4) .873(3) .902(1) .897(2) credit-a[653,15] .805(7) .874(5) .864(6) .894(3) .901(1) .884(4) .896(2) credit-g[1000,20] .591(7) .670(5) .670(6) .700(4) .711(2) .723(1) .705(3) diabetes[768,8] .640(7) .737(5) .737(6) .781(2) .801(1) .775(3) .757(4) german[1000,59] .587(7) .665(4) .665(5) .655(6) .691(2) .710(1) .669(3) kr-vs-kp[3196,36] .821(7) .917(6) .917(5) .928(4) .990(1) .985(2) .980(3) splice[3175,60] .678(7) .782(5) .782(6) .825(3) .899(1) .891(2) .823(4) svmguide3[1284,22] .605(7) .645(4) .643(5) .629(6) .769(2) .701(3) .771(1) Total rank 56 39 45","title":"AUC on UCI data sets without storage limit. The number in parentheses shows the relative rank of the algorithm on the corresponding data set. The smaller the rank, the better the relative performance. \"size\" means the number of instances; \"dim\" means the dimensionality. The best performance on each data set is bolded. Dataset [size","authors":null,"year":null,"venue":"KNN Harmonic CMN TSVM ClusK SoCK NysCK australian","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Avron et al., 2012] Haim Avron, Satyen Kale, Shiva Prasad Ka- siviswanathan, and Vikas Sindhwani. Efficient and practical stochastic subgradient descent for nuclear norm regularization. In ICML, 2012.","title":"Efficient and practical stochastic subgradient descent for nuclear norm regularization","authors":"Satyen Haim Avron, Shiva Kale, Vikas Prasad Kasiviswanathan, Sindhwani","year":"2012","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Belkin et al., 2004] Mikhail Belkin, Irina Matveeva, and Partha Niyogi. Regularization and semi-supervised learning on large graphs. In COLT, 2004.","title":"Regularization and semi-supervised learning on large graphs","authors":"Mikhail Belkin, Irina Matveeva, Partha Niyogi","year":"2004","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Blum and Chawla, 2001] Avrim Blum and Shuchi Chawla. Learn- ing from labeled and unlabeled data using graph mincuts. In ICML, 2001.","title":"Learning from labeled and unlabeled data using graph mincuts","authors":"Avrim Blum, Shuchi Chawla","year":"2001","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Boutsidis et al., 2009] Christos Boutsidis, Michael Mahoney, and Petros Drineas. An improved approximation algorithm for the column subset selection problem. In Proceedings of the twentieth Annual ACM-SIAM Symposium on Discrete Algorithms, 2009.","title":"An improved approximation algorithm for the column subset selection problem","authors":"Christos Boutsidis, Michael Mahoney, Petros Drineas","year":"2009","venue":"Proceedings of the twentieth Annual ACM-SIAM Symposium on Discrete Algorithms","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Brabanter et al., 2010] Kris De Brabanter, Jos De Brabanter, Johan Suykens, and Bart De Moor. Optimized fixed-size kernel models for large data sets. Computational Statistics & Data Analysis, 54(6):1484-1504, 2010.","title":"Optimized fixed-size kernel models for large data sets","authors":"Kris De Brabanter, Jos De Brabanter, Johan Suykens, Bart De Moor","year":"2010","venue":"Computational Statistics & Data Analysis","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cai et al., 2010] Jian-Feng Cai, Emmanuel Candès, and Zuowei Shen. A singular value thresholding algorithm for matrix comple- tion. SIAM Journal on Optimization, 20(4):1956-1982, 2010.","title":"A singular value thresholding algorithm for matrix completion","authors":"Jian-Feng Cai, Emmanuel Candès, Zuowei Shen","year":"2010","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chapelle et al., 2002] Olivier Chapelle, Jason Weston, and Bern- hard Schölkopf. Cluster kernels for semi-supervised learning. In NIPS, 2002.","title":"Cluster kernels for semi-supervised learning","authors":"Chapelle","year":"2002","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Cozman et al., 2003] Fábio Gagliardi Cozman, Ira Cohen, and Marcelo Cesar Cirelo. Semi-supervised learning of mixture mod- els. In ICML, 2003.","title":"Semi-supervised learning of mixture models","authors":"Fábio Gagliardi Cozman, Ira Cohen, Marcelo Cesar Cirelo","year":"2003","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Delalleau et al., 2006] Olivier Delalleau, Yoshua Bengio, and Nico- las Le Roux. Large-Scale Algorithms, pages 333-341. MIT Press, 2006.","title":"Large-Scale Algorithms","authors":"Delalleau","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Fowlkes et al., 2004] Charless Fowlkes, Serge Belongie, Fan Chung, and Jitendra Malik. Spectral grouping using the Nyström method. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(2):214-225, 2004.","title":"Spectral grouping using the Nyström method","authors":"Charless Fowlkes, Serge Belongie, Fan Chung, Jitendra Malik","year":"2004","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Grady and Funka-Lea, 2004] Leo Grady and Gareth Funka-Lea. Multi-label image segmentation for medical applications based on graph-theoretic electrical potentials. In ECCV, 2004.","title":"Multi-label image segmentation for medical applications based on graph-theoretic electrical potentials","authors":"Grady, Funka-Lea ;, Leo Grady, Gareth Funka-Lea","year":"2004","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Han et al., 2016] Yufei Han, Yun Shen, and Maurizio Filippone. Mini-batch spectral clustering. CoRR, abs/1607.02024, 2016.","title":"Mini-batch spectral clustering","authors":"Yufei Han, Yun Shen, Maurizio Filippone","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Joachims, 1999] Thorsten Joachims. Transductive inference for text classification using support vector machines. In ICML, 1999.","title":"Transductive inference for text classification using support vector machines","authors":"Thorsten Joachims","year":"1999","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Joachims, 2003] Thorsten Joachims. Transductive learning via spectral graph partitioning. In ICML, 2003.","title":"Transductive learning via spectral graph partitioning","authors":"Thorsten Joachims","year":"2003","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Jumutc et al., 2013] Vilen Jumutc, Xiaolin Huang, and Johan Suykens. Fixed-size Pegasos for hinge and pinball loss SVM. In IJCNN, 2013.","title":"Fixed-size Pegasos for hinge and pinball loss SVM","authors":"Vilen Jumutc, Xiaolin Huang, Johan Suykens","year":"2013","venue":"IJCNN","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kemp et al., 2003] Charles Kemp, Thomas Griffiths, Sean Strom- sten, and Joshua Tenenbaum. Semi-supervised learning with trees. In NIPS, 2003.","title":"Semi-supervised learning with trees","authors":"Charles Kemp, Thomas Griffiths, Sean Stromsten, Joshua Tenenbaum","year":"2003","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kumar et al., 2009] Sanjiv Kumar, Mehryar Mohri, and Ameet Tal- walkar. Ensemble nystrom method. In NIPS, 2009.","title":"Ensemble nystrom method","authors":"Sanjiv Kumar, Mehryar Mohri, Ameet Talwalkar","year":"2009","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kumar et al., 2012] Sanjiv Kumar, Mehryar Mohri, and Ameet Tal- walkar. Sampling methods for the Nyström method. Journal of Machine Learning Research, 13:981-1006, 2012.","title":"Sampling methods for the Nyström method","authors":"Sanjiv Kumar, Mehryar Mohri, Ameet Talwalkar","year":"2012","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Levin et al., 2004] Anat Levin, Dani Lischinski, and Yair Weiss. Colorization using optimization. ACM Transactions on Graphics, 23(3):689-694, 2004.","title":"Colorization using optimization","authors":"Levin","year":"2004","venue":"ACM Transactions on Graphics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Li and Zhou, 2015] Yu-Feng Li and Zhi-Hua Zhou. Towards mak- ing unlabeled data never hurt. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(1):175-188, 2015.","title":"Towards making unlabeled data never hurt","authors":"Li, Zhou ; ] Yu-Feng Li, Zhi-Hua Zhou","year":"2015","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Li et al., 2016] Yeqing Li, Junzhou Huang, and Wei Liu. Scalable sequential spectral clustering. In AAAI, 2016.","title":"Scalable sequential spectral clustering","authors":"Li","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Liu et al., 2012] Wei Liu, Jun Wang, and Shih-Fu Chang. Robust and scalable graph-based semisupervised learning. Proceedings of the IEEE, 100(9):2624-2638, 2012.","title":"Robust and scalable graph-based semisupervised learning","authors":"Wei Liu, Jun Wang, Shih-Fu Chang","year":"2012","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Mehrkanoon and Suykens, 2014] Siamak Mehrkanoon and Johan A. K. Suykens. Large scale semi-supervised learning using KSC based model. In IJCNN, 2014.","title":"Large scale semi-supervised learning using KSC based model","authors":"Siamak Mehrkanoon, Johan A K Suykens","year":"2014","venue":"IJCNN","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Nesterov, 2013] Yurii Nesterov. Gradient methods for minimizing composite functions. Mathematical Programming, 140(1):125- 161, 2013.","title":"Gradient methods for minimizing composite functions","authors":"Yurii Nesterov","year":"2013","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Pan and Chen, 1999] Victor Y. Pan and Zhao Q. Chen. The com- plexity of the matrix eigenproblem. In STOC, 1999.","title":"The complexity of the matrix eigenproblem","authors":"Y Victor, Pan, Q Zhao, Chen","year":"1999","venue":"STOC","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Pfahringer et al., 2007] Bernhard Pfahringer, Claire Leschi, and Pe- ter Reutemann. Scaling up semi-supervised learning: An efficient and effective LLGC variant. In PAKDD, 2007.","title":"Scaling up semi-supervised learning: An efficient and effective LLGC variant","authors":"Pfahringer","year":"2007","venue":"PAKDD","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Rahimi and Recht, 2007] Ali Rahimi and Benjamin Recht. Ran- dom features for large-scale kernel machines. In NIPS, 2007.","title":"Random features for large-scale kernel machines","authors":"Recht Rahimi, Ali Rahimi, Benjamin Recht","year":"2007","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sánchez et al., 2013] Jorge Sánchez, Florent Perronnin, Thomas Mensink, and Jakob Verbeek. Image classification with the fisher vector: Theory and practice. IJCV, 105(3):222-245, 2013.","title":"Image classification with the fisher vector: Theory and practice","authors":"Jorge Sánchez, Florent Perronnin, Thomas Mensink, Jakob Verbeek","year":"2013","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Williams and Seeger, 2001] Christopher Williams and Matthias Seeger. Using the Nyström method to speed up kernel machines. In NIPS, 2001.","title":"Using the Nyström method to speed up kernel machines","authors":"Christopher Williams, Matthias Seeger","year":"2001","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yang et al., 2012] Tianbao Yang, Yu-Feng Li, Mehrdad Mahdavi, Rong Jin, and Zhi-Hua Zhou. Nyström method vs random Fourier features: A theoretical and empirical comparison. In NIPS, 2012.","title":"Nyström method vs random Fourier features: A theoretical and empirical comparison","authors":"Tianbao Yang, Yu-Feng Li, Mehrdad Mahdavi, Rong Jin, Zhi-Hua Zhou","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang and Kwok, 2010] Kai Zhang and James Kwok. Clustered Nyström method for large scale manifold learning and dimension reduction. IEEE Transactions on Neural Networks, 21(10):1576- 1587, 2010.","title":"Clustered Nyström method for large scale manifold learning and dimension reduction","authors":"Kwok ; ] Kai Zhang, James Zhang, Kwok","year":"2010","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2012] Kai Zhang, Liang Lan, Zhuang Wang, and Fabian Moerchen. Scaling up kernel SVM on limited resources: A low-rank linearization approach. In AISTATS, 2012.","title":"Scaling up kernel SVM on limited resources: A low-rank linearization approach","authors":"Kai Zhang, Liang Lan, Zhuang Wang, Fabian Moerchen","year":"2012","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhang et al., 2016] Lijun Zhang, Tianbao Yang, Jin-Feng Yi, Rong Jin, and Zhi-Hua Zhou. Stochastic optimization for kernel PCA. In AAAI, 2016.","title":"Stochastic optimization for kernel PCA","authors":"Lijun Zhang, Tianbao Yang, Jin-Feng Yi, Rong Jin, Zhi-Hua Zhou","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhou et al., 2003] Dengyong Zhou, Olivier Bousquet, Thomas Navin Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. In NIPS, 2003.","title":"Learning with local and global consistency","authors":"Dengyong Zhou, Olivier Bousquet, Thomas Navin Lal, Jason Weston, Bernhard Schölkopf","year":"2003","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhou et al., 2009] Zhi-Hua Zhou, Michael K. Ng, Qiao-Qiao She, and Yuan Jiang. Budget semi-supervised learning. In PAKDD, 2009.","title":"Budget semi-supervised learning","authors":"Zhi-Hua Zhou, Michael K Ng, Qiao-Qiao She, Yuan Jiang","year":"2009","venue":"PAKDD","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhu et al., 2003] Xiaojin Zhu, Zoubin Ghahramani, and John Laf- ferty. Semi-supervised learning using Gaussian fields and har- monic functions. In ICML, 2003.","title":"Semi-supervised learning using Gaussian fields and harmonic functions","authors":"Xiaojin Zhu, Zoubin Ghahramani, John Lafferty","year":"2003","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhu et al., 2004] Xiaojin Zhu, Jaz Kandola, Zoubin Ghahramani, and John Lafferty. Nonparametric transforms of graph kernels for semi-supervised learning. In NIPS, 2004.","title":"Nonparametric transforms of graph kernels for semi-supervised learning","authors":"Xiaojin Zhu, Jaz Kandola, Zoubin Ghahramani, John Lafferty","year":"2004","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zhu et al., 2005] Xiaojin Zhu, John Lafferty, and Ronald Rosen- feld. Semi-supervised learning with graphs. Carnegie Mellon University, School of Computer Science, 2005.","title":"Semi-supervised learning with graphs","authors":"Xiaojin Zhu, John Lafferty, Ronald Rosenfeld","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zhu, 2007] Xiaojin Zhu. Semi-supervised learning literature survey. Technical report, University of Wisconsin-Madison, 2007.","title":"Semi-supervised learning literature survey","authors":"Xiaojin Zhu","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}