{"type":"ScholarlyArticle","title":"Robust Learning from Noisy Side-information by Semidefinite Programming","authors":[{"name":"En-Liang Hu","slug":"en-liang-hu","given":"En-Liang","family":"Hu","affiliation":"Department of Mathematic , Yunnan Normal University"},{"name":"Quanming Yao","slug":"quanming-yao","given":"Quanming","family":"Yao","affiliation":"4Paradigm Inc; Department of Computer Science and Engineering , Hong Kong University of Science and Technology"}],"abstract":"Robustness recently becomes one of the major concerns among machine learning community, since learning algorithms are usually vulnerable to outliers or corruptions. Motivated by such a trend and needs, we pursue robustness in semi-definite programming (SDP) in this paper.  Specifically, this is done by replacing the commonly used squared loss with the more robust L1-loss in the low-rank SDP. However, the resulting objective becomes neither convex nor smooth. As no existing algorithms can be applied, we design an efficient algorithm, based on majorization-minimization, to optimize the objective. The proposed algorithm not only has cheap iterations and low space complexity but also theoretically converges to some critical points. Finally, empirical study shows that the new objective armed with proposed algorithm outperforms state-of-the-art in terms of both speed and accuracy.","doi":"10.24963/ijcai.2019/349","url":"https://ijcai.org/proceedings/2019/349","pdf":"https://ijcai.org/proceedings/2019/0349.pdf","markdown":"https://ijcai.org/proceedings/2019/349.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/349","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":349,"pages":{"first":2514,"last":2520},"keywords":["Machine Learning: Feature Selection ; Learning Sparse Models","Machine Learning: Dimensionality Reduction and Manifold Learning","Machine Learning Applications: Big data ; Scalability","Machine Learning: Semi-Supervised Learning"],"videos":[],"references":[{"position":1,"raw":"[Beck and Teboulle, 2009] A. Beck and M. Teboulle. A fast iterative shrinkage-thresholding algorithm for linear in- verse problems. SIJIS, 2(1):183-202, 2009.","title":"A fast iterative shrinkage-thresholding algorithm for linear inverse problems","authors":"A Beck, M Teboulle","year":"2009","venue":"SIJIS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Boyd and Vandenberghe, 2004] S. Boyd and L. Vanden- berghe. Convex Optimization. Cambridge University Press, 2004.","title":"Convex Optimization","authors":"S Boyd, L Vandenberghe","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Boyd et al., 2011] S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. Distributed optimization and statistical learning via the alternating direction method of multiplier- s. Foundations and Trends in Machine Learning, 3(1):1- 122, 2011.","title":"Distributed optimization and statistical learning via the alternating direction method of multipliers","authors":"S Boyd, N Parikh, E Chu, B Peleato, J Eckstein","year":"2011","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Burer and Monteiro, 2003] S. Burer and R.D.C. Monteiro. A nonlinear programming algorithm for solving semidef- inite programs via low-rank factorization. MathProg, 95:329-357, 2003.","title":"A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization","authors":"S Burer, R D C Monteiro","year":"2003","venue":"MathProg","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"A. D'aspremont, El L. Ghaoui, M. I. Jordan, and G. R. G. Lanckriet. A direct formulation for sparse PCA using semidefinite programming. SIAM Review, 49(3):434-48, 2007.","title":"A direct formulation for sparse PCA using semidefinite programming","authors":"A El, L Ghaoui, M I Jordan, G R G Lanckriet","year":"2007","venue":"SIAM Review","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dekel et al., 2010] Ofer Dekel, Ohad Shamir, and Lin Xiao. Learning to classify with missing and corrupted features. ML, 2010.","title":"Learning to classify with missing and corrupted features","authors":"Ofer Dekel, Ohad Shamir, Lin Xiao","year":"2010","venue":"ML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Helmberg et al., 1996] C. Helmberg, F. Rendl, R.J. Vander- bei, and H. Wolkowicz. An interior-point method for semidefinite programming. SIOPT, 1996.","title":"An interior-point method for semidefinite programming","authors":"C Helmberg, F Rendl, R J Vanderbei, H Wolkowicz","year":"1996","venue":"SIOPT","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hoi et al., 2007] S. Hoi, R. Jin, and M.R. Lyu. Learning nonparametric kernel matrices from pairwise constraints. In ICML, pages 361-368, 2007.","title":"Learning nonparametric kernel matrices from pairwise constraints","authors":"S Hoi, R Jin, M R Lyu","year":"2007","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hu et al., 2011] E.-L. Hu, B. Wang, and S.-C. Chen. BCD- NPKL: Scalable non-parametric kernel learning using block coordinate descent. In ICML, pages 209-216, 2011.","title":"BCD-NPKL: Scalable non-parametric kernel learning using block coordinate descent","authors":"E.-L Hu, B Wang, S.-C Chen","year":"2011","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hunter and Lange, 2004] D. Hunter and K. Lange. A tu- torial on MM algorithms. The American Statistician, 58(1):30-37, 2004.","title":"A tutorial on MM algorithms","authors":"D Hunter, K Lange","year":"2004","venue":"The American Statistician","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jaggi, 2013] M. Jaggi. Revisiting Frank-Wolfe: Projection- free sparse convex optimization. In ICML, 2013.","title":"Revisiting Frank-Wolfe: Projectionfree sparse convex optimization","authors":"M Jaggi","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kulis et al., 2007] B. Kulis, A. C. Surendran, and J. C. Plat- t. Fast low-rank semidefinite programming for embedding and clustering. In AISTAT, pages 235-242, 2007.","title":"Fast low-rank semidefinite programming for embedding and clustering","authors":"B Kulis, A C Surendran, J C Platt","year":"2007","venue":"AISTAT","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lange et al., 2000] K. Lange, R. Hunter, and I. Yang. Op- timization transfer using surrogate objective functions. JCGS, 9(1):1-20, 2000.","title":"Optimization transfer using surrogate objective functions","authors":"K Lange, R Hunter, I Yang","year":"2000","venue":"JCGS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Laue, 2012] S. Laue. A hybrid algorithm for convex semidefinite optimization. In ICML, pages 177-184.","title":"A hybrid algorithm for convex semidefinite optimization","authors":"S Laue","year":null,"venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"Om- nipress, 2012.","title":"Omnipress","authors":null,"year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lemon et al., 2016] A. Lemon, A. So, and Y. Ye. Low- rank semidefinite programming: Theory and application- s. Foundations and Trends in Optimization, 2(1-2):1-156, 2016.","title":"Lowrank semidefinite programming: Theory and applications","authors":"A Lemon, A So, Y Ye","year":"2016","venue":"Foundations and Trends in Optimization","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li and Lin, 2015] H. Li and Z. Lin. Accelerated proximal gradient methods for nonconvex programming. In NeurIP- S, pages 379-387. 2015.","title":"Accelerated proximal gradient methods for nonconvex programming","authors":"H Li, Z Lin","year":"2015","venue":"NeurIP-S","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2008] Z. Li, J. Liu, and X. Tang. Pairwise con- straint propagation by semidefinite programming for semi- supervised classification. In ICML, pages 576-583, 2008.","title":"Pairwise constraint propagation by semidefinite programming for semisupervised classification","authors":"Li","year":"2008","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lin et al., 2017] Z. Lin, C. Xu, and H. Zha. Robust matrix factorization by majorization minimization. TPAMI, (99), 2017.","title":"Robust matrix factorization by majorization minimization","authors":"Z Lin, C Xu, H Zha","year":"2017","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nocedal and Wright, 2006] J. Nocedal and S.J. Wright. Nu- merical Optimization. Springer, 2006.","title":"Numerical Optimization","authors":"J Nocedal, S J Wright","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pumir et al., 2018] T. Pumir, S. Jelassi, and N. Boumal. S- moothed analysis of the low-rank approach for smooth semidefinite programs. In NeurIPS, 2018.","title":"Smoothed analysis of the low-rank approach for smooth semidefinite programs","authors":"T Pumir, S Jelassi, N Boumal","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Raykar et al., 2010] V. Raykar, S. Yu, L. Zhao, G. Valadez, C. Florin, L. Bogoni, and L. Moy. Learning from crowds. JMLR, 2010.","title":"Learning from crowds","authors":"V Raykar, S Yu, L Zhao, G Valadez, C Florin, L Bogoni, L Moy","year":"2010","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Royer, 2017] M. Royer. Adaptive clustering through semidefinite programming. In NeurIPS, pages 1793-1801, 2017.","title":"Adaptive clustering through semidefinite programming","authors":"M Royer","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Song et al., 2008] L. Song, A. Smola, K. Borgwardt, and A. Gretton. Colored maximum variance unfolding. In NeurIPS, 2008.","title":"Colored maximum variance unfolding","authors":"L Song, A Smola, K Borgwardt, A Gretton","year":"2008","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Srinadh et al., 2016] B. Srinadh, K. Anastasios, and Sujay S. Dropping convexity for faster semidefinite optimiza- tion. Preprint arXiv:1509.03917, 2016.","title":"Dropping convexity for faster semidefinite optimization","authors":"B Srinadh, K Anastasios, Sujay S","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Toh et al., 2015] K.-C. Toh, L. Yang, and D. Sun. SDP- NAL+: a majorized semismooth Newton-CG augment- ed Lagrangian method for semidefinite programming with nonnegative constraints. MPC, 2015.","title":"SDP-NAL+: a majorized semismooth Newton-CG augmented Lagrangian method for semidefinite programming with nonnegative constraints","authors":"K.-C Toh, L Yang, D Sun","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Vandenberghe and Boyd, 1996] L. Vandenberghe and S. Boyd. Semidefinite programming. SIAM Review, 38(1):49-95, 1996.","title":"Semidefinite programming","authors":"L Vandenberghe, S Boyd","year":"1996","venue":"SIAM Review","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Weinberger and Saul, 2006] K.Q. Weinberger and L.K. Saul. Unsupervised learning of image manifolds by semidefinite programming. IJCV, 2006.","title":"Unsupervised learning of image manifolds by semidefinite programming","authors":"K Q Weinberger, L K Saul","year":"2006","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Weinberger et al., 2004] K. Q. Weinberger, F. Sha, and L.K. Saul. Learning a kernel matrix for nonlinear dimensional- ity reduction. In ICML, pages 839-846, 2004.","title":"Learning a kernel matrix for nonlinear dimensionality reduction","authors":"K Q Weinberger, F Sha, L K Saul","year":"2004","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yao and Kwok, 2018] Q. Yao and J. Kwok. Scalable robust matrix factorization with nonconvex loss. In NeurIPS 31, pages 5066-5075. 2018.","title":"Scalable robust matrix factorization with nonconvex loss","authors":"Kwok Yao, ] Q Yao, J Kwok","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yao et al., 2018] Q. Yao, M. Wang, Y. Chen, W. Dai, Y. Hu, Y. Li, W. Tu, Q. Yang, and Y. Yu. Taking human out of learning applications: A survey on automated machine learning. Technical report, arXiv preprint, 2018.","title":"Taking human out of learning applications: A survey on automated machine learning","authors":"Q Yao, M Wang, Y Chen, W Dai, Y Hu, Y Li, W Tu, Q Yang, Y Yu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zheng and Lafferty, 2015] Q. Zheng and J. Lafferty. A con- vergent gradient descent algorithm for rank minimization and semidefinite programming from random linear mea- surements. In NeurIPS, 2015.","title":"A convergent gradient descent algorithm for rank minimization and semidefinite programming from random linear measurements","authors":"Q Zheng, J Lafferty","year":"2015","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhuang et al., 2011] J. Zhuang, I. Tsang, and S. Hoi. A fam- ily of simple non-parametric kernel learning algorithms. JMLR, 12:1313-1347, 2011.","title":"A family of simple non-parametric kernel learning algorithms","authors":"J Zhuang, I Tsang, S Hoi","year":"2011","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zou and Xue, 2018] H. Zou and L.Z. Xue. A selective overview of sparse principal component analysis. Pro- ceedings of the IEEE, 2018.","title":"A selective overview of sparse principal component analysis","authors":"H Zou, L Z Xue","year":"2018","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}