{"type":"ScholarlyArticle","title":"Non-smooth Optimization over Stiefel Manifolds with Applications to Dimensionality Reduction and Graph Clustering","authors":[{"name":"Fariba Zohrizadeh","slug":"fariba-zohrizadeh","given":"Fariba","family":"Zohrizadeh","affiliation":"The University of Texas at Arlington"},{"name":"Mohsen Kheirandishfard","slug":"mohsen-kheirandishfard","given":"Mohsen","family":"Kheirandishfard","affiliation":"The University of Texas at Arlington"},{"name":"Farhad Kamangar","slug":"farhad-kamangar","given":"Farhad","family":"Kamangar","affiliation":"The University of Texas at Arlington"},{"name":"Ramtin Madani","slug":"ramtin-madani","given":"Ramtin","family":"Madani","affiliation":"The University of Texas at Arlington"}],"abstract":"This paper is concerned with the class of non-convex optimization problems with orthogonality constraints. We develop computationally efficient relaxations that transform non-convex orthogonality constrained problems into polynomial-time solvable surrogates. A novel penalization technique is used to enforce feasibility and derive certain conditions under which the constraints of the original non-convex problem are guaranteed to be satisfied. Moreover, we extend our approach to a feasibility-preserving sequential scheme that solves penalized relaxation to obtain near-globally optimal points. Experimental results on synthetic and real datasets demonstrate the effectiveness of the proposed approach on two practical applications in machine learning.","doi":"10.24963/ijcai.2019/183","url":"https://ijcai.org/proceedings/2019/183","pdf":"https://ijcai.org/proceedings/2019/0183.pdf","markdown":"https://ijcai.org/proceedings/2019/183.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/183","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Heuristic Search and Game Playing","track":"Main track","number":183,"pages":{"first":1319,"last":1326},"keywords":["Machine Learning: Explainable Machine Learning","Heuristic Search and Game Playing: Combinatorial Search and Optimisation"],"videos":[],"references":[{"position":1,"raw":"[Abrudan et al., 2008] Traian E Abrudan, Jan Eriksson, and Visa Koivunen. Steepest descent algorithms for optimiza- tion under unitary matrix constraint. IEEE T SIGNAL PROCES, 56(3):1134-1147, 2008.","title":"Steepest descent algorithms for optimization under unitary matrix constraint","authors":"Jan Traian E Abrudan, Visa Eriksson, Koivunen","year":"2008","venue":"IEEE T SIGNAL PROCES","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Absil et al., 2009] P-A Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization algorithms on matrix manifolds. Princeton University Press, 2009.","title":"Optimization algorithms on matrix manifolds","authors":"P-A Absil, Robert Mahony, Rodolphe Sepulchre","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Attouch et al., 2013] Hedy Attouch, Jérôme Bolte, and Be- nar Fux Svaiter. Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss-Seidel methods. MATH PROGRAM, 137(1-2):91-129, 2013.","title":"Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss-Seidel methods","authors":"Hedy Attouch, Jérôme Bolte, Benar Fux Svaiter","year":"2013","venue":"MATH PROGRAM","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Belhumeur et al., 1997] Peter N Belhumeur, João P Hes- panha, and David J Kriegman. Eigenfaces vs. fisherfaces: Recognition using class specific linear projection. IEEE TPAMI, (7):711-720, 1997.","title":"Eigenfaces vs. fisherfaces: Recognition using class specific linear projection","authors":"João P Peter N Belhumeur, David J Hespanha, Kriegman","year":"1997","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bian and Tao, 2011] Wei Bian and Dacheng Tao. Max- min distance analysis by using sequential SDP relax- ation for dimension reduction. IEEE T PATTERN ANAL, 33(5):1037-1050, 2011.","title":"Maxmin distance analysis by using sequential SDP relaxation for dimension reduction","authors":"Wei Bian, Dacheng Tao","year":"2011","venue":"IEEE T PATTERN ANAL","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Boyd and Vandenberghe, 2004] Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge univer- sity press, 2004.","title":"Convex optimization","authors":"Vandenberghe Boyd","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Boyd et al., 2011] Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein. Distributed opti- mization and statistical learning via the alternating direc- tion method of multipliers. Foundations and Trends R in Machine Learning, 3(1):1-122, 2011.","title":"Distributed optimization and statistical learning via the alternating direction method of multipliers","authors":"Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein","year":"2011","venue":"Foundations and Trends R in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chang and Yeung, 2008] Hong Chang and Dit-Yan Ye- ung. Robust path-based spectral clustering. PATTERN RECOGN, 41(1):191-203, 2008.","title":"Robust path-based spectral clustering","authors":"Chang, Yeung, Chang, Dit-Yan Yeung","year":"2008","venue":"PATTERN RECOGN","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chen et al., 2016] Weiqiang Chen, Hui Ji, and Yanfei You. An augmented Lagrangian method for ℓ 1 -regularized op- timization problems with orthogonality constraints. SIAM J SCI COMPUT, 38(4):B570-B592, 2016.","title":"An augmented Lagrangian method for ℓ 1 -regularized optimization problems with orthogonality constraints","authors":"Weiqiang Chen, Hui Ji, Yanfei You","year":"2016","venue":"SIAM J SCI COMPUT","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dua and Graff, 2017] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.","title":"Dheeru Dua and Casey Graff. UCI machine learning repository","authors":"Graff Dua","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Edelman et al., 1998] Alan Edelman, Tomás A Arias, and Steven T Smith. The geometry of algorithms with orthog- onality constraints. SIAM J MATH ANAL, 20(2):303-353, 1998.","title":"The geometry of algorithms with orthogonality constraints","authors":"Alan Edelman, Tomás A Arias, Steven T Smith","year":"1998","venue":"SIAM J MATH ANAL","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Fortin and Glowinski, 2000] Michel Fortin and Roland Glowinski. Augmented Lagrangian methods: applica- tions to the numerical solution of boundary-value prob- lems, volume 15. Elsevier, 2000.","title":"Augmented Lagrangian methods: applications to the numerical solution of boundary-value problems","authors":"Glowinski Fortin, Michel Fortin, Roland Glowinski","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gao et al., 2018] Bin Gao, Xin Liu, Xiaojun Chen, and Ya- xiang Yuan. A new first-order algorithmic framework for optimization problems with orthogonality constraints. SIAM J OPTIMIZ, 28(1):302-332, 2018.","title":"A new first-order algorithmic framework for optimization problems with orthogonality constraints","authors":"Gao","year":"2018","venue":"SIAM J OPTIMIZ","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Georghiades et al., 2001] Athinodoros S. Georghiades, Pe- ter N. Belhumeur, and David J. Kriegman. From few to many: Illumination cone models for face recognition un- der variable lighting and pose. IEEE T PATTERN ANAL, 23(6):643-660, 2001.","title":"From few to many: Illumination cone models for face recognition under variable lighting and pose","authors":"Athinodoros S Georghiades, Peter N Belhumeur, David J Kriegman","year":"2001","venue":"IEEE T PATTERN ANAL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Gionis et al., 2007] Aristides Gionis, Heikki Mannila, and Panayiotis Tsaparas. Clustering aggregation. ACM T KNOWL DISCOV D, 1(1):4, 2007.","title":"Clustering aggregation","authors":"Gionis","year":"2007","venue":"ACM T KNOWL DISCOV D","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Han et al., 2017] Junwei Han, Kai Xiong, and Feiping Nie. Orthogonal and nonnegative graph reconstruction for large scale clustering. In IJCAI, 2017. [Ibaraki and Tomizuka, 2001] Soichi Ibaraki and Masayoshi Tomizuka. Rank minimization approach for solving BMI problems with random search. In ACC, volume 3, pages 1870-1875. IEEE, 2001.","title":"Orthogonal and nonnegative graph reconstruction for large scale clustering","authors":"Han","year":"2001","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/251"},{"position":17,"raw":"[Jain and Law, 2005] Anil K Jain and Martin HC Law. Data clustering: A user's dilemma. In ICPRAI, 2005.","title":"Data clustering: A user's dilemma","authors":"K Anil, Martin Hc Jain, Law","year":"2005","venue":"ICPRAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Jiang et al., 2017] Bo Jiang, Jin Tang, Chris HQ Ding, and Bin Luo. Nonnegative orthogonal graph matching. In AAAI, 2017.","title":"Nonnegative orthogonal graph matching","authors":"Bo Jiang, Jin Tang, Chris Hq Ding, Bin Luo","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kheirandishfard et al., 2018a] Mohsen Kheirandishfard, Fariba Zohrizadeh, Muhammad Adil, and Ramtin Madani. Convex relaxation of bilinear matrix inequalities part II: Applications to optimal control synthesis. In CDC, 2018.","title":"Convex relaxation of bilinear matrix inequalities part II: Applications to optimal control synthesis","authors":"Kheirandishfard","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kheirandishfard et al., 2018b] Mohsen Kheirandishfard, Fariba Zohrizadeh, and Ramtin Madani. Convex relax- ation of bilinear matrix inequalities part I: Theoretical results. In CDC, 2018. [Kovnatsky et al., 2016] Artiom Kovnatsky, Klaus Glashoff, and Michael M Bronstein. MADMM: a generic algorithm for non-smooth optimization on manifolds. In ECCV, 2016.","title":"Convex relaxation of bilinear matrix inequalities part I: Theoretical results","authors":"Kheirandishfard","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lai and Osher, 2014] Rongjie Lai and Stanley Osher. A splitting method for orthogonality constrained problems. SIAM J SCI COMPUT, 58(2):431-449, 2014.","title":"Rongjie Lai and Stanley Osher. A splitting method for orthogonality constrained problems","authors":"Osher Lai","year":"2014","venue":"SIAM J SCI COMPUT","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Mosek, 2015] ApS Mosek. The MOSEK optimization tool- box for MATLAB manual. Version 7.1, 2015.","title":"ApS Mosek. The MOSEK optimization toolbox for MATLAB manual","authors":"Mosek","year":"2015","venue":"Version","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Nakata et al., 2003] Kazuhide Nakata, Katsuki Fujisawa, Mituhiro Fukuda, Masakazu Kojima, and Kazuo Murota. Exploiting sparsity in semidefinite programming via ma- trix completion II: Implementation and numerical results. Mathematical Programming, 95(2):303-327, 2003.","title":"Exploiting sparsity in semidefinite programming via matrix completion II: Implementation and numerical results","authors":"Kazuhide Nakata, Katsuki Fujisawa, Mituhiro Fukuda, Masakazu Kojima, Kazuo Murota","year":"2003","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ng et al., 2002] Andrew Y Ng, Michael I Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm. In NIPS, pages 849-856, 2002.","title":"On spectral clustering: Analysis and an algorithm","authors":"Y Andrew, Michael I Ng, Yair Jordan, Weiss","year":"2002","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ozolin ¸š et al., 2013] Vidvuds Ozolin ¸š, Rongjie Lai, Rus- sel Caflisch, and Stanley Osher. Compressed modes for variational problems in mathematics and physics. PNAS, 110(46):18368-18373, 2013.","title":"Vidvuds Ozolin ¸š, Rongjie Lai, Russel Caflisch, and Stanley Osher. Compressed modes for variational problems in mathematics and physics","authors":"Ozolin ¸š","year":"2013","venue":"PNAS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tang and Liu, 2012] Jiliang Tang and Huan Liu. Unsuper- vised feature selection for linked social media data. In SIGKDD, 2012.","title":"Unsupervised feature selection for linked social media data","authors":"Liu Tang, Jiliang Tang, Huan Liu","year":"2012","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wen and Yin, 2013] Zaiwen Wen and Wotao Yin. A fea- sible method for optimization with orthogonality con- straints. MATH PROGRAM, 142(1-2):397-434, 2013.","title":"A feasible method for optimization with orthogonality constraints","authors":"Wen, Yin ;, Zaiwen Wen, Wotao Yin","year":"2013","venue":"MATH PROGRAM","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yang et al., 2011] Yi Yang, Heng Tao Shen, Zhigang Ma, Zi Huang, and Xiaofang Zhou. ℓ 2,1 -norm regularized dis- criminative feature selection for unsupervised learning. In IJCAI, 2011.","title":"ℓ 2,1 -norm regularized discriminative feature selection for unsupervised learning","authors":"Yi Yang, Heng Tao Shen, Zhigang Ma, Zi Huang, Xiaofang Zhou","year":"2011","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yin et al., 2008] Wotao Yin, Stanley Osher, Donald Gold- farb, and Jerome Darbon. Bregman iterative algorithms for ℓ 1 -minimization with applications to compressed sens- ing. SIAM J IMAGING SCI, 1(1):143-168, 2008.","title":"Bregman iterative algorithms for ℓ 1 -minimization with applications to compressed sensing","authors":"Yin","year":"2008","venue":"SIAM J IMAGING SCI","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhu et al., 2017] Hong Zhu, Xiaowei Zhang, Delin Chu, and Li-Zhi Liao. Nonconvex and nonsmooth optimiza- tion with generalized orthogonality constraints: An ap- proximate augmented Lagrangian method. SIAM J SCI COMPUT, pages 1-42, 2017.","title":"Nonconvex and nonsmooth optimization with generalized orthogonality constraints: An approximate augmented Lagrangian method","authors":"Zhu","year":"2017","venue":"SIAM J SCI COMPUT","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}